20.500.12556/DiRROS-30747 Accuracy and knowledge base evaluation of ChatGPT-4o, Gemini-2.0-Flash, and DeepSeek-V3 in metabolic and bariatric surgery an expert-rated blinded study Background: Large language models (LLMs) are increasingly applied in medicine; however, their accuracy in guideline-driven, high-stakes specialties, such as metabolic and bariatric surgery (MBS), remains uncertain. This study evaluates the performance of ChatGPT-4o, Gemini 2.0 Flash, and DeepSeek-V3 in generating guideline-concordant responses to MBS clinical questions. Methods: Thirty standardized, guideline-based MBS questions were presented to each model. Responses were randomized in order, anonymized (blinded as Model A/B/C), and evaluated by 93 MBS experts using a validated 0–3 scale (0 = inaccurate; 3 = fully guideline-concordant). A repeated-measures ANOVA with Bonferroni correction tested model differences; reliability was assessed with Cronbach’s α and intraclass correlation coefficients (ICC). Results: DeepSeek-V3 achieved the highest mean score (2.44 ± 0.40), followed by ChatGPT-4o (1.79 ± 0.46) and Gemini 2.0 Flash (1.63 ± 0.47) (p < 0.001). Fully guideline-concordant ratings (score = 3) were most frequent for DeepSeek (80%) vs. ChatGPT (0%) and Gemini (3.3%). Internal consistency was excellent (α > 0.90), and inter-rater reliability was strong (ICC > 0.88). When mapped against the QUEST evaluation framework, the study addressed Quality and Understanding but did not fully capture Expression, Safety, or Trust dimensions. Conclusions: DeepSeek-V3 outperformed ChatGPT-4o and Gemini 2.0 Flash in generating guideline-concordant responses in MBS. These results highlight the need for ongoing, domain-focused validation before clinical use. large language models metabolic and bariatric surgery artificial intelligence evaluation ChatGPT DeepSeek-V3 Gemini true false true Angleški jezik Ni določen Neznano 2026-07-01 14:02:52 2026-07-01 14:02:52 2026-07-02 03:59:46 0000-00-00 00:00:00 2026 0 0 Nasl. z nasl. zaslona; Opis vira z dne 30. 3. 2026; Sodelavka pri raziskavi iz Slovenije: Tadeja Pintar; str. 2160–2171 Vol. 36 2026 0000-00-00 Zaloznikova Objavljeno NiDoloceno 0000-00-00 0000-00-00 0000-00-00 616-089 1708-0428 10.1007/s11695-026-08562-z 273536771 RAZ_Hany_Mohamed_2026.pdf RAZ_Hany_Mohamed_2026.pdf 1 DB4CDCE74AEE21152963F8D95468AE89 e22c8f18c3f1129d93081addb12c932da14174ec6755acd189ec13262def5522 f2d421a0-7544-11f1-80d9-001a4af901a5 https://dirros.openscience.si/Dokument.php?lang=slv&id=46948 https://link.springer.com/article/10.1007/s11695-026-08562-z 1 cbb5951f-7544-11f1-80d9-001a4af901a5 https://dirros.openscience.si/Dokument.php?lang=slv&id=46945 Univerzitetni klinični center Ljubljana 0 0 0