Comparative Evaluation of ChatGPT and DeepSeek in Oral Pathology Education and Evidence-Based Diagnostic Decision Support: A Pilot Study

Karima Alfitory; Najwa  Awn; Basma; Mouna  Abdullah; Weam Hraisha; Omalkhair Hassan

doi:10.54361/ajmas.261032

Authors

Karima Alfitory Department of Oral Medicine, Oral Pathology and Oral and Maxillofacial Surgery, Faculty of Dentistry, Sirt University, Sirt, Libya https://orcid.org/0009-0007-7031-0927
Najwa Awn Department of Oral Medicine, Oral Pathology and Oral and Maxillofacial Surgery, Faculty of Dentistry, Al-Tahadi University, Tripoli, Libya https://orcid.org/0000-0001-7935-7448
Basma Elhabbasi Department of Oral Medicine, Oral Pathology and Oral and Maxillofacial Surgery, Faculty of Dentistry, University of Tripoli, Tripoli, Libya https://orcid.org/0009-0000-1168-3108
Mouna Abdullah Department of Oral Medicine, Oral Pathology and Oral and Maxillofacial Surgery, Faculty of Dentistry, University of Tripoli, Tripoli, Libya https://orcid.org/0000-0002-1961-0090
Weam Hraisha Department of Oral Medicine, Oral Pathology and Oral and Maxillofacial Surgery, Faculty of Dentistry, University of Tripoli, Tripoli, Libya https://orcid.org/0009-0003-0712-3353
Omalkhair Hassan Department of Oral Medicine and Diagnosis and Radiology, Faculty of Dentistry, Sirt University, Sirt, Libya https://orcid.org/0009-0007-1000-1612

DOI:

https://doi.org/10.54361/ajmas.261032

Keywords:

Artificial Intelligence; Large Language Models; Chatgpt; Deepseek; Oral Pathology; Dental Education

Abstract

Large language models (LLMs) are increasingly being integrated into healthcare education and clinical decision-support systems. While ChatGPT has demonstrated promising performance in medical and dental domains, comparative evidence regarding newer models such as DeepSeek remains limited, particularly in oral pathology. This pilot study aimed to compare the performance of ChatGPT and DeepSeek in oral pathology education and evidence-based diagnostic decision support. We conducted a cross-sectional comparative study from January to December 2025. We developed ten oral pathology educational questions and five standardised clinical case scenarios based on established dental curricula and authoritative oral pathology references. Identical prompts were submitted independently to ChatGPT and DeepSeek. Responses were anonymised and evaluated by three board-certified oral pathologists using a structured rubric assessing accuracy, completeness, clinical relevance, diagnostic reasoning, and evidence-based consistency. Each domain was scored on a 5-point Likert scale, with a maximum total score of 25. Statistical analyses included independent-samples t-tests, Cohen’s d effect sizes, and intraclass correlation coefficient (ICC) assessment. ChatGPT achieved a significantly higher mean total composite score than DeepSeek (23.2 ± 1.8 vs. 20.6 ± 2.4; p = 0.006; Cohen’s d = 1.15). ChatGPT also outperformed DeepSeek across all evaluation domains, including accuracy (4.72 ± 0.31 vs. 4.18 ± 0.52; p = 0.003), completeness (4.68 ± 0.37 vs. 4.10 ± 0.55; p = 0.005), clinical relevance (4.61 ± 0.42 vs. 4.09 ± 0.58; p = 0.011), diagnostic reasoning (4.58 ± 0.46 vs. 3.98 ± 0.63; p = 0.008), and evidence-based consistency (4.75 ± 0.29 vs. 4.16 ± 0.49; p = 0.002). Large effect sizes were observed across all domains. Inter-rater reliability demonstrated excellent agreement among evaluators (ICC = 0.87; 95% CI: 0.81–0.92). ChatGPT demonstrated superior performance compared with DeepSeek in oral pathology education and evidence-based diagnostic decision support. Its higher accuracy, completeness, clinical relevance, and consistency with established evidence suggest considerable potential as an adjunctive educational and clinical support tool. Nevertheless, human oversight remains essential to ensure accuracy and mitigate the risks associated with AI-generated outputs. Further large-scale studies incorporating real clinical cases and multimodal diagnostic data are warranted.