Large Language Model Support for CTEPH Differential Diagnosis

Pulmonary artery filling defects can result from several conditions, including chronic thromboembolic pulmonary hypertension (CTEPH), acute pulmonary embolism (PE), and pulmonary artery sarcoma (PAS). Distinguishing these conditions can be challenging for physicians who do not specialize in pulmonary vascular diseases. This prospective randomized controlled trial aims to assess whether a clinical decision support system using a locked large language model (LLM) can improve diagnostic performance and reduce diagnostic errors. The study will enroll up to 66 physicians, with a target of 60 participants completing the trial. After providing informed consent and completing a baseline assessment, participants will be stratified by professional seniority and randomly assigned in a 1:1 ratio to either an LLM assisted group or a control group. Randomization will be performed using a computer generated sequence, with allocation concealed until baseline assessments are completed. All participants will independently evaluate the same 30 deidentified clinical cases, including 10 cases each of CTEPH, acute PE, and PAS. The diagnoses will be confirmed by relevant specialists using established diagnostic criteria. These cases will not have been used for model development or optimization. Each participant will complete two sessions of 15 cases, separated by 3 to 7 days, with case order randomized for each participant. Clinical information will be presented in four stages: clinical history and physical examination, laboratory findings, echocardiography, and computed tomography pulmonary angiography (CTPA). Participants will record their diagnosis at each stage before receiving additional information. The control group will make diagnostic decisions based on the available clinical information, while the LLM assisted group will also receive the model's most likely diagnosis, supporting and opposing evidence, and referral recommendations. The model version, prompts, parameters, and case specific outputs will be fixed before randomization and remain unchanged throughout the study. The primary outcome is sensitivity for identifying CTEPH after all four stages. Secondary outcomes include diagnostic accuracy across all three conditions, diagnostic performance at each stage, time to diagnosis, appropriateness of referral recommendations, and errors associated with reliance on incorrect LLM recommendations. Analyses will follow the intention to treat principle. Mixed effects logistic regression will be used to compare diagnostic performance between groups, with physician seniority included as a fixed effect and physician and case included as random effects. The study will use deidentified clinical cases and will not involve direct patient participation or influence clinical care.