Back to AI Research

AI Research

SymptomAI: Towards a Conversational AI Agent for Ev... | AI Research

Key Takeaways

  • SymptomAI: Towards a Conversational AI Agent for Everyday Symptom Assessment This paper introduces SymptomAI, a conversational AI system designed to conduct...
  • Language models excel at diagnostic assessments on currated medical case-studies and vignettes, performing on par with, or better than, clinical professionals.
  • However, existing studies focus on complex scenarios with rich context making it difficult to draw conclusions about how these systems perform for patients reporting symptoms in everyday life.
  • This corpus captures diverse communication and a realistic distribution of illnesses from a real world population.
  • A subset of 1,228 participants reported a clinician-provided diagnosis, and 517 of these were further evaluated by a panel of clinicians during over 250 hours of annotation.
Paper AbstractExpand

Language models excel at diagnostic assessments on currated medical case-studies and vignettes, performing on par with, or better than, clinical professionals. However, existing studies focus on complex scenarios with rich context making it difficult to draw conclusions about how these systems perform for patients reporting symptoms in everyday life. We deployed SymptomAI, a set of conversational AI agents for end-to-end patient interviewing and differential diagnosis (DDx), via the Fitbit app in a study that randomized participants (N=13,917) to interact with five AI agents. This corpus captures diverse communication and a realistic distribution of illnesses from a real world population. A subset of 1,228 participants reported a clinician-provided diagnosis, and 517 of these were further evaluated by a panel of clinicians during over 250 hours of annotation. SymptomAI DDx were significantly more accurate (OR = 2.47, p < 0.001) than those from independent clinicians given the same dialogue in a blinded randomized comparison. Moreover, agentic strategies which conduct a dedicated symptom interview that elicit additional symptom information before providing a diagnosis, perform substantially better than baseline, user-guided conversations (p < 0.001). An auxiliary analysis on 1,509 conversations from a general US population panel validated that these results generalize beyond wearable device users. We used SymptomAI diagnoses as labels for all 13,917 participants to analyze over 500,000 days of wearable metrics across nearly 400 unique conditions. We identified strong associations between acute infections and physiological shifts (e.g., OR > 7 for influenza). While limited by self-reported ground truth, these results demonstrate the benefits of a dedicated and complete symptom interview compared to a user-guided symptom discussion, which is the default of most consumer LLMs.

SymptomAI: Towards a Conversational AI Agent for Everyday Symptom Assessment

This paper introduces SymptomAI, a conversational AI system designed to conduct patient interviews and provide differential diagnoses for everyday health concerns. While large language models have shown promise in medical vignettes, their performance in real-world, daily life scenarios remains under-researched. By deploying these agents through the Fitbit app, the researchers aimed to bridge the gap between controlled clinical studies and the messy, diverse reality of patient-reported symptoms.

A Real-World Diagnostic Approach

The researchers deployed five different AI agents to 13,917 participants to capture a realistic distribution of illnesses. Unlike many consumer-facing AI tools that rely on user-guided, open-ended discussions, the SymptomAI agents utilized "agentic strategies." These strategies involve a dedicated, structured interview process where the AI actively elicits specific information from the patient before offering a potential diagnosis. This method ensures that the AI gathers the necessary context to make a more informed assessment.

Performance and Accuracy

To evaluate the system, the team compared the AI’s differential diagnoses (DDx) against those provided by independent clinicians who reviewed the same patient dialogues. The results showed that SymptomAI was significantly more accurate than the human clinicians in this blinded comparison. Furthermore, the study found that the structured, agentic interview approach performed substantially better than the baseline, user-guided conversational models typically found in current consumer AI products.

Connecting Symptoms to Wearable Data

Beyond diagnostic accuracy, the study leveraged the large participant pool to analyze over 500,000 days of wearable device metrics. By using the AI-generated diagnoses as labels, the researchers identified strong correlations between specific conditions and physiological shifts. For example, they observed a strong association between acute infections, such as influenza, and changes in wearable health data. An auxiliary analysis of a general US population panel confirmed that these findings are not limited to wearable device users, suggesting broader applicability.

Considerations and Limitations

While the results demonstrate the effectiveness of a proactive, structured interview style for AI-driven health assessments, the authors note a key limitation: the study relied on self-reported ground truth for diagnoses. Despite this, the research provides a clear demonstration that a dedicated, complete symptom interview process is superior to the passive, user-guided discussions currently common in many AI health applications.

Comments (0)

No comments yet

Be the first to share your thoughts!