Self-Optimizing Prompts for Clinical Symptom Detection

Many of the findings that matter most for clinical AI appear only in the clinical note. Dyspnea, cognitive complaints, joint pain, and night sweats are documented in narrative text far more often than in structured fields, so any model that depends on them must first read the notes.

Each of the three conventional methods for this task carries a cost. Curated lexicons are fast, yet they flag each mention of a term, including “denies chest pain” and “mother had a stroke.” Fine-tuned language models need labeled examples for each concept and degrade on rare findings. Manual prompt engineering for large language models performs well, although someone must write and tune a prompt for each of dozens of concepts. In a new preprint, we describe a multi-agent system, released as the open-source engine Pythia, that writes, tests, and revises its own prompts with no fine-tuning and no manual prompt engineering.

How the system works

For each sign or symptom, an evaluation step scores the current prompt against a development set of annotated notes. A specialist agent then analyzes the errors in aggregate, an improver agent drafts and ranks revised prompts, and a controller decides whether to continue, return to the best earlier version, or stop. The loop selects prompts by sensitivity and specificity, the two operating characteristics clinicians use to judge a screening test. Because the system runs on a locally hosted open-weights model, clinical notes stay on institutional infrastructure.

Evaluation

We compared the system with a curated lexicon across 72 signs and symptoms in 400 clinical notes from 387 patients. For each concept, we split the notes independently into a development set of 300 notes and a validation set of 100 notes that played no part in optimization.

Results

Mean specificity was 0.95 for the agentic system and 0.76 for the lexicon, while mean sensitivity was 0.76 and 0.82, respectively. The difference in specificity was largest for 14 concepts that the lexicon marked positive in all notes. On those concepts, the agentic system reached a mean specificity of 0.97, since the optimized prompts required a current finding attributed to the patient. Among 62 concepts with a direct comparison, the system matched or exceeded the lexicon on both sensitivity and specificity for 20.

Specificity carried over intact from the development notes to the validation notes. Sensitivity fell for rare concepts, and below 2% prevalence the mean gap between development and validation sensitivity reached 0.25. A BERT classifier fine-tuned for each concept on the development set reached a mean sensitivity of 0.23, falling to zero for concepts with prevalence below about 5%.

Implications

Autonomous prompt optimization gives clinical teams a third option between lexicons and fine-tuned models. The system reads clinical context on local hardware and scales to dozens of concepts with no dedicated prompt engineer. Rare findings remain its main limitation, since a method that learns from examples has few to learn from when a finding appears in a handful of notes. We characterized one consequence of this limitation earlier this year, when we showed that self-improving agents can degrade their own performance under class imbalance. The system extends our agentic workflow for cognitive concerns and now extracts note-derived features for our studies of cardiovascular risk and dementia.

Read the preprint

Cagan C, Fard P, Tian J, Cheng J, Murphy SN, Estiri H. A Multi-Agent System for Autonomous, Fine-Tuning-Free Clinical Symptom Detection: Development and Validation Study. arXiv. 2026. arXiv:2607.12886