Thyroid Biology and Disease

SAT-406 - Thyroidbx: Automated Bethesda Classification of Thyroid Fine-Needle Aspiration Biopsy Reports using Specialized Small Large Language Models

Saturday, June 13, 2026

9:30 AM - 4:30 PM CT

Location: ENDOExpo; Poster Floor

Abstract Poster Presenter(s)

Felipe Larios Madrid, MD

Research Fellow
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Lima, Peru

Co-Author(s)

David Toro Tobon, MD (he/him/his)

Principal Investigator
Mayo Clinic
Rochester, Minnesota, United States
OP

Oscar Ponce-Ponte, MD, MSc

Derriford Hospital
Plymouth, United Kingdom
Naykky Maruquel Singh Ospina, MD

University of Florida
Gainesville, FL, United States
KG

Kerly Guevara maldonado, MD

Research Fellow
Mayo Clinic
Rochester, Minnesota, United States
LV

Luis Vilatuna Andrango, MD

Research Trainee
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States
AC

Ana Gabriela Claros, MD

Mayo Clinic
Rochester, Minnesota, United States
ML

Maria Lizarazo Jimenez, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
EG

Esteban Gomez Alvarez, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
MM

Marcelo Montero, MD

Research Trainee
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, Minnesota, United States
IA

Ivan Mateo Alzamora, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
RL

Ricardo Loor-Torres, MD

Research Collaborator
Ascension Saint Joseph
Chicago, Illinois, United States
CS

Cristian Soto Jacome, MD

Mayo Clinic
Rochester, United States
YW

Yuqi Wu, PhD

Mayo Clinic
Rochester, United States
MA

Misk Al Zahidy, MS

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
JF

Jungwei W. Fan, PhD

Mayo Clinic
Rochester, United States
MB

Megan E. Branda, MS

Principal Biostatistician
Mayo Clinic
Rochester, Minnesota, United States

Senior Author/Primary Investigator(s)

Juan P. Brito, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States

Disclosure(s):

Felipe Larios Madrid, MD: No financial relationships to disclose

David Toro Tobon, MD: Immunovant Inc: Consultant ()

Background
The Bethesda System standardizes thyroid cytopathology terminology, yet real world, large-scale research in clinical outcomes is limited by the unstructured narrative format of FNAB reports and manual data extraction, which is labor-intensive, costly, and error prone. Large language models (LLMs) may offer scalable automation of data extraction, but generic cloud-based models often raise privacy and accuracy concerns in specialized domains. We present ThyroidBx, an AI tool developed to identify Bethesda categories and cytologic subtypes in FNAB reports to support clinical quality control and research.
Methods
We evaluated three small, privacy-preserving, open-weight LLMs (Mistral-7B, Qwen-2.5-7B, Llama-3.1-8B) against a proprietary LLM (Gemini-2.5-Flash). The dataset comprised 444 de-identified cytology reports (Jan 2010–Feb 2023) from four Mayo Clinic sites, balanced across all six Bethesda categories and subtypes. A Prompt Development Set (n=46) was utilized for expert rule refinement and output calibration, then incorporated into the Training Set (n=383) used for Supervised Fine-Tuning (SFT) through Low-Rank Adaptation. We compared progressive prompting strategies including Zero-Shot, Expert-Rules (injecting definitions and tie-breaking logic), and 3-Shot, against SFT. Gemini-2.5-Flash was evaluated using Many-Shot Prompting (48 exemplars) to test context-window scaling. Performance was assessed on a strictly sequestered Held-Out Test Set (n=61). Classification performance was evaluated with accuracy and F1-score.
Results
Prompt engineering strategies (Zero-Shot, Expert-Rules, 3-Shot) yielded variable accuracy (45.9%–90.2%), failing to consistently surpass the baseline or handle complex cases. Notably, these generic approaches frequently misclassified clinically critical indeterminate nodules (Bethesda III and IV) and prompting yielded lower subtype classification accuracy (68.9%–75.4%). Even the massive Gemini-2.5-Flash (Many-Shot) peaked at 77.0% accuracy. In contrast, SFT significantly corrected these errors. The specialized Mistral-7B model reached 98.4% accuracy (F1 0.99) and Qwen-2.5 reached 96.7% (F1 0.97). SFT models demonstrated superior subtype classification (85.2%–86.9%), enabling reliable separation of relevant subtypes, particularly within the indeterminate categories where clinical management diverges.
Conclusion
Domain SFT of small LLMs outperforms Gemini-2.5-Flash, one of the largest general-purpose LLMs, in interpreting FNAB reports. These findings suggest that local deployment of specialized small LLMs could be an efficient, accurate, privacy-preserving strategy for modernizing cancer registries. This approach is particularly valuable in settings where reports are non-standardized, supporting automated clinical interpretation and large-scale epidemiological research.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*