Thyroid Biology and Disease

SUN-378 - Assessing Large Language Models for Generating Lay-Language Summaries of Thyroid Cancer

Sunday, June 14, 2026

9:00 AM - 4:00 PM CT

Location: ENDOExpo; Poster Floor

Abstract Poster Presenter(s)

Ana Gabriela Claros, MD

Visiting Research Fellow
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States

Co-Author(s)

ML

Maria Lizarazo Jimenez, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States

Senior Author/Primary Investigator(s)

David Toro Tobon, MD (he/him/his)

Principal Investigator
Mayo Clinic
Rochester, Minnesota, United States

Co-Author(s)

David Toro Tobon, MD (he/him/his)

Principal Investigator
Mayo Clinic
Rochester, Minnesota, United States
Felipe Larios Madrid, MD

Research Fellow
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Lima, Peru
MA

Misk Al Zahidy, MS

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
LV

Luis Vilatuna Andrango, MD

Research Trainee
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States
IA

Ivan Mateo Alzamora, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States
MM

Marcelo Montero, MD

Research Trainee
Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, Minnesota, United States
EG

Esteban Gomez Alvarez, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, United States

Senior Author/Primary Investigator(s)

OP

Oscar J Ponce-Ponte, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Plymouth, United Kingdom

Co-Author(s)

OP

Oscar J Ponce-Ponte, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Plymouth, United Kingdom

Senior Author/Primary Investigator(s)

Juan P. Brito, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States

Co-Author(s)

Juan P. Brito, MD

Care and AI Laboratory, Knowledge and Evaluation Research Unit, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic
Rochester, MN, United States

Disclosure(s):

Ana Gabriela Claros, MD: No financial relationships to disclose

David Toro Tobon, MD: Immunovant Inc: Consultant ()

Felipe Larios Madrid, MD: No financial relationships to disclose

Background
In the U.S. patients often access pathology reports without clinician review. As these reports are highly technical, lay-language summaries (LLS) are essential to improve understanding, reduce anxiety, and support shared decision-making. This study evaluates a state-of-the-art large language model’s (LLM) ability to generate LLS from thyroid cancer pathology reports.

Methods
This cross-sectional study compared Gemini-2.5-pro LLS against human-annotated gold standard LLS. Four medical experts and seven thyroid specialists developed the guidelines for gold-standard LLS generation for papillary, follicular, and medullary thyroid cancer using a standardized, question-based format covering surgical findings, diagnosis, tumor characteristics, staging, and estimated recurrence risk for a lay audience. Using pathology reports from the Mayo Clinic Network, a total of 150 gold-standard LLS were generated (50 papillary, 50 follicular, 50 medullary). Four prompting strategies were evaluated using a progressively constrained in-context learning approach, in which prompt instructions were incrementally expanded to include structured output requirements, subsection-level guidance, and estimation of the American Thyroid Association (ATA) risk of recurrence. Prompts were refined using 10% of the gold-standard summaries (five per cancer type). The remaining 90% were used to evaluate performance in semantic similarity (BERTScore-F1), structural overlap (Rouge-L), readability (Flesch-Kincaid Grade Level or FKGL), and factual accuracy in diagnosis, tumor size, and recurrence risk estimation.

Results
The best-performing prompting strategy included the most detailed instructions regarding subsection guidance, and methodology for generating ATA recurrence risk estimates. Gemini-generated LLS showed low semantic similarity (0.29 ±0.09) and structural overlap (0.33 ±0.01) compared with gold-standard LLS. These low scores were primarily driven by the inclusion of additional unimportant pathological details and unnecessary clinical explanations. However, Gemini-generated LLS achieved more optimal readability (FKGL 6.47±0.37) than gold-standard summaries (FKGL 7.06 ±0.41). Accuracy reached 98% for diagnosis, 96% for tumor size, and 86% for two-subtype recurrence risk, with errors concentrated in cases of lymph node involvement or multiple tumor types.

Conclusions
Gemini-2.5-pro can generate readable and accurate LLS of thyroid cancer pathology reports. Differences from gold-standards reflect added explanation rather than errors. Careful prompting is required to achieve acceptable performance. Further work is needed to develop LLMs capable of generating high-quality LLS without human involvement and to evaluate their clinical impact.

*Unless otherwise noted, all abstracts presented at ENDO must not be released to the press or the public until the date and time of presentation. For oral presentations, the abstracts are embargoed until the session begins. The Endocrine Society reserves the right to lift the embargo on specific abstracts that are selected for promotion prior to or during ENDO.*