Molecular Testing for Indeterminate Thyroid Nodules—When the Rubber Meets the Road

Jennifer L. Marti, MD¹; Ashok R. Shaha, MD²

Author Affiliations Article Information

• ¹Department of Surgery, Endocrine & Breast Surgery, Weill Cornell Medical Center, New York, New York
• ²Head and Neck Service, Department of Surgery, Memorial Sloan Kettering Cancer Center, New York, New York

JAMA Otolaryngol Head Neck Surg. Published online July 18, 2019. doi:10.1001/jamaoto.2019.1465

related articles icon 

Related

Articles

In this issue of JAMA Otolaryngology Head & Neck Surgery, Valderrabano et al¹ report provocative findings on the performance of a widely used molecular test for thyroid nodules. The authors’ goal was to determine whether the performance of this assay, the Afirma gene expression classifier (GEC; Veracyte), in the real world has matched the figures in the initial validation study that led to the approval and widespread adoption of the GEC. Undertaking a systematic review of 19 studies encompassing 2568 nodules, the authors found that the sensitivity and specificity of the test seemed to differ markedly from those reported in the initial study.²

Cytopathological analysis of thyroid fine-needle aspiration specimens is informative for most patients, but for the 15% to 30% of nodules with indeterminate cytological results, the prevalence of malignancy in these nodules varies across different populations and practice settings. It has been unclear how to manage these patients most appropriately, many of whom were traditionally subjected to thyroidectomy for diagnostic purposes. The development of the Afirma GEC, a gene expression–based test, seemed to be the answer many surgeons and endocrinologists were hoping for. With a negative predictive value (NPV) reported to be 94% to 95%, many patients with GEC-negative nodules could avoid an operation.² The reported positive predictive value (PPV) of 38% seemed reasonably high enough to justify operating on GEC-positive (so-called suspicious) nodules.²

However, after the introduction of the Afirma GEC, investigators at several institutions reported that performance of the test seemed to vary widely across institutions. The PPV rates of the GEC were reported to be lower than expected, as low as 17% to 30%.³ It became evident that clinicians who used the test needed to know the prevalence of malignancy in indeterminate nodules in their practice setting, but collecting these numbers may not always be easily achievable. Despite this challenge, the reportedly high NPV was unquestioned, and most GEC-negative (so-called benign) nodules have been observed, with the reassurance of the statistics quoted in the test report. The findings of Valderrabano et al¹ now call into question the reliability of the NPV, which they estimated to be at 88%.

The authors used a novel statistical technique, evaluating performance by analyzing the correlation of PPV and benign call rate across the studies in comparison with the initial validation study. These analyses found that the test’s performance was different from what would be expected according to the initial study. These data raise important questions about the reliability of the seemingly definitive NPV and PPV numbers that appear on the Afirma GEC reports sent to clinicians.

Ultimately, this discrepancy illuminates the difference between efficacy (the performance of a diagnostic test under ideal and controlled circumstances) and effectiveness (the performance in real-world conditions). Often, effectiveness underperforms efficacy because patients and physicians participating in initial prospective trials do not always perfectly represent the patients and physicians who end up using a drug, device, or diagnostic test. This dissimilarity is one of the reasons that postmarketing surveillance is often required of drugs, devices, and some diagnostic tests after approval.

Why would the NPV and PPV vary by institution and not be similar everywhere? Some factors include the subjectivity of the cytological and histological, pathological interpretation of thyroid fine-needle aspiration as well as the varying prevalence of malignancy among institutions and practice settings. In addition, PPV values can be erroneously increased, and NPVs decreased, if incidental small papillary thyroid carcinomas separate from the biopsied nodule, or if NIFTPs (noninvasive follicular thyroid neoplasms with papillary-like nuclear features), which are not cancer, are counted as true-positives.

These findings underscore the value of more rigorous evaluation of new diagnostic tests. Larger, multicenter pivotal studies prior to approval would be helpful, as would more real-world analyses conducted independently of industry sponsorship. As data emerge, requiring modifications to the language used in clinical reports may be appropriate, to ensure that patients and clinicians interpret the results correctly.

It is possible that newer versions of gene expression–based testing may perform better. Historically, the Afirma GEC classified many benign oncocytic and Hurthle cell lesions as suspicious, which may now be improved with the newer Afirma genomic sequencing classifier (Veracyte), which has been reported to have higher specificity and similar sensitivity.^4,5 As the genomic sequencing classifier is more widely used, independent analyses will have similar value in evaluating this test in real-world settings.

With an understanding that the NPV of a GEC-benign result and the PPV of a GEC-suspicious result may not always match the number cited on the test report, clinicians may want to keep in mind the substantial costs of these assays to health systems, payers, and patients. These tests should not be sent reflexively. With increasing standardization of radiographic and cytological classifications, risk stratification based on clinical examination, ultrasonography characteristics, and cytological features can now be more informed than in years past. Before ordering a molecular assay for a thyroid nodule, clinicians should consider the patient’s age and comorbidities, the duration and stability of the thyroid nodule, the nodule size, and imaging characteristics. The increasingly adopted TI-RADS (Thyroid Imaging Reporting and Data System) can be helpful in assessing the risk of malignancy. Details of the cytological results under the Bethesda classification, and consideration of a repeat fine-needle aspiration biopsy after an interval, may also assist in decision-making for some indeterminate thyroid nodules.⁶

When should a clinician consider a molecular test for an indeterminate thyroid nodule? If the patient is not going to have surgery (eg, the nodule is subcentimeter and should not have undergone biopsy in the first place, or the patient is not a surgical candidate owing to medical comorbidities), do not order it. If the patient will be recommended an operation for other indications (eg, a young patient with a >4 cm enlarging indeterminate nodule, or a patient who is symptomatic with compressive goiter or hyperthyroidism), do not order it. The test should be used selectively when it will potentially alter management, and it should not be used to guide the extent of the surgical procedure. However, if sent, the GEC results need to be interpreted cautiously, as the PPV and NPV will vary by institution and may be lower than reported. One should consider all of these factors and should order and interpret the test results with a critical eye.

We congratulate Valderrabano and colleagues¹ for a comprehensive analysis of the literature assessing the varying performance of the Afirma GEC. Their findings demonstrate the difference between efficacy and effectiveness as well as the value of a body of research generated by physicians studying their own patients.

Back to top

Article Information

Corresponding Author: Jennifer L. Marti, MD, Department of Surgery, Endocrine & Breast Surgery, Weill Cornell Medicine, 420 E 70th St, 2nd Floor, New York, NY 10021 (jem9080@med.cornell.edu).

Published Online: July 18, 2019. doi:10.1001/jamaoto.2019.1465

Conflict of Interest Disclosures: None reported.

References

References

1.

Valderrabano  P, Hallanger-Johnson  JE, Thapa  R, Wang  X, McIver  B.  Comparison of postmarketing findings vs the clinical validation findings of a thyroid nodule gene expression classifier: a systematic review and meta-analysis  [published online July 18, 2019].  JAMA Otolaryngol Head Neck Surg. doi:10.1001/jamaoto.2019.1449
ArticleGoogle Scholar

2.

Alexander  EK, Kennedy  GC, Baloch  ZW,  et al.  Preoperative diagnosis of benign thyroid nodules with indeterminate cytology.  N Engl J Med. 2012;367(8):705-715. doi:10.1056/NEJMoa1203208PubMedGoogle ScholarCrossref

3.

Marti  JL, Avadhani  V, Donatelli  LA,  et al.  Wide inter-institutional variation in performance of a molecular classifier for indeterminate thyroid nodules.  Ann Surg Oncol. 2015;22(12):3996-4001. doi:10.1245/s10434-015-4486-3PubMedGoogle ScholarCrossref

4.

Harrell  RM, Eyerly-Webb  SA, Golding  AC, Edwards  CM, Bimston  DN.  Statistical comparison of Afirma GSC and Afirma GEC outcomes in a community endocrine surgical practice: early findings.  Endocr Pract. 2019;25(2):161-164. doi:10.4158/EP-2018-0395PubMedGoogle ScholarCrossref

5.

Patel  KN, Angell  TE, Babiarz  J,  et al.  Performance of a genomic sequencing classifier for the preoperative diagnosis of cytologically indeterminate thyroid nodules.  JAMA Surg. 2018;153(9):817-824. doi:10.1001/jamasurg.2018.1153
ArticlePubMedGoogle ScholarCrossref

6.

Saieg  MA, Barbosa  B, Nishi  J, Ferrari  A, Costa  F.  The impact of repeat FNA in non-diagnostic and indeterminate thyroid nodules: a 5-year single-centre experience.  Cytopathology. 2018;29(2):196-200. doi:10.1111/cyt.12508PubMedGoogle ScholarCrossref