Skip to main content
Log in

Performance of automatic machine learning versus radiologists in the evaluation of endometrium on computed tomography

  • Pelvis
  • Published:
Abdominal Radiology Aims and scope Submit manuscript

Abstract

Purpose

In this study, we developed radiomic models that utilize a combination of imaging features and clinical variables to distinguish endometrial cancer (EC) from normal endometrium on routine computed tomography (CT).

Methods

A total of 926 patients consisting of 416 endometrial cancer (EC) and 510 normal endometrium were included. The CT images of these patients were segmented manually, and divided into training, validation, testing and external testing sets. Non-texture and texture features of these images with endometrium or uterus as region of interest were extracted. The clinical feature “age” was also included in the feature set. Feature selection and machine learning classifier were applied to normalized feature set. This manual optimized combination was then compared with the best pipeline exported by Tree-Based Pipeline Optimization Tool (TPOT) on testing and external testing set. The performances of these machine learning pipelines were compared to that of radiologists.

Results

The manual expert optimized pipeline using the “reliefF” feature selection method and “Bagging” classifier on the external testing set achieved a test ROC AUC of 0.73, accuracy of 0.73 (95% CI 0.62–0.82), sensitivity of 0.64 (95% CI 0.45–0.79), and specificity of 0.78 (95% CI 0.65–0.87), while TPOT achieved a test ROC AUC of 0.79, accuracy of 0.80 (95% CI 0.70–0.87), sensitivity of 0.61 (95% CI 0.43–0.77), and specificity of 0.90 (95% CI 0.78–0.96). When compared to average radiologist performance, the TPOT achieved higher test accuracy (0.80 vs. 0.49, p < 0.001) and specificity (0.90 vs. 0.51, p < 0.001), with comparable sensitivity (0.61 vs. 0.46, p = 0.130).

Conclusion

Our results demonstrate that automatic machine learning can distinguish EC from normal endometrium on routine CT imaging with higher accuracy and specificity than radiologists.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Code availability

This code is available at https://github.com/HurongCSU/Endoproj.

Data availability

The images and clinical data are not available for public use to protect patient information.

References

  1. Siegel R.L., Miller K.D., Jemal A (2017) Cancer Statistics. CA Cancer J Clin. 67(1), 7-30 (2017).

    Article  Google Scholar 

  2. Moore K., Brewer M.A. Endometrial Cancer: Is This a New Disease?, Am Soc Clin Oncol Educ Book. 37, 435–442.

  3. Lee YC, Lheureux S, Oza AM (2017) Treatment strategies for endometrial cancer: current practice and perspective, Curr Opin Obstet Gynecol. 29(1), 47-58.

    Article  Google Scholar 

  4. Barwick T.D, Rockall AG, Barton DP, Sohaib SA (2006) Imaging of endometrial adenocarcinoma. Clinical Radiology, 61(7), 545-555.

    Article  CAS  Google Scholar 

  5. Coussoou C, Laigle-Quérat V, Loussouarn D, Vaucel E, Frampas E (2020) Magnetic Resonance Imaging for local preoperative staging in endometrial cancer: Nantes local experience, Gynecol Obstet Fertil Senol.

  6. Hardesty L.A, Sumkin JH, Hakim C, Johns C, Nath M (2001) The Ability of Helical CT to Preoperatively Stage Endometrial Carcinoma, American Journal of Roentgenology. 176(3), 603-606.

    Article  CAS  Google Scholar 

  7. Braun M.M., Overbeek-Wager E.A., Grumbo R.J (2016) Diagnosis and Management of Endometrial Cancer, Am Fam Physician. 93(6), 468-74.

    PubMed  Google Scholar 

  8. Rizzo S, et al (2018). Radiomics: the facts and the challenges of image analysis, European Radiology Experimental. 2(1), 36.

    Article  Google Scholar 

  9. Gillies R.J., Kinahan P.E., Hricak H (2016) Radiomics: Images Are More than Pictures, They Are Data, Radiology. 278(2), 563-577.

    Google Scholar 

  10. Parmar C, Grossmann P, Bussink J, Lambin P, Aerts HJWL (2015) Machine Learning methods for Quantitative Radiomic Biomarkers, Sci Rep. 5, 13087.

    Article  CAS  Google Scholar 

  11. Tongtong Liu, et al (2017) A mRMRMSRC feature selection method for radiomics approach, Conf Proc IEEE Eng Med Biol Soc. 616–619.

  12. Mayerhoefer ME, et al (2020) Introduction to Radiomics, J Nucl Med.

  13. Sidey-Gibbons, J.A.M., Sidey-Gibbons C.J (2019) Machine learning in medicine: a practical introduction, BMC Medical Research Methodology. 19(1), 64.

    Article  Google Scholar 

  14. Günakan E, et al (2019) A novel prediction method for lymph node involvement in endometrial cancer: machine learning, Int J Gynecol Cancer. 29(2), 320-324.

    Article  Google Scholar 

  15. Olson R.S., et al (2016) Automating Biomedical Data Science Through Tree-Based Pipeline Optimization, Lecture Notes in Computer Science. vol 9597.

  16. Le T.T., Fu W, Moore J.H (2020) Scaling tree-based automated machine learning to biomedical big data with a feature set selector, Bioinformatics. 36(1), 250-256.

    Article  CAS  Google Scholar 

  17. Su X, et al (2019) Automated Machine Learning Based on Radiomics Features Predicts H3 K27M Mutation in Midline Gliomas of the Brain, Neuro Oncol.

  18. Orlenko A, et al (2019) Model selection for metabolomics: predicting diagnosis of coronary artery disease using automated machine learning (AutoML), Bioinformatics.

  19. Andriy Fedorov, et al (2012) 3D Slicer as an image computing platform for the Quantitative Imaging Network, Magnetic Resonance Imaging. 30(9), 1323-1341.

    Article  Google Scholar 

  20. Zwanenburg A, L.S., Vallières M, Löck S (2016) Image biomarker standardisation initiative, arXiv preprint. arXiv:161207003.

  21. Zwanenburg A, et al (2020) The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping, Radiology. 10, 191145.

  22. Jundong Li, et al (2018) Feature selection: A data perspective, ACM Computing Surveys (CSUR). 50(6), 94.

    Article  Google Scholar 

  23. Pedregosa, et al (2011) Scikit-learn: Machine Learning in Python. JMLR. 12, 2825-2830.

    Google Scholar 

  24. Alan Agresti, Brent A. Coull (1998) Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions, The American Statistician. 52(2), 119-126.

    Google Scholar 

  25. Vallières, M., et al (2017). Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer, Scientific Reports. 7(1), 10117.

    Article  Google Scholar 

  26. Vallières, M, Freeman CR, Skamene SR, El Naqa I (2015) A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities, Phys Med Biol. 60(14), 5471-96.

    Article  Google Scholar 

  27. Bi Q, et al (2019). The Diagnostic Value of MRI for Preoperative Staging in Patients with Endometrial Cancer: A Meta-Analysis, Acad Radiol.

  28. Alcazar J.L, et al (2015) Transvaginal ultrasound for preoperative assessment of myometrial invasion in patients with endometrial cancer: a systematic review and meta-analysis. Ultrasound Obstet Gynecol. 46(4), 405-13.

    Article  CAS  Google Scholar 

  29. Alcázar JL, et al (2019) Diagnostic Performance of Transvaginal Ultrasound for Detecting Cervical Invasion In Women With Endometrial Carcinoma: A Systematic Review and Meta-analysis, J Ultrasound Med. 38(1), 179-189.

    Article  Google Scholar 

  30. Bollineni VR, Ytre-Hauge S, Bollineni-Balabay O, Salvesen HB, Haldorsen IS (2016) High Diagnostic Value of 18F-FDG PET/CT in Endometrial Cancer: Systematic Review and Meta-Analysis of the Literature, J Nucl Med. 57(6), 879-85.

    Article  CAS  Google Scholar 

  31. Deng L, et al (2015) The Combination of Diffusion- and T2-Weighted Imaging in Predicting Deep Myometrial Invasion of Endometrial Cancer: A Systematic Review and Meta-Analysis, J Comput Assist Tomogr. 39(5), 661-73.

    Article  Google Scholar 

  32. Alcázar JL, et al (2017) Transvaginal ultrasound versus magnetic resonance imaging for preoperative assessment of myometrial infiltration in patients with endometrial cancer: a systematic review and meta-analysis, J Gynecol Oncol. 28(6), e86.

  33. Luomaranta A, Leminen A, Loukovaara M (2015) Magnetic resonance imaging in the assessment of high-risk features of endometrial carcinoma: a meta-analysis, Int J Gynecol Cancer. 25(5), 837-42.

    Article  Google Scholar 

  34. Xu X, et al (2019) Multiplanar MRI-Based Predictive Model for Preoperative Assessment of Lymph Node Metastasis in Endometrial Cancer, Front Oncol. 9, 1007.

    Article  Google Scholar 

  35. Xie H, et al (2019) Preliminary utilization of radiomics in differentiating uterine sarcoma from atypical leiomyoma: Comparison on diagnostic efficacy of MRI features and radiomic features, Eur J Radiol. 115, 39-45.

    Article  Google Scholar 

  36. De Bernardi E, et al (2018) Radiomics of the primary tumour as a tool to improve 18F-FDG-PET sensitivity in detecting nodal metastases in endometrial cancer, EJNMMI Res. 8(1), 86.

    Article  Google Scholar 

  37. Neacşu A, et al (2018) Clinical and morphological correlations in early diagnosis of endometrial cancer, Rom J Morphol Embryol. 59(2), 527-531.

    PubMed  Google Scholar 

  38. Colombo N, et al (2016) ESMO-ESGO-ESTRO Consensus Conference on Endometrial Cancer: diagnosis, treatment and follow-up. Ann Oncol. 27(1), 16–41.

  39. Gans SL, Pols MA, Stoker J, Boermeester MA, expert steering group (2015) Guideline for the diagnostic pathway in patients with acute abdominal pain, Dig Surg. 32(1), 23-31.

    Article  Google Scholar 

  40. Paolantonio P, Rengo M, Ferrari R, Laghi A (2016) Multidetector CT in emergency radiology: acute and generalized non-traumatic abdominal pain, Br J Radiol. 89(1061), 20150859.

    Article  Google Scholar 

  41. Karia M, Seager M, Rafique A, Sheth H (2017) The Diagnostic Utility and Clinical Impact of After-Hours CT Scans of the Abdomen and Pelvis Investigating Abdominal Pain, ScientificWorldJournal. 2017, 4028352.

    Article  Google Scholar 

  42. Haldorsen I.S, Salvesen H.B (2016) What Is the Best Preoperative Imaging for Endometrial Cancer?, Current oncology reports. 18(4), 25-25.

    Article  Google Scholar 

  43. Lin M.Y, Dobrotwir A, McNally O, Abu-Rustum NR, Narayan K (2018) Role of imaging in the routine management of endometrial cancer, Int J Gynaecol Obstet. 143 Suppl 2(Suppl 2), 109–117.

  44. Trojano G, Olivieri C, Tinelli R, Damiani GR, Pellegrino A, Cicinelli E (2019) Conservative treatment in early stage endometrial cancer: a review. Acta Biomed. 90(4), 405-410.

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Gressel G.M., Parkash V., L. Pal (2015) Management options and fertility-preserving therapy for premenopausal endometrial hyperplasia and early-stage endometrial cancer, Int J Gynaecol Obstet. 131(3), 234-9.

    Article  Google Scholar 

  46. Rizzo S, et al (2018) Endometrial cancer: an overview of novelties in treatment and related imaging keypoints for local staging, Cancer Imaging. 18(1), 45.

    Article  Google Scholar 

  47. Matteson, K.A., K. Robison, V.L. Jacoby (2018) Opportunities for Early Detection of Endometrial Cancer in Women with Postmenopausal Bleeding, JAMA Intern Med. 178(9), 1222-1223.

    Article  Google Scholar 

  48. Banzhaf W, Nordin P, Keller RE, Francone FD (1998) Genetic programming: an introduction: on the automatic evolution of computer programs and its applications, Morgan Kaufmann Publishers Inc. 470.

  49. Orlenko A, et al (2018) Considerations for automated machine learning in clinical metabolic profiling: Altered homocysteine plasma concentration associated with metformin exposure, Pacific Symposium on Biocomputing. 23, 460-471.

    PubMed  Google Scholar 

  50. Ken Chang, et al (2019) Automatic assessment of glioma burden: a deep learning algorithm for fully automated volumetric and bidimensional measurement, Neuro-Oncology. 21(11), 1412–1422.

    Article  Google Scholar 

  51. Cignini P, et al (2017) Preoperative work-up for definition of lymph node risk involvement in early stage endometrial cancer: 5-year follow-up, Updates Surg. 69(1), 75-82.

    Article  Google Scholar 

  52. Ortoft G, et al (2013) Preoperative staging of endometrial cancer using TVS, MRI, and hysteroscopy, Acta Obstet Gynecol Scand. 92(5), 536-45.

    Article  Google Scholar 

Download references

Funding

This work is supported by the National Key Research and Development Program of China (No. 2018AAA0102100), the Scientific and Technological Innovation Leading Plan of High-tech Industry of Hunan Province (2020GK2021), the National Natural Science Foundation of China (No. 61902434) and the 111 project under grant (No. B18059).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chengzhang Zhu.

Ethics declarations

Conflict of interest

The authors declare no potential conflicts of interest.

Ethical approval

The study was approved by the Institutional Review Boards from respective institutions.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 95 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, D., Hu, R., Li, H. et al. Performance of automatic machine learning versus radiologists in the evaluation of endometrium on computed tomography. Abdom Radiol 46, 5316–5324 (2021). https://doi.org/10.1007/s00261-021-03210-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00261-021-03210-9

Keywords

Navigation