Advertisement

Why validation of prognostic models matters?

  • Alex Zwanenburg
    Affiliations
    OncoRay – National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden – Rossendorf, Germany

    National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany; Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany, and; Helmholtz Association/Helmholtz-Zentrum Dresden - Rossendorf (HZDR), Dresden, Germany

    German Cancer Consortium (DKTK), Partner Site Dresden, and German Cancer Research Center (DKFZ), Heidelberg, Germany
    Search for articles by this author
  • Steffen Löck
    Correspondence
    Corresponding author at: OncoRay – National Center for Radiation Research in Oncology, Medical Faculty Carl Gustav Carus, Technische Universität Dresden, Fetscherstr. 74, 01307 Dresden, Germany.
    Affiliations
    OncoRay – National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden – Rossendorf, Germany

    German Cancer Consortium (DKTK), Partner Site Dresden, and German Cancer Research Center (DKFZ), Heidelberg, Germany

    Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Germany
    Search for articles by this author

      Summary

      Prognostic models are powerful tools for treatment personalisation. However, not all proposed models work well when validated using new data, despite impressive results being reported initially. Here, we will use a hands-on approach to highlight important aspects of prognostic modelling, as well as to demonstrate methods to generate generalisable models.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Radiotherapy and Oncology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Baumann M.
        • Krause M.
        • Overgaard J.
        • Debus J.
        • Bentzen S.M.
        • Daartz J.
        • et al.
        Radiation oncology in the era of precision medicine.
        Nat Rev Cancer. 2016; 16: 234-249
        • Collins G.S.
        • Reitsma J.B.
        • Altman D.G.
        • Moons K.G.M.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement.
        Ann Intern Med. 2015; 162: 55
        • Moons K.G.M.
        • Altman D.G.
        • Reitsma J.B.
        • Ioannidis J.P.A.
        • Macaskill P.
        • Steyerberg E.W.
        • et al.
        Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and Elaboration.
        Ann Intern Med. 2015; 162: W1
        • Cancer Genome Atlas Network
        Comprehensive genomic characterization of head and neck squamous cell carcinomas.
        Nature. 2015; 517: 576-582
        • Cerami E.
        • Gao J.
        • Dogrusoz U.
        • Gross B.E.
        • Sumer S.O.
        • Aksoy B.A.
        • et al.
        The cBio cancer genomics portal: an open platform for exploring multidimensional cancer genomics data.
        Cancer Discov. 2012; 2: 401-404
        • Gao J.
        • Aksoy B.A.
        • Dogrusoz U.
        • Dresdner G.
        • Gross B.
        • Sumer S.O.
        • et al.
        Integrative analysis of complex cancer genomics and clinical profiles using the cBioPortal.
        Sci Signal. 2013; 6: l1
        • Harrell F.E.
        • Lee K.L.
        • Mark D.B.
        Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
        Stat Med. 1996; 15: 361-387
        • Pencina M.J.
        • D’Agostino R.B.
        Overall C as a measure of discrimination in survival analysis: model specific population value and confidence interval estimation.
        Stat Med. 2004; 23: 2109-2123
        • Tibshirani R.
        Regression selection and shrinkage via the lasso.
        J R Stat Soc B. 1996; 58: 267-288
        • Friedman J.
        • Hastie T.
        • Tibshirani R.
        Regularization Paths for Generalized Linear Models via Coordinate Descent.
        J Stat Softw. 2010; 33: 1-22
        • Hutter F.
        • Hoos H.H.
        • Leyton-Brown K.
        Sequential model-based optimization for general algorithm configuration.
        Lect Notes Comput Sci. 2011; 6683 (LNCS): 507-523
        • Kohavi R.
        A study of cross-validation and bootstrap for accuracy estimation and model selection.
        Int Jt Conf Artif Intell. 1995; 14: 1137-1143
        • Parker B.J.
        • Günter S.
        • Bedo J.
        Stratification bias in low signal microarray studies.
        BMC Bioinform. 2007; 8: 326
        • Lever J.
        • Krzywinski M.
        • Altman N.
        Points of significance: model selection and overfitting.
        Nat Methods. 2016; 13: 703-704
        • Dupuy A.
        • Simon R.M.
        Critical review of published microarray studies for cancer outcome and guidelines on statistical analysis and reporting.
        J Natl Cancer Inst. 2007; 99: 147-157
        • Hastie T.
        • Tibshirani R.
        • Friedman J.
        The elements of statistical learning: data mining, inference, and prediction.
        2nd ed. Springer Science+Business Media, LLC, New York, New York, USA2009
        • Babyak M.A.
        What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models.
        Psychosom Med. 2004; 66: 411-421
        • Chalkidou A.
        • O’Doherty M.J.
        • Marsden P.K.
        False discovery rates in PET and CT studies with texture features: a systematic review.
        PLoS One. 2015; 10: e0124165