Synthetic MRI-aided multi-organ segmentation on male pelvic CT using cycle consistent deep attention network

Published:October 17, 2019DOI:


      • We use CT-based synthetic MRI (SMRI) to aid multi-organ segmentation in male pelvic CT.
      • SMRI with a superior soft-tissue contrast can significantly improve CT segmentation accuracy.
      • The deep attention mechanism helps identify the most relevant features to differentiate organs.
      • The proposed SMRI-aided strategy has a great potential to facilitate radiotherapy planning.


      Background and purpose

      Manual contouring is labor intensive, and subject to variations in operator knowledge, experience and technique. This work aims to develop an automated computed tomography (CT) multi-organ segmentation method for prostate cancer treatment planning.

      Methods and materials

      The proposed method exploits the superior soft-tissue information provided by synthetic MRI (sMRI) to aid the multi-organ segmentation on pelvic CT images. A cycle generative adversarial network (CycleGAN) was used to estimate sMRIs from CT images. A deep attention U-Net (DAUnet) was trained on sMRI and corresponding multi-organ contours for auto-segmentation. The deep attention strategy was introduced to identify the most relevant features to differentiate different organs. Deep supervision was incorporated into the DAUnet to enhance the features’ discriminative ability. Segmented contours of a patient were obtained by feeding CT image into the trained CycleGAN to generate sMRI, which was then fed to the trained DAUnet to generate organ contours. We trained and evaluated our model with 140 datasets from prostate patients.


      The Dice similarity coefficient and mean surface distance between our segmented and bladder, prostate, and rectum manual contours were 0.95 ± 0.03, 0.52 ± 0.22 mm; 0.87 ± 0.04, 0.93 ± 0.51 mm; and 0.89 ± 0.04, 0.92 ± 1.03 mm, respectively.


      We proposed a sMRI-aided multi-organ automatic segmentation method on pelvic CT images. By integrating deep attention and deep supervision strategy, the proposed network provides accurate and consistent prostate, bladder and rectum segmentation, and has the potential to facilitate routine prostate-cancer radiotherapy treatment planning.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic and Personal
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Radiotherapy and Oncology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Sharp G.
        • Fritscher K.D.
        • Pekar V.
        • et al.
        Vision 20/20: perspectives on automated image segmentation for radiotherapy.
        Med Phys. 2014; 41050902
        • Rasch C.
        • Steenbakkers R.
        • van Herk M.
        Target definition in prostate, head, and neck.
        Semin Radiat Oncol. 2005; 15: 136-145
        • Hurkmans C.W.
        • Borger J.H.
        • Pieters B.R.
        • Russell N.S.
        • Jansen E.P.
        • Mijnheer B.J.
        Variability in target volume delineation on CT scans of the breast.
        Int J Radiat Oncol Biol Phys. 2001; 50: 1366-1372
        • Vinod S.K.
        • Jameson M.G.
        • Min M.
        • Holloway L.C.
        Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies.
        Radiother Oncol. 2016; 121: 169-179
        • Breunig J.
        • Hernandez S.
        • Lin J.
        • et al.
        A system for continual quality improvement of normal tissue delineation for radiation therapy treatment planning.
        Int J Radiat Oncol Biol Phys. 2012; 83: e703-e708
        • Nelms B.E.
        • Tome W.A.
        • Robinson G.
        • Wheeler J.
        Variations in the contouring of organs at risk: test case from a patient with oropharyngeal cancer.
        Int J Radiat Oncol Biol Phys. 2012; 82: 368-378
        • Villeirs G.M.
        • Vaerenbergh K.V.
        • Vakaet L.
        • et al.
        Interobserver delineation variation using CT versus combined CT + MRI in intensity-modulated radiotherapy for prostate cancer.
        Strahlenther Onkol. 2005; 181: 424-430
        • Smith W.
        • Lewis C.
        • Bauman G.
        • et al.
        Prostate volume contouring: a 3D analysis of segmentation using 3DTRUS, CT, and MR.
        Int J Radiat Oncol Biol Phys. 2007; 67: 1238-1247
        • Isgum I.
        • Staring M.
        • Rutten A.
        • Prokop M.
        • Viergever M.A.
        • van Ginneken B.
        Multi-atlas-based segmentation with local decision fusion–application to cardiac and aortic segmentation in CT scans.
        IEEE Trans Med Imaging. 2009; 28: 1000-1010
        • Aljabar P.
        • Heckemann R.A.
        • Hammers A.
        • Hajnal J.V.
        • Rueckert D.
        Multi-atlas based segmentation of brain images: atlas selection and its effect on accuracy.
        Neuroimage. 2009; 46: 726-738
        • Iglesias J.E.
        • Sabuncu M.R.
        Multi-atlas segmentation of biomedical images: a survey.
        Med Image Anal. 2015; 24: 205-219
        • Ecabert O.
        • Peters J.
        • Schramm H.
        • et al.
        Automatic model-based segmentation of the heart in CT images.
        IEEE Trans Med Imaging. 2008; 27: 1189-1201
        • Qazi A.A.
        • Pekar V.
        • Kim J.
        • Xie J.
        • Breen S.L.
        • Jaffray D.A.
        Auto-segmentation of normal and target structures in head and neck CT images: a feature-driven model-based approach.
        Med Phys. 2011; 38: 6160-6170
        • Sun S.
        • Bauer C.
        • Beichel R.
        Automated 3-D segmentation of lungs with lung cancer in CT data using a novel robust active shape model approach.
        IEEE Trans Med Imaging. 2012; 31: 449-460
        • Wang T.
        • Lei Y.
        • Tang H.
        • et al.
        A learning-based automatic segmentation and quantification method on left ventricle in gated myocardial perfusion SPECT imaging: a feasibility study.
        J Nucl Cardiol. 2019; : 1-12
        • Yang X.
        • Huang J.
        • Lei Y.
        • et al.
        Med-A-nets: segmentation of multiple organs in chest CT image with deep adversarial networks.
        Paper presented at: Med Phys. 2018
        • Yang X.
        • Lei Y.
        • Wang T.
        • et al.
        3D prostate segmentation in MR image using 3D deeply supervised convolutional neural networks.
        Med Phys. 2018; 45: E582-E583
        • Wang B.
        • Lei Y.
        • Tian S.
        • et al.
        Deeply supervised 3D FCN with group dilated convolution for automatic MRI prostate segmentation.
        Med Phys. 2019;
        • Dong X.
        • Lei Y.
        • Wang T.
        • et al.
        Automatic multiorgan segmentation in thorax CT images using U-net-GAN.
        Med Phys. 2019; 46: 2157-2168
        • Kazemifar S.
        • Balagopal A.
        • Nguyen D.
        • et al.
        Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning.
        Biomed Phys Eng Express. 2018; : 4
        • Samaneh Kazemifar A.B.
        • Nguyen Dan
        • McGuire Sarah
        • Hannan Raquibul
        • Jiang Steve
        • Owrangi Amir
        . Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning.
        Biomed Phys Eng Express. 2018; : 4
        • Liu C.
        • Gardner S.J.
        • Wen N.
        • et al.
        Automatic segmentation of the prostate on CT images using deep neural networks (DNN).
        Int J Radiat Oncol Biol Phys. 2019;
        • Lei Y.
        • Harms J.
        • Wang T.
        • et al.
        MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks.
        Med Phys. 2019;
        • Mishra D.
        • Chaudhury S.
        • Sarkar M.
        • Soin A.S.
        Ultrasound image segmentation: a deeply supervised network with attention to boundaries.
        IEEE Trans Biomed Eng. 2018; (in press)
        • Wang B.
        • Lei Y.
        • Tian S.
        • et al.
        Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation.
        Med Phys. 2019; 46: 1707-1718
        • Lei Y.
        • Tian S.
        • He X.
        • et al.
        Ultrasound prostate segmentation based on multi-directional deeply supervised V-net.
        Med Phys. 2019; (in press)
        • Wang T.
        • Lei Y.
        • Tian S.
        • et al.
        Learning-based automatic segmentation of arteriovenous malformations on contrast CT images in brain stereotactic radiosurgery.
        Med Phys. 2019; (in press)
        • Balagopal A.
        • Kazemifar S.
        • Nguyen D.
        • et al.
        Fully automated organ segmentation in male pelvic CT images.
        Phys Med Biol. 2018; : 63
        • Huyskens D.P.
        • Maingon P.
        • Vanuytsel L.
        • et al.
        A qualitative and a quantitative analysis of an auto-segmentation module for prostate cancer.
        Radiother Oncol. 2009; 90: 337-345
        • Geraghty J.P.
        • Grogan G.
        • Ebert M.A.
        Automatic segmentation of male pelvic anatomy on computed tomography images: a comparison with multiple observers in the context of a multicentre clinical trial.
        Radiat Oncol. 2013; 8: 106
        • Simmat I.
        • Georg P.
        • Georg D.
        • Birkfellner W.
        • Goldner G.
        • Stock M.
        Assessment of accuracy and efficiency of atlas-based autosegmentation for prostate radiotherapy in a variety of clinical conditions.
        Strahlenther Onkol. 2012; 188: 807-815