Diagnostic accuracy of MRI techniques for treatment response evaluation in patients with brain metastasis: A systematic review and meta-analysis

Background: Treatment response assessment in patients with brain metastasis uses contrast enhanced T1-weighted MRI. Advanced MRI techniques have been studied, but the diagnostic accuracy is not well known. Therefore, we performed a metaanalysis to assess the diagnostic accuracy of the currently avail- able MRI techniques for treatment response. Methods: A systematic literature search was done. Study selection and data extraction were done by two authors independently. Meta-analysis was performed using a bivariate random effects model. An independent cohort was used for DSC perfusion external validation of diagnostic accuracy. Results: Anatomical MRI (16 studies, 726 lesions) showed a pooled sensitivity of 79% and a speciﬁcity of 76%. DCE perfusion (4 studies, 114 lesions) showed a pooled sensitivity of 74% and a speciﬁcity of 92%. DSC perfusion (12 studies, 418 lesions) showed a pooled sensitivity was 83% with a speciﬁcity of 78%. Diffusion weighted imaging (7 studies, 288 lesions) showed a pooled sensitivity of 67% and a speciﬁcity of 79%. MRS (4 studies, 54 lesions) showed a pooled sensitivity of 80% and a speciﬁcity of 78%. Combined techniques (6 studies, 375 lesions) showed a pooled sensitivity of 84% and a speciﬁcity of 88%. External validation of DSC showed a lower sensitivity and a higher speciﬁcity for the reported cut-off values included in this metaanalysis. Conclusion: A combination of techniques shows the highest diagnostic accuracy differentiating tumor progression from treatment induced abnormalities. External validation of imaging results is important to better deﬁne the reliability of imaging results with the different techniques.

Background: Treatment response assessment in patients with brain metastasis uses contrast enhanced T1-weighted MRI. Advanced MRI techniques have been studied, but the diagnostic accuracy is not well known. Therefore, we performed a metaanalysis to assess the diagnostic accuracy of the currently available MRI techniques for treatment response. Methods: A systematic literature search was done. Study selection and data extraction were done by two authors independently. Meta-analysis was performed using a bivariate random effects model. An independent cohort was used for DSC perfusion external validation of diagnostic accuracy. Results: Anatomical MRI (16 studies, 726 lesions) showed a pooled sensitivity of 79% and a specificity of 76%. DCE perfusion (4 studies, 114 lesions) showed a pooled sensitivity of 74% and a specificity of 92%. DSC perfusion (12 studies, 418 lesions) showed a pooled sensitivity was 83% with a specificity of 78%. Diffusion weighted imaging (7 studies, 288 lesions) showed a pooled sensitivity of 67% and a specificity of 79%. MRS (4 studies, 54 lesions) showed a pooled sensitivity of 80% and a specificity of 78%. Combined techniques (6 studies, 375 lesions) showed a pooled sensitivity of 84% and a specificity of 88%. External validation of DSC showed a lower sensitivity and a higher specificity for the reported cut-off values included in this metaanalysis. Conclusion: A combination of techniques shows the highest diagnostic accuracy differentiating tumor progression from treatment induced abnormalities. External validation of imaging results is important to better define the reliability of imaging results with the different techniques. Brain metastasis are the most common intracranial malignant tumors with a high mortality rate [1]. Although treatment for the primary tumor may also have an effect on brain metastasis (e.g. immunotherapy), the most important treatment of brain metastasis is high dose focal radiation therapy, while surgical resection is also an option in selected patients [2]. Magnetic resonance imaging (MRI) of the brain is the common radiological modality to monitor treatment effect during follow-up.
One of the major radiological challenges in patients with brain metastasis treated with focal high dose radiation therapy is the differentiation between tumor progression (PD) and treatment induced radiological abnormalities, also called pseudoprogression (PsP). This phenomenon is called radiation necrosis when occurring (late) after radiation therapy only [3]. Conventional T2weighted and contrast-enhanced T1-weighted MRI techniques are considered unreliable to differentiate between PD and PsP [4]. Therefore, advanced MRI techniques such as diffusion weighted imaging (DWI) and perfusion weighted imaging (PWI) can be added to the scanning protocol to assist in differentiating PD and PsP. PWI is an umbrella term that encompasses different available perfusion MRI techniques: dynamic susceptibility contrast (DSC), dynamic contrast enhanced (DCE) and arterial spin labelling (ASL). DSC measures T2-weighted or T2*-weighted signal changes over time after administration of an exogenous gadolinium based contrast agent (GBCA). DCE measures T1-weighted signal changes over time after administration of exogenous GBCA, while ASL uses magnetically labelled blood to measure blood flow without the need of exogenous contrast. The ability to accurately distinguish between PD and PsP after focal high dose radiation therapy for brain metastasis has important clinical implications for patients and clinicians. In case of PD switch to a different treatment or reirradiation might be indicated while in case of PsP the current treatment can be judged as effective and should be continued.
Multiple studies have determined the sensitivity and specificity of the available (advanced) MRI techniques in the assessment of brain metastasis treatment response, but patient numbers are usually small and studies are heterogeneous in terms of patient population and imaging techniques, warranting pooling of data to obtain more reliable results. Also, previous reviews did not include all available MRI techniques [5]. There is a wide variety in the use of different (advanced) MRI techniques in current daily clinical practice. Dynamic susceptibility contrast (DSC) perfusion is currently the most commonly used perfusion MRI technique to differentiate between PD and PsP based on a relative cerebral blood volume (rCBV) [6]. However, the optimum rCBV value for differentiating between PD and PsP is unknown and there is a lack of external validation of the reported thresholds [7]. This external validation is a requirement for the generalisability of such thresholds.
The aim of this systematic review and meta-analysis was to provide an up to date overview of the current MRI techniques and their diagnostic accuracies for monitoring the response of treated brain metastasis, focusing on advanced MRI techniques like PWI, DWI, and spectroscopy. The main research question was to investigate which MRI technique has the best sensitivity and specificity to distinguish PD from treatment induced abnormalities during follow-up in brain metastasis patients. In addition, we performed an external validation of the various reported cut-off values for DSC perfusion in an independent patient cohort. This will provide insight into the generalisability of rCBV cut-off values that can be used in the clinic or further research.

Methods
The systematic review and meta-analysis were performed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) criteria [8]. In addition, the Cochrane handbook for review of diagnostic test accuracy and the AMSTAR guidelines for the assessment of the methodological quality of systematic reviews were used [9]. This review and meta-analysis was not registered in a register.

Search strategy
The search strategy was designed under the supervision of information specialists of the Erasmus MC Medical Library, Rotterdam, The Netherlands. The database search was conducted in Embase, Medline, Web Of Science, Cochrane and Google Scholar. We used the following keywords: brain metastasis, radiation therapy, treatment response, MRI, perfusion MRI, diffusion MRI, MR spectroscopy, chemical exchange saturation transfer (CEST) and blood oxygenation level dependent (BOLD) imaging. The full details on the search strategy can be found in the supplementary material. The first search was performed on May 1st 2020 and updated on November 8th 2021. There were no limits regarding the year of publication.

Selection criteria
Studies eligible for inclusion consisted of patients with intraaxial brain metastasis of extracranial primary cancer treated with focal (high dose) radiation therapy. Treatment response had to be evaluated by MRI, i.e. conventional and/or advanced MRI. The definitive diagnosis, on either treatment induced abnormalities or true tumor progression, was assessed by histopathological assessment of resected tissue, imaging follow-up, clinical followup or a combination of these. All consecutive series of patients, both retrospective and prospective, were included. Case reports, case series and (systematic) reviews were excluded. Only articles written in English were considered. Studies were included for meta-analysis if information for the 2 Â 2 tables could be extracted. The 2 Â 2 tables were populated with the numbers of true positive (TP), true negative (TN), false positive (FP) and false negative (FN) MRI-based diagnoses. We defined true progression as the presence of disease and radiation induced abnormalities as absence of disease.

Study selection, data extraction and quality assessment
After deduplication of the identified articles, two independent reviewers (WT, CG) screened all titles and abstracts according to the predefined selection criteria. A third reviewer (AH) was consulted in case of discrepancies. In the same way, full text screening was performed by the two independent reviewers. To limit possible publication bias, conference abstracts were considered for data extraction if sufficient information was available.
A data extraction form was used to systematically extract the general study information (i.e. study design, total number of included patients, mean age, gender, primary tumor type, MRI characteristics and type and timing of follow-up) as well as numbers of TP, FP, TN and FN. 2 Â 2 tables were made per imaging technique. If more than one technique was used and authors provided TP, FP, TN and FN for a combination of techniques, these were used in the pooled analysis for the 'combined techniques group'. Authors were contacted for further information if the data for the 2 Â 2 table could not be extracted. If no further information could be provided or the authors did not respond, these studies were excluded for data extraction and not further considered. To perform the external validation, thresholds to distinguish PD from PsP were additionally extracted in studies reporting DSC techniques.
The risk of bias was assessed according to the quality assessment of diagnostic accuracy studies (QUADAS-2), consisting of the following domains: patient selection, index test, reference standard, flow and timing [10]. Data extraction and quality assessment were performed by the same two independent reviewers (WT, CG) and discrepancies were resolved by a third reviewer (AH). No studies were excluded from the analyses based on the QUADAS-2 assessment.

Statistical analysis
To provide a graphical overview of the results of the included studies and their results per MRI technique, forest plots were created with RevMan 5.4.1 (Cochrane collaboration, Copenhagen, Denmark). Pooled sensitivity and specificity for all studies were calculated with the ''MADA package" in Rstudio [11]. As heterogeneity between the studies was expected, a random effects model was used to calculate pooled sensitivities and specificities for each MRI technique [12]. For an overview of the pooled data and their heterogeneity per imaging technique, summary receiver operating characteristic (SROC) curves were made.

External validation of DSC perfusion MRI
The rCBV ratio thresholds reported in the studies describing DSC were validated in an external data set, consisting of a retrospective convenience sample of 39 patients with brain metastasis from the Erasmus MC. This external validation study was reviewed MRI for response evaluation of brain metastases by the Erasmus MC Medical Ethics Committee and all patients had either provided written informed consent or were already deceased at the time of inclusion. All patients had undergone follow-up with 1.5 or 3 T MRI including DSC perfusion with a preload bolus between 2013 and 2019. All patients had received focal high dose radiation therapy. Final diagnosis, PD or PsP, was determined by histopathology (n = 5) or follow-up imaging (n = 34) with mean follow-up time of 5.4 months. rCBV maps were computed with Intellispace Portal (Philips Healthcare) using leakage correction. First, a region of interest (ROI) of at least 70 mm 2 was drawn manually in a representative part of the tumor and subsequently copied to the contralateral normal appearing white matter (NAWM). Then, the mean tumor values and mean NAWM values were used to calculate the ratios between tumor and NAWM. Sensitivity and specificity values were calculated for each threshold identified in the systematic review.

Selection and general description of studies
After deduplication, a total of 8318 unique titles were identified through our electronic database search (see Fig. 1 for the flowchart). After screening titles and abstracts, 245 studies were deemed eligible for full text screening. This resulted in the identification of 53 possibly relevant studies. From 37/53 studies, data for the 2 Â 2 tables could be extracted, making them eligible for quantitative analysis (see Table 1 for study characteristics). Characteristics of the sixteen studies [13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28] which could not be included are shown in supplementary table 1. Patient and imaging characteristics between studies included or not included in the quantitative analysis did not differ (see Table 1 and supplementary table 1).     A total of 50 2 Â 2 tables from the eligible studies (comprising 1989 lesions; Table 2) could be created for the quantitative analysis, because some studies reported on more than one MRI technique. The vast majority (35/37) of the analyzed studies had a retrospective study design; only two studies had a prospective design. Sixteen authors were contacted for more information because information for a 2 Â 2 table could not be extracted, which resulted in one extra 2 Â 2 table. The other authors did not respond (75 %) or could not provide this extra information (19 %).
Forest plots for each MRI technique with information on the sensitivity, specificity and true and false positives / negatives rates are shown in Fig. 2. The pooled results of the meta-analyses are shown in Table 2. SROC curves are shown in Fig. 3.

Combined perfusion and diffusion techniques
Four studies reported data to construct six different 2 Â 2 tables (375 lesions) [33,34,50,62], in which a combination of DWI and DSC perfusion was used. One of these studies [34] also used DWI with DCE perfusion and a combination of DWI, DSC and DCE. In addition, one study reported only a combination of techniques [50]. Forest plots of combined techniques all showed a high sensitivity (range: 86-92 %) and specificity (range: 65-100 %) with small confidence intervals. The calculated heterogeneity (I 2 ) of the included studies was 99.6, indicating considerable heterogeneity. Overall, the combination of techniques had a pooled sensitivity of 0.84 (95 % CI: 0.78-0.89) and specificity of 0.88 (95 % CI: 0.77-0.94).

External validation of DSC-rCBV thresholds
The mean age of patients included for external validation was 60.8 (standard deviation: 10.6) years and 21 out of 39 (54 %) patients were female. Twenty-five patients had brain metastasis from lung cancer, seven from breast cancer, three from melanoma and four patients with other cancers or unknown primary tumors. The reported rCBV thresholds of 1.5-2.6 have a sensitivity between 0.47 and 0.77 and specificity between 0.86 and 0.95 (Table 3) in our validation cohort. The previously determined threshold of 2.0-2.1 would be optimal based on the external validation.

Discussion
This systematic review and meta-analysis comprises 37 unique studies (resulting in 50 2 Â 2 tables with relevant information) on the diagnostic accuracy of conventional and advanced MRI techniques for evaluating treatment response in patients with brain metastasis after focal high dose radiotherapy. We demonstrated that among the available advanced MRI techniques, DSC and DCE perfusion had the best diagnostic accuracy. Moreover, the combination of multiple perfusion and diffusion techniques outperformed the diagnostic accuracy of these techniques separately, with a pooled sensitivity of 0.84 and specificity of 0.88.
The findings in this meta-analysis in brain metastasis patients are consistent with the findings in comparable cohorts of patients with high grade glioma [65], in which similar values of sensitivity and specificity for conventional MRI, DWI, MRS, DSC and DCE perfusion were reported.
In general, if we compare the pooled sensitivities with specificities identified in this meta-analysis, slightly higher specificities were observed. The observed high specificity has important clinical implications, as it is relevant to identify those patients with PsP. If PsP occurs, this usually means that follow-up with MR imaging or systemic anti-tumor therapy will be continued, whereas PD usually means a switch or discontinuation of systemic anti-tumor treatment or that salvage radiation therapy will be administered.
Our results suggest that perfusion MRI and DWI as advanced MRI technique additional to conventional MRI have the best diagnostic performance. Although DWI is routinely used, PWI is used less frequently in clinical practice. Nevertheless, based on the currently available literature it might be worth considering to add PWI to conventional MRI in daily clinical practice for patients with an unclear treatment response on conventional imaging. These PWI results should subsequently be combined with the other sequences to draw a definite conclusion on the treatment response and thus the management of the patient. PWI could be done using a DSC, DCE or ASL technique of which DSC is most commonly performed. We showed that DCE and DSC have comparable diagnostic accuracy. The diagnostic accuracy of ASL remains largely unknown with only one study being available. However, ASL might be a promising technique to distinguish PD from PsP, because of the previously shown diagnostic accuracy in glioma [66]. Further research is thus warranted to assess its value in the context of brain metastasis. Moreover, DCE and ASL perfusion have several advantages over DSC perfusion, such as fewer issues with susceptibility artefacts and leakage effects. Furthermore, ASL has less issues with a tumor location near major blood vessels [67]. On the other hand, DSC is considered more easy to implement, resulting in it being currently the most commonly used technique. This meta-analysis focusses on MRI only, but positron emission tomography (PET) is also used to differentiate between PD and PsP. Different PET tracers can be used, such as 18F-2-fluoro-2-deoxy-Dglucose (FDG) and different amino acid PET tracers, such as methionine (MET). As described in a systematic review by Galldiks et al. from 2019 [68], studies on FDG PET show a lower diagnostic performance than MRI, while amino acid PET shows a high diagnostic performance. Nevertheless, amino acid PET scans are more expensive and less widely available compared to MRI.
This meta-analysis has several limitations. First, the quality of many included studies is some cases questionable, because of an unknown or high risk of bias. A large part of this high risk of bias was due to bias in selection of patients and performing the index test (e.g. lack of predefined cut-off values). For determining true and generalisable clinical value it is important to either use predefined cut-off values or to externally validate cut-off values determined within the study. Previous data [69] has shown that a considerable number of lesions are difficult to interpret due to their location or artifacts (15 %) or due to unmeasurable residual metastases (31 %). Exclusion of these patients may results in an overestimation of the value of perfusion MRI. Only one study reported excluding patients on the basis of susceptibility artefacts, but it is not certain whether other studies included in this meta- The columns below ''Reported study data" describe the reported TP, FP, TN, FN, sensitivity and specificity of these eight studies. Below ''External validation" TP, FP, TN, FN, sensitivity and specificity are calculated. These calculations are made by using the thresholds mentioned in the included studies, applied on an external data set of 39 patients with metastasis as described in the methods.
MRI for response evaluation of brain metastases analysis did or did not do the same. Another limitation is that not all possible eligible studies could be analysed because relevant information was lacking and could not be retrieved. Nevertheless, we believe that these missing results did not influence the results because the excluded studies were not different from the included studies with respect to imaging techniques and patient characteristics. Another limitation is the low number of studies using histopathology as the gold standard for determining the final diagnosis. Ideally, only studies with a histopathological diagnosis should be included, as histopathology is generally considered the gold standard, but this would limit the number of eligible studies. In addition, this would not reflect daily clinical practice, where information on histopathology is also not always available and would thus induce an inclusion bias. Through the mechanism of ''diagnosis by indication" the use of histopathology tends to lead to a larger proportion of PD and the use of radiology to a larger proportion of PsP, because surgery would be considered more appropriate for PD than for PsP. While the calculated sensitivity for DSC perfusion in the external validation dataset was lower than the reported pooled sensitivity (0.83 [95 % CI: 0.73-0.89]), the specificity was higher than the pooled specificity of 0.78 (95 % CI: 0.65-0.88). Particularly the number of false-negatives was slightly higher in our validation cohort, resulting in lower sensitivity values (between 0.47 and 0.77). This finding has implications for clinical practice, because this would result in patients with PD being falsely classified as having PsP, for whom follow-up with MRI or systemic anti-tumor therapy would be continued unnecessarily, with potentially detrimental consequences. This finding also exemplifies the need for external validation of study results in independent datasets, to better define the reliability of the imaging results with the different techniques.
In conclusion, this meta-analysis shows the highest pooled sensitivity and specificity when combining perfusion and diffusion MRI to differentiate between PD and PsP in brain metastasis patients during follow-up after focal high dose radiation therapy. Despite the limitations of the studies included in this metaanalysis, the pooled sensitivity and specificity of all MRI techniques are important results in optimising treatment evaluation for brain metastasis. For future research we would suggest using larger cohorts investigating the diagnostic accuracy for combined MRI techniques. As there is a lot of heterogeneity of the reviewed studies regarding imaging protocols and follow-up methods, we encourage harmonization of imaging protocols [6] and also systematic reporting of treatment response within international collaborations. We would also suggest using predefined cut-offs to prevent a high risk of bias in the index test domain, or to externally validate any newly defined cut-off values. Lastly, we would suggest research on the use of ASL perfusion MRI.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.