Advertisement

A virtual dosimetry audit – Towards transferability of gamma index analysis between clinical trial QA groups

Published:October 31, 2017DOI:https://doi.org/10.1016/j.radonc.2017.10.012

      Abstract

      Purpose

      Quality assurance (QA) for clinical trials is important. Lack of compliance can affect trial outcome. Clinical trial QA groups have different methods of dose distribution verification and analysis, all with the ultimate aim of ensuring trial compliance. The aim of this study was to gain a better understanding of different processes to inform future dosimetry audit reciprocity.

      Materials

      Six clinical trial QA groups participated. Intensity modulated treatment plans were generated for three different cases. A range of 17 virtual ‘measurements’ were generated by introducing a variety of simulated perturbations (such as MLC position deviations, dose differences, gantry rotation errors, Gaussian noise) to three different treatment plan cases. Participants were blinded to the ‘measured’ data details. Each group analysed the datasets using their own gamma index (γ) technique and using standardised parameters for passing criteria, lower dose threshold, γ normalisation and global γ.

      Results

      For the same virtual ‘measured’ datasets, different results were observed using local techniques. For the standardised γ, differences in the percentage of points passing with γ < 1 were also found, however these differences were less pronounced than for each clinical trial QA group’s analysis. These variations may be due to different software implementations of γ.

      Conclusions

      This virtual dosimetry audit has been an informative step in understanding differences in the verification of measured dose distributions between different clinical trial QA groups. This work lays the foundations for audit reciprocity between groups, particularly with more clinical trials being open to international recruitment.

      Keywords

      Radiotherapy dosimetry audits allow for the testing of procedures and the identification of deviations. Dosimetry audits range in complexity from measuring machine output under reference conditions to complex radiotherapy such as intensity modulated radiotherapy (IMRT) measurements [
      • Clark C.H.
      • Aird E.G.
      • Bolton S.
      • Miles E.A.
      • Nisbet A.
      • Snaith J.A.
      • et al.
      Radiotherapy dosimetry audit: three decades of improving standards and accuracy in UK clinical practice and trials.
      ,
      • Johansson K.-A.
      • Nilsson P.
      • Zackrisson B.
      • Ohlson B.
      • Kjellén E.
      • Mercke C.
      • et al.
      The quality assurance process for the ARTSCAN head and neck study – a practical interactive approach for QA in 3DCRT and IMRT.
      ,
      • Schiefer H.
      • Fogliata A.
      • Nicolini G.
      • Cozzi L.
      • Seelentag W.W.
      • Born E.
      • et al.
      The Swiss IMRT dosimetry intercomparison using a thorax phantom.
      ,
      • Budgell G.
      • Berresford J.
      • Trainer M.
      • Bradshaw E.
      • Sharpe P.
      • Williams P.
      A national dosimetric audit of IMRT.
      ,
      • Clark C.H.
      • Hussein M.
      • Tsang Y.
      • Thomas R.
      • Wilkinson D.
      • Bass G.
      • et al.
      A multi-institutional dosimetry audit of rotational intensity-modulated radiotherapy.
      ,
      • Gershkevitsh E.
      • Pesznyak C.
      • Petrovic B.
      • Grezdo J.
      • Chelminski K.
      • do Carmo Lopes M.
      • et al.
      Dosimetric inter-institutional comparison in European radiotherapy centres: Results of IAEA supported treatment planning system audit.
      ,
      • Izewska J.
      • Wesolowska P.
      • Azangwe G.
      • Followill D.S.
      • Thwaites D.I.
      • Arib M.
      • et al.
      Testing the methodology for dosimetry audit of heterogeneity corrections and small MLC-shaped fields: results of IAEA multi-center studies.
      ,
      • Jurado-Bruggeman D.
      • Hernández V.
      • Sáez J.
      • Navarro D.
      • Pino F.
      • Martínez T.
      • et al.
      Multi-centre audit of VMAT planning and pre-treatment verification.
      ,
      • Distefano G.
      • Lee J.
      • Jafari S.
      • Gouldstone C.
      • Baker C.
      • Mayles H.
      • et al.
      A national dosimetry audit for stereotactic ablative radiotherapy in lung.
      ]. Currently the verification of the measured dose distribution can vary largely with multiple different commercial hardware and software systems available. There are also different methods of the analysis of the dose distribution such as dose difference and distance-to-agreement (DTA). One of the most widely used techniques is the gamma index method [
      • Low D.A.
      • Harms W.B.
      • Mutic S.
      • Purdy J.A.
      A technique for the quantitative evaluation of dose distributions.
      ]. Various studies have evaluated the response of the gamma index in different commercial systems and shown that it can respond in different ways between different systems [
      • Hussein M.
      • Rowshanfarzad P.
      • Ebert M.A.
      • Nisbet A.
      • Clark C.H.
      A comparison of the gamma index analysis in various commercial IMRT/VMAT QA systems.
      ,
      • Zhen H.
      • Nelms B.E.
      • Tome W.A.
      Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.
      ,
      • Crowe S.B.
      • Sutherland B.
      • Wilks R.
      • Seshadri V.
      • Sylvander S.
      • Trapp J.V.
      • et al.
      Technical Note: Relationships between gamma criteria and action levels: Results of a multicenter audit of gamma agreement index results.
      ].
      Quality assurance (QA) in clinical trials is crucial as lack of compliance can affect trial outcome [
      • Haworth A.
      • Kearvell R.
      • Greer P.B.
      • Hooton B.
      • Denham J.W.
      • Lamb D.
      • et al.
      Assuring high quality treatment delivery in clinical trials – results from the Trans-Tasman Radiation Oncology Group (TROG) study 03.04 “RADAR” set-up accuracy study.
      ,
      • Weber D.C.
      • Poortmans P.M.
      • Hurkmans C.W.
      • Aird E.
      • Gulyban A.
      • Fairchild A.
      Quality assurance for prospective EORTC radiation oncology trials: the challenges of advanced technology in a multicenter international setting.
      ,
      • Weber D.C.
      • Tomsej M.
      • Melidis C.
      • Hurkmans C.W.
      QA makes a clinical trial stronger: Evidence-based medicine in radiation therapy.
      ,
      • Ebert M.A.
      • Harrison K.M.
      • Cornes D.
      • Howlett S.J.
      • Joseph D.J.
      • Kron T.
      • et al.
      Comprehensive Australasian multicentre dosimetric intercomparison: issues, logistics and recommendations.
      ,
      • Peters L.J.
      • O’Sullivan B.
      • Giralt J.
      • Fitzgerald T.J.
      • Trotti A.
      • Bernier J.
      • et al.
      Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02.
      ]. Different international radiotherapy clinical trial QA groups have developed independent methods of measured dose distribution verification and analysis for various historical and other logistical reasons and the particular systems they had access to, all with the ultimate aim of ensuring compliance [
      • Eaton D.J.
      • Tyler J.
      • Backshall A.
      • Bernstein D.
      • Carver A.
      • Gasnier A.
      • et al.
      An external dosimetry audit programme to credential static and rotational IMRT delivery for clinical trials quality assurance.
      ,
      • Weber D.C.
      • Vallet V.
      • Molineu A.
      • Melidis C.
      • Teglas V.
      • Naudy S.
      • et al.
      IMRT credentialing for prospective trials using institutional virtual phantoms: results of a joint European Organization for the Research and Treatment of Cancer and Radiological Physics Center project.
      ,
      • Lye J.
      • Kenny J.
      • Lehmann J.
      • Dunn L.
      • Kron T.
      • Alves A.
      • et al.
      A 2D ion chamber array audit of wedged and asymmetric fields in an inhomogeneous lung phantom.
      ,
      • Miri N.
      • Lehmann J.
      • Legge K.
      • Vial P.
      • Greer P.B.
      Virtual EPID standard phantom audit (VESPA) for remote IMRT and VMAT credentialing.
      ].
      Individual clinical trial QA groups have methods for streamlining the trial QA for multiple trials to avoid duplication. For example, a centre that has had the dosimetry credentialed for a particular trial may be exempted from repeating the dosimetry QA for the same clinical site or other similar (or less complex) clinical trials. Some clinical trials are now open to international recruitment to increase patient numbers and limit the time to full accrual. Streamlining dosimetry QA in the international setting, such that an institution credentialed by one QA group may be accepted by another, is therefore of interest. To be able to achieve this, it is important to understand how different analysis techniques and tolerances translate between different groups, and the challenges involved. The Global Quality Assurance of Radiation Therapy Clinical Trials Harmonisation Group (GHG) has been established to facilitate the harmonisation and reciprocity of clinical trial QA between different groups and consistency in the dose delivery in the trials [
      • Melidis C.
      • Bosch W.R.
      • Izewska J.
      • Fidarova E.
      • Zubizarreta E.
      • Ishikura S.
      • et al.
      Radiation therapy quality assurance in clinical trials – global harmonisation group.
      ,

      Global Quality Assurance of Radiation Therapy Clinical Trials Harmonisation Group – Ensuring Quality Cancer Treatment Worldwide n.d. https://rtqaharmonization.com/ (accessed May 29, 2017).

      ,
      • Melidis C.
      • Bosch W.R.
      • Izewska J.
      • Fidarova E.
      • Zubizarreta E.
      • Ulin K.
      • et al.
      Global harmonization of quality assurance naming conventions in radiation therapy clinical trials.
      ,
      • Clark C.H.
      • Hurkmans C.W.
      • Kry S.F.
      The role of dosimetry audit in lung SBRT multi-centre clinical trials.
      ].
      This study focuses on the verification of measured dose distributions for complex techniques such as IMRT and volumetric modulated arc therapy (VMAT). The aim was to gain a better understanding of the different gamma index analysis processes between international clinical trial QA groups and to inform potential future dosimetry audit reciprocity within and outside clinical trials.

      Methods and materials

      Six international radiotherapy clinical trial QA groups, which are members of the GHG, participated in this study. These were the Radiotherapy Trials QA (RTTQA) group in the United Kingdom, the European Organization for Research and Treatment of Cancer (EORTC) Radiation Oncology QA group, the Imaging and Radiation Oncology Core (IROC) in the United States, the Japan Clinical Oncology Group (JCOG), the Trans-Tasman Radiation Oncology Group (TROG), and the Australian Clinical Dosimetry Service (ACDS).

      Virtual ‘measured’ plan creation

      Three individual cases were chosen for the study. These were the three-dimensional treatment planning system (3DTPS) test developed by RTTQA for VMAT & Tomotherapy benchmarking [
      • Tsang Y.
      • Ciurlionis L.
      • Clark C.
      • Venables K.
      Development of a novel treatment planning test for credentialing rotational intensity-modulated radiotherapy techniques in the UK.
      ], a prostate cancer case, and a head & neck (H&N) cancer case. For the 3DTPS case, a 2 × 360° arc volumetric modulated arc therapy (VMAT) plan was generated. The prostate and H&N cases respectively were planned with 5 and 7 fixed field IMRT fields respectively. All plans were generated in the Varian Eclipse TPS (Varian Medical Systems, Palo Alto, CA) and calculated using the AAA v 11.3 algorithm with 2.5 mm dose grid spacing. Screenshots of these cases are shown in Fig. 1.
      Figure thumbnail gr1
      Fig. 1The three different cases used; (a) the 3DTPS test, (b) a prostate cancer case and (c) a head & neck cancer case.
      Using a similar methodology as has been reported previously [
      • Hussein M.
      • Rowshanfarzad P.
      • Ebert M.A.
      • Nisbet A.
      • Clark C.H.
      A comparison of the gamma index analysis in various commercial IMRT/VMAT QA systems.
      ,
      • Zhen H.
      • Nelms B.E.
      • Tome W.A.
      Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.
      ,
      • Hussein M.
      • Adams E.J.
      • Jordan T.J.
      • Clark C.H.
      • Nisbet A.
      A critical evaluation of the PTW 2D-Array seven29 and Octavius II phantom for IMRT and VMAT verification.
      ,
      • Nelms B.E.
      • Zhen H.
      • Tomé W.A.
      Per-beam, planar IMRT QA passing rates do not predict clinically relevant patient dose errors.
      ], plans were copied and a range of deliberate errors were introduced to perturb the dose distribution. These included a varying combination of single and whole bank MLC errors ranging from 1 to 5 mm, dose difference errors of +3% and −3% and gantry and collimator errors of 0.5 and 1 degrees. In some of the plans, gravity effects were introduced into the MLC positions based on Carver et al. [
      • Carver A.
      • Gilmore M.
      • Riley S.
      • Uzan J.
      • Mayles P.
      An analytical approach to acceptance criteria for quality assurance of Intensity Modulated Radiotherapy.
      ] using Eq. (1):
      MLCmod=MLCorig+Asin(θ)
      (1)


      Where MLCmod is the modified MLC position, MLCorig is the original position, and A is the specified maximum MLC position change (in this case we used 1–5 mm), and θ is gantry angle [
      • Carver A.
      • Gilmore M.
      • Riley S.
      • Uzan J.
      • Mayles P.
      An analytical approach to acceptance criteria for quality assurance of Intensity Modulated Radiotherapy.
      ]. Some plans also had subtle positional errors into the MLC using a Gaussian random number generator in MATLAB. The overall result was a range of virtual datasets that appeared to have simulated ‘measured’ features. In some of the plans the errors were such that dose-volume histogram constraints are pushed out of tolerance according to the corresponding author’s institutional objectives; for example the rectum tolerance for the prostate cancer case and spinal cord for the head & neck case.
      In total there were 5 ‘measured’ virtual datasets for the 3DTPS plan, 5 for the prostate plan and 7 for the H&N plan; a short description of the errors introduced into each one is given in Table 1. To ensure consistency all plans were recalculated on the same water-equivalent cylindrical phantom measuring 30 cm diameter by 30 cm length. Example gamma index distributions for a virtual measured plan from each of the three cases are shown in Supplementary Fig. 1.
      Table 1Percentage of points passing with γ < 1 for each virtual ‘measurement’ by each group according to their own analysis. Analysis that failed each Group’s local technique is highlighted in bold text and with an asterisk (*) by the number.
      Virtual planErrors introducedGroup 1Group 2Group 3Group 3 (repeat)Group 4Group 5Group 6
      3DTPS 1+4 mm single MLC error10099.499.999.410099.5100
      3DTPS 2+3% dose difference10095.4100.094.9*10096.099.2
      3DTPS 3+2 mm single MLC error10099.999.999.910099.9100
      3DTPS 4−3% dose difference, Col 1 deg10097.399.995.710097.799.2
      3DTPS 5Gravitational MLC errors using Eq. 1, with A = 4 mm10097.8100.096.210098.099.2
      Prost 1Single +3 mm MLC error, 1° gantry error, and Gaussian noise10091.392.1*88.5*98.092.497.7
      Prost 2Gaussian MLC error with maximum allowed error = 5 mm10097.584.7*93.6*10098.097.6
      Prost 3Gravitational MLC errors using Eq. 1, with A = 2 mm, and +2% dose difference10097.475.8*93.5*10097.597.6
      Prost 4Gravitational MLC errors using Eq. 1, with A = 1 mm, and -2% dose difference10099.579.8*97.910099.598.7
      Prost 5+3% dose difference100100100.0100100100100
      H&N 1Gaussian MLC error with maximum allowed error = 2 mm99.795.988.3*91.6*10096.196.8
      H&N 21° collimator error, and −3% dose difference99.999.898.79510099.897.2
      H&N 3Gravitational MLC errors using Eq. 1, with A = 1 mm99.810083.8*99.210010099.8
      H&N 4Single +3 mm MLC error, 1° gantry error, and Gaussian noise99.899.591.3*99.410099.699.8
      H&N 5Gaussian MLC error with maximum allowed error = 1 mm99.810090.6*99.810010099.8
      H&N 6Gravitational MLC errors using Eq. 1, with A = 2 mm99.899.977.2*98.610010099.5
      H&N 7Single −3 mm MLC error, 1° gantry error, and Gaussian noise99.899.887.8*99.210099.999.7

      Gamma index analysis

      Each clinical trial QA group was sent the original unedited dose distribution labelled ‘TPS dose’ and the edited distributions labelled ‘Measured Dose 1’, ‘Measured Dose 2’ and so forth, for each of the individual cases. All users were blinded to the specific details of the perturbations (if any) within the ‘measured’ virtual datasets to avoid subjective bias.
      All datasets were sent in 3D DICOM format with 2.5 mm pixel-pixel spacing in the x and y coordinates, and 3 mm in the z (slice spacing) coordinate. Additional 2D coronal planes were sent to allow each clinical trial QA group to import the correct dataset as normal for their practice. For example to facilitate a group whose standard practice was to compare a coronal film measurement against a 2D calculated coronal dose plane.
      Gamma index analysis was performed in two ways as described below. All users reported the percentage of points passing with γ < 1.

      Gamma index analysis using each clinical trial QA group’s own routine settings

      Each clinical trial QA group performed a gamma index analysis with their own routine settings for the following:
      • Global or local γ analysis.
      • Whether the evaluated and reference dose distributions are rescaled or not
      • γ normalisation technique (e.g. max dose/point in high dose region etc.).
      • Lower dose threshold as a percentage of the normalisation.
      • Passing criteria (% and mm).
      Each QA group were requested to provide the details of what was used for the above points, as well as which software and version was utilised.

      Standardised gamma index analysis

      Each clinical trial QA group then repeated the gamma index analysis using their software with standardised gamma index parameters for the passing criteria, normalisation and low dose threshold. Analysis was performed for the following: 2%/2 mm, 3%/2 mm, 3%/3 mm, 5%/5 mm, 7%/4 mm, global gamma index, no rescaling of the datasets, gamma index normalisation set as the maximum dose point in the ‘measured’ dataset, and a 20% lower dose threshold. The passing criteria were chosen based on typical criteria used by different groups. Where possible users were asked to perform the gamma index where the reference distribution was the ‘measured’ dataset and the evaluated distribution (i.e. the distribution that was searched for the minimum γ) was set as the TPS dose and perform the gamma index search in 3D if the software allowed, otherwise the coronal plane was used and a 2D search was performed.

      Results

      Gamma index analysis using each clinical trial QA group’s own settings

      Supplementary Table 1 gives a summary of the different gamma index analysis techniques and software between the clinical trial QA groups. All groups used the global γ analysis, however there were variations in the γ normalisation method (i.e. whether to use the maximum dose, a point in a high dose low gradient region, etc.) and in pass/fail criteria. Two of the groups had three decision levels for analysis: optimal pass, mandatory pass and fail. The remainder had mandatory pass and fail decision criteria. For consistency, only the mandatory pass and fail criteria were used in the remaining analysis. Table 1 shows the percentage of points passing with γ < 1 for each group’s own technique for each virtual measurement, and also shows the plans that were recorded as either pass or fail according to each group’s own analysis. All groups used 3%/3 mm as the mandatory passing criteria, except for Group 1 which used 7%/4 mm.
      As can be seen in Table 1, Group 3 had some clearly different gamma index passing rates for some of the plans when using their own approach for the analysis. Further investigation into this discrepancy revealed that the software in question was performing a relative gamma index analysis (i.e. both datasets were renormalised where the maximum dose in each dataset was set to 100%) as the software version manual description for performing an absolute comparison did not give sufficient detail as to how this calculation was made and the distinction between image normalisation and gamma index normalisation functions was unclear. Table 1 shows the repeated analysis in the intended way, labelled ‘Group 3 repeat’.

      Standardised gamma index analysis

      The % points passing with γ < 1 for each plan for the standardised analysis approach (20% threshold and normalised to the maximum dose in the measured dose distribution) between the QA groups for 2%/2 mm, 3%/2 mm and 3%/3 mm are shown in Fig. 2a–c, respectively. For 5%/5 mm and 7%/4 mm, all groups achieved 100% pass-rates. All groups were able to set the three-dimensional TPS dose cube as the evaluated distribution for the gamma analysis in their software. All but Group 4 performed a 3D gamma search.
      Figure thumbnail gr2
      Fig. 2Plot showing percentage of points passing with γ < 1 on the y-axis, for each of the virtual measurements in the x-axis by each clinical trial QA group. The data shown are for the standardised gamma index approach. Plots a–c show results for 2%/2 mm, 3%/2 mm, and 3%/3 mm respectively.
      Fig. 3 shows gamma index distributions for Prostate Case 1 for 3%/3 mm using the standardised gamma index parameters from each of the QA groups. This case had the most significant error introduced into it: a single MLC error of +3 mm throughout all control points in all fields, a 1 degree gantry rotation error in all fields, a +1% systematic dose error, and with additional Gaussian noise.
      Figure thumbnail gr3
      Fig. 3Comparison of different gamma index coronal planes for Group 1–6 (a–f respectively) for the prostate virtual measurement 1. For (b) and (c), red indicates gamma index >1.

      Discussion

      Different groups for clinical trial QA have different dose distribution measurement and gamma analysis approaches. For the same virtual ‘measured’ datasets, different γ passing rates were observed using each group’s local technique as shown in Table 1. This shows that there are some underlying features that are often hidden in commercial gamma analysis software or not well described, resulting in different outcomes for identical inputs and apparently identical evaluation criteria. An example of the impact of this issue was demonstrated with the results of Group 3 which were originally largely different for some of the plans when using their own approach for the analysis. As described in the results, this was eventually found to be due to unclear description in the software manual for performing an absolute γ comparison which did not give detail as to how this calculation was made and the distinction between image normalisation and gamma index normalisation functions was not clear. This study has led to the group modifying the software settings to perform their routine analysis the intended way.
      The variations in passing rates seen in Table 1 were blurred when considering the binary pass/fail decisions. With the exception of Group 3, all groups reported a ‘Pass’ for all of the virtual plans according to their own analysis. This is interesting as some of the cases had serious errors, as discussed in the methodology section, which would be seen when reviewing the gamma index distribution as all groups currently undertake; one such example is shown in Fig. 3. However the passing rate metric can hide errors and it has been shown in various studies that it is difficult to correlate γ passing rates with clinical outcomes [
      • Zhen H.
      • Nelms B.E.
      • Tome W.A.
      Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.
      ,
      • Nelms B.E.
      • Zhen H.
      • Tomé W.A.
      Per-beam, planar IMRT QA passing rates do not predict clinically relevant patient dose errors.
      ,
      • Cozzolino M.
      • Oliviero C.
      • Califano G.
      • Clemente S.
      • Pedicini P.
      • Caivano R.
      • et al.
      Clinically relevant quality assurance (QA) for prostate RapidArc plans: gamma maps and DVH-based evaluation.
      ], this in turn may make the transferability of this metric between different groups complex, which will need to be addressed. The focus of this study was on the gamma index analysis as this is the most commonly used approach amongst both the QA groups and hospital departments. Furthermore the analysis focussed on comparing data for the % points passing with γ < 1 which is the most commonly reported metric in the literature and possibly the most understood between different departments [
      • Hussein M.
      • Clark C.H.
      • Nisbet A.
      Challenges in calculation of the gamma index in radiotherapy – towards good practice.
      ]. However, the results of this study suggest that other metrics or analysis should be investigated, aiming for a more robust behaviour with respect to pre-processing steps and calculation parameters. Alternately, for the purpose of interinstitutional comparisons, a set of standard pre-processing steps, parameters and even acquisition steps should be identified.
      For the standardised gamma index approach where all of the same key parameters were fixed, differences in the percentage of points passing with γ < 1 were also found between the groups, however these differences were less pronounced than for each groups own analysis. These variations were larger, as expected, for tighter passing criteria such as 2%/2 mm. However, they started to be reduced as the passing criteria increased. As has been shown in other studies, differences in passing rates can occur between individual software and even different versions of the same software [
      • Hussein M.
      • Rowshanfarzad P.
      • Ebert M.A.
      • Nisbet A.
      • Clark C.H.
      A comparison of the gamma index analysis in various commercial IMRT/VMAT QA systems.
      ,
      • Zhen H.
      • Nelms B.E.
      • Tome W.A.
      Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.
      ,
      • Nelms B.E.
      • Zhen H.
      • Tomé W.A.
      Per-beam, planar IMRT QA passing rates do not predict clinically relevant patient dose errors.
      ,
      • Agnew C.E.
      • McGarry C.K.
      A tool to include gamma analysis software into a quality assurance program.
      ,
      • Masi L.
      • Casamassima F.
      • Doro R.
      • Francescon P.
      Quality assurance of volumetric modulated arc therapy: Evaluation and comparison of different dosimetric systems.
      ,
      • Fredh A.
      • Scherman J.B.
      • Fog L.S.
      Rosenschold PM af. Patient QA systems for rotational radiation therapy: a comparative experimental study with intentional errors.
      ]. These variations in results are potentially due to differences in software implementations of the gamma index [
      • Hussein M.
      • Clark C.H.
      • Nisbet A.
      Challenges in calculation of the gamma index in radiotherapy – towards good practice.
      ]. These individual approaches may have been implemented to speed up the gamma index calculation, which is inherently computationally expensive. Differences in the programming of the gamma index analysis have been previously shown to affect results [
      • Hussein M.
      • Clark C.H.
      • Nisbet A.
      Challenges in calculation of the gamma index in radiotherapy – towards good practice.
      ]. It is worth noting that it is often difficult to know exactly how the gamma index has been computed in a commercial software, as generally not enough detail is given. Determination of the cause of these differences would require manufacturers to disclose software implementation details such as describing whether one or both of the measurement and TPS dose are interpolated and to what level, what type of interpolation algorithm is used, and whether the search distance for finding the minimum gamma index is limited. The differences between gamma index implementations could be minimised if solutions allowed users to adjust these parameters.
      It is important to note that this investigation only focussed on the software being used. Each group has a different way of performing the dose distribution measurement which is reflected in their own passing criteria as this includes measurement uncertainty and spatial resolution. A useful next step is for each group to perform a physical audit by measuring the plans with the deliberately introduced errors to determine if these differences in results persist when real measurement uncertainty and spatial resolution differences are included [
      • Huang J.Y.
      • Pulliam K.B.
      • McKenzie E.M.
      • Followill D.S.
      • Kry S.F.
      Effects of spatial resolution and noise on gamma analysis for IMRT QA.
      ]. This is the subject of ongoing future work.
      The virtual datasets were only calculated in one TPS which is a potential limitation of this work. Another TPS may have calculated the dose distribution differently depending on the modelling of the linac, e.g. different low dose modelling, difference in MLC penumbra modelling leading to differences in the steepness of dose gradients. Additionally, the virtual measurements were on one linac type (Varian TrueBeam) and we have not investigated how the size of the MLC may affect the errors introduced and resulting analysis; both re-simulating the virtual measurements in a different TPS and different MLC sizes should be investigated. Since both the virtual measurements and the original unperturbed plans originated from the same TPS, it is unlikely that the results would be significantly different if a second or alternative TPS was used. Additionally the main aim was to develop a suite of tests with varying levels of discrepancy built into them to create a deliberate spread of results.
      Based on the results and some of the above discussion areas, there is a future need to develop an independent software to be able to handle different measurement approaches (e.g. film, different detector array configurations etc.) in a standardised manner. This is particularly important for the remote audit approach, for example where centres wishing to enter a clinical trial may send their own measured data to the responsible QA group for independent analysis [
      • Weber D.C.
      • Vallet V.
      • Molineu A.
      • Melidis C.
      • Teglas V.
      • Naudy S.
      • et al.
      IMRT credentialing for prospective trials using institutional virtual phantoms: results of a joint European Organization for the Research and Treatment of Cancer and Radiological Physics Center project.
      ,
      • Jornet N.
      • Carrasco P.
      • Beltrán M.
      • Calvo J.F.
      • Escudé L.
      • Hernández V.
      • et al.
      Multicentre validation of IMRT pre-treatment verification: comparison of in-house and external audit.
      ]. This may also have a wider impact in being able to facilitate the transferability in routine QA in departments changing from one system to another (e.g. one commercial system to another).
      This virtual dosimetry audit has been an informative step in beginning to understand differences in the verification of measured dose distributions between different clinical trial QA groups. This work lays the foundations for future studies in moving towards audit reciprocity between groups, particularly in light of more clinical trials being open to international recruitment. This is a step-by-step process, and future work will also involve looking at the ways different clinical trial QA groups report results of measured dose distribution verification. This study has also highlighted the challenges that are involved and informs future work focussing on analysis techniques that are transferable between different clinical trial QA groups.

      Conflict of interest statement

      The authors have nothing to declare.

      Appendix A. Supplementary data

      References

        • Clark C.H.
        • Aird E.G.
        • Bolton S.
        • Miles E.A.
        • Nisbet A.
        • Snaith J.A.
        • et al.
        Radiotherapy dosimetry audit: three decades of improving standards and accuracy in UK clinical practice and trials.
        Br J Radiol. 2015; 8820150251
        • Johansson K.-A.
        • Nilsson P.
        • Zackrisson B.
        • Ohlson B.
        • Kjellén E.
        • Mercke C.
        • et al.
        The quality assurance process for the ARTSCAN head and neck study – a practical interactive approach for QA in 3DCRT and IMRT.
        Radiother Oncol. 2008; 87: 290-299
        • Schiefer H.
        • Fogliata A.
        • Nicolini G.
        • Cozzi L.
        • Seelentag W.W.
        • Born E.
        • et al.
        The Swiss IMRT dosimetry intercomparison using a thorax phantom.
        Med Phys. 2010; 37: 4424
        • Budgell G.
        • Berresford J.
        • Trainer M.
        • Bradshaw E.
        • Sharpe P.
        • Williams P.
        A national dosimetric audit of IMRT.
        Radiother Oncol. 2011; 99: 246-252
        • Clark C.H.
        • Hussein M.
        • Tsang Y.
        • Thomas R.
        • Wilkinson D.
        • Bass G.
        • et al.
        A multi-institutional dosimetry audit of rotational intensity-modulated radiotherapy.
        Radiother Oncol. 2014; 113: 272-278
        • Gershkevitsh E.
        • Pesznyak C.
        • Petrovic B.
        • Grezdo J.
        • Chelminski K.
        • do Carmo Lopes M.
        • et al.
        Dosimetric inter-institutional comparison in European radiotherapy centres: Results of IAEA supported treatment planning system audit.
        Acta Oncol. 2014; 53: 628-636
        • Izewska J.
        • Wesolowska P.
        • Azangwe G.
        • Followill D.S.
        • Thwaites D.I.
        • Arib M.
        • et al.
        Testing the methodology for dosimetry audit of heterogeneity corrections and small MLC-shaped fields: results of IAEA multi-center studies.
        Acta Oncol. 2016; 55: 909-916
        • Jurado-Bruggeman D.
        • Hernández V.
        • Sáez J.
        • Navarro D.
        • Pino F.
        • Martínez T.
        • et al.
        Multi-centre audit of VMAT planning and pre-treatment verification.
        Radiother Oncol. 2017; 124: 302-310
        • Distefano G.
        • Lee J.
        • Jafari S.
        • Gouldstone C.
        • Baker C.
        • Mayles H.
        • et al.
        A national dosimetry audit for stereotactic ablative radiotherapy in lung.
        Radiother Oncol. 2017; 122: 406-410
        • Low D.A.
        • Harms W.B.
        • Mutic S.
        • Purdy J.A.
        A technique for the quantitative evaluation of dose distributions.
        Med Phys. 1998; 25: 656-661
        • Hussein M.
        • Rowshanfarzad P.
        • Ebert M.A.
        • Nisbet A.
        • Clark C.H.
        A comparison of the gamma index analysis in various commercial IMRT/VMAT QA systems.
        Radiother Oncol. 2013; 109: 370-376
        • Zhen H.
        • Nelms B.E.
        • Tome W.A.
        Moving from gamma passing rates to patient DVH-based QA metrics in pretreatment dose QA.
        Med Phys. 2011; 38: 5477-5489
        • Crowe S.B.
        • Sutherland B.
        • Wilks R.
        • Seshadri V.
        • Sylvander S.
        • Trapp J.V.
        • et al.
        Technical Note: Relationships between gamma criteria and action levels: Results of a multicenter audit of gamma agreement index results.
        Med Phys. 2016; 43: 1501
        • Haworth A.
        • Kearvell R.
        • Greer P.B.
        • Hooton B.
        • Denham J.W.
        • Lamb D.
        • et al.
        Assuring high quality treatment delivery in clinical trials – results from the Trans-Tasman Radiation Oncology Group (TROG) study 03.04 “RADAR” set-up accuracy study.
        Radiother Oncol. 2009; 90: 299-306
        • Weber D.C.
        • Poortmans P.M.
        • Hurkmans C.W.
        • Aird E.
        • Gulyban A.
        • Fairchild A.
        Quality assurance for prospective EORTC radiation oncology trials: the challenges of advanced technology in a multicenter international setting.
        Radiother Oncol. 2011; 100: 150-156
        • Weber D.C.
        • Tomsej M.
        • Melidis C.
        • Hurkmans C.W.
        QA makes a clinical trial stronger: Evidence-based medicine in radiation therapy.
        Radiother Oncol. 2012; 105: 4-8
        • Ebert M.A.
        • Harrison K.M.
        • Cornes D.
        • Howlett S.J.
        • Joseph D.J.
        • Kron T.
        • et al.
        Comprehensive Australasian multicentre dosimetric intercomparison: issues, logistics and recommendations.
        J Med Imaging Radiat Oncol. 2009; 53: 119-131
        • Peters L.J.
        • O’Sullivan B.
        • Giralt J.
        • Fitzgerald T.J.
        • Trotti A.
        • Bernier J.
        • et al.
        Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02.
        J Clin Oncol. 2010; 28: 2996-3001
        • Eaton D.J.
        • Tyler J.
        • Backshall A.
        • Bernstein D.
        • Carver A.
        • Gasnier A.
        • et al.
        An external dosimetry audit programme to credential static and rotational IMRT delivery for clinical trials quality assurance.
        Phys Med. 2017; 35: 25-30
        • Weber D.C.
        • Vallet V.
        • Molineu A.
        • Melidis C.
        • Teglas V.
        • Naudy S.
        • et al.
        IMRT credentialing for prospective trials using institutional virtual phantoms: results of a joint European Organization for the Research and Treatment of Cancer and Radiological Physics Center project.
        Radiat Oncol. 2014; 9: 123
        • Lye J.
        • Kenny J.
        • Lehmann J.
        • Dunn L.
        • Kron T.
        • Alves A.
        • et al.
        A 2D ion chamber array audit of wedged and asymmetric fields in an inhomogeneous lung phantom.
        Med Phys. 2014; 41: 101712
        • Miri N.
        • Lehmann J.
        • Legge K.
        • Vial P.
        • Greer P.B.
        Virtual EPID standard phantom audit (VESPA) for remote IMRT and VMAT credentialing.
        Phys Med Biol. 2017; 62: 4293
        • Melidis C.
        • Bosch W.R.
        • Izewska J.
        • Fidarova E.
        • Zubizarreta E.
        • Ishikura S.
        • et al.
        Radiation therapy quality assurance in clinical trials – global harmonisation group.
        Radiother Oncol. 2014; 111: 327-329
      1. Global Quality Assurance of Radiation Therapy Clinical Trials Harmonisation Group – Ensuring Quality Cancer Treatment Worldwide n.d. https://rtqaharmonization.com/ (accessed May 29, 2017).

        • Melidis C.
        • Bosch W.R.
        • Izewska J.
        • Fidarova E.
        • Zubizarreta E.
        • Ulin K.
        • et al.
        Global harmonization of quality assurance naming conventions in radiation therapy clinical trials.
        Int J Radiat Oncol Biol Phys. 2017; 90: 1242-1249
        • Clark C.H.
        • Hurkmans C.W.
        • Kry S.F.
        The role of dosimetry audit in lung SBRT multi-centre clinical trials.
        Phys Med. 2017;
        • Tsang Y.
        • Ciurlionis L.
        • Clark C.
        • Venables K.
        Development of a novel treatment planning test for credentialing rotational intensity-modulated radiotherapy techniques in the UK.
        Br J Radiol. 2012; 86: 20120315
        • Hussein M.
        • Adams E.J.
        • Jordan T.J.
        • Clark C.H.
        • Nisbet A.
        A critical evaluation of the PTW 2D-Array seven29 and Octavius II phantom for IMRT and VMAT verification.
        J Appl Clin Med Phys. 2013; 14: 4460
        • Nelms B.E.
        • Zhen H.
        • Tomé W.A.
        Per-beam, planar IMRT QA passing rates do not predict clinically relevant patient dose errors.
        Med Phys. 2011; 38: 1037
        • Carver A.
        • Gilmore M.
        • Riley S.
        • Uzan J.
        • Mayles P.
        An analytical approach to acceptance criteria for quality assurance of Intensity Modulated Radiotherapy.
        Radiother Oncol. 2011; 100: 453-455
        • Cozzolino M.
        • Oliviero C.
        • Califano G.
        • Clemente S.
        • Pedicini P.
        • Caivano R.
        • et al.
        Clinically relevant quality assurance (QA) for prostate RapidArc plans: gamma maps and DVH-based evaluation.
        Phys Medica. 2014; 30: 462-472
        • Hussein M.
        • Clark C.H.
        • Nisbet A.
        Challenges in calculation of the gamma index in radiotherapy – towards good practice.
        Phys Med. 2017; 36: 1-11
        • Agnew C.E.
        • McGarry C.K.
        A tool to include gamma analysis software into a quality assurance program.
        Radiother Oncol. 2016; 118: 568-573
        • Masi L.
        • Casamassima F.
        • Doro R.
        • Francescon P.
        Quality assurance of volumetric modulated arc therapy: Evaluation and comparison of different dosimetric systems.
        Med Phys. 2011; 38: 612
        • Fredh A.
        • Scherman J.B.
        • Fog L.S.
        Rosenschold PM af. Patient QA systems for rotational radiation therapy: a comparative experimental study with intentional errors.
        Med Phys. 2013; 40: 31716-31719
        • Huang J.Y.
        • Pulliam K.B.
        • McKenzie E.M.
        • Followill D.S.
        • Kry S.F.
        Effects of spatial resolution and noise on gamma analysis for IMRT QA.
        J Appl Clin Med Phys. 2014; 15: 4690
        • Jornet N.
        • Carrasco P.
        • Beltrán M.
        • Calvo J.F.
        • Escudé L.
        • Hernández V.
        • et al.
        Multicentre validation of IMRT pre-treatment verification: comparison of in-house and external audit.
        Radiother Oncol. 2014; 112: 381-388