**5. Addressing between-study heterogeneity in meta-analyses of PTSD RCTs**

Meta-analyses on the effectiveness of psychotherapeutic treatments for PTSD revealed between-study heterogeneity in any kind of RCTs—those that compared psychotherapy with wait list, with a psychological placebo control and those that compared two types of rival PTSD psychotherapies. But not all meta-analyses reporting heterogeneous results made attempts to explore potential sources of the observed heterogeneity (e.g. several of the comparisons in Bisson and Andrew [23]).

A typical approach in meta-analyses to deal with the presence of between-study heterogeneity is to identify characteristics of the included studies that systematically differentiate studies with larger or smaller effect sizes (i.e. so-called moderators or effect modifiers [43, 47]. Every characteristic of a study that is associated with the treatment effect may also act as a moderator in a meta-analysis. Two statistical approaches are used in order to identify relevant moderators: stratification of analyses by potential moderators and meta-regression analyses [35]. In stratified meta-analysis, the effect estimates of the subgroups of studies with and without a particular characteristic are contrasted. If effect estimates differ significantly in the contrasted subgroups or heterogeneity is reduced in at least one of the contrasted subgroups, the respective study characteristic may be interpreted as relevant moderator. Meta-regression analysis provides a statistical test for the exploration of sources of heterogeneity in metaanalyses [48].

The following paragraphs will give examples of different kinds of study characteristics that have been shown to moderate the pooled effect estimates in meta-analyses on PTSD treatments. We will focus on meta-analytic findings that summarized data from placebo-controlled and comparative RCTs as those two designs are informative with respect to the identification of *characteristic* treatment components in PTSD treatments and thus to the two highlighted research questions.

#### **5.1. Moderators in placebo-controlled PTSD RCTs**

already been present in a subgroup of poor-quality studies. The pooled effect estimate across all studies as well as the pooled effect estimate of the poor-quality subgroup of studies will be biased. In this case, only the effect estimate of the high-quality subgroup of studies may be regarded as valid. Second, the *apples-and-oranges problem* reflects meaningful variation between effect estimates due to dissimilarity between studies on relevant study characteristics (i.e. genuine diversity). That is, the pooled effect estimate across all included studies may be invalid whereas the pooled effect estimates in each subgroup of studies may be valid. Third, if the *filedrawer problem* was present, the pooled effect estimate of published studies is expected to differ from the pooled effect estimate of unpublished studies, as study results may more likely be published if they have statistically significant results. Many of the unpublished studies may, therefore, have non-significant results. Thus, a meta-analysis restricted to the published studies would probably provide a higher result compared to a meta-analysis restricted to the unpublished studies [45]. Only including both, published and unpublished studies, would warrant the validity of the effect-estimate. The difference between published and unpublished studies should particularly be the case if the study samples are small [46], which further complicates the issue. If a meta-analysis considers only published studies and non-significant results are most likely to be lacking if a study was small in scale, publication bias should be most potent in the small-scale studies and less pronounced or even not present in the large-scale studies. In this case (i.e. when including only published studies), the pooled effect estimate including all studies as well as the pooled effect estimate restricted to small-scale studies might be biased, whereas the effect estimate in the large-scale studies will be most valid with respect to publication bias. It is important to note, however, that the presence of any of the three problems indicates only an increased probability of bias in a meta-analysis rather than being necessarily

236 A Multidimensional Approach to Post-Traumatic Stress Disorder - from Theory to Practice

associated with bias in the meta-analytically pooled effect estimates.

**RCTs**

Bisson and Andrew [23]).

**5. Addressing between-study heterogeneity in meta-analyses of PTSD**

Meta-analyses on the effectiveness of psychotherapeutic treatments for PTSD revealed between-study heterogeneity in any kind of RCTs—those that compared psychotherapy with wait list, with a psychological placebo control and those that compared two types of rival PTSD psychotherapies. But not all meta-analyses reporting heterogeneous results made attempts to explore potential sources of the observed heterogeneity (e.g. several of the comparisons in

A typical approach in meta-analyses to deal with the presence of between-study heterogeneity is to identify characteristics of the included studies that systematically differentiate studies with larger or smaller effect sizes (i.e. so-called moderators or effect modifiers [43, 47]. Every characteristic of a study that is associated with the treatment effect may also act as a moderator in a meta-analysis. Two statistical approaches are used in order to identify relevant moderators: stratification of analyses by potential moderators and meta-regression analyses [35]. In stratified meta-analysis, the effect estimates of the subgroups of studies with and without a particular characteristic are contrasted. If effect estimates differ significantly in the contrasted One meta-analysis [7] summarized RCTs which compared treatments that somehow focused on the trauma with treatments lacking a trauma focus. The initial overall analysis showed a moderate superiority of the trauma-focused over the non-trauma-focused treatments with moderate between-study heterogeneity. On closer examination, the extent of structural equivalence (i.e. that therapists in both treatment conditions were equally trained and supervised and that the number of sessions was equivalent in both treatment conditions) substantially moderated the initially observed differences when all studies were included in the analysis. In the stratified meta-analyses, the superiority of trauma-focused treatments over the placebo controls was larger in studies without equivalence between the two treatment conditions, which was most likely due to an underestimation of the efficacy of the placebo control. Heterogeneity was considerably reduced in the stratified analyses. Furthermore, the initially observed superiority of trauma-focused treatments over placebo controls was moderated by patient characteristics: by combining several indicators of more complex clinical problems (e.g. the presence of comorbid disorders in addition to the PTSD symptoms or trauma history, as suggested by Cloitre and colleagues [49]), the moderator analysis conducted by Gerger and colleagues [7] revealed that patients with more complex clinical problems benefited equally from trauma-focused PTSD treatments as well as from the psychological placebo control. In contrast, in studies with less complex problems (e.g. PTSD symptoms without comorbid symptoms following a single trauma), patients benefited more from the traumafocused treatments than from the placebo control. Again, the inclusion of the moderator reduced heterogeneity, and only a small amount of between-study heterogeneity remained unexplained. Importantly, in studies with less complex clinical problems and structural inequivalence, a clear superiority of trauma-focused over placebo control treatments was observed (ES = 0.93; *p* = 0.001), whereas this was not the case in studies with complex clinical problems and structural equivalence of the trauma-focused treatment and placebo control (ES = 0.11; *p* = 0.28).

#### **5.2. Moderators in comparative PTSD RCTs**

A concrete example of the presence of between-study heterogeneity, i.e. contradicting findings from individual studies that compared two rival PTSD treatments, can be found in the review by Bisson and Andrew [23]. In one of the conducted meta-analyses out of six studies that compared CBT and EMDR, three studies reported moderate to large superiority of CBT on clinician-rated PTSD scores, while the remaining three studies reported the exact opposite effect, namely moderate to large superiority of EMDR over CBT. Overall, the meta-analysis indicated no difference between the effects of the two treatments, but a large amount of between-study heterogeneity (ES = 0.03, *p* = 0.92, *τ*<sup>2</sup> = 0.28). Without further exploration of the observed heterogeneity, no valid conclusions may be drawn from such data [35].

Several meta-analyses aimed at explaining such heterogeneity between individual study estimates in PTSD RCTs. Two meta-analyses [14, 34] included different types of PTSD treatments, but found no evidence for the type of psychotherapeutic PTSD treatment to explain between-study heterogeneity. Rather, Gerger et al. [14] found evidence for the presence of publication bias with respect to the trauma-focused PTSD treatments: a meta-analysis that was restricted to large-scale studies demonstrated considerably reduced treatment effects compared to the effects found in the overall analysis or in an analysis that was restricted to smallscale studies. The between-study heterogeneity, which was very large in the initial analysis (*τ*<sup>2</sup> = 0.29), was considerably reduced in the analysis restricted to large-scale trials (*τ*<sup>2</sup> = 0.08).

One possible explanation for the striking differences in the direction of effects between two treatments as in the EMDR-CBT comparison by Bisson and Andrew [23] is the presence of researchers' preferences for one over the other treatment, the so-called researcher allegiance [50]. Accordingly, the intriguing pattern of results in the EMDR-CBT meta-analysis by Bisson and Andrew [23] could simply be explained by the fact that in one half of the studies researchers preferred CBT and in the other half researchers preferred EMDR. While, by chance, in this particular case, the distribution of researcher allegiance appears balanced across the six included studies, an unbalanced preference for one particular treatment may be more problematic. In fact, a meta-analysis on trauma-focused PTSD treatments found researcher allegiance to significantly correlate with effect-size differences between the trauma-focused PTSD treatments (*r* = 0.35) and to explain between-study heterogeneity [15]. Further, Munder and colleagues presented evidence for the assumption that the association between researcher allegiance and outcome was due to bias [16] and against the assumption that true differences in the effectiveness of different types of PTSD treatments explained the association between researcher allegiance and outcome [15].

Thus, meta-analyses on comparative PTSD RCTs failed to demonstrate the superiority of particular *characteristic* components, but demonstrated the relevance of researcher allegiance —a factor that is *incidental* to the treatment—in explaining differences between individual study results. Thus, in the case of PTSD outcome studies, comparative RCTs run a considerable risk of providing biased estimates of the contribution of *characteristic* treatment components to the entire treatment effect. Furthermore, while on first sight meta-analyses of placebocontrolled PTSD RCTs appeared to support the claim that focusing on the trauma is necessary for successful PTSD treatment, a closer examination of potential moderators of treatment effects indicated that a trauma focus might be necessary for some but not all patient samples. A thorough implementation of the assumed psychological placebo might further enhance its effectiveness and, hence, reduce the superiority of trauma-focused treatments over placebo controls. The finding of only a small and non-significant superiority of established PTSD treatments over present-centered therapy in a meta-analysis [6], as well as a recent metaanalysis on counseling treatments for PTSD [5], confirm the objection regarding a general necessity of a trauma focus in psychotherapeutic PTSD treatment.
