Addiction 2017 (in press). [doi: 10.1111/add.13789]
Ryoka Suskida, PhD, Rosa M. Crum, MD, Cyrus Ebnesajjad, MA, Elizabeth A. Stuart, PhD, Ramin Mojtabai, MD (all from Johns Hopkins).
There is growing concern that the results from randomized controlled trials (RCTs) might not generalize to real world settings, particularly in the context of RCTs of treatments for substance use disorders (SUDs). Limitations in generalizability of the findings from RCTs pose major clinical and policy concerns because RCTs are considered the most accepted study design for choosing evidence-based practices. The randomized study design does not necessarily ensure external validity, which means that the findings of an RCT may not be applicable to all individuals for whom treatment or intervention is intended. Individuals who volunteer to participate in RCTs are typically different from those who refuse to participate. Furthermore, strict eligibility criteria are likely to make the findings less applicable to subgroups who are excluded from trials.
This study had two main aims: 1) to estimate sample treatment effects and the population effects of RCTs of SUD treatment, and 2) to examine the treatment effect heterogeneity by subgroups that are under- or over-represented in a set of RCTs conducted by the NIDA Clinical Trials Network.
Statistical weighting was used to re-compute the effects from ten CTN RCTs such that the participants in the trials had characteristics that resembled those of patients in the target populations. Three outcomes of SUD treatment were examined: retention, urine toxicology, and abstinence. The RCT sample treatment effects were weighted using propensity scores representing the conditional probability of participating in RCTs.
Weighting the samples changed the significance of estimated sample treatment effects. Most commonly, positive effects of trials became statistically non-significant after weighting (three trials for retention and urine toxicology, and one trial for abstinence); but also, non-significant effects became significantly positive (one trial for abstinence), and significantly negative effects became non-significant (two trials for abstinence). There was suggestive evidence of treatment effect heterogeneity in subgroups that are under- or over-represented in the trials, some of which were consistent with the differences in average treatment effects between weighted and unweighted results.
Conclusions: The findings of randomized controlled trials (RCTs) for substance use disorder treatment do not appear to be directly generalizable to target populations when the RCT samples do not adequately reflect the target populations and there is treatment effect heterogeneity across patient subgroups. Results from this study provide a first insight into whether and how deviations in RCT sample representativeness from target populations influence the observed outcomes of SUD RCTs. It is critical for future CTN studies to place greater emphasis on external validity of RCTs, particularly because a primary goal of the NIDA CTN was to provide data on SUD treatment that can be disseminated in usual care settings. (Article (Peer-Reviewed), PDF, English, 2017)
Keywords: CTN platform/ancillary study | Evidence-based treatment | Research design | Research participation | Addiction (journal)
Document No: 1256.
Submitted by the CTN Dissemination Librarians (2/22/2017).