Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review
1 Nordic Cochrane Centre, Rigshospitalet, DK-2100 Copenhagen , Denmark,2 Department of Biostatistics, Panum Institute, University of Copenhagen, Denmark
Abstract
Objective To compare the methodological quality and conclusions in Cochrane reviews with those in industry supported meta-analyses and other meta-analyses of the same drugs.
Design Systematic review comparing pairs of meta-analyses that studied the same two drugs in the same disease and were published within two years of each other.
, http://www.100md.com
Data sources Cochrane Database of Systematic Reviews (2003, issue 1), PubMed, and Embase.
Data extraction Two observers independently extracted data and used a validated scale to judge the methodological quality of the reviews.
Results 175 of 1596 Cochrane reviews had a meta-analysis that compared two drugs. Twenty four meta-analyses that matched the Cochrane reviews were found: eight were industry supported, nine had undeclared support, and seven had no support or were supported by non-industry sources. On a 0-7 scale, the median quality score was 7 for Cochrane reviews and 3 for other reviews (P < 0.01). Compared with industry supported reviews and reviews with undeclared support, Cochrane reviews had more often considered the potential for bias in the review—for example, by describing the method of concealment of allocation and describing excluded patients or studies. The seven industry supported reviews that had conclusions recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P = 0.02), although the estimated treatment effect was similar on average (z = 0.46, P = 0.64). Reviews with undeclared support and reviews with not for profit support or no support had conclusions that were similar in cautiousness to the Cochrane reviews.
, 百拇医药
Conclusions Industry supported reviews of drugs should be read with caution as they were less transparent, had few reservations about methodological limitations of the included trials, and had more favourable conclusions than the corresponding Cochrane reviews.
Introduction
Bias in drug trials is common and often favours the sponsor's product.1-3 Critical, systematic reviews that aggregate the available information in a neutral manner are therefore essential. Cochrane reviews aim to minimise bias and avoid conflicts of interest,4 and, on average, they may have greater methodological rigour than systematic reviews published in paper based journals.5 6 We therefore hypothesised that Cochrane reviews would be more transparent and less biased than industry supported systematic reviews. We aimed to compare Cochrane reviews with other meta-analyses of the same drugs, which we divided into those that had industry support, those with undeclared support, and those that had non-profit support or no support.
, http://www.100md.com
Methods
We searched for pairs that consisted of a Cochrane review and a similar review in a paper based journal. A Cochrane review was eligible if it used meta-analysis to compare at least two different drugs or classes of drugs; was published in the Cochrane Database of Systematic Reviews 2003, issue 1; could be matched with a metaanalysis of the same drugs and diseases published in full in a paper based journal within two years before or after the most recent substantive amendment of the Cochrane review; and had no authors in common with the Cochrane review.
, 百拇医药
We defined support by the pharmaceutical industry as provision of grants, authorship, or other major assistance such as help with the statistical analysis. We did not consider provision of references or unpublished trial reports as support.
One investigator (AWJ) hand searched all reviews in the Cochrane Database of Systematic Reviews 2003, issue 1 for drug comparisons. For each potentially eligible Cochrane review, we sought possibly eligible paper based reviews by searching PubMed (January 1966 to July 2003) for the same diseases and drugs combined with "meta-analysis" or meta-analysis[pt]. From online inspection of titles and abstracts, we selected metaanalyses for examination of the full text. When we found no match in PubMed, we searched Embase (WebSPIRS 5) (1980 to August 2003). When we found more than one match with the same type of support, we chose the one with the closest publication date to the Cochrane review.
, http://www.100md.com
AWJ and PCG independently assessed each pair of reviews in random order, by reading the Cochrane review first in half of the pairs. We used a pilot tested data sheet and resolved disagreements by discussion. We were not blinded. We extracted data on the date of the most recent substantive amendment to the Cochrane review and of the publication of the paper based review; names of relevant drugs and diseases; types of support; number and type of sources used to identify trials for the review; searches for unpublished trials; and descriptions of concealment of allocation, details of blinding, and excluded patients and trials.
, http://www.100md.com
We assessed the methodological quality of the reviews with Oxman and Guyatt's index, which is a validated tool with nine items considering the potential for bias and an overall assessment on a 0-7 scale.7-9 We judged the review authors' conclusions by assessing whether the experimental intervention was recommended without reservations or whether it was not recommended or recommended only with reservations.3 As our data were paired, we compared quantitative data with the Wilcoxon-Pratt one sample rank sum test and binary data with a sign test; P values are two sided.
, 百拇医药
For the industry supported reviews, we also assessed whether the estimated treatment effects were different from those reported in the Cochrane reviews. We used the first reported outcome with data in the industry supported review that was also presented in the Cochrane review, and we also did the corresponding analysis in which we started with the first reported outcome in the Cochrane review. We calculated pooled comparative z scores, after adjustment for the number of patients contributing to the outcome and for the number of patients that were common to the two analyses (see appendix on bmj.com for details).
, http://www.100md.com
Results
The Cochrane Database of Systematic Reviews 2003, issue 1 contained 1596 Cochrane reviews, of which we excluded 1421, mostly because they did not compare drugs (figure). In 72 of the remaining 175 reviews, the Medline and Embase searches identified potentially eligible paper based reviews, of which we excluded 48, mostly because their publication date differed by more than two years from the last substantive update of the Cochrane review, resulting in 24 matched pairs.w1-w48 One Cochrane revieww1 was paired with a paper based review with industry supportw2 and also with one with undeclared supportw3; one of these reviewsw2 presented only a subgroup analysis, and we therefore substituted it with an older review that contained the same trials.w4 In eight of the 24 pairs the paper based reviews had industry support,w1 w4-w18 in nine they had undeclared support,w1 w3 w19-w34 and in seven they had non-profit or no support.w35-w48
, http://www.100md.com
[PowerPoint Slide for Teaching]
Searches for pairs of reviews (first or most obvious reason for exclusion indicated)
The overall median quality score was 7 for the 24 Cochrane reviews and 3 for the other reviews (P < 0.001; table A on bmj.com). The mean years of publication for the three sets of pairs were 2000 versus 2000 for Cochrane reviews versus industry supported reviews, 2000 versus 1999 for Cochrane reviews versus reviews with undeclared support, and 2000 versus 2000 for Cochrane reviews versus reviews with non-profit or no support; the median differences in number of included trials were 0, 1, and 1 (table B on bmj.com).
, 百拇医药
Cochrane reviews versus industry supported reviews (eight pairs)
Cochrane reviews were of higher quality than industry supported reviews (P < 0.01). They also more often stated the search methods used to find studies (P = 0.06), searched comprehensively (P = 0.06), avoided bias in the selection of studies (P = 0.03), reported criteria for assessing the validity of the studies (P = 0.03), used appropriate criteria in assessing the studies (P < 0.01) (table A), described methods of concealment of allocation (P = 0.02), and described excluded patients (P = 0.03) and studies (P = 0.03), and they used more sources to identify studies (P = 0.02) (table B).
, 百拇医药
One of the industry supported reviews had no conclusion, as it referred to physiological characteristics of the drug.w6 The other seven reviews supported by industry all recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P < 0.01). This difference was related to interpretation of the data (table) and consideration of costs. The authors of six of the eight Cochrane reviews had reservations about the quality or relevance of the trials or their findings,w1 w5 w7 w9 w11 w17 and two noted that the effect decreased with increasing sample size.w5 w9 Seven mentioned the higher cost of the experimental drug as a problem compared with none of the industry supported reviews, of which two claimed that economic analyses had shown that the experimental drug was cost effective.w4 w18 In contrast to these interpretations, the estimated treatment effect was similar, on average, in the pairs of reviews (pooled z = 0.46, P = 0.64; appendix and table C on bmj.com). However, the scatter of the comparative z scores was high (2 = 19.4, df = 6, P = 0.004), which partly reflects differential inclusion of trials and patients despite our close matching (table C) and partly could be caused by selective or biased handling of data.
, 百拇医药
Characteristics of pairs of Cochrane reviews (C) and industry supported paper based reviews (I) of the same drugs
Cochrane reviews versus reviews with undeclared support (nine pairs)
The results for the comparison with reviews with undeclared support were similar to those for the industry supported reviews (table A), except that no significant differences existed for the stated search methods (P = 1.00) or efforts to avoid bias in the selection of studies (P = 0.22) and the recommendations were without reservations in one Cochrane review and in two other reviews (P = 1.00).
, http://www.100md.com
The paper based reviews were often biased and poorly done. One seemed to have included non-randomised studies.w22 Another included retrospective studies, had arbitrary entry criteria, and seemed to have preferentially selected those studies and data that were in favour of the experimental drug; the outcome was analysed in eight different ways, and the biggest difference was emphasised.w26 A third review presented a highly misleading result in the abstract that was based on an indirect comparison of treatment arms,w30 and, in contrast to the Cochrane review,w29 the authors failed to include all eligible studies and failed to detect sample size bias. A fourth review found no differences between the drugs but stated, with reference to an economic analysis: "Coupled with potential cost savings driven by the reduced need for hospitalization and revascularization procedures, it is not difficult to understand why clinicians, administrators and medical organizations representing the interests of physicians and the welfare of patients look favourably upon [low molecular weight heparin]."w3
, 百拇医药
Cochrane reviews versus reviews with non-profit or no support (seven pairs)
We found no significant differences between Cochrane reviews and reviews with non-profit or no support (table A). The recommendations were without reservations in two Cochrane reviews and in one other review.
However, in three pairs of reviews the conclusions favoured different drugs. In one set, although the same trials were included, the authors had different views on the trade-off between benefits and harms. The Cochrane review strongly supported the use of warfarin in patients with atrial fibrillation with average or greater risk of stroke,w35 whereas the paper based review strongly favoured antiplatelet drugs.w36 In another set, the paper review found better control of haemorrhage with octreotide.w40 The Cochrane review did not find any differences but referred to another meta-analysis when it emphasised that terlipressin is the only drug for which a reduction in mortality has been found compared with no treatment.w39 We have published a comment with this review, as we doubt that the effect on mortality is reliable.
, 百拇医药
In the third set, the paper review included only three double blind randomised trials and concluded that methadone should be used for heroin dependence; it noted in an addendum that LAAM (levomethadyl acetate hydrochloride) had been withdrawn from the market because of cardiotoxicity.w46 The Cochrane review included 18 studies, three of which were cohort studies that were not analysed separately from the randomised trials, and concluded that LAAM seems to be more effective than methadone.w45 A fourth Cochrane review found a major sample size bias but failed to make reservations in the abstract as regards the mortality benefit, which was based on all trials.w44
, 百拇医药
Discussion
We found that although some Cochrane reviews had clear methodological deficiencies, these were fewer, on average, than in reviews published in paper based journals.
Limitations
A minor limitation was that we could not be blinded, as the layout of Cochrane reviews is unique; blinding has little impact on extraction of data for reviews.10 A more important limitation is that our sample was small and needs to be replicated. Furthermore, we are affiliated with a Cochrane centre, and Cochrane reviews are done according to a handbook11 that was developed partly by Andy Oxman, who also participated in the development of the validated index that we used for evaluating methodological quality.9 12 To help readers to make their own judgments, we have provided details on the comparison with industry supported reviews (table) and added items to our data extraction sheet that are indisputably important for the reliability of reviews, such as adequate concealment of allocation.13 14 Our findings correspond to another recent finding that Cochrane reviews assess methodological quality more often than do other reviews,15 although because of space constraints some paper based reviews might have been conducted better than was reported.
, http://www.100md.com
Industry supported review
The estimated treatment effects in industry supported reviews were similar to those of Cochrane reviews, but the former had uniformly positive recommendations for the experimental drug, without reservations about methodological limitations of the trials or costs, in contrast to none of the Cochrane reviews. This suggests that the main problem with industry supported reviews lies in how conclusions are formulated.
, 百拇医药
We compared pairs of similar reviews published within a narrow time frame. Other such pairwise comparisons have been anecdotal. A meta-analysis found a similar drop-out rate for fluoxetine as for tricyclic antidepressants (P = 0.4),16 17 whereas a company employee reported a marked difference in favour of fluoxetine (P < 0.001)18 in subsequent correspondence. This is surprising, as the industry supported meta-analysis contained fewer patients and included "data on file" reports, which are usually less favourable than published ones.19 20 In contrast to industry supported authors, authors of Cochrane reviews often cannot get access to such data, as indicated in two of the reviews.w11 w37
, http://www.100md.com
As another example, a meta-analysis supported by Merck concluded in 2001 that no increased risk of arterial thrombosis existed with the company's drug rofecoxib,21 but a meta-analysis not supported by industry showed an increased risk,22 which was apparent in publications available to the authors of the industry supported meta-analysis. Rofecoxib was withdrawn because of thromboses in 2004.
The influence of industry on trial reports is similar to our findings. A survey found that none of 56 trials of non-steroidal anti-inflammatory drugs supported by the manufacturer presented results that were unfavourable to the company.23 Another survey found that the conclusions recommended the experimental drug as the drug of choice five times as often if the trial was funded by for profit organisations, even after adjustment for the effect size.3
, 百拇医药
Reviews with undeclared support
The conclusions of paper based reviews with undeclared support were more cautious than those for industry supported reviews. We contacted the authors after we had assembled our data. Eight declared that they had not received any external funding or other type of support (which we exemplified as help with the statistical analyses), and one replied that he had not received any other financial support.w3 We do not know whether these replies were comprehensive, and we suspect that some authors had received undeclared support or had allowed the company to review the paper and insert text, as suggested by the recommendation for low molecular weight heparin above.w3
, http://www.100md.com
Interpretation of financial support
The interpretation of financial support is not always straightforward. The authors of a paper based review that we classified as "non-profit support" noted that it was supported in part by public sources but did not describe the nature of the other part, and two of the authors had previously received "unrestricted grants" from the manufacturer of octreotide.w40 The authors of the matching Cochrane review declared that they had no conflicts of interest but added that they had "no permanent financial contracts" with companies producing the comparator, terlipressin.w39 Finally, one of the Cochrane reviews had industry support, as Upjohn had funded secondary analyses of the author's own trial for use in the review.w47 The current policy in the Cochrane Collaboration is that industry support of Cochrane reviews is not acceptable.24
, 百拇医药
Other problems with reviews
Less rigorously controlled studies than ours have reported on discrepant conclusions between systematic reviews assessing the same subject. The major reasons were incomplete searches, differential inclusion of trials, insufficient attention to the quality of the trials and to bias detection, and differences in interpretation.25-32 Some reviews missed more than half of the available trials,26-28 and a review of meta-analyses of analgesic interventions found that those with positive conclusions had lower quality scores on the Oxman and Guyatt index.8 A recent study of antihypertensive drugs found that the conclusions of meta-analyses were positive in 91% of the papers with financial ties and in 72% of other papers.33 However, the conclusions of Cochrane reviews also tend to be too positive.34
, http://www.100md.com
Our examination of the comparative z scores revealed more scatter than expected, which indicates that some effect estimates might have been biased, and, furthermore, that the confidence interval in a meta-analysis generally exaggerates the precision in the underlying data.
Conclusions
Industry supported reviews of drugs are less transparent than Cochrane reviews and have few reservations about methodological limitations of the included trials; their conclusions should be read with caution. We believe that details of concealment of allocation, blinding, inclusion and exclusion criteria for trials, search strategies, and estimated effects in each included trial need to be reported to allow readers to judge the reliability of reviews. To improve transparency, access to the protocol should be available. Protocols for Cochrane reviews are published in the Cochrane Library, and protocols for other systematic reviews can be registered free of charge at the UK national research register through the Centre for Reviews and Dissemination in York, UK.
, 百拇医药
What is already known on this topic
Bias commonly occurs in trials of healthcare interventions and often favours the sponsor's product
Anecdotal reports have suggested that industry supported meta-analyses may also be more flawed than other meta-analyses
What this study adds
Industry supported reviews were of lesser quality than Cochrane reviews of the same drugs and always recommended the experimental drug without reservations, which none of the Cochrane reviews did
, 百拇医药
Industry supported meta-analyses of drugs were less transparent and had few reservations about methodological limitations of included trials
Reviews with undeclared support and those with not for profit support or no support had similarly cautious conclusions to matched Cochrane reviews
An appendix, three extra tables, and extra references w1-w48 are on bmj.com
We thank Stefan Leucht, Deborah J Cook, Deborah Goebert, Michael Ludwig, Max H Pittler, Chiel Springer, Steven L West, Frederick A Spencer, and Patrick Chien for providing information on support and funding of their studies and the Iberoamerican Cochrane Centre, Barcelona for providing study facilities.
, 百拇医药
Contributors: AWJ wrote the draft protocol and manuscript, and PCG contributed. AWJ and PCG extracted data. JH did the statistical analyses that compared estimated treatment effects. All authors commented on the final manuscript. PCG is the guarantor.
Funding: None.
Competing interests: AWJ and PCG are affiliated with the Nordic Cochrane Centre. The views expressed in this article represent those of the authors and are not necessarily the views or the official policy of the Cochrane Collaboration.
, 百拇医药
References
Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326: 1167-70.
Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA 2003;289: 454-65.
Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events JAMA 2003;290: 921-8.
, http://www.100md.com
The Cochrane Collaboration. The Cochrane Collaboration—principles. www.cochrane.org/docs/tenprinciples.htm (accessed 18 Sept 2006).
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280: 278-80.
Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, et al. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320: 537-40.
, 百拇医药
Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Assoc J 1988;138: 697-703.
Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol 1996;49: 235-43.
Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44: 1271-8.
Berlin JA. Does blinding of readers affect the results of meta-analyses Lancet 1997;350: 185-6.
, 百拇医药
Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions 4.2.5 [updated May 2005]. www.cochrane.org/resources/handbook/hbook.htm (accessed 19 Sept 2005).
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, et al. Agreement among reviewers of review articles. J Clin Epidemiol 1991;44: 91-8.
Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323: 42-6.
, 百拇医药
Juni P, Egger M. Allocation concealment in clinical trials. JAMA 2002;288: 2407-8.
Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 2005;330: 1053.
Song F, Freemantle N, Sheldon TA, House A, Watson P, Long A, et al. Selective serotonin reuptake inhibitors: meta-analysis of efficacy and acceptability. BMJ 1993;306: 683-7.
, http://www.100md.com
Smith G, Egger M. Meta-analysis: unresolved issues and future developments. BMJ 1998;316: 221-5.
Nakielny J. Effective and acceptable treatment for depression [letter]. BMJ 1993;306: 1125.
Stern JM, Simes JR. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640-5.
Hemminki E. Study of information submitted by drug companies to licensing authorities. BMJ 1980;280: 833-6.
, 百拇医药
Konstam MA, Weir MR, Reicin A, Shapiro D, Sperling RS, Barr E, et al. Cardiovascular thrombotic events in controlled, clinical trials of rofecoxib. Circulation 2001;104: 2280-8.
Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet 2004;364: 2021-9.
Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, Minaker KL, et al. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 1994;154: 157-63.
, 百拇医药
Cochrane Collaboration. Commercial sponsorship and the Cochrane Collaboration. www.cochrane.org/docs/commercialsponsorship.htm (accessed 16 June 2005).
Gtzsche PC. Steroids and peptic ulcer: an end to the controversy J Intern Med 1994;236: 599-601.
Katerndahl DA, Lawler WR. Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. Am J Epidemiol 1999;149: 429-41.
, 百拇医药
Chalmers TC, Berrier J, Sacks HS, Levin H, Reitman D, Nagalingam R. Meta-analysis of clinical trials as a scientific discipline. II: replicate variability and comparison of studies that agree and disagree. Stat Med 1987;6: 733-44.
Linde K, Willich SN. How objective are systematic reviews Differences between reviews on complementary medicine. J R Soc Med 2003;96: 17-22.
Gtzsche PC, Pdenphant J, Olesen M, Halberg P. Meta-analysis of second-line antirheumatic drugs: sample size bias and uncertain benefit. J Clin Epidemiol 1992;45: 587-94.
, http://www.100md.com
Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, et al. Stress ulcer prophylaxis in critically ill patients: resolving discordant meta-analyses. JAMA 1996;275: 308-14.
Olsen O, Gtzsche PC. Screening for breast cancer with mammography. Cochrane Database Syst Rev 2001;(4): CD001877.
Hopayian K. The need for caution in interpreting high quality systematic reviews. BMJ 2001;323: 681-4.
, 百拇医药
Yank V, Rennie D, Bero LA. Are authors' financial ties with pharmaceutical companies associated with positive results or conclusions in meta-analyses on antihypertensive medications 5th International Congress on Peer Review and Biomedical Publication, Chicago, 16-18 September 2005:16 (www.jama-peer.org).
Olsen O, Middleton P, Ezzo J, Gtzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H. Quality of Cochrane reviews: assessment of sample from 1998. BMJ 2001;323: 829-32., 百拇医药(Anders W Jrgensen, Jrgen )
Abstract
Objective To compare the methodological quality and conclusions in Cochrane reviews with those in industry supported meta-analyses and other meta-analyses of the same drugs.
Design Systematic review comparing pairs of meta-analyses that studied the same two drugs in the same disease and were published within two years of each other.
, http://www.100md.com
Data sources Cochrane Database of Systematic Reviews (2003, issue 1), PubMed, and Embase.
Data extraction Two observers independently extracted data and used a validated scale to judge the methodological quality of the reviews.
Results 175 of 1596 Cochrane reviews had a meta-analysis that compared two drugs. Twenty four meta-analyses that matched the Cochrane reviews were found: eight were industry supported, nine had undeclared support, and seven had no support or were supported by non-industry sources. On a 0-7 scale, the median quality score was 7 for Cochrane reviews and 3 for other reviews (P < 0.01). Compared with industry supported reviews and reviews with undeclared support, Cochrane reviews had more often considered the potential for bias in the review—for example, by describing the method of concealment of allocation and describing excluded patients or studies. The seven industry supported reviews that had conclusions recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P = 0.02), although the estimated treatment effect was similar on average (z = 0.46, P = 0.64). Reviews with undeclared support and reviews with not for profit support or no support had conclusions that were similar in cautiousness to the Cochrane reviews.
, 百拇医药
Conclusions Industry supported reviews of drugs should be read with caution as they were less transparent, had few reservations about methodological limitations of the included trials, and had more favourable conclusions than the corresponding Cochrane reviews.
Introduction
Bias in drug trials is common and often favours the sponsor's product.1-3 Critical, systematic reviews that aggregate the available information in a neutral manner are therefore essential. Cochrane reviews aim to minimise bias and avoid conflicts of interest,4 and, on average, they may have greater methodological rigour than systematic reviews published in paper based journals.5 6 We therefore hypothesised that Cochrane reviews would be more transparent and less biased than industry supported systematic reviews. We aimed to compare Cochrane reviews with other meta-analyses of the same drugs, which we divided into those that had industry support, those with undeclared support, and those that had non-profit support or no support.
, http://www.100md.com
Methods
We searched for pairs that consisted of a Cochrane review and a similar review in a paper based journal. A Cochrane review was eligible if it used meta-analysis to compare at least two different drugs or classes of drugs; was published in the Cochrane Database of Systematic Reviews 2003, issue 1; could be matched with a metaanalysis of the same drugs and diseases published in full in a paper based journal within two years before or after the most recent substantive amendment of the Cochrane review; and had no authors in common with the Cochrane review.
, 百拇医药
We defined support by the pharmaceutical industry as provision of grants, authorship, or other major assistance such as help with the statistical analysis. We did not consider provision of references or unpublished trial reports as support.
One investigator (AWJ) hand searched all reviews in the Cochrane Database of Systematic Reviews 2003, issue 1 for drug comparisons. For each potentially eligible Cochrane review, we sought possibly eligible paper based reviews by searching PubMed (January 1966 to July 2003) for the same diseases and drugs combined with "meta-analysis" or meta-analysis[pt]. From online inspection of titles and abstracts, we selected metaanalyses for examination of the full text. When we found no match in PubMed, we searched Embase (WebSPIRS 5) (1980 to August 2003). When we found more than one match with the same type of support, we chose the one with the closest publication date to the Cochrane review.
, http://www.100md.com
AWJ and PCG independently assessed each pair of reviews in random order, by reading the Cochrane review first in half of the pairs. We used a pilot tested data sheet and resolved disagreements by discussion. We were not blinded. We extracted data on the date of the most recent substantive amendment to the Cochrane review and of the publication of the paper based review; names of relevant drugs and diseases; types of support; number and type of sources used to identify trials for the review; searches for unpublished trials; and descriptions of concealment of allocation, details of blinding, and excluded patients and trials.
, http://www.100md.com
We assessed the methodological quality of the reviews with Oxman and Guyatt's index, which is a validated tool with nine items considering the potential for bias and an overall assessment on a 0-7 scale.7-9 We judged the review authors' conclusions by assessing whether the experimental intervention was recommended without reservations or whether it was not recommended or recommended only with reservations.3 As our data were paired, we compared quantitative data with the Wilcoxon-Pratt one sample rank sum test and binary data with a sign test; P values are two sided.
, 百拇医药
For the industry supported reviews, we also assessed whether the estimated treatment effects were different from those reported in the Cochrane reviews. We used the first reported outcome with data in the industry supported review that was also presented in the Cochrane review, and we also did the corresponding analysis in which we started with the first reported outcome in the Cochrane review. We calculated pooled comparative z scores, after adjustment for the number of patients contributing to the outcome and for the number of patients that were common to the two analyses (see appendix on bmj.com for details).
, http://www.100md.com
Results
The Cochrane Database of Systematic Reviews 2003, issue 1 contained 1596 Cochrane reviews, of which we excluded 1421, mostly because they did not compare drugs (figure). In 72 of the remaining 175 reviews, the Medline and Embase searches identified potentially eligible paper based reviews, of which we excluded 48, mostly because their publication date differed by more than two years from the last substantive update of the Cochrane review, resulting in 24 matched pairs.w1-w48 One Cochrane revieww1 was paired with a paper based review with industry supportw2 and also with one with undeclared supportw3; one of these reviewsw2 presented only a subgroup analysis, and we therefore substituted it with an older review that contained the same trials.w4 In eight of the 24 pairs the paper based reviews had industry support,w1 w4-w18 in nine they had undeclared support,w1 w3 w19-w34 and in seven they had non-profit or no support.w35-w48
, http://www.100md.com
[PowerPoint Slide for Teaching]
Searches for pairs of reviews (first or most obvious reason for exclusion indicated)
The overall median quality score was 7 for the 24 Cochrane reviews and 3 for the other reviews (P < 0.001; table A on bmj.com). The mean years of publication for the three sets of pairs were 2000 versus 2000 for Cochrane reviews versus industry supported reviews, 2000 versus 1999 for Cochrane reviews versus reviews with undeclared support, and 2000 versus 2000 for Cochrane reviews versus reviews with non-profit or no support; the median differences in number of included trials were 0, 1, and 1 (table B on bmj.com).
, 百拇医药
Cochrane reviews versus industry supported reviews (eight pairs)
Cochrane reviews were of higher quality than industry supported reviews (P < 0.01). They also more often stated the search methods used to find studies (P = 0.06), searched comprehensively (P = 0.06), avoided bias in the selection of studies (P = 0.03), reported criteria for assessing the validity of the studies (P = 0.03), used appropriate criteria in assessing the studies (P < 0.01) (table A), described methods of concealment of allocation (P = 0.02), and described excluded patients (P = 0.03) and studies (P = 0.03), and they used more sources to identify studies (P = 0.02) (table B).
, 百拇医药
One of the industry supported reviews had no conclusion, as it referred to physiological characteristics of the drug.w6 The other seven reviews supported by industry all recommended the experimental drug without reservations, compared with none of the Cochrane reviews (P < 0.01). This difference was related to interpretation of the data (table) and consideration of costs. The authors of six of the eight Cochrane reviews had reservations about the quality or relevance of the trials or their findings,w1 w5 w7 w9 w11 w17 and two noted that the effect decreased with increasing sample size.w5 w9 Seven mentioned the higher cost of the experimental drug as a problem compared with none of the industry supported reviews, of which two claimed that economic analyses had shown that the experimental drug was cost effective.w4 w18 In contrast to these interpretations, the estimated treatment effect was similar, on average, in the pairs of reviews (pooled z = 0.46, P = 0.64; appendix and table C on bmj.com). However, the scatter of the comparative z scores was high (2 = 19.4, df = 6, P = 0.004), which partly reflects differential inclusion of trials and patients despite our close matching (table C) and partly could be caused by selective or biased handling of data.
, 百拇医药
Characteristics of pairs of Cochrane reviews (C) and industry supported paper based reviews (I) of the same drugs
Cochrane reviews versus reviews with undeclared support (nine pairs)
The results for the comparison with reviews with undeclared support were similar to those for the industry supported reviews (table A), except that no significant differences existed for the stated search methods (P = 1.00) or efforts to avoid bias in the selection of studies (P = 0.22) and the recommendations were without reservations in one Cochrane review and in two other reviews (P = 1.00).
, http://www.100md.com
The paper based reviews were often biased and poorly done. One seemed to have included non-randomised studies.w22 Another included retrospective studies, had arbitrary entry criteria, and seemed to have preferentially selected those studies and data that were in favour of the experimental drug; the outcome was analysed in eight different ways, and the biggest difference was emphasised.w26 A third review presented a highly misleading result in the abstract that was based on an indirect comparison of treatment arms,w30 and, in contrast to the Cochrane review,w29 the authors failed to include all eligible studies and failed to detect sample size bias. A fourth review found no differences between the drugs but stated, with reference to an economic analysis: "Coupled with potential cost savings driven by the reduced need for hospitalization and revascularization procedures, it is not difficult to understand why clinicians, administrators and medical organizations representing the interests of physicians and the welfare of patients look favourably upon [low molecular weight heparin]."w3
, 百拇医药
Cochrane reviews versus reviews with non-profit or no support (seven pairs)
We found no significant differences between Cochrane reviews and reviews with non-profit or no support (table A). The recommendations were without reservations in two Cochrane reviews and in one other review.
However, in three pairs of reviews the conclusions favoured different drugs. In one set, although the same trials were included, the authors had different views on the trade-off between benefits and harms. The Cochrane review strongly supported the use of warfarin in patients with atrial fibrillation with average or greater risk of stroke,w35 whereas the paper based review strongly favoured antiplatelet drugs.w36 In another set, the paper review found better control of haemorrhage with octreotide.w40 The Cochrane review did not find any differences but referred to another meta-analysis when it emphasised that terlipressin is the only drug for which a reduction in mortality has been found compared with no treatment.w39 We have published a comment with this review, as we doubt that the effect on mortality is reliable.
, 百拇医药
In the third set, the paper review included only three double blind randomised trials and concluded that methadone should be used for heroin dependence; it noted in an addendum that LAAM (levomethadyl acetate hydrochloride) had been withdrawn from the market because of cardiotoxicity.w46 The Cochrane review included 18 studies, three of which were cohort studies that were not analysed separately from the randomised trials, and concluded that LAAM seems to be more effective than methadone.w45 A fourth Cochrane review found a major sample size bias but failed to make reservations in the abstract as regards the mortality benefit, which was based on all trials.w44
, 百拇医药
Discussion
We found that although some Cochrane reviews had clear methodological deficiencies, these were fewer, on average, than in reviews published in paper based journals.
Limitations
A minor limitation was that we could not be blinded, as the layout of Cochrane reviews is unique; blinding has little impact on extraction of data for reviews.10 A more important limitation is that our sample was small and needs to be replicated. Furthermore, we are affiliated with a Cochrane centre, and Cochrane reviews are done according to a handbook11 that was developed partly by Andy Oxman, who also participated in the development of the validated index that we used for evaluating methodological quality.9 12 To help readers to make their own judgments, we have provided details on the comparison with industry supported reviews (table) and added items to our data extraction sheet that are indisputably important for the reliability of reviews, such as adequate concealment of allocation.13 14 Our findings correspond to another recent finding that Cochrane reviews assess methodological quality more often than do other reviews,15 although because of space constraints some paper based reviews might have been conducted better than was reported.
, http://www.100md.com
Industry supported review
The estimated treatment effects in industry supported reviews were similar to those of Cochrane reviews, but the former had uniformly positive recommendations for the experimental drug, without reservations about methodological limitations of the trials or costs, in contrast to none of the Cochrane reviews. This suggests that the main problem with industry supported reviews lies in how conclusions are formulated.
, 百拇医药
We compared pairs of similar reviews published within a narrow time frame. Other such pairwise comparisons have been anecdotal. A meta-analysis found a similar drop-out rate for fluoxetine as for tricyclic antidepressants (P = 0.4),16 17 whereas a company employee reported a marked difference in favour of fluoxetine (P < 0.001)18 in subsequent correspondence. This is surprising, as the industry supported meta-analysis contained fewer patients and included "data on file" reports, which are usually less favourable than published ones.19 20 In contrast to industry supported authors, authors of Cochrane reviews often cannot get access to such data, as indicated in two of the reviews.w11 w37
, http://www.100md.com
As another example, a meta-analysis supported by Merck concluded in 2001 that no increased risk of arterial thrombosis existed with the company's drug rofecoxib,21 but a meta-analysis not supported by industry showed an increased risk,22 which was apparent in publications available to the authors of the industry supported meta-analysis. Rofecoxib was withdrawn because of thromboses in 2004.
The influence of industry on trial reports is similar to our findings. A survey found that none of 56 trials of non-steroidal anti-inflammatory drugs supported by the manufacturer presented results that were unfavourable to the company.23 Another survey found that the conclusions recommended the experimental drug as the drug of choice five times as often if the trial was funded by for profit organisations, even after adjustment for the effect size.3
, 百拇医药
Reviews with undeclared support
The conclusions of paper based reviews with undeclared support were more cautious than those for industry supported reviews. We contacted the authors after we had assembled our data. Eight declared that they had not received any external funding or other type of support (which we exemplified as help with the statistical analyses), and one replied that he had not received any other financial support.w3 We do not know whether these replies were comprehensive, and we suspect that some authors had received undeclared support or had allowed the company to review the paper and insert text, as suggested by the recommendation for low molecular weight heparin above.w3
, http://www.100md.com
Interpretation of financial support
The interpretation of financial support is not always straightforward. The authors of a paper based review that we classified as "non-profit support" noted that it was supported in part by public sources but did not describe the nature of the other part, and two of the authors had previously received "unrestricted grants" from the manufacturer of octreotide.w40 The authors of the matching Cochrane review declared that they had no conflicts of interest but added that they had "no permanent financial contracts" with companies producing the comparator, terlipressin.w39 Finally, one of the Cochrane reviews had industry support, as Upjohn had funded secondary analyses of the author's own trial for use in the review.w47 The current policy in the Cochrane Collaboration is that industry support of Cochrane reviews is not acceptable.24
, 百拇医药
Other problems with reviews
Less rigorously controlled studies than ours have reported on discrepant conclusions between systematic reviews assessing the same subject. The major reasons were incomplete searches, differential inclusion of trials, insufficient attention to the quality of the trials and to bias detection, and differences in interpretation.25-32 Some reviews missed more than half of the available trials,26-28 and a review of meta-analyses of analgesic interventions found that those with positive conclusions had lower quality scores on the Oxman and Guyatt index.8 A recent study of antihypertensive drugs found that the conclusions of meta-analyses were positive in 91% of the papers with financial ties and in 72% of other papers.33 However, the conclusions of Cochrane reviews also tend to be too positive.34
, http://www.100md.com
Our examination of the comparative z scores revealed more scatter than expected, which indicates that some effect estimates might have been biased, and, furthermore, that the confidence interval in a meta-analysis generally exaggerates the precision in the underlying data.
Conclusions
Industry supported reviews of drugs are less transparent than Cochrane reviews and have few reservations about methodological limitations of the included trials; their conclusions should be read with caution. We believe that details of concealment of allocation, blinding, inclusion and exclusion criteria for trials, search strategies, and estimated effects in each included trial need to be reported to allow readers to judge the reliability of reviews. To improve transparency, access to the protocol should be available. Protocols for Cochrane reviews are published in the Cochrane Library, and protocols for other systematic reviews can be registered free of charge at the UK national research register through the Centre for Reviews and Dissemination in York, UK.
, 百拇医药
What is already known on this topic
Bias commonly occurs in trials of healthcare interventions and often favours the sponsor's product
Anecdotal reports have suggested that industry supported meta-analyses may also be more flawed than other meta-analyses
What this study adds
Industry supported reviews were of lesser quality than Cochrane reviews of the same drugs and always recommended the experimental drug without reservations, which none of the Cochrane reviews did
, 百拇医药
Industry supported meta-analyses of drugs were less transparent and had few reservations about methodological limitations of included trials
Reviews with undeclared support and those with not for profit support or no support had similarly cautious conclusions to matched Cochrane reviews
An appendix, three extra tables, and extra references w1-w48 are on bmj.com
We thank Stefan Leucht, Deborah J Cook, Deborah Goebert, Michael Ludwig, Max H Pittler, Chiel Springer, Steven L West, Frederick A Spencer, and Patrick Chien for providing information on support and funding of their studies and the Iberoamerican Cochrane Centre, Barcelona for providing study facilities.
, 百拇医药
Contributors: AWJ wrote the draft protocol and manuscript, and PCG contributed. AWJ and PCG extracted data. JH did the statistical analyses that compared estimated treatment effects. All authors commented on the final manuscript. PCG is the guarantor.
Funding: None.
Competing interests: AWJ and PCG are affiliated with the Nordic Cochrane Centre. The views expressed in this article represent those of the authors and are not necessarily the views or the official policy of the Cochrane Collaboration.
, 百拇医药
References
Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326: 1167-70.
Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA 2003;289: 454-65.
Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events JAMA 2003;290: 921-8.
, http://www.100md.com
The Cochrane Collaboration. The Cochrane Collaboration—principles. www.cochrane.org/docs/tenprinciples.htm (accessed 18 Sept 2006).
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA 1998;280: 278-80.
Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, et al. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ 2000;320: 537-40.
, 百拇医药
Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Assoc J 1988;138: 697-703.
Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol 1996;49: 235-43.
Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol 1991;44: 1271-8.
Berlin JA. Does blinding of readers affect the results of meta-analyses Lancet 1997;350: 185-6.
, 百拇医药
Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions 4.2.5 [updated May 2005]. www.cochrane.org/resources/handbook/hbook.htm (accessed 19 Sept 2005).
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, et al. Agreement among reviewers of review articles. J Clin Epidemiol 1991;44: 91-8.
Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323: 42-6.
, 百拇医药
Juni P, Egger M. Allocation concealment in clinical trials. JAMA 2002;288: 2407-8.
Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, Liberati A. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ 2005;330: 1053.
Song F, Freemantle N, Sheldon TA, House A, Watson P, Long A, et al. Selective serotonin reuptake inhibitors: meta-analysis of efficacy and acceptability. BMJ 1993;306: 683-7.
, http://www.100md.com
Smith G, Egger M. Meta-analysis: unresolved issues and future developments. BMJ 1998;316: 221-5.
Nakielny J. Effective and acceptable treatment for depression [letter]. BMJ 1993;306: 1125.
Stern JM, Simes JR. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640-5.
Hemminki E. Study of information submitted by drug companies to licensing authorities. BMJ 1980;280: 833-6.
, 百拇医药
Konstam MA, Weir MR, Reicin A, Shapiro D, Sperling RS, Barr E, et al. Cardiovascular thrombotic events in controlled, clinical trials of rofecoxib. Circulation 2001;104: 2280-8.
Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet 2004;364: 2021-9.
Rochon PA, Gurwitz JH, Simms RW, Fortin PR, Felson DT, Minaker KL, et al. A study of manufacturer-supported trials of nonsteroidal anti-inflammatory drugs in the treatment of arthritis. Arch Intern Med 1994;154: 157-63.
, 百拇医药
Cochrane Collaboration. Commercial sponsorship and the Cochrane Collaboration. www.cochrane.org/docs/commercialsponsorship.htm (accessed 16 June 2005).
Gtzsche PC. Steroids and peptic ulcer: an end to the controversy J Intern Med 1994;236: 599-601.
Katerndahl DA, Lawler WR. Variability in meta-analytic results concerning the value of cholesterol reduction in coronary heart disease: a meta-meta-analysis. Am J Epidemiol 1999;149: 429-41.
, 百拇医药
Chalmers TC, Berrier J, Sacks HS, Levin H, Reitman D, Nagalingam R. Meta-analysis of clinical trials as a scientific discipline. II: replicate variability and comparison of studies that agree and disagree. Stat Med 1987;6: 733-44.
Linde K, Willich SN. How objective are systematic reviews Differences between reviews on complementary medicine. J R Soc Med 2003;96: 17-22.
Gtzsche PC, Pdenphant J, Olesen M, Halberg P. Meta-analysis of second-line antirheumatic drugs: sample size bias and uncertain benefit. J Clin Epidemiol 1992;45: 587-94.
, http://www.100md.com
Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, et al. Stress ulcer prophylaxis in critically ill patients: resolving discordant meta-analyses. JAMA 1996;275: 308-14.
Olsen O, Gtzsche PC. Screening for breast cancer with mammography. Cochrane Database Syst Rev 2001;(4): CD001877.
Hopayian K. The need for caution in interpreting high quality systematic reviews. BMJ 2001;323: 681-4.
, 百拇医药
Yank V, Rennie D, Bero LA. Are authors' financial ties with pharmaceutical companies associated with positive results or conclusions in meta-analyses on antihypertensive medications 5th International Congress on Peer Review and Biomedical Publication, Chicago, 16-18 September 2005:16 (www.jama-peer.org).
Olsen O, Middleton P, Ezzo J, Gtzsche PC, Hadhazy V, Herxheimer A, Kleijnen J, McIntosh H. Quality of Cochrane reviews: assessment of sample from 1998. BMJ 2001;323: 829-32., 百拇医药(Anders W Jrgensen, Jrgen )