Fate of biomedical research protocols and publication bias in France:
http://www.100md.com
《英国医生杂志》
1 Clinical Research Unit, DIM des Hospices Civils de Lyon, 162 avenue Lacassagne, 69424 Lyon cedex 03, France, 2 CCPPRB Lyon B - H?pital Hotel-Dieu, place de l'Hopital, 69002 Lyon, 3 French National Confederation of Research Ethics Committees - H?pital Hotel-Dieu, 69002 Lyon
Correspondence to: F Chapuis francois.chapuis@chu-lyon.fr
Objectives To describe the fate of protocols approved by the French research ethics committees, a national system created by the French 1988 Huriet-Sérusclat Act; to assess publication bias at a national level.
Design Retrospective cohort study.
Setting Representative sample of 25/48 French research ethics committees in 1994.
Protocols 649 research protocols approved by committees, with follow-up information.
Main outcome measures Protocols' initial characteristics (design, study size, investigator) ed from committees' archives; follow-up information (rates of initiation, completion, and publication) obtained from mailed questionnaire to principal investigators.
Results Completed questionnaires were available for 649/976 (69%) protocols. Of these, 581 (90%) studies were initiated, 501/581 (86%) were completed, and 190/501 (38%) were published. Studies with confirmatory results were more likely to be published as scientific papers than were studies with inconclusive results (adjusted odds ratio 4.59, 95% confidence interval 2.21 to 9.54). Moreover, studies with confirmatory results were published more quickly than studies with inconclusive results (hazard ratio 2.48, 1.36 to 4.55).
Conclusion At a national level, too many research studies are not completed, and among those completed too many are not published. We suggest capitalising on research ethics committees to register and follow all authorised research on human participants on a systematic and prospective basis.
Biomedical research protocols, once approved by a research ethics committee, do not have one typical fate. Some protocols have a linear course—approval, initiation, completion, and publication—whereas others may fail at any step. Information about the fate of studies is useful for funders, society, the scientific community, and patients.1 2 Whether publication is influenced by characteristics of the study such as the direction and strength of findings is of particular interest. Publication bias—defined as the tendency on the parts of investigators, editors, and others to favour publication of research with confirmatory results over research with inconclusive or invalidating results3—threatens the reliability of reviews focusing on the published literature.4
Four papers have reported on follow-up of protocols approved by research ethics committees: in Barcelona, Oxford, Sydney, and Baltimore.5-8 In these studies, 79-93% of approved protocols were initiated and 64-74% of the initiated studies proceeded to completion. Two other studies have reported on follow-up of trials funded by the US National Institutes of Health.9 10
Three of the studies based on research ethics committees also assessed publication bias and showed that confirmatory results are associated with publication.6-8 Odds ratios were highly consistent, ranging from 2.32 to 2.93, and main reason for non-publication was that investigators considered their results not interesting. A survey of authors publishing in psychology in 1973 showed that in the case of statistically non-significant results, the probability of submission was only 6%.11
In France, the 1988 Huriet-Sérusclat Act created a national system of 48 research ethics committees (committees for protection of human beings involved in biomedical research), which contribute to a national confederation of research ethics committees.12 Every protocol involving humans in France must be approved by one of the French committees. The network of structured committees provides prospective and exhaustive recording, but this information had not previously been used for research purposes. Our objective was to describe the fate of clinical protocols after approval and to assess publication bias.
Methods
We surveyed a sample of 25/48 (54%) committees, randomly chosen to ensure a geographical cross section representative of the French administrative areas (the number of committees in each area depends on population size). All invited committees agreed to participate in the study. We assessed three main outcomes: study initiation, study completion, and publication as a scientific paper (table 1). Our main hypothesis was that studies with confirmatory results were more likely to be published than those either inconclusive or invalidating results. All protocols newly approved between 1 January 1994 and 31 December 1994 by any of the 25 participating French committees were eligible.
Table 1 Data collected for analysis
Definitions
We refer to "protocols" up until the time of initiation, from which time we refer to "studies." We collected data either from the committee files or from questionnaires mailed to the principal investigator. We classified study results as "confirmatory," "invalidating," or "inconclusive" (table 1). When the investigator did not respond to questions about publication status, we considered this as missing data.
We classified studies published in formats other than scientific papers as "grey literature"—that is, not generally accessible through libraries (internal reports, theses, s, posters).13 We classified as "confidential" protocols describing research that the investigator reported was not intended to be published.
Data collection
Research assistants attended a formal training session on ion of study characteristics in June 2000. We assigned an identification number to each protocol to ensure anonymity of the investigator, and completed forms were sent to the coordinating centre.
Research assistants were also locally responsible for obtaining follow-up data from the principal investigator of each protocol by using a mailed questionnaire. In the case of non-response, principal investigators were contacted up to six times by mail or phone. When no answer could be obtained, the local committee contacted the sponsor in summer 2002. When no follow-up response was obtained at all, we classified the reason (refusal, investigator retired, deceased, moved away).
Ethical considerations
We conducted this study according to the French law on epidemiological and descriptive studies. We collected data anonymously, and no consent was needed as we retrieved no individual information. For research confidentiality, we assigned an identification number and the researchers' names were not mentioned. Therefore, we did not check publication status on any bibliographic database.
Statistical methods
We obtained frequency distributions for all variables (means, percentages, and 95% confidence intervals). When assessment of association was needed, we used 2 tests.
To build explicative models for the three outcomes (initiation, completion, and publication), we introduced variables significant at the 0.25 level in univariate analysis in a forward stepwise logistic regression (P value for entry = 0.25, P value for remaining = 0.15).14 We restricted analysis of publication to the cohort of completed studies. We excluded studies from the analysis when their results were not known by the investigator and when they were declared to be not aimed at publication (confidential results or phase I studies).
We calculated the time between the date of approval by the committee and the date of first publication and did a Kaplan-Meier survival analysis.15 We used a log-rank test to compare survival curves. We excluded studies with an unknown date of first publication. We censored unpublished studies at the date when the questionnaire was completed; we analysed studies described as "in press" as if published at the date of the completion of the questionnaire. We used a Cox univariate analysis to obtain hazard ratios.16
We used SAS software for all analyses. We considered associations to be statistically significant when P values were less than 0.05.
Results
In 1994 the 25 committees evaluated a total of 1143 protocols. We did not include protocols that were approved in 1993 (n = 19, 2%) or 1995 (n = 82, 7%), that were dropped by the investigator before formal approval (n = 48, 4%), or for which committee submission was not required by the law (n = 12, 1%). Among the 982 protocols included, initial characteristics (available on request) were fully described for 976 (99.4%) protocols.
We did not receive investigators' follow-up answers to the mailed questionnaire for 305 (31%), and 22 (2%) were not suitable for statistical analysis (empty questionnaires or empty pages). This left 649 approved protocols to be included.
Study of non-responses
Seventeen volunteer committees provided complementary information on reasons for investigator's non-response and gathered data for 185/305 studies (61% of non-respondents). The reasons were refusal to fill in a follow-up form (n = 74, 40%), unable to find the original file (n = 56, 30%), and investigator not located because he or she had moved (n = 42, 23%) or had retired or died and nobody could locate the protocol archives (n = 13, 7%).
Protocols with missing follow-up data (n = 305) did not differ from the included protocols (n = 649) by either type of sponsor or study design, but they more often needed modifications to gain approval (relative risk 1.25, 95% confidence interval 1.01 to 1.55), were more often multicentre (2.04, 1.66 to 2.50), and were more often international (1.45, 1.18 to 1.78).
Characteristics of approved protocols
The most common characteristics of the 649 approved protocols were drug testing topic (68%), private funding (73%), and conducted nationally only (82%) (table 2). Experimental designs were most frequent, and 62% of them were randomised. Planned study size was less than 20 patients in 34% of studies, and expected duration of study was less than 18 months in 56% of the studies
Table 2 Characteristics of 649 protocols, by status at follow-up. Values are numbers (percentages)
Fate of approved protocols
Figure 1 shows the fate of biomedical research protocols for the three study outcomes. Ninety per cent (581/649) of approved protocols were initiated at the time of our study, and 86% (501/581) of these were completed. Protocols not initiated tended more often to be national (91% v 82%), to be testing medical devices (9% v 5%), and to have no funding (21% v 8%) (table 2).
Fig 1 Fate of biomedical research protocols
Initiation of protocols
Table 3 shows the factors associated with study initiation. Phase I protocols were about three times more likely to begin than others. Protocols with mixed funding were also most likely to be initiated, as were multinational ones.
Table 3 Multivariate analysis of study initiation, completion, and publication*
Among the 68 (10%) protocols that were not initiated, reasons given for non-initiation were refusal of the legal sponsor (n = 21, 31%), problems with recruitment of patients (n = 15, 22%), technical aspects and feasibility (n = 9, 13%), absence of funding (n = 8, 12%), decision of the investigator (n = 8, 12%), and a similar study having been published (n = 2, 3%). No reason was given for five studies (7%).
Completion of studies
Among the 581 protocols initiated, 16 were ongoing. We found in the logistic regression that phase I studies and studies without adverse effects were more likely to be completed (table 3). Investigators gave several reasons for stopping 64 studies before their planned completion, including patient recruitment problems (n = 28, 44%), results found in the interim analysis (n = 13, 20%), incidence of adverse effects (n = 8, 12%), sponsor's decision (n = 8; 13%), and other (n = 7; 11%).
Publication of results
Results were published in a scientific paper for 190/501 (38%) of completed studies, for 7/16 (44%) of ongoing studies, and for 8/64 (12%) of stopped studies. Among stopped studies, publication rates varied from 0% for studies with recruitment difficulties to 3/8 (37%) for studies with adverse events. Among the 501 completed studies, the publication rate was also heterogeneous; it was lower for the subgroup of phase I studies—21/127 (17%) compared with 169/374 (45%) for others.
Publication bias
Among the 501 completed studies, 127 (25%) phase I studies and 54 (11%) other studies were deemed confidential and were not included in our analyses of publication bias. We also excluded those protocols in which no hypothesis was tested (n = 32) and those for which the investigator did not know the study results (n = 20) or did not provide information about the direction of results (n = 8) or whether the results were published (n = 12). Thus 248 completed studies were included.
Four variables remained in the final model (table 3): direction of results, international versus national scope of the study, study design, and presence of an interim analysis. The stepwise regression confirmed the existence of publication bias; studies with confirmatory results were significantly more likely to be published (odds ratio 4.59, 95% confidence interval 2.21 to 9.54).
Fig 2 Time elapsed to publication
Investigators' reasons for non-publication
The main reason for non-publication given by the investigator was invalidating results (table 4). Some studies had manuscripts still in the writing or submission stage. Rejection of manuscript was cited for only 5% of unpublished studies. The reasons given by the investigator for non-publication corroborate the logistic regression results (confirmatory results were the strongest predictor of publication).
Table 4 Reasons given by investigators for not publishing (n=102)
Delayed publication of invalidating results
We estimated the effect of direction of results on time to publication for the 248 completed studies. For this analysis, we excluded 61 more protocols because of missing date of first publication. Mean time to publication was significantly associated with direction of results (P < 0.001; fig 2): 5.2 years (n = 139, 95% confidence interval 4.8 to 5.6) for confirmatory results compared with 6.9 years (n = 13; 5.9 to 7.9) for invalidating results, and 6.5 years (n = 35, 5.8 to 7.2) for inconclusive results. Cox univariate analysis yielded hazard ratios of 2.48 (1.36 to 4.55) for confirmatory results versus inconclusive results and 0.64 (0.18 to 2.27) for invalidating results versus inconclusive results.
Dissemination of results
Among the 248 protocols used for the analysis of publication bias, 146 (59%) led to scientific papers. Only 26% of these resulted in more than one paper (table 5), and 92% of studies with multiple publications had confirmatory results. However, the association between multiple publication and direction of results was not significant.
Table 5 Reporting of 248 completed studies
Ninety one per cent of studies were reported to be published in international journals. Moreover, 55% of the studies reported in scientific papers were also presented orally. The 102 remaining studies were not published as scientific papers. Forty (39%) resulted in neither publication nor oral presentation, 13 (13%) resulted in an oral presentation only, 23 (23%) appeared in the grey literature only, and 26 (25%) were reported in both an oral presentation and the grey literature. In total, 49 (48%) studies resulted at least in grey literature (table 5).
Discussion
Shields PG. Publication bias is a scientific problem with adverse ethical outcomes: the case for a section for null results. Cancer Epidemiol Biomarkers Prev 2000;9: 771-2.
Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263: 1405-8.
Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263: 1385-9.
Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4: 1529-41.
Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J. Role of a research ethics committee in follow-up and publication of results. Lancet 2003;361: 1015-6.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640-5.
Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267: 374-8.
Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279: 281-6.
Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;doc no 50(1 suppl):.
Greenwald AG. Consequences of prejudice against the null hypothesis. Psychol Bull 1975;82: 1-20.
Loi 88-1138 du 20 Décembre 1988 dite Loi Huriet-Sérusclat. Loi relative à la protection des personnes qui se prêtent à des recherches biomédicales.
New York Academy Of Medicine. What is grey literature? www.nyam.org/library/greywhat.shtml (accessed 18 April 2005).
Hosmer DW, Lemeshow S. Applied logistic regression. New York: John Wiley, 2001.
Kaplan-Meier. Non parametric estimation from incomplete observations. J Am Stat Assoc 1958;53: 457-81.
Cox DR. Regression models and life-tables (with discussion). J R Statist Soc B 1972;34: 187-220.
Easterbrook PJ, Matthews DR. Fate of research studies. J R Soc Med 1992;85: 71-6.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291: 2457-65.
Dickersin K. Why register clinical trials?—Revisited. Control Clin Trials 1992;13: 170-7.
Easterbrook P. Reducing publication bias. BMJ (Clin Res Ed) 1987;295: 1347.
Horton R, Smith R. Time to register randomised trials: the case is now unanswerable. BMJ 1999;319: 865-6.
Abbasi K. Compulsory registration of clinical trials. BMJ 2004;329: 637-8.
DeAngelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Med J Aust 2004;181: 293-4.
Directive 2001/20/EC of the European Parliament and of the Council of 4 April 2001 on the approximation of the laws, regulations and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. Official journal L121, 01/05/2001, 2001: 34-44.(Evelyne Decullier, research fellow1, Vér)
Correspondence to: F Chapuis francois.chapuis@chu-lyon.fr
Objectives To describe the fate of protocols approved by the French research ethics committees, a national system created by the French 1988 Huriet-Sérusclat Act; to assess publication bias at a national level.
Design Retrospective cohort study.
Setting Representative sample of 25/48 French research ethics committees in 1994.
Protocols 649 research protocols approved by committees, with follow-up information.
Main outcome measures Protocols' initial characteristics (design, study size, investigator) ed from committees' archives; follow-up information (rates of initiation, completion, and publication) obtained from mailed questionnaire to principal investigators.
Results Completed questionnaires were available for 649/976 (69%) protocols. Of these, 581 (90%) studies were initiated, 501/581 (86%) were completed, and 190/501 (38%) were published. Studies with confirmatory results were more likely to be published as scientific papers than were studies with inconclusive results (adjusted odds ratio 4.59, 95% confidence interval 2.21 to 9.54). Moreover, studies with confirmatory results were published more quickly than studies with inconclusive results (hazard ratio 2.48, 1.36 to 4.55).
Conclusion At a national level, too many research studies are not completed, and among those completed too many are not published. We suggest capitalising on research ethics committees to register and follow all authorised research on human participants on a systematic and prospective basis.
Biomedical research protocols, once approved by a research ethics committee, do not have one typical fate. Some protocols have a linear course—approval, initiation, completion, and publication—whereas others may fail at any step. Information about the fate of studies is useful for funders, society, the scientific community, and patients.1 2 Whether publication is influenced by characteristics of the study such as the direction and strength of findings is of particular interest. Publication bias—defined as the tendency on the parts of investigators, editors, and others to favour publication of research with confirmatory results over research with inconclusive or invalidating results3—threatens the reliability of reviews focusing on the published literature.4
Four papers have reported on follow-up of protocols approved by research ethics committees: in Barcelona, Oxford, Sydney, and Baltimore.5-8 In these studies, 79-93% of approved protocols were initiated and 64-74% of the initiated studies proceeded to completion. Two other studies have reported on follow-up of trials funded by the US National Institutes of Health.9 10
Three of the studies based on research ethics committees also assessed publication bias and showed that confirmatory results are associated with publication.6-8 Odds ratios were highly consistent, ranging from 2.32 to 2.93, and main reason for non-publication was that investigators considered their results not interesting. A survey of authors publishing in psychology in 1973 showed that in the case of statistically non-significant results, the probability of submission was only 6%.11
In France, the 1988 Huriet-Sérusclat Act created a national system of 48 research ethics committees (committees for protection of human beings involved in biomedical research), which contribute to a national confederation of research ethics committees.12 Every protocol involving humans in France must be approved by one of the French committees. The network of structured committees provides prospective and exhaustive recording, but this information had not previously been used for research purposes. Our objective was to describe the fate of clinical protocols after approval and to assess publication bias.
Methods
We surveyed a sample of 25/48 (54%) committees, randomly chosen to ensure a geographical cross section representative of the French administrative areas (the number of committees in each area depends on population size). All invited committees agreed to participate in the study. We assessed three main outcomes: study initiation, study completion, and publication as a scientific paper (table 1). Our main hypothesis was that studies with confirmatory results were more likely to be published than those either inconclusive or invalidating results. All protocols newly approved between 1 January 1994 and 31 December 1994 by any of the 25 participating French committees were eligible.
Table 1 Data collected for analysis
Definitions
We refer to "protocols" up until the time of initiation, from which time we refer to "studies." We collected data either from the committee files or from questionnaires mailed to the principal investigator. We classified study results as "confirmatory," "invalidating," or "inconclusive" (table 1). When the investigator did not respond to questions about publication status, we considered this as missing data.
We classified studies published in formats other than scientific papers as "grey literature"—that is, not generally accessible through libraries (internal reports, theses, s, posters).13 We classified as "confidential" protocols describing research that the investigator reported was not intended to be published.
Data collection
Research assistants attended a formal training session on ion of study characteristics in June 2000. We assigned an identification number to each protocol to ensure anonymity of the investigator, and completed forms were sent to the coordinating centre.
Research assistants were also locally responsible for obtaining follow-up data from the principal investigator of each protocol by using a mailed questionnaire. In the case of non-response, principal investigators were contacted up to six times by mail or phone. When no answer could be obtained, the local committee contacted the sponsor in summer 2002. When no follow-up response was obtained at all, we classified the reason (refusal, investigator retired, deceased, moved away).
Ethical considerations
We conducted this study according to the French law on epidemiological and descriptive studies. We collected data anonymously, and no consent was needed as we retrieved no individual information. For research confidentiality, we assigned an identification number and the researchers' names were not mentioned. Therefore, we did not check publication status on any bibliographic database.
Statistical methods
We obtained frequency distributions for all variables (means, percentages, and 95% confidence intervals). When assessment of association was needed, we used 2 tests.
To build explicative models for the three outcomes (initiation, completion, and publication), we introduced variables significant at the 0.25 level in univariate analysis in a forward stepwise logistic regression (P value for entry = 0.25, P value for remaining = 0.15).14 We restricted analysis of publication to the cohort of completed studies. We excluded studies from the analysis when their results were not known by the investigator and when they were declared to be not aimed at publication (confidential results or phase I studies).
We calculated the time between the date of approval by the committee and the date of first publication and did a Kaplan-Meier survival analysis.15 We used a log-rank test to compare survival curves. We excluded studies with an unknown date of first publication. We censored unpublished studies at the date when the questionnaire was completed; we analysed studies described as "in press" as if published at the date of the completion of the questionnaire. We used a Cox univariate analysis to obtain hazard ratios.16
We used SAS software for all analyses. We considered associations to be statistically significant when P values were less than 0.05.
Results
In 1994 the 25 committees evaluated a total of 1143 protocols. We did not include protocols that were approved in 1993 (n = 19, 2%) or 1995 (n = 82, 7%), that were dropped by the investigator before formal approval (n = 48, 4%), or for which committee submission was not required by the law (n = 12, 1%). Among the 982 protocols included, initial characteristics (available on request) were fully described for 976 (99.4%) protocols.
We did not receive investigators' follow-up answers to the mailed questionnaire for 305 (31%), and 22 (2%) were not suitable for statistical analysis (empty questionnaires or empty pages). This left 649 approved protocols to be included.
Study of non-responses
Seventeen volunteer committees provided complementary information on reasons for investigator's non-response and gathered data for 185/305 studies (61% of non-respondents). The reasons were refusal to fill in a follow-up form (n = 74, 40%), unable to find the original file (n = 56, 30%), and investigator not located because he or she had moved (n = 42, 23%) or had retired or died and nobody could locate the protocol archives (n = 13, 7%).
Protocols with missing follow-up data (n = 305) did not differ from the included protocols (n = 649) by either type of sponsor or study design, but they more often needed modifications to gain approval (relative risk 1.25, 95% confidence interval 1.01 to 1.55), were more often multicentre (2.04, 1.66 to 2.50), and were more often international (1.45, 1.18 to 1.78).
Characteristics of approved protocols
The most common characteristics of the 649 approved protocols were drug testing topic (68%), private funding (73%), and conducted nationally only (82%) (table 2). Experimental designs were most frequent, and 62% of them were randomised. Planned study size was less than 20 patients in 34% of studies, and expected duration of study was less than 18 months in 56% of the studies
Table 2 Characteristics of 649 protocols, by status at follow-up. Values are numbers (percentages)
Fate of approved protocols
Figure 1 shows the fate of biomedical research protocols for the three study outcomes. Ninety per cent (581/649) of approved protocols were initiated at the time of our study, and 86% (501/581) of these were completed. Protocols not initiated tended more often to be national (91% v 82%), to be testing medical devices (9% v 5%), and to have no funding (21% v 8%) (table 2).
Fig 1 Fate of biomedical research protocols
Initiation of protocols
Table 3 shows the factors associated with study initiation. Phase I protocols were about three times more likely to begin than others. Protocols with mixed funding were also most likely to be initiated, as were multinational ones.
Table 3 Multivariate analysis of study initiation, completion, and publication*
Among the 68 (10%) protocols that were not initiated, reasons given for non-initiation were refusal of the legal sponsor (n = 21, 31%), problems with recruitment of patients (n = 15, 22%), technical aspects and feasibility (n = 9, 13%), absence of funding (n = 8, 12%), decision of the investigator (n = 8, 12%), and a similar study having been published (n = 2, 3%). No reason was given for five studies (7%).
Completion of studies
Among the 581 protocols initiated, 16 were ongoing. We found in the logistic regression that phase I studies and studies without adverse effects were more likely to be completed (table 3). Investigators gave several reasons for stopping 64 studies before their planned completion, including patient recruitment problems (n = 28, 44%), results found in the interim analysis (n = 13, 20%), incidence of adverse effects (n = 8, 12%), sponsor's decision (n = 8; 13%), and other (n = 7; 11%).
Publication of results
Results were published in a scientific paper for 190/501 (38%) of completed studies, for 7/16 (44%) of ongoing studies, and for 8/64 (12%) of stopped studies. Among stopped studies, publication rates varied from 0% for studies with recruitment difficulties to 3/8 (37%) for studies with adverse events. Among the 501 completed studies, the publication rate was also heterogeneous; it was lower for the subgroup of phase I studies—21/127 (17%) compared with 169/374 (45%) for others.
Publication bias
Among the 501 completed studies, 127 (25%) phase I studies and 54 (11%) other studies were deemed confidential and were not included in our analyses of publication bias. We also excluded those protocols in which no hypothesis was tested (n = 32) and those for which the investigator did not know the study results (n = 20) or did not provide information about the direction of results (n = 8) or whether the results were published (n = 12). Thus 248 completed studies were included.
Four variables remained in the final model (table 3): direction of results, international versus national scope of the study, study design, and presence of an interim analysis. The stepwise regression confirmed the existence of publication bias; studies with confirmatory results were significantly more likely to be published (odds ratio 4.59, 95% confidence interval 2.21 to 9.54).
Fig 2 Time elapsed to publication
Investigators' reasons for non-publication
The main reason for non-publication given by the investigator was invalidating results (table 4). Some studies had manuscripts still in the writing or submission stage. Rejection of manuscript was cited for only 5% of unpublished studies. The reasons given by the investigator for non-publication corroborate the logistic regression results (confirmatory results were the strongest predictor of publication).
Table 4 Reasons given by investigators for not publishing (n=102)
Delayed publication of invalidating results
We estimated the effect of direction of results on time to publication for the 248 completed studies. For this analysis, we excluded 61 more protocols because of missing date of first publication. Mean time to publication was significantly associated with direction of results (P < 0.001; fig 2): 5.2 years (n = 139, 95% confidence interval 4.8 to 5.6) for confirmatory results compared with 6.9 years (n = 13; 5.9 to 7.9) for invalidating results, and 6.5 years (n = 35, 5.8 to 7.2) for inconclusive results. Cox univariate analysis yielded hazard ratios of 2.48 (1.36 to 4.55) for confirmatory results versus inconclusive results and 0.64 (0.18 to 2.27) for invalidating results versus inconclusive results.
Dissemination of results
Among the 248 protocols used for the analysis of publication bias, 146 (59%) led to scientific papers. Only 26% of these resulted in more than one paper (table 5), and 92% of studies with multiple publications had confirmatory results. However, the association between multiple publication and direction of results was not significant.
Table 5 Reporting of 248 completed studies
Ninety one per cent of studies were reported to be published in international journals. Moreover, 55% of the studies reported in scientific papers were also presented orally. The 102 remaining studies were not published as scientific papers. Forty (39%) resulted in neither publication nor oral presentation, 13 (13%) resulted in an oral presentation only, 23 (23%) appeared in the grey literature only, and 26 (25%) were reported in both an oral presentation and the grey literature. In total, 49 (48%) studies resulted at least in grey literature (table 5).
Discussion
Shields PG. Publication bias is a scientific problem with adverse ethical outcomes: the case for a section for null results. Cancer Epidemiol Biomarkers Prev 2000;9: 771-2.
Chalmers I. Underreporting research is scientific misconduct. JAMA 1990;263: 1405-8.
Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263: 1385-9.
Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4: 1529-41.
Pich J, Carne X, Arnaiz JA, Gomez B, Trilla A, Rodes J. Role of a research ethics committee in follow-up and publication of results. Lancet 2003;361: 1015-6.
Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337: 867-72.
Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640-5.
Dickersin K, Min YI, Meinert CL. Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992;267: 374-8.
Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279: 281-6.
Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;doc no 50(1 suppl):.
Greenwald AG. Consequences of prejudice against the null hypothesis. Psychol Bull 1975;82: 1-20.
Loi 88-1138 du 20 Décembre 1988 dite Loi Huriet-Sérusclat. Loi relative à la protection des personnes qui se prêtent à des recherches biomédicales.
New York Academy Of Medicine. What is grey literature? www.nyam.org/library/greywhat.shtml (accessed 18 April 2005).
Hosmer DW, Lemeshow S. Applied logistic regression. New York: John Wiley, 2001.
Kaplan-Meier. Non parametric estimation from incomplete observations. J Am Stat Assoc 1958;53: 457-81.
Cox DR. Regression models and life-tables (with discussion). J R Statist Soc B 1972;34: 187-220.
Easterbrook PJ, Matthews DR. Fate of research studies. J R Soc Med 1992;85: 71-6.
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004;291: 2457-65.
Dickersin K. Why register clinical trials?—Revisited. Control Clin Trials 1992;13: 170-7.
Easterbrook P. Reducing publication bias. BMJ (Clin Res Ed) 1987;295: 1347.
Horton R, Smith R. Time to register randomised trials: the case is now unanswerable. BMJ 1999;319: 865-6.
Abbasi K. Compulsory registration of clinical trials. BMJ 2004;329: 637-8.
DeAngelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Med J Aust 2004;181: 293-4.
Directive 2001/20/EC of the European Parliament and of the Council of 4 April 2001 on the approximation of the laws, regulations and administrative provisions of the member states relating to the implementation of good clinical practice in the conduct of clinical trials on medicinal products for human use. Official journal L121, 01/05/2001, 2001: 34-44.(Evelyne Decullier, research fellow1, Vér)