A comparison of handsearching versus EMBASE searching of the Archives of Iranian Medicine to identify reports of randomized controlled trial
http://www.100md.com
《伊朗医学》
Dental Research Center, Tehran University of Medical Sciences, Tehran, Iran,The Bahrain Branch of the UK Cochrane Center, the Cochrane Collaboration, Bahrain.
It is increasingly recognized that health care decision-making around the world needs to be informed by high quality and timely research evidence. The randomized controlled trial (RCT) has long been considered the “gold standard” in the hierarchy of evidence; randomized trials, involving sufficient numbers of participants, are essential to distinguish reliably between the effects of health care interventions and the effects of bias or chance. The synthesis of the results of these trials in systematic reviews can provide reliable evidence about the effects of these interventions. The Cochrane Collaboration is an international organization dedicated to improving health care for the world’s population by preparing, maintaining, and promoting the accessibility of Cochrane systematic reviews of the evidence of the effects of health care interventions.
The validity of the results of a systematic review is highly dependent on the data included and a prerequisite is to include as unbiased and complete a set of relevant studies as possible. The Cochrane Collaboration has focused on the systematic electronic searching of MEDLINE and EMBASE and the systematic handsearching of currently over 2000 general and specialized health care journals for reports of RCTs.1 This handsearching involves reading each document in a journal to decide, according to the set Cochrane eligibility criteria,2 if it might be a report of a randomized trial. The efforts of the many volunteers working within The Cochrane Collaboration have added a substantial number of previously “buried” reports of RCTs to The Cochrane Central Register of Controlled Trials (CENTRAL) published in The Cochrane Library. Some of these reports of trials may have been “buried” as a result of inconsistencies in indexing (indexing bias),3 a lack of cover-to-cover indexing, or because they have been published in journals not indexed in the major health care databases such as MEDLINE and EMBASE (database bias)4 or in journals published in languages other than English (language bias).5, 6
There is a heightened interest among the scientific community in charting the publication of medical research by geographical region. Although no Iranian health care journals were indexed in MEDLINE at the time, a study of the number of MEDLINE-indexed publications originating from Iranian researchers from 1991 to 2002 identified 2060 articles. The number of these articles increased from 44 in 1991 to 508 in 2002.7
Four Iranian biomedical publications, namely Archives of Iranian Medicine, Iranian Journal of Medical Sciences, Iranian Biomedical Journal, and Daru were indexed in Excerpta Medica Database (EMBASE)8 in January 2002. An analysis conducted one year after indexing showed that these journals contributed almost 1/6 of the Iranian biomedical research output.9 The number of indexed journals in Iran has steadily increased in recent years and as of April 2006, 15 journals are indexed in EMBASE and one in MEDLINE.
The Bahrain Branch of the UK Cochrane Center (UKCC) together with colleagues in Iran is actively seeking to minimize the effects of bias by addressing problems of study identification through a comprehensive handsearching program, which it is conducting throughout the region. This handsearching program, which is a collaborative project between a registered Cochrane handsearcher based at Tehran University of Medical Sciences and the Bahrain Branch of the UKCC, is the first in-country handsearching of an Iranian journal which has been registered on the Cochrane Collaboration’s Master List of Journals being handsearched (http://www.cochrane.us/Masterlist.asp), maintained by the US Cochrane Center.
Objectives
This study seeks to identify reports of RCTs by handsearching Archives of Iranian Medicine which is indexed in EMBASE. It will also determine the added value, if any, of the handsearch in minimizing potential effects of bias by comparing the reports found by the handsearch with what would have been found by searching EMBASE to examine the sensitivity of the EMBASE index term RCT. Any reports of trials found by the electronic search and missed by the handsearch will also be determined.
Methods
In January 2006, all issues of the Archives of Iranian Medicine (October 1998 to January 2006) were searched by hand from cover to cover for reports of trials. These were classified as RCTs or controlled clinical trials (CCTs) according to the Cochrane eligibility criteria for reports of randomized trials2 in which participants were definitely or possibly assigned prospectively to one of two or more alternative forms of health care using random allocation or some quasi-random method of allocation such as alternation, date of birth, or medical record number. The handsearcher classified reports of trials as RCTs if the groups compared in the trial were established by random allocation. If the author(s) did not state explicitly that the trial was randomized but randomization could not be ruled out, the report was classified as a CCT. CCT was also applied to quasi-randomized studies where the method of allocation was known but not considered strictly random (e.g. date of birth), and for possibly quasi-randomized studies.
Photocopies of the bibliographic details and of the pages describing the study design of the reports identified were sent to the UK CC for verification and processing for submission to CENTRAL in The Cochrane Library.
EMBASE (via OvidWeb) was searched to identify if the reports found by the handsearching were already included in this database, using a combination of words from the title and abstract of the relevant reports.
The sensitivity of the EMBASE index term RCT was also assessed to see what proportion of the total number of known reports of randomized trials (found by the handsearch) were retrieved by this index term.
The “gold standard” for the calculation of sensitivity was the handsearch.
Results
We checked 376 articles in the Archives of Iranian Medicine and found reports of 20 RCTs and 1 CCT. A full list of the 21 articles is available from the corresponding author. Only thirteen (65%) of the reports of RCTs, found by the handsearch were indexed as RCTs EMBASE. Seven were not identified, of which four were published before this journal was indexed in EMBASE and three were not indexed with the term RCT. The added value of the handsearch in relation to EMBASE for those records indexed in EMBASE (from 2002) is therefore 3 of 16 (19%). The overall added value of the handsearch in relation to EMBASE for those records not indexed as RCTs in EMBASE (n = 3) or published before the journal was indexed (before 2002) in EMBASE (n = 4) is therefore 7 of 20 (35%).
The electronic search of EMBASE, using the index term RCT found 14 reports of RCTs, of which 13 were also found by the handsearch. One report of an RCT was missed by the handsearch.
A total of 21 reports of RCTs were found by using a combination of both handsearching and EMBASE searching. Of these 7 (33%) were identified only by the handsearch and 1 (5%) was identified only by the EMBASE search.
Discussion
To minimize bias due to the selective availability of data, systematic reviewers need to identify as many relevant studies as possible to provide reliable evidence on which to base health care decisions. It has been shown previously that the identification of trials from bibliographic databases can be problematic.4 In an effort to minimize the effects of lack of availability of appropriate indexing terms and inconsistencies in indexing (indexing bias), The Cochrane Collaboration has carried out systematic electronic searches of MEDLINE and EMBASE using extensive search strategies designed to be sensitive i.e. to avoid missing reports of trials. The reports of trials, which were identified by an assessment of the titles and abstracts only using these sensitive search strategies, are included in CENTRAL.
However, despite sensitive searching of electronic databases, it has been found that handsearching still provides additional reports of trials missed by the electronic searches.10, 11 Our study revealed that for the reports of randomized trials found by the handsearch and also indexed in EMBASE, the indexing was largely consistent, with high sensitivity (13 of 16, 81%). This finding contrasts with two recent studies,12,13 which compared the handsearch of journals published in Arab countries with electronic searches of EMBASE, where the indexing was found to be inconsistent and led to a number of reports being missed by the electronic search.
Four of the seven reports missed by the electronic search in our study came from issues of the journal which were published before the journal was indexed. The Archives of Iranian Medicine has been indexed in EMBASE since January 20028 and in MEDLINE only since the beginning of 2006. This finding confirms the importance of handsearching journals to make available the reports of trials from issues of journals published before the journal was indexed in electronic databases. The electronic search of EMBASE contributed an additional report of an RCT not found by the handsearch, which is consistent with similar findings10 that a small number of reports of RCTs are missed by handsearching.
Implications for practice
The results of this study suggest that a combination of EMBASE and handsearching is required to identify adequately reports of randomized trials. Handsearching will identify reports of trials not found by electronic searches. If resources for handsearching are limited, it will prove beneficial to target issues of journals published before the journal was indexed in major electronic databases or journals that are not indexed.
Implications for research
Further research is required to assess the quality of the trials identified and to assess how many trials were duplicated. Additionally, comparisons need to be made in the quality of trials and the treatment effects of trials reported in Farsi with those reported in English to determine whether there might be differences, which could lead to bias being introduced into reviews based exclusively on English language reports.14
Conclusion
A combination of electronic searching and handsearching for reports of RCTs should help to close the gap between the number of reports of trials that exist and the number of reports of trials accessible to authors of Cochrane reviews and others needing to make informed decisions about the effects of health care interventions.
The handsearching of the Archives of Iranian Medicine and the consistent indexing in EMBASE of this journal should help reviewers to minimize the effects of publication bias by providing reports of trials not previously identified. It will also contribute to a more comprehensive assessment of the biomedical research output of Iran.
Disclaimer
The views expressed in this paper represent those of the authors and are not necessarily the views or the official policy of the Cochrane Collaboration.
Acknowledgment
The authors are grateful to Dr. Zbys Fedorowicz, the Director of the Bahrain Branch of The UK Cochrane Center and Anne Eisinga, Information Specialist of The UK Cochrane Centre for their comments on an earlier version of this manuscript.
References
1 Dickersin K, Manheimer E, Wieland S, Robinson K, Lefebvre C, McDonald S. Development of The Cochrane Collaboration’s CENTRAL Register of controlled clinical trials. Eva Health Prof. 2002; 25: 38 – 64.
2 Higgins JPT, Green S. Cochrane and National Library of Medicine Randomized Controlled Trial and Controlled Clinical Trial Criteria. Cochrane Handbook of Systematic Reviews of Interventions 4.2.5. Issue 3. Chichester, UK: John Wiley and Sons; 2005: Appendix 5a.
3 Lefebvre C, Clarke M. Identifying randomized trials. In: Egger M, Smith GD, Altman DG, eds. Systematic Reviews in Health Care: Metaanalysis in Context. 2nd ed. London: BMJ Books; 2001: 69 – 86.
4 Egger M, Smith GD. Bias in location and selection of studies. BMJ. 1998; 316: 61 – 66.
5 Marti J, Bonfill X, Urrutia G, Lacalle J, Bravo R. The identification and description of clinical trials published in Spanish journals of general and internal medicine during the period of 1971 – 1995 [in Spanish]. Medicina Clinica (Barcelona). 1999; 112 (suppl 1): 28 – 34.
6 Turp JC, Schulte JM, Antes G. Nearly half of dental randomized controlled trials published in German are not included in MEDLINE. Eur J Oral Sci. 2002; 110: 405 – 411.
7 Azizi F, Etemadi A, Hajipoor R, Mortazavi N. Trends of publication in medical journals in Iran in the 1990s. Saudi Med J. 2004; 25: S34 – S34.
8 Malekzadeh R, Azarmina P. Archives of Iranian Medicine: challenges of becoming indexed. Arch Iranian Med. 2002; 5: 71 – 72.
9 Radmard AR, Khademi H, Azarmina P, Safavi MS, Nouraie M, Kolahdouzan S, et al. Iran’s biomedical science’s research output in 2003: a bibliographic analysis of MEDLINE and Excerpta Medica databases. Arch Iranian Med. 2005; 8: 180 – 183.
10 Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 2002; 21: 1625 – 1634.
11 Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. The Cochrane Database of Methodology Reviews 2002; Issue 4: Art. No.:MR000001.DOI:10.1002/14651858.MR000001.
12 Al Hajeri A, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Bahrain Medical Bulletin to identify reports of randomized controlled trials. Bahrain Med Bull. 2005; 27: 160 – 163.
13 Cathey JT, Al Hajeri AA, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Annals of Saudi Medicine to identify reports of randomized controlled trials. Ann Saudi Medicine. 2006; 26: 49 – 51.
14 Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in metaanalyses of controlled trials: empirical study. Int J Epidemiol. 2002; 31: 115 – 123.(Mona Nasser DDS, Amani Al)
It is increasingly recognized that health care decision-making around the world needs to be informed by high quality and timely research evidence. The randomized controlled trial (RCT) has long been considered the “gold standard” in the hierarchy of evidence; randomized trials, involving sufficient numbers of participants, are essential to distinguish reliably between the effects of health care interventions and the effects of bias or chance. The synthesis of the results of these trials in systematic reviews can provide reliable evidence about the effects of these interventions. The Cochrane Collaboration is an international organization dedicated to improving health care for the world’s population by preparing, maintaining, and promoting the accessibility of Cochrane systematic reviews of the evidence of the effects of health care interventions.
The validity of the results of a systematic review is highly dependent on the data included and a prerequisite is to include as unbiased and complete a set of relevant studies as possible. The Cochrane Collaboration has focused on the systematic electronic searching of MEDLINE and EMBASE and the systematic handsearching of currently over 2000 general and specialized health care journals for reports of RCTs.1 This handsearching involves reading each document in a journal to decide, according to the set Cochrane eligibility criteria,2 if it might be a report of a randomized trial. The efforts of the many volunteers working within The Cochrane Collaboration have added a substantial number of previously “buried” reports of RCTs to The Cochrane Central Register of Controlled Trials (CENTRAL) published in The Cochrane Library. Some of these reports of trials may have been “buried” as a result of inconsistencies in indexing (indexing bias),3 a lack of cover-to-cover indexing, or because they have been published in journals not indexed in the major health care databases such as MEDLINE and EMBASE (database bias)4 or in journals published in languages other than English (language bias).5, 6
There is a heightened interest among the scientific community in charting the publication of medical research by geographical region. Although no Iranian health care journals were indexed in MEDLINE at the time, a study of the number of MEDLINE-indexed publications originating from Iranian researchers from 1991 to 2002 identified 2060 articles. The number of these articles increased from 44 in 1991 to 508 in 2002.7
Four Iranian biomedical publications, namely Archives of Iranian Medicine, Iranian Journal of Medical Sciences, Iranian Biomedical Journal, and Daru were indexed in Excerpta Medica Database (EMBASE)8 in January 2002. An analysis conducted one year after indexing showed that these journals contributed almost 1/6 of the Iranian biomedical research output.9 The number of indexed journals in Iran has steadily increased in recent years and as of April 2006, 15 journals are indexed in EMBASE and one in MEDLINE.
The Bahrain Branch of the UK Cochrane Center (UKCC) together with colleagues in Iran is actively seeking to minimize the effects of bias by addressing problems of study identification through a comprehensive handsearching program, which it is conducting throughout the region. This handsearching program, which is a collaborative project between a registered Cochrane handsearcher based at Tehran University of Medical Sciences and the Bahrain Branch of the UKCC, is the first in-country handsearching of an Iranian journal which has been registered on the Cochrane Collaboration’s Master List of Journals being handsearched (http://www.cochrane.us/Masterlist.asp), maintained by the US Cochrane Center.
Objectives
This study seeks to identify reports of RCTs by handsearching Archives of Iranian Medicine which is indexed in EMBASE. It will also determine the added value, if any, of the handsearch in minimizing potential effects of bias by comparing the reports found by the handsearch with what would have been found by searching EMBASE to examine the sensitivity of the EMBASE index term RCT. Any reports of trials found by the electronic search and missed by the handsearch will also be determined.
Methods
In January 2006, all issues of the Archives of Iranian Medicine (October 1998 to January 2006) were searched by hand from cover to cover for reports of trials. These were classified as RCTs or controlled clinical trials (CCTs) according to the Cochrane eligibility criteria for reports of randomized trials2 in which participants were definitely or possibly assigned prospectively to one of two or more alternative forms of health care using random allocation or some quasi-random method of allocation such as alternation, date of birth, or medical record number. The handsearcher classified reports of trials as RCTs if the groups compared in the trial were established by random allocation. If the author(s) did not state explicitly that the trial was randomized but randomization could not be ruled out, the report was classified as a CCT. CCT was also applied to quasi-randomized studies where the method of allocation was known but not considered strictly random (e.g. date of birth), and for possibly quasi-randomized studies.
Photocopies of the bibliographic details and of the pages describing the study design of the reports identified were sent to the UK CC for verification and processing for submission to CENTRAL in The Cochrane Library.
EMBASE (via OvidWeb) was searched to identify if the reports found by the handsearching were already included in this database, using a combination of words from the title and abstract of the relevant reports.
The sensitivity of the EMBASE index term RCT was also assessed to see what proportion of the total number of known reports of randomized trials (found by the handsearch) were retrieved by this index term.
The “gold standard” for the calculation of sensitivity was the handsearch.
Results
We checked 376 articles in the Archives of Iranian Medicine and found reports of 20 RCTs and 1 CCT. A full list of the 21 articles is available from the corresponding author. Only thirteen (65%) of the reports of RCTs, found by the handsearch were indexed as RCTs EMBASE. Seven were not identified, of which four were published before this journal was indexed in EMBASE and three were not indexed with the term RCT. The added value of the handsearch in relation to EMBASE for those records indexed in EMBASE (from 2002) is therefore 3 of 16 (19%). The overall added value of the handsearch in relation to EMBASE for those records not indexed as RCTs in EMBASE (n = 3) or published before the journal was indexed (before 2002) in EMBASE (n = 4) is therefore 7 of 20 (35%).
The electronic search of EMBASE, using the index term RCT found 14 reports of RCTs, of which 13 were also found by the handsearch. One report of an RCT was missed by the handsearch.
A total of 21 reports of RCTs were found by using a combination of both handsearching and EMBASE searching. Of these 7 (33%) were identified only by the handsearch and 1 (5%) was identified only by the EMBASE search.
Discussion
To minimize bias due to the selective availability of data, systematic reviewers need to identify as many relevant studies as possible to provide reliable evidence on which to base health care decisions. It has been shown previously that the identification of trials from bibliographic databases can be problematic.4 In an effort to minimize the effects of lack of availability of appropriate indexing terms and inconsistencies in indexing (indexing bias), The Cochrane Collaboration has carried out systematic electronic searches of MEDLINE and EMBASE using extensive search strategies designed to be sensitive i.e. to avoid missing reports of trials. The reports of trials, which were identified by an assessment of the titles and abstracts only using these sensitive search strategies, are included in CENTRAL.
However, despite sensitive searching of electronic databases, it has been found that handsearching still provides additional reports of trials missed by the electronic searches.10, 11 Our study revealed that for the reports of randomized trials found by the handsearch and also indexed in EMBASE, the indexing was largely consistent, with high sensitivity (13 of 16, 81%). This finding contrasts with two recent studies,12,13 which compared the handsearch of journals published in Arab countries with electronic searches of EMBASE, where the indexing was found to be inconsistent and led to a number of reports being missed by the electronic search.
Four of the seven reports missed by the electronic search in our study came from issues of the journal which were published before the journal was indexed. The Archives of Iranian Medicine has been indexed in EMBASE since January 20028 and in MEDLINE only since the beginning of 2006. This finding confirms the importance of handsearching journals to make available the reports of trials from issues of journals published before the journal was indexed in electronic databases. The electronic search of EMBASE contributed an additional report of an RCT not found by the handsearch, which is consistent with similar findings10 that a small number of reports of RCTs are missed by handsearching.
Implications for practice
The results of this study suggest that a combination of EMBASE and handsearching is required to identify adequately reports of randomized trials. Handsearching will identify reports of trials not found by electronic searches. If resources for handsearching are limited, it will prove beneficial to target issues of journals published before the journal was indexed in major electronic databases or journals that are not indexed.
Implications for research
Further research is required to assess the quality of the trials identified and to assess how many trials were duplicated. Additionally, comparisons need to be made in the quality of trials and the treatment effects of trials reported in Farsi with those reported in English to determine whether there might be differences, which could lead to bias being introduced into reviews based exclusively on English language reports.14
Conclusion
A combination of electronic searching and handsearching for reports of RCTs should help to close the gap between the number of reports of trials that exist and the number of reports of trials accessible to authors of Cochrane reviews and others needing to make informed decisions about the effects of health care interventions.
The handsearching of the Archives of Iranian Medicine and the consistent indexing in EMBASE of this journal should help reviewers to minimize the effects of publication bias by providing reports of trials not previously identified. It will also contribute to a more comprehensive assessment of the biomedical research output of Iran.
Disclaimer
The views expressed in this paper represent those of the authors and are not necessarily the views or the official policy of the Cochrane Collaboration.
Acknowledgment
The authors are grateful to Dr. Zbys Fedorowicz, the Director of the Bahrain Branch of The UK Cochrane Center and Anne Eisinga, Information Specialist of The UK Cochrane Centre for their comments on an earlier version of this manuscript.
References
1 Dickersin K, Manheimer E, Wieland S, Robinson K, Lefebvre C, McDonald S. Development of The Cochrane Collaboration’s CENTRAL Register of controlled clinical trials. Eva Health Prof. 2002; 25: 38 – 64.
2 Higgins JPT, Green S. Cochrane and National Library of Medicine Randomized Controlled Trial and Controlled Clinical Trial Criteria. Cochrane Handbook of Systematic Reviews of Interventions 4.2.5. Issue 3. Chichester, UK: John Wiley and Sons; 2005: Appendix 5a.
3 Lefebvre C, Clarke M. Identifying randomized trials. In: Egger M, Smith GD, Altman DG, eds. Systematic Reviews in Health Care: Metaanalysis in Context. 2nd ed. London: BMJ Books; 2001: 69 – 86.
4 Egger M, Smith GD. Bias in location and selection of studies. BMJ. 1998; 316: 61 – 66.
5 Marti J, Bonfill X, Urrutia G, Lacalle J, Bravo R. The identification and description of clinical trials published in Spanish journals of general and internal medicine during the period of 1971 – 1995 [in Spanish]. Medicina Clinica (Barcelona). 1999; 112 (suppl 1): 28 – 34.
6 Turp JC, Schulte JM, Antes G. Nearly half of dental randomized controlled trials published in German are not included in MEDLINE. Eur J Oral Sci. 2002; 110: 405 – 411.
7 Azizi F, Etemadi A, Hajipoor R, Mortazavi N. Trends of publication in medical journals in Iran in the 1990s. Saudi Med J. 2004; 25: S34 – S34.
8 Malekzadeh R, Azarmina P. Archives of Iranian Medicine: challenges of becoming indexed. Arch Iranian Med. 2002; 5: 71 – 72.
9 Radmard AR, Khademi H, Azarmina P, Safavi MS, Nouraie M, Kolahdouzan S, et al. Iran’s biomedical science’s research output in 2003: a bibliographic analysis of MEDLINE and Excerpta Medica databases. Arch Iranian Med. 2005; 8: 180 – 183.
10 Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. A comparison of handsearching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med. 2002; 21: 1625 – 1634.
11 Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. The Cochrane Database of Methodology Reviews 2002; Issue 4: Art. No.:MR000001.DOI:10.1002/14651858.MR000001.
12 Al Hajeri A, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Bahrain Medical Bulletin to identify reports of randomized controlled trials. Bahrain Med Bull. 2005; 27: 160 – 163.
13 Cathey JT, Al Hajeri AA, Fedorowicz Z. A comparison of handsearching versus EMBASE searching of the Annals of Saudi Medicine to identify reports of randomized controlled trials. Ann Saudi Medicine. 2006; 26: 49 – 51.
14 Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in metaanalyses of controlled trials: empirical study. Int J Epidemiol. 2002; 31: 115 – 123.(Mona Nasser DDS, Amani Al)