“Systematic Review”: Is it Different from the “Tra
http://www.100md.com
《伊朗医学》
Digestive Disease Research Center, Tehran University of Medical Sciences, Tehran, Iran.
Over the past half century we have witnessed exponential growth of scientific research and its application in industry as well as daily life. Medicine is not an exception. The medical literature has grown dramatically with over 2 million articles published yearly in more than 20,000 journals.1 Keeping up with this is impossible for medical professionals, namely physicians. In addition to being unable to access all the relevant original articles published on a single topic, it has been shown that the main body of practicing and even academic and executive physicians, lack adequate method-ological knowledge and expertise to take care of the published research works, even if they have adequate access.2 Therefore, “review articles” have gained wide popularity among doctors.
These “traditional reviews” are usually written by an expert in the field, either on a personal interest or more commonly on a contract by major journals. These “traditional reviews” provide useful information on general aspects of diseases or new progress. They usually have a general title, are extensively referenced, and reflect points of view of their author(s). The problem rises when you look at different major articles or major reviews on a single topic, which in many instances end up with different conclusions.1 What is the source of dispute, as these authors are expected to have looked at the same literature? Is there any difference in their resources? Have some of them missed some important publications? Are the authors biased towards their own research and points of view? Have they looked differently to the same research work? Are there research works not published in major English language journals, therefore not available to many who need it, i.e., is there some sort of publication bias? Can the statements in the review be generalized to other (especially our own) conditions? What can be done to minimize all these?
So we know the following: 1) information is produced at a rapid rate, 2) summarizing the new information at a timely manner is essential to modern medical practice, 3) good reviews are necessary to update users (i.e., medical professionals), and 4) traditional reviews, despite their usefulness, suffer from several major and sometimes fatal drawbacks. “systematic reviews” have been born to respond to these needs.
What is a “systematic review” and how it is different from the “traditional review”?
As mentioned, “traditional reviews” usually have a general topic like “Systemic Lupus Erythematosus” or “Liver Transplantation”. The first step for a “systematic review” is developing a focused, answerable question as its topic.3 A focused, answerable question is usually made of 3 or 4 parts: 1) target population, 2) measurable clinical outcome, 3) intervention, and 4) comparison. This is an example: “Does liver transplantation improve quality of life in middle-aged Asian patients with end-stage liver disease (ESLD) due to chronic hepatitis B?” As you can see the question is dissectable. The “target population” is “middle-aged, Asian patients with ESLD due to HBV”, the “clinical outcome” is “improvement in quality of life”, and the “intervention” is “liver transplantation”. The question does not aim to tackle liver transplantation as a whole, but it clearly wants to clarify a practical aspect of the subject.
Now that the “focused, answerable question” is developed, one should go for a “systematic search of the literature.”4 Generally, when we want to look up something in the literature, we go to the “PubMed”, type a couple of keywords, look at the first few pages of our search, and end up with numerous abstracts or full-text articles, many of which sounding absurd and irrelevant. This has its drawbacks: many journals are not indexed in PubMed, searching the PubMed in this way yields 30 – 40% of the relevant articles in this data-base at most, and unstructured keywords and search strategy will miss many of the related works. A systematic search is different in the way that keywords, which will yield the most relevant articles, are first selected through looking at related available articles, the “MeSH” directory, and a consensus reached by the investigators. They can be revised at any time if new information calls for that. Then several databases, in addition to PubMed, are searched and the authors of relevant articles are contacted for any unpublished, relevant information. Thereafter, the abstracts are carefully screened and full-text articles are selected according to set criteria. In this way, it is guaranteed that the best effort to find all relevant works has been done and that papers are ruled in or ruled out in a systematic and reproducible way, not just according to one’s preference.
The next step is structured extraction of data from the full-texts, again following set criteria. And the final step is to summarize data of each article.3 – 5 These latter two steps are performed by at least two different authors and discrepancies fixed by consensus. It is possible to use specific statistical methods to summarize the information and if this is used then the “systematic review” will turn into a “metaanalysis”.6 Steps for development of a systematic review are summarized in Table 1.
To summarize, a systematic review is not just a compilation of data according to selection and interpretation of an authority, but it is a research work where the subjects are published or unpublished trials (mostly randomized controlled trials [RCTs]), have a specific methodology with their own inclusion and exclusion criteria and systematic search for outcomes of interest (instead of endpoints used in primary research). Therefore, a systematic review is actually a “secondary research”. In this way available literature on a focused, answerable question is summarized systematically, trying to avoid bias as much as possible. This is why systematic reviews are considered as one of the highest levels of evidence1 in the hierarchy of evidence (Table 2).
Are all systematic reviews reliable, good, and useful?
It has been made clear that systematic reviews are researches done on primary research works in order to compile them systematically, summarizing all information as if just one large study has been done and avoid bias as much as possible. That is why they are considered among the highest levels of evidence. But, are all systematic reviews the same? Do all of them provide the reader with good and reliable information? The answer is a definite “no”. As systematic reviews are research works, like all other research jobs they depend on the rigor of their methodology, inclusion and exclusion criteria, the quality of included studies, and the way they are performed. If a systematic review has flaws in its methodology (e.g., poor search of the literature and poor extraction and compilation of data, including poor quality studies), then it would be misleading rather than useful; therefore, the famous statement about them: “garbage in, garbage out”. This is just like an original RCT, where if the methodology of research is poor, the results would be unreliable.7 Systematic reviews should first be screened for their rigor of methodology and report, and if these are adequate, then they are true systematic reviews with the mentioned value.8 A checklist for evaluating the quality of systematic reviews (and metaanalyses) is provided in Table 3 (the QUOROM statement checklist).
Abstract
Use a structured format
Objective
Describe the clinical question explicitly
Data sources
Describe the databases (i.e., list) and other sources
Review methods
Describe the selection criteria (i.e., population, intervention, outcome, and study design); methods for validity assessment, data abstraction, and study characteristics; and quantitative data synthesis in sufficient detail to permit replication
Results
Describe characteristics of the RCTs included and excluded; qualitative and quantitative findings (i.e., point estimates and confidence intervals); and subgroup analyses
Conclusion
Describe the main results
Introduction
Describe the explicit clinical problem, biological rationale for the intervention, and rationale for review
Methods
Searching
Describe the information sources in detail (e.g., databases, registers, personal files, expert informants, agencies, and hand-searching) and any restrictions (years considered, publication status, and language of publication)
Selection
The inclusion and exclusion criteria (defining population, intervention, principal outcomes, and study design)
Validity assessment
The criteria and process used (e.g., masked conditions, quality assessment, and their findings)
Data abstraction
The process or processes used (e.g., completed independently, in duplicate)
Study characteristics
The type of study design, participants’ characteristics, details of intervention, outcome definitions, and how clinical heterogeneity was assessed
Quantitative data analysis
The principal measures of effect (e.g., relative risk), method of combining results, (statistical testing and confidence intervals), and handling of missing data; how statistical heterogeneity was assessed; a rationale for any a-priori sensitivity and subgroup analyses; and any assessment of publication bias
Results
Trial flow
Provide a metaanalysis profile summarizing trial flow
Study characteristics
Present descriptive data for each trial (e.g., age, sample size, intervention, dose, duration, and follow-up period)
Quantitative data synthesis
Report agreement on the selection and validity assessment; present simple summary results (for each treatment group in each trial, and for each primary outcome); present data needed to calculate effect sizes and confidence intervals in intention-to-treat analyses (e.g., 2 ′ 2 tables of counts, means and SDs, and proportions)
Discussion
Summarize key findings; discuss clinical inferences based on internal and external validity; interpret the results in light of the totality of available evidence; describe potential biases in the review process (e.g., publication bias); and suggest a future research agenda
Useful links to find systematic reviews are as follows:
· Systematic Reviews Training Unit http://www.ich.ucl.ac.uk/srtu
· Cochrane Collaboration http://hiru.mcmaster.ca/cochrane/default.htm
· NHS Center for Reviews and Dissemination http://www.york.ac.uk/inst/crd/welcome.htm
· Centre for Evidence-Based Medicine at Oxford http://cebm.jr2.ox.ac.uk/
· Bandolier http://www.jr2.ox.ac.uk/bandolier/index.html
· Hayward Medical Communications' Evidence-Based Medicine (EBM) Web Pages http://www.evidence-based-medicine.co.uk
All of these sites provide useful systematic reviews. The Cochrane database, which has been established with the aim of maintaining, developing, and disseminating systematic reviews, is one of the most useful ones. In addition to the Cochrane database of reviews, there is a handbook on the Cochrane’s website, which explains various steps of development of a systematic review in detail for those who may be interested in developing them.
To conclude, systematic reviews are structured research to find the answer to a well-formulated, focused clinical question. They try to have an unbiased look to the whole literature, including the “grey literature”, i.e., the unpublished information. They have a research methodology and have clear and reproducible steps for themselves. A quantitative summary that includes a specific statistical synthesis is a metaanalysis. High quality systematic reviews provide high levels of evidence necessary for medical practice of excellence. They have major differences with “narrative” or “traditional” reviews as depicted in Table 4.
References
1 Davies HTO, Crombie IK. What is a systematic review? 2001. Available from: URL: http://www.evidence-based-medicine.co.uk/What_is_series.html.
2 Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985; 103: 596 – 599.
3 Counsell C. Formulating questions and locating primary studies for inclusion of systematic reviews. Ann Intern Med. 1997; 127: 380 – 387.
4 Hunt AL, McKibbon KA. Locating and appraising systematic reviews. Ann Intern Med. 1997; 126: 532 – 538.
5 Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Ann Intern Med. 1997; 127: 531 – 537.
6 Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997; 127: 820 – 826.
7 Moher D, Schulz KF, Altman DG; CONSORT Group (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. J Am Podiatr Med Assoc. 2001; 91: 437 – 442.
8 Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999; 354: 1896 – 1900.(Siavosh Nasseri-Moghaddam)
Over the past half century we have witnessed exponential growth of scientific research and its application in industry as well as daily life. Medicine is not an exception. The medical literature has grown dramatically with over 2 million articles published yearly in more than 20,000 journals.1 Keeping up with this is impossible for medical professionals, namely physicians. In addition to being unable to access all the relevant original articles published on a single topic, it has been shown that the main body of practicing and even academic and executive physicians, lack adequate method-ological knowledge and expertise to take care of the published research works, even if they have adequate access.2 Therefore, “review articles” have gained wide popularity among doctors.
These “traditional reviews” are usually written by an expert in the field, either on a personal interest or more commonly on a contract by major journals. These “traditional reviews” provide useful information on general aspects of diseases or new progress. They usually have a general title, are extensively referenced, and reflect points of view of their author(s). The problem rises when you look at different major articles or major reviews on a single topic, which in many instances end up with different conclusions.1 What is the source of dispute, as these authors are expected to have looked at the same literature? Is there any difference in their resources? Have some of them missed some important publications? Are the authors biased towards their own research and points of view? Have they looked differently to the same research work? Are there research works not published in major English language journals, therefore not available to many who need it, i.e., is there some sort of publication bias? Can the statements in the review be generalized to other (especially our own) conditions? What can be done to minimize all these?
So we know the following: 1) information is produced at a rapid rate, 2) summarizing the new information at a timely manner is essential to modern medical practice, 3) good reviews are necessary to update users (i.e., medical professionals), and 4) traditional reviews, despite their usefulness, suffer from several major and sometimes fatal drawbacks. “systematic reviews” have been born to respond to these needs.
What is a “systematic review” and how it is different from the “traditional review”?
As mentioned, “traditional reviews” usually have a general topic like “Systemic Lupus Erythematosus” or “Liver Transplantation”. The first step for a “systematic review” is developing a focused, answerable question as its topic.3 A focused, answerable question is usually made of 3 or 4 parts: 1) target population, 2) measurable clinical outcome, 3) intervention, and 4) comparison. This is an example: “Does liver transplantation improve quality of life in middle-aged Asian patients with end-stage liver disease (ESLD) due to chronic hepatitis B?” As you can see the question is dissectable. The “target population” is “middle-aged, Asian patients with ESLD due to HBV”, the “clinical outcome” is “improvement in quality of life”, and the “intervention” is “liver transplantation”. The question does not aim to tackle liver transplantation as a whole, but it clearly wants to clarify a practical aspect of the subject.
Now that the “focused, answerable question” is developed, one should go for a “systematic search of the literature.”4 Generally, when we want to look up something in the literature, we go to the “PubMed”, type a couple of keywords, look at the first few pages of our search, and end up with numerous abstracts or full-text articles, many of which sounding absurd and irrelevant. This has its drawbacks: many journals are not indexed in PubMed, searching the PubMed in this way yields 30 – 40% of the relevant articles in this data-base at most, and unstructured keywords and search strategy will miss many of the related works. A systematic search is different in the way that keywords, which will yield the most relevant articles, are first selected through looking at related available articles, the “MeSH” directory, and a consensus reached by the investigators. They can be revised at any time if new information calls for that. Then several databases, in addition to PubMed, are searched and the authors of relevant articles are contacted for any unpublished, relevant information. Thereafter, the abstracts are carefully screened and full-text articles are selected according to set criteria. In this way, it is guaranteed that the best effort to find all relevant works has been done and that papers are ruled in or ruled out in a systematic and reproducible way, not just according to one’s preference.
The next step is structured extraction of data from the full-texts, again following set criteria. And the final step is to summarize data of each article.3 – 5 These latter two steps are performed by at least two different authors and discrepancies fixed by consensus. It is possible to use specific statistical methods to summarize the information and if this is used then the “systematic review” will turn into a “metaanalysis”.6 Steps for development of a systematic review are summarized in Table 1.
To summarize, a systematic review is not just a compilation of data according to selection and interpretation of an authority, but it is a research work where the subjects are published or unpublished trials (mostly randomized controlled trials [RCTs]), have a specific methodology with their own inclusion and exclusion criteria and systematic search for outcomes of interest (instead of endpoints used in primary research). Therefore, a systematic review is actually a “secondary research”. In this way available literature on a focused, answerable question is summarized systematically, trying to avoid bias as much as possible. This is why systematic reviews are considered as one of the highest levels of evidence1 in the hierarchy of evidence (Table 2).
Are all systematic reviews reliable, good, and useful?
It has been made clear that systematic reviews are researches done on primary research works in order to compile them systematically, summarizing all information as if just one large study has been done and avoid bias as much as possible. That is why they are considered among the highest levels of evidence. But, are all systematic reviews the same? Do all of them provide the reader with good and reliable information? The answer is a definite “no”. As systematic reviews are research works, like all other research jobs they depend on the rigor of their methodology, inclusion and exclusion criteria, the quality of included studies, and the way they are performed. If a systematic review has flaws in its methodology (e.g., poor search of the literature and poor extraction and compilation of data, including poor quality studies), then it would be misleading rather than useful; therefore, the famous statement about them: “garbage in, garbage out”. This is just like an original RCT, where if the methodology of research is poor, the results would be unreliable.7 Systematic reviews should first be screened for their rigor of methodology and report, and if these are adequate, then they are true systematic reviews with the mentioned value.8 A checklist for evaluating the quality of systematic reviews (and metaanalyses) is provided in Table 3 (the QUOROM statement checklist).
Abstract
Use a structured format
Objective
Describe the clinical question explicitly
Data sources
Describe the databases (i.e., list) and other sources
Review methods
Describe the selection criteria (i.e., population, intervention, outcome, and study design); methods for validity assessment, data abstraction, and study characteristics; and quantitative data synthesis in sufficient detail to permit replication
Results
Describe characteristics of the RCTs included and excluded; qualitative and quantitative findings (i.e., point estimates and confidence intervals); and subgroup analyses
Conclusion
Describe the main results
Introduction
Describe the explicit clinical problem, biological rationale for the intervention, and rationale for review
Methods
Searching
Describe the information sources in detail (e.g., databases, registers, personal files, expert informants, agencies, and hand-searching) and any restrictions (years considered, publication status, and language of publication)
Selection
The inclusion and exclusion criteria (defining population, intervention, principal outcomes, and study design)
Validity assessment
The criteria and process used (e.g., masked conditions, quality assessment, and their findings)
Data abstraction
The process or processes used (e.g., completed independently, in duplicate)
Study characteristics
The type of study design, participants’ characteristics, details of intervention, outcome definitions, and how clinical heterogeneity was assessed
Quantitative data analysis
The principal measures of effect (e.g., relative risk), method of combining results, (statistical testing and confidence intervals), and handling of missing data; how statistical heterogeneity was assessed; a rationale for any a-priori sensitivity and subgroup analyses; and any assessment of publication bias
Results
Trial flow
Provide a metaanalysis profile summarizing trial flow
Study characteristics
Present descriptive data for each trial (e.g., age, sample size, intervention, dose, duration, and follow-up period)
Quantitative data synthesis
Report agreement on the selection and validity assessment; present simple summary results (for each treatment group in each trial, and for each primary outcome); present data needed to calculate effect sizes and confidence intervals in intention-to-treat analyses (e.g., 2 ′ 2 tables of counts, means and SDs, and proportions)
Discussion
Summarize key findings; discuss clinical inferences based on internal and external validity; interpret the results in light of the totality of available evidence; describe potential biases in the review process (e.g., publication bias); and suggest a future research agenda
Useful links to find systematic reviews are as follows:
· Systematic Reviews Training Unit http://www.ich.ucl.ac.uk/srtu
· Cochrane Collaboration http://hiru.mcmaster.ca/cochrane/default.htm
· NHS Center for Reviews and Dissemination http://www.york.ac.uk/inst/crd/welcome.htm
· Centre for Evidence-Based Medicine at Oxford http://cebm.jr2.ox.ac.uk/
· Bandolier http://www.jr2.ox.ac.uk/bandolier/index.html
· Hayward Medical Communications' Evidence-Based Medicine (EBM) Web Pages http://www.evidence-based-medicine.co.uk
All of these sites provide useful systematic reviews. The Cochrane database, which has been established with the aim of maintaining, developing, and disseminating systematic reviews, is one of the most useful ones. In addition to the Cochrane database of reviews, there is a handbook on the Cochrane’s website, which explains various steps of development of a systematic review in detail for those who may be interested in developing them.
To conclude, systematic reviews are structured research to find the answer to a well-formulated, focused clinical question. They try to have an unbiased look to the whole literature, including the “grey literature”, i.e., the unpublished information. They have a research methodology and have clear and reproducible steps for themselves. A quantitative summary that includes a specific statistical synthesis is a metaanalysis. High quality systematic reviews provide high levels of evidence necessary for medical practice of excellence. They have major differences with “narrative” or “traditional” reviews as depicted in Table 4.
References
1 Davies HTO, Crombie IK. What is a systematic review? 2001. Available from: URL: http://www.evidence-based-medicine.co.uk/What_is_series.html.
2 Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann Intern Med. 1985; 103: 596 – 599.
3 Counsell C. Formulating questions and locating primary studies for inclusion of systematic reviews. Ann Intern Med. 1997; 127: 380 – 387.
4 Hunt AL, McKibbon KA. Locating and appraising systematic reviews. Ann Intern Med. 1997; 126: 532 – 538.
5 Meade MO, Richardson WS. Selecting and appraising studies for a systematic review. Ann Intern Med. 1997; 127: 531 – 537.
6 Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997; 127: 820 – 826.
7 Moher D, Schulz KF, Altman DG; CONSORT Group (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. J Am Podiatr Med Assoc. 2001; 91: 437 – 442.
8 Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet. 1999; 354: 1896 – 1900.(Siavosh Nasseri-Moghaddam)