Peer Review and Federal Regulations
http://www.100md.com
《新英格兰医药杂志》
Federal regulations that affect science and health are frequently controversial. Whether the topic is air pollution, stem-cell research, clean needles for injection-drug users, or pharmaceutical advertising, powerful economic and social interests are often at stake. It is hard to dispute the view that federal regulations — or the information disseminated by the federal government, more generally — should be based on data of the highest quality. It is far more difficult, however, for scientists and government officials to agree about how to achieve this goal and to keep political considerations from trumping science when policies are created. Former president Bill Clinton, for example, has said that he regretted having continued a ban on using federal funds to provide clean needles to drug users, given the evidence that needle-exchange programs could reduce the spread of the human immunodeficiency virus.
During the past few decades, epidemiologic studies linking urban air pollution with mortality have been challenged repeatedly. One of the most notable and extended controversies involves the Six Cities Study, which showed an association between air pollution and mortality in six U.S. cities. Data from this study were used by the Environmental Protection Agency (EPA) in 1997 as the basis for air-quality regulations. Enrollment in the study began in 1974, and subjects were followed for up to 16 years; the findings were reported in 1993.1 Subsequently, there were various challenges to the validity of the study; as a result, the Health Effects Institute, an independent, nonprofit corporation in Cambridge, Massachusetts, that is supported jointly by the EPA and industry, organized a reanalysis of the data. The audit and validation, as reported in this issue of the Journal (pages 198–199), generally confirmed the original findings.
It would be highly impractical — and of uncertain benefit — to subject all the scientific studies that inform federal regulations to exhaustive scrutiny like the reanalysis of the Six Cities Study. Peer review is an alternative approach to refereeing what is sometimes called regulatory science.2 Many federal agencies already have extensive requirements for peer review, but there are no government-wide minimal standards.
In 2000, Congress enacted the Data Quality Act and directed the White House Office of Management and Budget (OMB) to develop guidelines to ensure the quality of data disseminated by the federal government. The final guidelines encourage, but do not require, peer reviews. In August 2003, the OMB's Office of Information and Regulatory Affairs (OIRA) issued a "Proposed Bulletin on Peer Review and Information Quality." The bulletin would establish government-wide standards for peer review that would apply to information that could be used to support a major regulatory action or that might affect important public policies or decisions in the private sector "with a possible impact of more than $100 million in any year."3 The proposal has other features, such as the disclosure of the names, organizational affiliations, and qualifications of all peer reviewers, and information about their previous peer reviews. The public record would include the final peer-review report, dissents from group reports, and written responses by agencies.
Peer review could be of value to policymakers who must often evaluate conflicting data, expert opinion, and the arguments of passionate advocates on both sides of a question. In a press release, John D. Graham, the administrator of OIRA, said, "Peer review is an effective way to further engage the scientific community in the regulatory process. A more uniform peer review policy promises to make regulatory science more competent and credible, thereby advancing the administration's `smart-regulation' agenda. The goal is fewer lawsuits and a more consistent regulatory environment, which is good for consumers and businesses." At a workshop about the proposal at the National Academies in November, Graham said that he expects the new requirements would apply to "several hundred information products" each year.4
There are, however, substantial concerns about the proposal. At the workshop, several participants asked the OMB to define the problem the bulletin is intended to solve. According to David M. Michaels, who was an assistant secretary in the Department of Energy in the Clinton administration and is now a professor at the George Washington University School of Public Health, "We have not heard of widespread examples of inappropriate or flawed federal regulations being promulgated as a result of failure to peer review. In fact, we have not heard of a single example." Because peer reviewers are "asked to review scientific and technical matters, leaving policy determinations for the agency," the proposal may have little effect on the long battles over the way in which science is reflected in regulations.
A second concern involves the selection of peer reviewers. Agencies would be instructed to "appoint experts who, in addition to possessing the necessary scientific and technical expertise, are independent of the agency, do not possess real or perceived conflicts of interest, and are capable of approaching the subject matter in an open-minded and unbiased manner." The relevant factors would include whether "an individual has any financial interests in the matter at issue," has "advocated a position on the specific matter," is "currently receiving or seeking substantial funding from the agency," or has conducted other peer reviews for the same agency in recent years. Critics contend that if all these criteria are used, it will often happen that all the qualified experts are excluded. Moreover, reviews by experts with known biases or critical points of view may be essential in establishing the most serious weaknesses of a study. Although the proposal allows for the use of peer review undertaken by a scientific journal, many of the studies that provide support for regulations have not been submitted for publication.
A third concern is that the proposal is politically motivated. The bulletin might help industry, organizations supported by industry, or activist groups to contest the data that are used to support regulations or other government actions that they dislike. The result might be a form of gridlock, in which nothing gets accomplished because the scientific basis of any potentially controversial piece of information or regulation is continually being challenged. In many instances, maintaining the status quo is not the best way to protect public health. According to Michaels, "There is now a whole industry that has sprung up that reanalyzes data to make results go away. Not surprisingly, these reanalyses — I don't like to call them studies — are commissioned when regulation appears on the horizon. The companies that do this work are hired guns working for dirty companies — they have the same relationship to epidemiology and toxicology as the Arthur Anderson Company has to accounting."
Consider the recent controversy about the National Cancer Institute (NCI) Web site and the relation between abortion and breast cancer. Would the peer-review proposal abet such disputes, prevent them, or have no effect? Compelling peer-reviewed data have established that "induced abortions have no overall effect on the risk of breast cancer."5 Yet the belief persists that there may be a relation. In 2002, a fact sheet on the NCI Web site was changed to suggest that whether abortion caused breast cancer was an open question. After a public outcry, the NCI organized a workshop involving leading experts on pregnancy and the risk of breast cancer. The NCI fact sheet now states the conclusion of the workshop, that "having an abortion or miscarriage does not increase a woman's subsequent risk of developing breast cancer."
The OMB bulletin is likely to be revised after public comment and review within the government. Are uniform, government-wide standards for peer review, coordinated by the OMB, desirable? Or is a more flexible approach better, one that gives more authority to individual federal agencies to establish their own procedures? Are there other approaches that might be less onerous, expensive, and susceptible to abuse? The Bush administration has raised important issues, but there is little consensus that its specific proposal is the best one.
References
Dockery DW, Pope A III, Xu X, et al. An association between air pollution and mortality in six U.S. cities. N Engl J Med 1993;329:1753-1759.
Jasanoff S. The fifth branch: science advisers as policymakers. Cambridge, Mass.: Harvard University Press, 1990.
Office of Management and Budget. Proposed bulletin on peer review and information quality, September 15, 2003 (68FR 54023-9).
November 18, 2003, Workshop on Peer Review Standards for Regulatory Science and Technical Information. Washington, D.C.: National Academies, 2003. (Accessed December 10, 2003, at http://www.nationalacademies.org/stl.)
Melbye M, Wohlfahrt J, Olsen JH, et al. Induced abortion and the risk of breast cancer. N Engl J Med 1997;336:81-85.
Related Letters:
Six Cities Revisited
Bruner J. M.(Robert Steinbrook, M.D.)
During the past few decades, epidemiologic studies linking urban air pollution with mortality have been challenged repeatedly. One of the most notable and extended controversies involves the Six Cities Study, which showed an association between air pollution and mortality in six U.S. cities. Data from this study were used by the Environmental Protection Agency (EPA) in 1997 as the basis for air-quality regulations. Enrollment in the study began in 1974, and subjects were followed for up to 16 years; the findings were reported in 1993.1 Subsequently, there were various challenges to the validity of the study; as a result, the Health Effects Institute, an independent, nonprofit corporation in Cambridge, Massachusetts, that is supported jointly by the EPA and industry, organized a reanalysis of the data. The audit and validation, as reported in this issue of the Journal (pages 198–199), generally confirmed the original findings.
It would be highly impractical — and of uncertain benefit — to subject all the scientific studies that inform federal regulations to exhaustive scrutiny like the reanalysis of the Six Cities Study. Peer review is an alternative approach to refereeing what is sometimes called regulatory science.2 Many federal agencies already have extensive requirements for peer review, but there are no government-wide minimal standards.
In 2000, Congress enacted the Data Quality Act and directed the White House Office of Management and Budget (OMB) to develop guidelines to ensure the quality of data disseminated by the federal government. The final guidelines encourage, but do not require, peer reviews. In August 2003, the OMB's Office of Information and Regulatory Affairs (OIRA) issued a "Proposed Bulletin on Peer Review and Information Quality." The bulletin would establish government-wide standards for peer review that would apply to information that could be used to support a major regulatory action or that might affect important public policies or decisions in the private sector "with a possible impact of more than $100 million in any year."3 The proposal has other features, such as the disclosure of the names, organizational affiliations, and qualifications of all peer reviewers, and information about their previous peer reviews. The public record would include the final peer-review report, dissents from group reports, and written responses by agencies.
Peer review could be of value to policymakers who must often evaluate conflicting data, expert opinion, and the arguments of passionate advocates on both sides of a question. In a press release, John D. Graham, the administrator of OIRA, said, "Peer review is an effective way to further engage the scientific community in the regulatory process. A more uniform peer review policy promises to make regulatory science more competent and credible, thereby advancing the administration's `smart-regulation' agenda. The goal is fewer lawsuits and a more consistent regulatory environment, which is good for consumers and businesses." At a workshop about the proposal at the National Academies in November, Graham said that he expects the new requirements would apply to "several hundred information products" each year.4
There are, however, substantial concerns about the proposal. At the workshop, several participants asked the OMB to define the problem the bulletin is intended to solve. According to David M. Michaels, who was an assistant secretary in the Department of Energy in the Clinton administration and is now a professor at the George Washington University School of Public Health, "We have not heard of widespread examples of inappropriate or flawed federal regulations being promulgated as a result of failure to peer review. In fact, we have not heard of a single example." Because peer reviewers are "asked to review scientific and technical matters, leaving policy determinations for the agency," the proposal may have little effect on the long battles over the way in which science is reflected in regulations.
A second concern involves the selection of peer reviewers. Agencies would be instructed to "appoint experts who, in addition to possessing the necessary scientific and technical expertise, are independent of the agency, do not possess real or perceived conflicts of interest, and are capable of approaching the subject matter in an open-minded and unbiased manner." The relevant factors would include whether "an individual has any financial interests in the matter at issue," has "advocated a position on the specific matter," is "currently receiving or seeking substantial funding from the agency," or has conducted other peer reviews for the same agency in recent years. Critics contend that if all these criteria are used, it will often happen that all the qualified experts are excluded. Moreover, reviews by experts with known biases or critical points of view may be essential in establishing the most serious weaknesses of a study. Although the proposal allows for the use of peer review undertaken by a scientific journal, many of the studies that provide support for regulations have not been submitted for publication.
A third concern is that the proposal is politically motivated. The bulletin might help industry, organizations supported by industry, or activist groups to contest the data that are used to support regulations or other government actions that they dislike. The result might be a form of gridlock, in which nothing gets accomplished because the scientific basis of any potentially controversial piece of information or regulation is continually being challenged. In many instances, maintaining the status quo is not the best way to protect public health. According to Michaels, "There is now a whole industry that has sprung up that reanalyzes data to make results go away. Not surprisingly, these reanalyses — I don't like to call them studies — are commissioned when regulation appears on the horizon. The companies that do this work are hired guns working for dirty companies — they have the same relationship to epidemiology and toxicology as the Arthur Anderson Company has to accounting."
Consider the recent controversy about the National Cancer Institute (NCI) Web site and the relation between abortion and breast cancer. Would the peer-review proposal abet such disputes, prevent them, or have no effect? Compelling peer-reviewed data have established that "induced abortions have no overall effect on the risk of breast cancer."5 Yet the belief persists that there may be a relation. In 2002, a fact sheet on the NCI Web site was changed to suggest that whether abortion caused breast cancer was an open question. After a public outcry, the NCI organized a workshop involving leading experts on pregnancy and the risk of breast cancer. The NCI fact sheet now states the conclusion of the workshop, that "having an abortion or miscarriage does not increase a woman's subsequent risk of developing breast cancer."
The OMB bulletin is likely to be revised after public comment and review within the government. Are uniform, government-wide standards for peer review, coordinated by the OMB, desirable? Or is a more flexible approach better, one that gives more authority to individual federal agencies to establish their own procedures? Are there other approaches that might be less onerous, expensive, and susceptible to abuse? The Bush administration has raised important issues, but there is little consensus that its specific proposal is the best one.
References
Dockery DW, Pope A III, Xu X, et al. An association between air pollution and mortality in six U.S. cities. N Engl J Med 1993;329:1753-1759.
Jasanoff S. The fifth branch: science advisers as policymakers. Cambridge, Mass.: Harvard University Press, 1990.
Office of Management and Budget. Proposed bulletin on peer review and information quality, September 15, 2003 (68FR 54023-9).
November 18, 2003, Workshop on Peer Review Standards for Regulatory Science and Technical Information. Washington, D.C.: National Academies, 2003. (Accessed December 10, 2003, at http://www.nationalacademies.org/stl.)
Melbye M, Wohlfahrt J, Olsen JH, et al. Induced abortion and the risk of breast cancer. N Engl J Med 1997;336:81-85.
Related Letters:
Six Cities Revisited
Bruner J. M.(Robert Steinbrook, M.D.)