Strategies for Planning and Launching PBRN Research Studies: A Project of the Academy of Family Physicians National Research Netwo
http://www.100md.com
《家庭医学美国版医学期刊》
the American Academy of Family Physicians National Research Network, Leawood, KS
Abstract
Research conducted in a practice-based research network (PBRN) differs from other multisite research and presents particular planning challenges. The American Academy of Family Physicians National Research Network (AAFP NRN) has developed a number of procedures used for planning and implementing studies, which address the challenges of national PBRN studies. In this study, we highlight challenges common to PBRN research and describe the methods used by the AAFP NRN to address those challenges. The following tasks were identified as important to implementing PBRN research studies: (1) selecting fundable, feasible studies that interest members and have the potential to improve quality of care; (2) creating a practical budget that covers the costs of the study; (3) composing study teams and securing written agreements between team members; (4) recruiting and selecting study sites; and (5) training practice staff and physicians. Striking the balance of scientific rigor with practical application of PBRN studies must be addressed throughout these tasks. Proper planning for PBRN studies significantly affects the success of study implementation. Although developed by a national PBRN, the planning procedures described in this study may be adapted for state or regional PBRNs.
A PBRN is a group of ambulatory practices devoted principally to providing quality patient care. These practices affiliate with one another to investigate questions related to both improving the care they provide and improving their discipline.1 PBRN research studies take place primarily in the office and the research questions are generally those of interest to the participating practices.
The history of primary care PBRNs in the United States began in the 1970s, with the appearance of the earliest regional networks.2,3 In 1978, the creation of a national family medicine PBRN was set in motion, leading to development of the Ambulatory Sentinel Practice Network (ASPN). The AAFP NRN was established in 2000 to replace ASPN as the national family medicine PBRN.4,5 Since the AAFP NRN's inception, primary care PBRNs have grown significantly in size and number. The growth of PBRNs over the past 2 decades has been supported by a number of funding opportunities from the Agency for Healthcare Research Quality (AHRQ) specifically directed toward primary care PBRNs.6 In 2004, AHRQ identified 111 primary care PBRNs operating in the United States.3,6,7
PBRNs emphasize a close collaboration between practicing clinicians and researchers.1 Engaging network members in reflective inquiries can lead to practice improvement as well as new researchable questions for the network.8 However, the engagement of busy practices in practice-based research along with its growth in popularity and complexity of studies have led to increasing challenges in planning and implementing research studies in busy practices. Key challenges in this process include selecting studies that meet the goals and objectives of the network and its members; creating a practical, accurate, and sufficient study budget; developing study teams and agreements between team members; recruiting and selecting study sites; and training the practice staff for participation in network studies.9–12
AAFP NRN membership includes approximately 350 clinicians and study coordinators from 180 practices in 50 states and Canadian provinces. To date, the AAFP NRN has completed data collection in 17 studies and is currently working on 8 active projects. AAFP NRN studies range from simple physician surveys to complex randomized control trials and have included studies on topics such as bioterrorism preparedness of family physicians,13 patient safety in family physician offices,14–20 diabetes,21 alcohol screening,22 patient communication, practice change, and depression screening and care.
In this study, we discuss the 5 strategies that have been developed in the AAFP NRN through an iterative process of adjusting and improving procedures to plan and launch new studies. We expect that these processes and procedures, as well as the lessons that we have learned, will be useful for both new and experienced networks interested in successfully implementing PBRN studies.
Selecting Fundable, Feasible Studies
Because of the large number of projects that may be under consideration at any one time in the AAFP NRN, a multistep process for "vetting" studies has been developed. In the true spirit of PBRNs, this process incorporates input from both AAFP NRN staff and network members. A variety of issues influence study selection. A proposed study must be a topic of interest to members, of importance to the discipline, and feasible for the NRN. Figure 1 outlines the general process for evaluating study ideas and launching new studies, although this process may vary for a particular study.
Study ideas in the AAFP NRN come from a variety of sources, including directly from clinician members (the traditional bottom-up concept), principal investigators external to the AAFP NRN, potential funders, AAFP NRN staff, other PBRNs, and indirectly from requests for applications/proposals. Each study concept, regardless of the source of the idea, is first evaluated by the AAFP NRN's senior leaders (network director, research director, associate research director, and senior scientists). They review available documents including protocols, outlines, and abstracts, and address 4 main questions: (1) Is this a researchable study question (2) Is the AAFP NRN the best place to answer the question (3) Does this study fit with the goals and objectives of the NRN and (4) Is this study fundable/Do we have the available resources
If the senior leadership agrees that the idea meets these 4 criteria, a 1-page concept overview is presented to 1 of the 2 AAFP NRN Scientific Review Committees (SRC). Each SRC consists of 6 AAFP NRN physicians and 1 study coordinator (usually a nurse or medical assistant). Proposals are distributed to the 2 SRCs on a rotating basis. The SRC’s objective is to determine whether the study proposal is suitable and feasible in a primary care practice. SRC members may also make suggestions on a study’s methodology to enhance its implementation in a busy family medicine practice. The physician serving as committee chair is responsible for collecting committee member input and transmitting the final decision back to the AAFP NRN leadership.
Next, we assess the interest of the AAFP NRN membership. This process occurs before practice recruitment to validate that there is sufficient member interest in a particular study topic to meet recruitment goals. An AAFP NRN staff member communicates with network members who are appropriate for a given study, provides them with a 1-page overview of the study concept and a description of the responsibilities of the clinician or practice, and requests that they let us know if they are potentially interested in participating.
After a study idea progresses to this point in the process, the senior management begins developing a complete study protocol and pursuing funding options. Ultimately, it is the final decision of the AAFP NRN leadership and staff as to whether a project will proceed. However, because the success of network studies is contingent on participation of members, we strive to assure that they have ample input into the design of a research project early in the process.
The AAFP NRN developed its Principles for Industry-Funded Research (available at www.aafp.org/natnet) to guide negotiations with potential external commercial funders who invite the AAFP NRN to collaborate or take the lead on protocol development. This document outlines the principles by which the NRN collaborates with such industry sponsors, and clarifies the roles of both the AAFP NRN and the intended funders in the development, data collection, analysis, and dissemination of the research project.
Creating a Practical Budget
We begin to create a detailed budget once a preliminary protocol has been developed and potential funding source(s) have been identified. The Network Director and Research Administrator begin the budget process by reviewing the preliminary protocol and assigning costs by category, starting with personnel. Historically, we have tended to underestimate the amount of time required by staff to implement network studies. Therefore, we now pay particular attention to adequate salary coverage for the project manager and research assistants when possible. Once personnel costs have been determined, all other project-related costs are added into the budget. Our experience indicates that 2 other cost categories for PBRN studies require particular attention: study site training costs and payments to participating practices.
Depending on the funding source, a draft budget is either submitted for approval in the case of commercial and most private philanthropic funding or, in the case of government-sponsored funding, reviewed to determine whether the proposed budget fits into the funder’s allotment for the project. If the budget exceeds the available funds, the AAFP NRN director will make adjustments that ensure the project can be conducted without compromising the integrity of the study.
Composing the Study Team
The dynamics of the research team are essential to the success of any study. The composition of the team offers distinctive challenges to PBRN research. Members of the team generally include the principal investigator (PI), coinvestigators, project manager, consultants, statistician, research assistants, senior scientist(s), practicing physicians, and practice study coordinators. In addition, PBRNs carrying out prospective cohort studies, clinical trials, or practice change interventions must also consider sophisticated information management resources and formal linkages with the statistical and methodologic expertise existing in academic centers.7 An overall description of the study team’s roles appears in Table 1.
Practice-based research is distinctive because physicians and study coordinators from the selected practices are essential members of the study team. Their roles are critical in the planning stage of a research study. They provide invaluable feedback on the feasibility of implementing the protocol in practice. These practice representatives provide feedback on whether the proposed study protocol is feasible given their specific local requirements. Early engagement of these team members always improves implementation of the study protocol. The AAFP NRN first involves these practice leaders during the face-to-face study training session, as described in the following sections.
The AAFP NRN has developed policies and procedures to standardize network operations. These policies strengthen the understanding of the research team, expedite our work, and eliminate future misunderstandings. In addition, we have developed model agreements designed to ensure a common understanding between the AAFP NRN and outside research team members. All the policies and agreements listed below are available at www.aafp.org/natnet.
"Writing Process for Manuscripts" includes the roles, expectations, timelines, and procedures for writing papers related to studies conducted in the AAFP NRN. This writing process was developed to facilitate the timely dissemination of research findings to practices and to the academic press. This agreement states that the AAFP NRN expects to publish the results of every study conducted in our network.
The Publication Policy outlines how investigators working with the AAFP NRN are to prepare manuscripts for primary and secondary publication. It covers topics such as authorship, manuscript preparation and titles, conflict resolution, and acknowledging the AAFP NRN and study sites in published papers.
The External PI Agreement provides a detailed description of the relationship between the AAFP NRN and PIs who are not staff of the AAFP. It clarifies that any projects brought to the network by external PIs must be consistent with the AAFP NRN's mission, objectives, and current research agenda. The document also outlines the development of project budgets, AAFP NRN policies for day-to-day project operations, financial responsibilities and authority of the AAFP NRN director, and expectations for the dissemination of findings.
The Data Sharing Agreement sets forth the terms, conditions, and obligations concerning data sharing between the AAFP NRN and any another network or person. It clarifies that if the AAFP NRN conducts a study resulting in a project dataset, then it owns such data.
The General Affiliation Agreement asserts that the AAFP NRN is interested in promoting collaborative arrangements with state and regional networks. It also delineates the conditions under which the AAFP NRN and regional networks can collaborate, including study selection, decision making, data collection, and IRB approval for collaborative studies.
Recruiting and Selecting Sites
Practice recruitment and selection is another challenge for PBRNs. The practice recruitment and selection strategy is developed by the central research team for each study based on the number of practices, physicians, staff, and/or patients needed, determined by the power analysis and augmented by expected attrition.23–26 Once the target sample size has been set, AAFP NRN staff assess whether a sufficient number of practices can be recruited exclusively from the AAFP NRN membership based on the study’s requirements for particular patient and/or practice characteristics.
In situations where AAFP NRN members will not provide adequate numbers, a decision is made whether affiliated state or regional networks will be invited to participate. The AAFP NRN staff generally works with affiliate network directors or administrators to identify members of those networks who may be appropriate for the study.
Once the recruitment strategy is developed, the project staff initiates the recruitment process. The chronology of practice recruitment and selection from AAFP NRN practices is generally the following:
A broadcast recruitment E-mail is sent to all eligible AAFP NRN members announcing the study and briefly describing the eligibility criteria.
One week later, a personalized letter that includes responsibilities and timeline, a 1-page study overview, a Study Interest Form, and a postage-paid return envelope are sent to all eligible AAFP NRN members.
Two days later, another broadcast E-mail is sent to all eligible members announcing that recruitment letters and invitations for the study have been mailed.
Affiliated networks recruit their own practices to participate in a study.
One to 2 weeks later, if adequate numbers of AAFP NRN and affiliated network practices have not volunteered to participate in the study, we send individual E-mails and make personal telephone calls to selected AAFP NRN members inviting them to participate. In addition, we may contact staff of the AAFP regular electronic newsletter, AAFP News Now, about publishing a recruitment article in an upcoming edition. If necessary, we may contact additional affiliated networks about their participation in the study.
We select practices from the pool of eligible and interested practices that meet inclusion criteria and maximize diversity based on geographic location, practice type, size, and any other pertinent characteristics. We also consider other factors such as whether a practice is already participating in another study, and a practice and study coordinator’s previous experience in network studies (in particular, if we are selecting practices for a more complex study). We have created a database to assist us in practice selection in which we store these data on all AAFP NRN member practices, physicians, and study coordinators.
For each participating practice, the lead physician, study coordinator, Practice Signatory Official (as designated by the practice), and the study PI sign a Practice Agreement. This document states that the practice agrees to carry out all designated responsibilities of the study and the AAFP NRN agrees to provide materials, provide and pay for training, and provide the designated stipend. Institutional Review Board (IRB) issues are spelled out in the Unaffiliated Investigator Agreement.
At times during the recruitment phase we must make tough decisions about balancing scientific rigor with the realities of "real-world" medical practice. For example, occasionally we are forced to modify the initial sampling plan to achieve the required sample size. In a recent study, we initially planned to include few residency practices. However, an eligibility requirement that a practice either had to deliver at least 50 babies or perform 50 well-baby visits in the previous year forced us to change our original recruitment strategy and include residency practices to get sufficient numbers for our study. In addition, although in an ideal world we would have the luxury of randomly selecting practices to increase the representativeness of our sample, this is not always possible in PBRN studies. Sometimes we need to include all interested practices to reach sample size requirements. Finally, although a study design may require equal numbers of practices with or without a particular characteristic (eg, those with and without electronic medical records), we sometimes learn after a study begins that a practice, in fact, did not really have that characteristic. Thus, when the budget permits, we oversample practices just as we oversample patients. For instance, if a study is randomized at the practice level and the power analysis indicates that 10 practices per arm are sufficient, we plan for attrition by budgeting for 12 practices per arm.
Training Practice Staff and Physicians
The AAFP NRN has adopted a strategy of providing face-to-face training for practices selected to participate in a research study. When funding permits, we require each participating practice to send a team, usually comprising the site’s lead physician and study coordinator, for a weekend of training. Sessions generally run all day Saturday and, occasionally, through Sunday morning. In this way, staff and physicians are not out of the office during a busy work week. Bringing these people together ensures that the lead physicians and study coordinators understand the protocol and all participating practice staff receive the same instructions for implementing the study in the practice. This increases the likelihood that the study will be implemented with fidelity in all participating practices.
The entire research team, including all investigators, consultants, project manager, and research assistants, conducts the weekend sessions. The sessions have evolved into both a successful strategy for implementing studies in practices and a place where the practices make significant contributions that improve the design and feasibility of study protocols. Discussions encourage brainstorming on best strategies for implementing the protocol in a practice. As mentioned previously, practice leadership provides a new perspective, which frequently leads to improved protocol design and implementation, and ultimately to high quality study data.27
Training sessions also provide the opportunity for site leaders and the research team to get to know one another. In contrast to many regional networks, our national staff has few opportunities to meet face-to-face with members. Training sessions may be the only time during the study period that the research team and practice staff meet. These personal connections between the practice members and the research team lead to better communication throughout the study period and eventually lead to a more successful research study.
When key personnel from study sites are unable to attend the face-to-face training, we provide telephone training or ask someone who did attend the training to orient those who did not. However, these techniques cannot replicate the experience of a face-to-face training weekend with all the practices.
In addition, we develop an in-depth training manual that includes copies of and instructions for completion of all forms; the AAFP IRB application; all consent forms; instructions for completion of human subjects training; contact information for the central research team, practice physicians, practice study coordinators, and other key people; the study protocol, as appropriate; suggestions for study implementation, including recruitment and consent process for subjects; and other pertinent information. The manual is the basis of the training curriculum and agenda. A template for the manual and schedule appears in Table 2.
The cost and logistic issues involved in planning training sessions has led to the development of some useful guidelines. In an effort to control expenses and simplify the process of making travel arrangements for the training sessions, we use a travel agency. All participants are asked to make their own travel arrangements through the agency within the provided parameters for arrival and departure times and maximum ticket price. In this way, we can also limit the number of participants who arrive late or leave early. In addition, all flight expenses are billed directly to the AAFP NRN, eliminating the inconvenience of reimbursing participants. This system has worked well for both participants and the research team.
After training, practice participants and the research staff are generally excited and motivated to implement the study. To preserve the momentum generated from the training, we try to minimize the time between training and implementation in practices. However, this is not always possible for reasons such as if numerous changes to the protocol are made as a result of feedback from training participants; if practices need to train their staff and physicians before study initiation; or if practice-specific implementation strategies need to be developed. In cases where there is an extended period of time between the training session and study implementation in the practice, we conduct one-on-one refresher telephone calls with the study coordinator or the lead physician before the start of data collection.
Conclusion
Although the nature of PBRNs presents numerous challenges to conducting quality studies, the AAFP NRN has identified a number of processes that facilitate the implementation of studies on a national level. These strategies include how to select fundable, feasible studies; compose the study team; recruit and select sites; and train practice staff and physicians. The planning procedures described in this paper can be modified to work for state or regional PBRNs, and can create a strong foundation on which to build rigorous PBRN studies. Proper planning for PBRN studies significantly improves the success of study implementation and the dissemination of scientifically sound results.
Acknowledgments
We thank Debbi Main for her assistance in editing this manuscript and all the members of the AAFP NRN for their dedication and support.
Notes
This article was externally peer reviewed.
Conflict of interest: The authors are employees of (DG, TS, MS, WP) or partially paid by (ES) the American Academy of Family Physicians.
References
Fraser I, Lanier D, Hellinger F, Eisenberg JM. Putting practice into research. Health Serv Res 2002; 37: 1–14.
Nelson EC, Kirk JW, Bise BW, Chapman RJ, et al. The cooperative information project: part 1: a sentinel practice network for service and research in primary care. J Fam Pract 1981; 13: 641–649.
Lanier D. Primary care practice-based research comes of age in the United States. Ann Fam Med 2005; 3(Suppl 1): S2–S4.
Green L. A, Hickner J. A short history of primary care practice-based research networks: from concept to essential research laboratories. J Am Board Fam Med 2006; 19: 1–10.
Nutting PA, Weasley JW, Werner JJ. Practice-based research networks answer primary care questions. JAMA 1999; 281: 686–88.
Agency for Healthcare Research and Quality. Available from www.ahrq.gov/research/pbrn/pbrnfact.htm. Accessed October 17, 2006.
Green LA, White LL, Barry HC, Nease DE Jr, Hudson BL. Infrastructure requirements for practice-based research networks. Ann Fam Med 2005; 3(Suppl 1): S5–S11.
Mold JW, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med 2005; 3(Suppl 1): S12–S20.
Croughan M. Factors influencing physician participation in practice-based research network studies: a call for further research. J Fam Pract 2001; 50: 978–9.
Asch S, Connor SE, Hamilton EG, Fox SA. Problems in recruiting community-based physicians for health services research. J Gen Intern Med 2000; 15: 591–9.
Bell-Syer SE, Moffet JA. Recruiting patients to randomized trials in primary care: principles and case study. J Fam Pract 2000; 17: 187–91.
Van Weel C, Smith H, Beasley JW. Family practice research networks: experiences from 3 countries. J Fam Pract 2000; 49: 938–43.
Chen FM, Hickner J, Finks KS, Galliher JM, Burstin H. On the front lines: family physicians’ preparedness for bioterrorism. J Fam Pact 2002; 51: 745–9.
Dovey SM, Meyers DS, Phillips RL Jr, et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care 2002; 11: 233–8.
Elder NC, Graham D, Hickner J. Can a national primary-care error reporting system make a difference in medical practice Focus Patient Saf 2004; 7: 5–6.
Elder NC, Graham D, Brandt E, Dovey S, et al. The testing process in family medicine: problems, solutions and barriers as seen by physicians and their staff: a study of the American Academy of Family Physicians’ National Research Network. J Patient Saf 2006; 2: 25–32.
Lindbloom EJ, Ewigman BG, Hickner J. Practice-based research networks: the laboratories of primary care research. Med Care 2004; 42(Suppl 4): III45–9.
Graham DG, Pace WD, Kappus JA, Holcomb S, et al. The challenges and successes of institutional review board review and approval of practice-based research network studies. In: Henriksen K. Advances in Patient Safety: From Research to Implementation. Vol. 3, AHRQ Publication No. 050021 (3). Agency for Healthcare Research and Quality, Rockville, MD; February 2005.
Phillips RL, Dovey SM, Hickner J, Graham DG, et al. The AAFP Patient Safety Reporting System: Needed bridges for the next chasm in patient safety. In Henriksen K. Advances in Patient Safety: From Research to Implementation. Volume 3, AHRQ Publication No. 050021 (3). Agency for Healthcare Research and Quality, Rockville, MD; February 2005.
Phillips RL, Dovey SM, Graham D, Elder NC, Hickner JS, Learning from different lenses: reports of medical errors in primary care by clinicians, staff and patients. A project of the AAFP National Research Network. J Patient Saf 2006; 2: 140–6.
Spann SJ, Nutting PA, Galliher JM, Peterson KA, et al. Management of type-2 diabetes in the primary care setting: a practice-based research network study. Ann Fam Med 2006; 4: 23–31.
Vinson DC, Galliher JM, Reidinger C, Kappus JA. Comfortably engaging: which approach to alcohol screening should we use Ann Fam Med 2004; 2: 398–404.
McBride PE, Massoth KM, Underbakke G, Solberg LI, Beasley JW, Plane MB. Recruitment of private practices for primary care research: experience in a preventive services clinical trial. J Fam Pract 1996; 43: 389–95.
Christoffel KK, Binns HJ, Stockman JA 3rd, et al. Practice-based research: opportunities and obstacles. Pediatrics 1988; 82(3 Pt 2): 399–406.
Carey TS, Kinsinger L, Keyserling T, Harris R. Research in the community: recruiting and retaining practices. J Community Health 1996; 21: 315–27.
LeBailly S, Ariza A, Bayldon B, Binns HJ. The origin and evolution of a regional pediatric practice-based research network: practical and methodological lessons from the Pediatric Practice Research Group. Curr Probl Pediatr Adolesc Health Care 2003; 33: 124–34.
Holtrop JS, Hickner J, Dosh S, Noel M, et al. Sticking to it – diabetes mellitus: a pilot study of an innovative behavior change program for women with type 2 diabetes. Am J Health Educ 2002; 33: 161–6.(Deborah G. Graham, MSPH, Mindy S. Spano,)
Abstract
Research conducted in a practice-based research network (PBRN) differs from other multisite research and presents particular planning challenges. The American Academy of Family Physicians National Research Network (AAFP NRN) has developed a number of procedures used for planning and implementing studies, which address the challenges of national PBRN studies. In this study, we highlight challenges common to PBRN research and describe the methods used by the AAFP NRN to address those challenges. The following tasks were identified as important to implementing PBRN research studies: (1) selecting fundable, feasible studies that interest members and have the potential to improve quality of care; (2) creating a practical budget that covers the costs of the study; (3) composing study teams and securing written agreements between team members; (4) recruiting and selecting study sites; and (5) training practice staff and physicians. Striking the balance of scientific rigor with practical application of PBRN studies must be addressed throughout these tasks. Proper planning for PBRN studies significantly affects the success of study implementation. Although developed by a national PBRN, the planning procedures described in this study may be adapted for state or regional PBRNs.
A PBRN is a group of ambulatory practices devoted principally to providing quality patient care. These practices affiliate with one another to investigate questions related to both improving the care they provide and improving their discipline.1 PBRN research studies take place primarily in the office and the research questions are generally those of interest to the participating practices.
The history of primary care PBRNs in the United States began in the 1970s, with the appearance of the earliest regional networks.2,3 In 1978, the creation of a national family medicine PBRN was set in motion, leading to development of the Ambulatory Sentinel Practice Network (ASPN). The AAFP NRN was established in 2000 to replace ASPN as the national family medicine PBRN.4,5 Since the AAFP NRN's inception, primary care PBRNs have grown significantly in size and number. The growth of PBRNs over the past 2 decades has been supported by a number of funding opportunities from the Agency for Healthcare Research Quality (AHRQ) specifically directed toward primary care PBRNs.6 In 2004, AHRQ identified 111 primary care PBRNs operating in the United States.3,6,7
PBRNs emphasize a close collaboration between practicing clinicians and researchers.1 Engaging network members in reflective inquiries can lead to practice improvement as well as new researchable questions for the network.8 However, the engagement of busy practices in practice-based research along with its growth in popularity and complexity of studies have led to increasing challenges in planning and implementing research studies in busy practices. Key challenges in this process include selecting studies that meet the goals and objectives of the network and its members; creating a practical, accurate, and sufficient study budget; developing study teams and agreements between team members; recruiting and selecting study sites; and training the practice staff for participation in network studies.9–12
AAFP NRN membership includes approximately 350 clinicians and study coordinators from 180 practices in 50 states and Canadian provinces. To date, the AAFP NRN has completed data collection in 17 studies and is currently working on 8 active projects. AAFP NRN studies range from simple physician surveys to complex randomized control trials and have included studies on topics such as bioterrorism preparedness of family physicians,13 patient safety in family physician offices,14–20 diabetes,21 alcohol screening,22 patient communication, practice change, and depression screening and care.
In this study, we discuss the 5 strategies that have been developed in the AAFP NRN through an iterative process of adjusting and improving procedures to plan and launch new studies. We expect that these processes and procedures, as well as the lessons that we have learned, will be useful for both new and experienced networks interested in successfully implementing PBRN studies.
Selecting Fundable, Feasible Studies
Because of the large number of projects that may be under consideration at any one time in the AAFP NRN, a multistep process for "vetting" studies has been developed. In the true spirit of PBRNs, this process incorporates input from both AAFP NRN staff and network members. A variety of issues influence study selection. A proposed study must be a topic of interest to members, of importance to the discipline, and feasible for the NRN. Figure 1 outlines the general process for evaluating study ideas and launching new studies, although this process may vary for a particular study.
Study ideas in the AAFP NRN come from a variety of sources, including directly from clinician members (the traditional bottom-up concept), principal investigators external to the AAFP NRN, potential funders, AAFP NRN staff, other PBRNs, and indirectly from requests for applications/proposals. Each study concept, regardless of the source of the idea, is first evaluated by the AAFP NRN's senior leaders (network director, research director, associate research director, and senior scientists). They review available documents including protocols, outlines, and abstracts, and address 4 main questions: (1) Is this a researchable study question (2) Is the AAFP NRN the best place to answer the question (3) Does this study fit with the goals and objectives of the NRN and (4) Is this study fundable/Do we have the available resources
If the senior leadership agrees that the idea meets these 4 criteria, a 1-page concept overview is presented to 1 of the 2 AAFP NRN Scientific Review Committees (SRC). Each SRC consists of 6 AAFP NRN physicians and 1 study coordinator (usually a nurse or medical assistant). Proposals are distributed to the 2 SRCs on a rotating basis. The SRC’s objective is to determine whether the study proposal is suitable and feasible in a primary care practice. SRC members may also make suggestions on a study’s methodology to enhance its implementation in a busy family medicine practice. The physician serving as committee chair is responsible for collecting committee member input and transmitting the final decision back to the AAFP NRN leadership.
Next, we assess the interest of the AAFP NRN membership. This process occurs before practice recruitment to validate that there is sufficient member interest in a particular study topic to meet recruitment goals. An AAFP NRN staff member communicates with network members who are appropriate for a given study, provides them with a 1-page overview of the study concept and a description of the responsibilities of the clinician or practice, and requests that they let us know if they are potentially interested in participating.
After a study idea progresses to this point in the process, the senior management begins developing a complete study protocol and pursuing funding options. Ultimately, it is the final decision of the AAFP NRN leadership and staff as to whether a project will proceed. However, because the success of network studies is contingent on participation of members, we strive to assure that they have ample input into the design of a research project early in the process.
The AAFP NRN developed its Principles for Industry-Funded Research (available at www.aafp.org/natnet) to guide negotiations with potential external commercial funders who invite the AAFP NRN to collaborate or take the lead on protocol development. This document outlines the principles by which the NRN collaborates with such industry sponsors, and clarifies the roles of both the AAFP NRN and the intended funders in the development, data collection, analysis, and dissemination of the research project.
Creating a Practical Budget
We begin to create a detailed budget once a preliminary protocol has been developed and potential funding source(s) have been identified. The Network Director and Research Administrator begin the budget process by reviewing the preliminary protocol and assigning costs by category, starting with personnel. Historically, we have tended to underestimate the amount of time required by staff to implement network studies. Therefore, we now pay particular attention to adequate salary coverage for the project manager and research assistants when possible. Once personnel costs have been determined, all other project-related costs are added into the budget. Our experience indicates that 2 other cost categories for PBRN studies require particular attention: study site training costs and payments to participating practices.
Depending on the funding source, a draft budget is either submitted for approval in the case of commercial and most private philanthropic funding or, in the case of government-sponsored funding, reviewed to determine whether the proposed budget fits into the funder’s allotment for the project. If the budget exceeds the available funds, the AAFP NRN director will make adjustments that ensure the project can be conducted without compromising the integrity of the study.
Composing the Study Team
The dynamics of the research team are essential to the success of any study. The composition of the team offers distinctive challenges to PBRN research. Members of the team generally include the principal investigator (PI), coinvestigators, project manager, consultants, statistician, research assistants, senior scientist(s), practicing physicians, and practice study coordinators. In addition, PBRNs carrying out prospective cohort studies, clinical trials, or practice change interventions must also consider sophisticated information management resources and formal linkages with the statistical and methodologic expertise existing in academic centers.7 An overall description of the study team’s roles appears in Table 1.
Practice-based research is distinctive because physicians and study coordinators from the selected practices are essential members of the study team. Their roles are critical in the planning stage of a research study. They provide invaluable feedback on the feasibility of implementing the protocol in practice. These practice representatives provide feedback on whether the proposed study protocol is feasible given their specific local requirements. Early engagement of these team members always improves implementation of the study protocol. The AAFP NRN first involves these practice leaders during the face-to-face study training session, as described in the following sections.
The AAFP NRN has developed policies and procedures to standardize network operations. These policies strengthen the understanding of the research team, expedite our work, and eliminate future misunderstandings. In addition, we have developed model agreements designed to ensure a common understanding between the AAFP NRN and outside research team members. All the policies and agreements listed below are available at www.aafp.org/natnet.
"Writing Process for Manuscripts" includes the roles, expectations, timelines, and procedures for writing papers related to studies conducted in the AAFP NRN. This writing process was developed to facilitate the timely dissemination of research findings to practices and to the academic press. This agreement states that the AAFP NRN expects to publish the results of every study conducted in our network.
The Publication Policy outlines how investigators working with the AAFP NRN are to prepare manuscripts for primary and secondary publication. It covers topics such as authorship, manuscript preparation and titles, conflict resolution, and acknowledging the AAFP NRN and study sites in published papers.
The External PI Agreement provides a detailed description of the relationship between the AAFP NRN and PIs who are not staff of the AAFP. It clarifies that any projects brought to the network by external PIs must be consistent with the AAFP NRN's mission, objectives, and current research agenda. The document also outlines the development of project budgets, AAFP NRN policies for day-to-day project operations, financial responsibilities and authority of the AAFP NRN director, and expectations for the dissemination of findings.
The Data Sharing Agreement sets forth the terms, conditions, and obligations concerning data sharing between the AAFP NRN and any another network or person. It clarifies that if the AAFP NRN conducts a study resulting in a project dataset, then it owns such data.
The General Affiliation Agreement asserts that the AAFP NRN is interested in promoting collaborative arrangements with state and regional networks. It also delineates the conditions under which the AAFP NRN and regional networks can collaborate, including study selection, decision making, data collection, and IRB approval for collaborative studies.
Recruiting and Selecting Sites
Practice recruitment and selection is another challenge for PBRNs. The practice recruitment and selection strategy is developed by the central research team for each study based on the number of practices, physicians, staff, and/or patients needed, determined by the power analysis and augmented by expected attrition.23–26 Once the target sample size has been set, AAFP NRN staff assess whether a sufficient number of practices can be recruited exclusively from the AAFP NRN membership based on the study’s requirements for particular patient and/or practice characteristics.
In situations where AAFP NRN members will not provide adequate numbers, a decision is made whether affiliated state or regional networks will be invited to participate. The AAFP NRN staff generally works with affiliate network directors or administrators to identify members of those networks who may be appropriate for the study.
Once the recruitment strategy is developed, the project staff initiates the recruitment process. The chronology of practice recruitment and selection from AAFP NRN practices is generally the following:
A broadcast recruitment E-mail is sent to all eligible AAFP NRN members announcing the study and briefly describing the eligibility criteria.
One week later, a personalized letter that includes responsibilities and timeline, a 1-page study overview, a Study Interest Form, and a postage-paid return envelope are sent to all eligible AAFP NRN members.
Two days later, another broadcast E-mail is sent to all eligible members announcing that recruitment letters and invitations for the study have been mailed.
Affiliated networks recruit their own practices to participate in a study.
One to 2 weeks later, if adequate numbers of AAFP NRN and affiliated network practices have not volunteered to participate in the study, we send individual E-mails and make personal telephone calls to selected AAFP NRN members inviting them to participate. In addition, we may contact staff of the AAFP regular electronic newsletter, AAFP News Now, about publishing a recruitment article in an upcoming edition. If necessary, we may contact additional affiliated networks about their participation in the study.
We select practices from the pool of eligible and interested practices that meet inclusion criteria and maximize diversity based on geographic location, practice type, size, and any other pertinent characteristics. We also consider other factors such as whether a practice is already participating in another study, and a practice and study coordinator’s previous experience in network studies (in particular, if we are selecting practices for a more complex study). We have created a database to assist us in practice selection in which we store these data on all AAFP NRN member practices, physicians, and study coordinators.
For each participating practice, the lead physician, study coordinator, Practice Signatory Official (as designated by the practice), and the study PI sign a Practice Agreement. This document states that the practice agrees to carry out all designated responsibilities of the study and the AAFP NRN agrees to provide materials, provide and pay for training, and provide the designated stipend. Institutional Review Board (IRB) issues are spelled out in the Unaffiliated Investigator Agreement.
At times during the recruitment phase we must make tough decisions about balancing scientific rigor with the realities of "real-world" medical practice. For example, occasionally we are forced to modify the initial sampling plan to achieve the required sample size. In a recent study, we initially planned to include few residency practices. However, an eligibility requirement that a practice either had to deliver at least 50 babies or perform 50 well-baby visits in the previous year forced us to change our original recruitment strategy and include residency practices to get sufficient numbers for our study. In addition, although in an ideal world we would have the luxury of randomly selecting practices to increase the representativeness of our sample, this is not always possible in PBRN studies. Sometimes we need to include all interested practices to reach sample size requirements. Finally, although a study design may require equal numbers of practices with or without a particular characteristic (eg, those with and without electronic medical records), we sometimes learn after a study begins that a practice, in fact, did not really have that characteristic. Thus, when the budget permits, we oversample practices just as we oversample patients. For instance, if a study is randomized at the practice level and the power analysis indicates that 10 practices per arm are sufficient, we plan for attrition by budgeting for 12 practices per arm.
Training Practice Staff and Physicians
The AAFP NRN has adopted a strategy of providing face-to-face training for practices selected to participate in a research study. When funding permits, we require each participating practice to send a team, usually comprising the site’s lead physician and study coordinator, for a weekend of training. Sessions generally run all day Saturday and, occasionally, through Sunday morning. In this way, staff and physicians are not out of the office during a busy work week. Bringing these people together ensures that the lead physicians and study coordinators understand the protocol and all participating practice staff receive the same instructions for implementing the study in the practice. This increases the likelihood that the study will be implemented with fidelity in all participating practices.
The entire research team, including all investigators, consultants, project manager, and research assistants, conducts the weekend sessions. The sessions have evolved into both a successful strategy for implementing studies in practices and a place where the practices make significant contributions that improve the design and feasibility of study protocols. Discussions encourage brainstorming on best strategies for implementing the protocol in a practice. As mentioned previously, practice leadership provides a new perspective, which frequently leads to improved protocol design and implementation, and ultimately to high quality study data.27
Training sessions also provide the opportunity for site leaders and the research team to get to know one another. In contrast to many regional networks, our national staff has few opportunities to meet face-to-face with members. Training sessions may be the only time during the study period that the research team and practice staff meet. These personal connections between the practice members and the research team lead to better communication throughout the study period and eventually lead to a more successful research study.
When key personnel from study sites are unable to attend the face-to-face training, we provide telephone training or ask someone who did attend the training to orient those who did not. However, these techniques cannot replicate the experience of a face-to-face training weekend with all the practices.
In addition, we develop an in-depth training manual that includes copies of and instructions for completion of all forms; the AAFP IRB application; all consent forms; instructions for completion of human subjects training; contact information for the central research team, practice physicians, practice study coordinators, and other key people; the study protocol, as appropriate; suggestions for study implementation, including recruitment and consent process for subjects; and other pertinent information. The manual is the basis of the training curriculum and agenda. A template for the manual and schedule appears in Table 2.
The cost and logistic issues involved in planning training sessions has led to the development of some useful guidelines. In an effort to control expenses and simplify the process of making travel arrangements for the training sessions, we use a travel agency. All participants are asked to make their own travel arrangements through the agency within the provided parameters for arrival and departure times and maximum ticket price. In this way, we can also limit the number of participants who arrive late or leave early. In addition, all flight expenses are billed directly to the AAFP NRN, eliminating the inconvenience of reimbursing participants. This system has worked well for both participants and the research team.
After training, practice participants and the research staff are generally excited and motivated to implement the study. To preserve the momentum generated from the training, we try to minimize the time between training and implementation in practices. However, this is not always possible for reasons such as if numerous changes to the protocol are made as a result of feedback from training participants; if practices need to train their staff and physicians before study initiation; or if practice-specific implementation strategies need to be developed. In cases where there is an extended period of time between the training session and study implementation in the practice, we conduct one-on-one refresher telephone calls with the study coordinator or the lead physician before the start of data collection.
Conclusion
Although the nature of PBRNs presents numerous challenges to conducting quality studies, the AAFP NRN has identified a number of processes that facilitate the implementation of studies on a national level. These strategies include how to select fundable, feasible studies; compose the study team; recruit and select sites; and train practice staff and physicians. The planning procedures described in this paper can be modified to work for state or regional PBRNs, and can create a strong foundation on which to build rigorous PBRN studies. Proper planning for PBRN studies significantly improves the success of study implementation and the dissemination of scientifically sound results.
Acknowledgments
We thank Debbi Main for her assistance in editing this manuscript and all the members of the AAFP NRN for their dedication and support.
Notes
This article was externally peer reviewed.
Conflict of interest: The authors are employees of (DG, TS, MS, WP) or partially paid by (ES) the American Academy of Family Physicians.
References
Fraser I, Lanier D, Hellinger F, Eisenberg JM. Putting practice into research. Health Serv Res 2002; 37: 1–14.
Nelson EC, Kirk JW, Bise BW, Chapman RJ, et al. The cooperative information project: part 1: a sentinel practice network for service and research in primary care. J Fam Pract 1981; 13: 641–649.
Lanier D. Primary care practice-based research comes of age in the United States. Ann Fam Med 2005; 3(Suppl 1): S2–S4.
Green L. A, Hickner J. A short history of primary care practice-based research networks: from concept to essential research laboratories. J Am Board Fam Med 2006; 19: 1–10.
Nutting PA, Weasley JW, Werner JJ. Practice-based research networks answer primary care questions. JAMA 1999; 281: 686–88.
Agency for Healthcare Research and Quality. Available from www.ahrq.gov/research/pbrn/pbrnfact.htm. Accessed October 17, 2006.
Green LA, White LL, Barry HC, Nease DE Jr, Hudson BL. Infrastructure requirements for practice-based research networks. Ann Fam Med 2005; 3(Suppl 1): S5–S11.
Mold JW, Peterson KA. Primary care practice-based research networks: working at the interface between research and quality improvement. Ann Fam Med 2005; 3(Suppl 1): S12–S20.
Croughan M. Factors influencing physician participation in practice-based research network studies: a call for further research. J Fam Pract 2001; 50: 978–9.
Asch S, Connor SE, Hamilton EG, Fox SA. Problems in recruiting community-based physicians for health services research. J Gen Intern Med 2000; 15: 591–9.
Bell-Syer SE, Moffet JA. Recruiting patients to randomized trials in primary care: principles and case study. J Fam Pract 2000; 17: 187–91.
Van Weel C, Smith H, Beasley JW. Family practice research networks: experiences from 3 countries. J Fam Pract 2000; 49: 938–43.
Chen FM, Hickner J, Finks KS, Galliher JM, Burstin H. On the front lines: family physicians’ preparedness for bioterrorism. J Fam Pact 2002; 51: 745–9.
Dovey SM, Meyers DS, Phillips RL Jr, et al. A preliminary taxonomy of medical errors in family practice. Qual Saf Health Care 2002; 11: 233–8.
Elder NC, Graham D, Hickner J. Can a national primary-care error reporting system make a difference in medical practice Focus Patient Saf 2004; 7: 5–6.
Elder NC, Graham D, Brandt E, Dovey S, et al. The testing process in family medicine: problems, solutions and barriers as seen by physicians and their staff: a study of the American Academy of Family Physicians’ National Research Network. J Patient Saf 2006; 2: 25–32.
Lindbloom EJ, Ewigman BG, Hickner J. Practice-based research networks: the laboratories of primary care research. Med Care 2004; 42(Suppl 4): III45–9.
Graham DG, Pace WD, Kappus JA, Holcomb S, et al. The challenges and successes of institutional review board review and approval of practice-based research network studies. In: Henriksen K. Advances in Patient Safety: From Research to Implementation. Vol. 3, AHRQ Publication No. 050021 (3). Agency for Healthcare Research and Quality, Rockville, MD; February 2005.
Phillips RL, Dovey SM, Hickner J, Graham DG, et al. The AAFP Patient Safety Reporting System: Needed bridges for the next chasm in patient safety. In Henriksen K. Advances in Patient Safety: From Research to Implementation. Volume 3, AHRQ Publication No. 050021 (3). Agency for Healthcare Research and Quality, Rockville, MD; February 2005.
Phillips RL, Dovey SM, Graham D, Elder NC, Hickner JS, Learning from different lenses: reports of medical errors in primary care by clinicians, staff and patients. A project of the AAFP National Research Network. J Patient Saf 2006; 2: 140–6.
Spann SJ, Nutting PA, Galliher JM, Peterson KA, et al. Management of type-2 diabetes in the primary care setting: a practice-based research network study. Ann Fam Med 2006; 4: 23–31.
Vinson DC, Galliher JM, Reidinger C, Kappus JA. Comfortably engaging: which approach to alcohol screening should we use Ann Fam Med 2004; 2: 398–404.
McBride PE, Massoth KM, Underbakke G, Solberg LI, Beasley JW, Plane MB. Recruitment of private practices for primary care research: experience in a preventive services clinical trial. J Fam Pract 1996; 43: 389–95.
Christoffel KK, Binns HJ, Stockman JA 3rd, et al. Practice-based research: opportunities and obstacles. Pediatrics 1988; 82(3 Pt 2): 399–406.
Carey TS, Kinsinger L, Keyserling T, Harris R. Research in the community: recruiting and retaining practices. J Community Health 1996; 21: 315–27.
LeBailly S, Ariza A, Bayldon B, Binns HJ. The origin and evolution of a regional pediatric practice-based research network: practical and methodological lessons from the Pediatric Practice Research Group. Curr Probl Pediatr Adolesc Health Care 2003; 33: 124–34.
Holtrop JS, Hickner J, Dosh S, Noel M, et al. Sticking to it – diabetes mellitus: a pilot study of an innovative behavior change program for women with type 2 diabetes. Am J Health Educ 2002; 33: 161–6.(Deborah G. Graham, MSPH, Mindy S. Spano,)