Maintenance of Certification for Family Physicians (MC-FP) Self Assessment Modules (SAMs): The First Year
http://www.100md.com
《家庭医学美国版》
Department of Family and Community Medicine, University of Kentucky College of Medicine
American Board of Family Medicine
Department of Family Medicine, University of Oklahoma-Tulsa
Department of Medicine, Washington University School of Medicine
Assessment Technologies, Inc
Abstract
Introduction: In 2000, the American Board of Medical Specialties adopted Maintenance of Certification (MOC) to replace intermittent, periodic recertification. MOC consists of 4 components: demonstration of professionalism (part I); commitment to life-long learning (part II); demonstration of cognitive expertise (part III); and evaluation of performance in practice (part IV). The American Board of Family Medicine (ABFM) implemented Maintenance of Certification for Family Physicians (MC-FP) in 2004, with its MC-FP part II self-assessment modules (SAMs) as the focus of the first year’s activities.
Methods: The SAMs use materials and resources provided at the ABFM’s website (www.theabfm.org). As of April 2005, 7000 Diplomates had successfully completed SAMs in essential hypertension (N = 2351) and type 2 diabetes mellitus (N = 4648). Participants completed categorical modified Likert scale evaluations to receive continuing education credit, and many offered unstructured free-text comments regarding the clinical simulation component. These free-text comments were entered into the AnSWR qualitative analysis program from the Centers for Disease Control and Prevention. Text coding was performed by 2 authors (MDH, DJI). As no inferential analyses or comparisons were anticipated, the authors conducted no studies of inter-rater consistency. Results are reported as means (SD) and medians for continuous data, and as frequencies for count data.
Results: Likert-scale ratings indicated generally favorable responses (predominantly 5 to 6 on a 6-point scale) to the hypertension and diabetes SAMs. In addition, over half (ie, 55% for hypertension and 54% for diabetes participants) of the respondents indicated that the experience would lead to changes in their practices. Navigation and system operation issues predominated in the free-text comments offered for the diabetes and hypertension simulations.
Conclusion: The MC-FP SAMs received generally favorable ratings in the program’s first year. The SAMs underwent a number of modifications and improvements during the first year, largely in response to feedback and suggestions from ABFM Diplomates.
Since its inception (as the American Board of Family Practice, ABFP), the American Board of Family Medicine (ABFM) has embraced ongoing continuing education requirements and periodic recertification for its Diplomates. The ABFM was for many years the only member of the American Board of Medical Specialties (ABMS) to require recertification. The creators of the ABFP selected 7 years as the time limit for its certificates. Although no data existed at the time of the Board’s creation to recommend a particular recertification interval, subsequent experience demonstrated predictable declines in Diplomate cognitive examination performance over the 7-year time frame.1
In response to concerns about the intermittent nature of recertification and public discussion about perceived health care quality problems,2 the ABMS embraced continuous Maintenance of Certification (MOC) in 2000 to replace the periodic recertification concept.3 ABMS member boards agreed to submit implementation plans for MOC by July 1, 2003. The ABFM developed a plan and created the materials and processes to implement Maintenance of Certification for Family Physicians (MC-FP) by January 1, 2004.
The MOC process to which all the ABMS member boards agreed includes 4 components: demonstration of professionalism, commitment to life-long learning, demonstration of cognitive expertise, and evaluation of performance in practice. Although these components comprise new evaluation dimensions for many boards, they represent only a modification and enrichment of processes that ABFM has used since 1969. This report will focus on the second component, commitment to life-long learning.
Whereas the Board has always had a continuing education requirement (which remains in the MC-FP process), participants in MC-FP must demonstrate commitment to life-long learning by completing annually a self-assessment module (SAM) that provides in-depth assessment of current knowledge in specific content areas (beginning in 2007, Diplomates will have the option of extending their certification to 10 years duration; those who take this option will need to complete only 2 SAMs in each 3-year stage of the process).4 The SAM consists of 2 parts: a 60 item knowledge assessment, with direct links to supporting references and critiques, and a clinical simulation that focuses on patient management (at the time of this writing, the Board has created SAMs for hypertension, type 2 diabetes mellitus, asthma, coronary artery disease, major depression, and heart failure). The SAM knowledge assessment includes several different item types: multiple choice single best answer, one or more correct answers, and uncued fill-in-the-blank styles. The 60 items are grouped in competency areas (eg, pharmacologic management of diabetes), and Diplomates must accomplish 80% correct in each competency area to complete the assessment successfully. Once the Diplomate has completed the knowledge assessment, she then proceeds to the clinical simulation (ClinSim). Both components are completed over the Internet via the ABFM website at http://www.theabfm.org. The Diplomates may download the knowledge assessment materials to work off-line at their leisure and subsequently submit their results through the website.
The remainder of this report will focus on Diplomate evaluations and feedback regarding the SAMs completed during MC-FP‘s first year.
Methods
Diplomates became eligible to participate in MC-FP in 2004 by certifying or recertifying in calendar year 2003. SAMs available for 2004 included type 2 diabetes mellitus and essential hypertension. As of April 14, 2005, 4648 Diplomates completed the diabetes SAM, and 2351 fulfilled the hypertension SAM requirements. On completion of both SAM components, the Diplomates executed an evaluation that included several dimensions: (1) a 6-point scale assay of relevance, currency, usefulness, and overall assessment of the module (1 = lowest possible, 6 = highest possible); (2) a query regarding whether or not the SAM experience would lead to changes in practice; (3) an estimate of time spent conducting the knowledge assessment; and (4) an estimate of time spent completing the clinical simulation component. Diplomates submitted 4047 evaluations for the diabetes SAM (87%), and 2317 (98%) for the hypertension SAM. In addition, the evaluation process provided Diplomates with the opportunity to submit free-text comments regarding their experience with the clinical simulation component. The Board received 1160 responses regarding the hypertension simulation, and 2229 comments for the diabetes simulation. Two of the authors (MDH for hypertension, DJI for diabetes) conducted a qualitative analysis of these comments along 5 dimensions: perceived expectations for performance, navigation, natural language parsing, system operation, and overall assessment. The analysis was performed using the AnSWR program from the Centers for Disease Control and Prevention (available free of charge at http://www.cdc.gov/hiv/software/answr.htm).5 The authors (MDH, DJI) did not conduct analyses of inter-rater reliability (see Limitations).
Results in the First Year
By April 2005, over 7000 Diplomates who entered the MC-FP process in 2004 (ie, either certified or recertified in 2003) had completed both SAM components. Two thirds of the Diplomates chose the diabetes module, and the remainder used the hypertension module.
Most users rated both the hypertension and diabetes modules at levels 5 and 6 across all 4 dimensions (Figures 1 and 2, respectively.) Furthermore, a majority of participants in both modules indicated that the experience would lead to changes in their practices (55% and 54% for the hypertension and diabetes SAMs, respectively).
Diplomates reported needing a mean of 9.96 hours (SD 5.87, median = 10 hours) to complete the hypertension knowledge assessment, and 11.4 hours (SD 7.15, median = 10 hours) for diabetes. The hypertension simulation took an average of 2.4 hours (SD 2.54, median = 2 hours) to complete; diabetes took slightly longer at 3.12 hours (SD 3.23, median = 2 hours).
Figure 3 displays the distribution of comments for the hypertension simulation in 5 major categories. Figure 4 displays similar information for the diabetes simulation. Most of the comments related to navigation issues (eg, "Had trouble finding the fundoscopic examination"), followed by observations about system operation (eg, "The system kept hanging up") and Diplomates’ overall assessment of the simulation experience. Although the analyses were performed independently by 2 of the authors, the figures exhibit substantially similar distributions. The free-text comments included a number of suggestions that have led to significant modifications of the simulator interface and operation (eg, additional "help" features, elimination of lengthy image downloads, and more informative consultation reports).
Limitations
Several limitations apply to this report. The SAMs underwent significant modification over the course of the year, but the authors have made no attempt to stratify the evaluation materials as a function of when Diplomates actually participated. Evaluations and comments from early in the year might therefore not directly correlate with those provided later.
The reports of change in practice as a result of SAM participation represent self-reports. The SAM process does not include outcomes assessments that could corroborate these reports. Data from MC-FP part IV, which became available in 2005, may provide the substrate for such corroboration in future reports.
The qualitative analyses of the diabetes and hypertension clinical simulation feedback were conducted by 2 of the authors (MDH, DJI) independently, without an assessment of inter-rater reliability. Differences in the summaries of the comments for these 2 domains could therefore represent inter-rater, rather than real, variation. However, as seen in Figures 3 and 4, the outcomes of these analyses appear quite similar, suggesting that inter-rater variation didn’t substantially affect the results.
Discussion
Recent reports have highlighted potential quality issues in the American health care system.2,6 In addition, evidence suggests that physician practice performance declines over time,7 as does cognitive expertise (at least as measured by proctored standardized examinations).1 These concerns have stimulated the medical specialty certifying boards to re-examine their certification processes to focus on ongoing cognitive development and quality improvement.3
The first year (2004) of MC-FP focused on introduction of the part II SAMs. The evaluations provided by users during this period were overall quite positive. The Diplomates additionally provided very substantive and helpful suggestions that have served to improve the process. For example, the Board has streamlined the process for accessing reference materials, provided earlier access to critiques in the knowledge assessment, and made a number of changes to the simulations that have improved performance for users who access the simulator via dial-up connections.
The ABFM does not yet have direct evidence that the MC-FP Part II process as presently organized will improve health care quality. However, the literature suggests that cognitive performance (the focus of the SAMs) does relate to patient outcomes that matter.8–11
The Future of Family Medicine report identified information management as a key skill for the family physician of the 21st century.12 In addition, the report advocates the integration of evidence-based clinical guidelines into a new model of care. The ABFM created the MC-FP process and the SAMs to facilitate these goals and plans to add at least 2 new SAMs per year. The process remains dynamic and the Board anticipates that feedback and suggestions provided by the Diplomates will continue to enhance these products for all users
Notes
Conflict of interest: none declared.
Received for publication September 2, 2005. Revision received April 7, 2006. Accepted for publication April 17, 2006.
References
Leigh T. Performance of Family Practice Diplomates on successive mandatory recertification examinations. Acad Med 1993; 68: 912–9.
Kohn L, Corrigan J, Donaldson M. To err is human: building a safer health system. Washington (DC): National Academy Press; 1999.
Statement on Commitment to Maintenance of Certification. 2000 [cited 4/5/05]. Available from: http://www.abms.org/downloads/publications/3-approved%20initiatives%20for%20MOC.pdf.
MC-FP Program Enhancements. 2006 [accessed 2006 March 28]; Description of the 10 year option for Maintenance of Certification]. Available from: https://www.theabfm.org/MOC/enhancements.aspx.
Analysis software for Word-based records. 2004, Centers for Disease Control and Prevention: Atlanta, GA. Available from: http://www.cdc.gov/hiv/software/answr.htm.
Committee on Quality of Health Care in America. Crossing the Quality Chasm. A New Health System for the 21st Century. Washington (DC): National Academy Press; 2001.
Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med 2005; 142: 260–73.
Norcini JJ, Kimball HR, Lipner RS. Certification and specialization: do they matter in the outcome of acute myocardial infarction? Acad Med 2000; 75: 1193–8.
Norcini JJ, Lipner RS. The relationship between the nature of practice and performance on a cognitive examination. Acad Med 2000; 75(10 Suppl): S68–70.
Norton PG, Dunn EV, Soberman L. What factors affect quality of care? Using the Peer Assessment Program in Ontario family practices. Can Fam Physician 1997; 43: 1739–44.
Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ 2002; 36: 853–9.
Committee, F.o.F.M.P.L., The future of family medicine: a collaborative project of the family medicine community. Ann Fam Med 2004; 2 Suppl 1: S3–32.(Michael D. Hagen, MD, Dou)
American Board of Family Medicine
Department of Family Medicine, University of Oklahoma-Tulsa
Department of Medicine, Washington University School of Medicine
Assessment Technologies, Inc
Abstract
Introduction: In 2000, the American Board of Medical Specialties adopted Maintenance of Certification (MOC) to replace intermittent, periodic recertification. MOC consists of 4 components: demonstration of professionalism (part I); commitment to life-long learning (part II); demonstration of cognitive expertise (part III); and evaluation of performance in practice (part IV). The American Board of Family Medicine (ABFM) implemented Maintenance of Certification for Family Physicians (MC-FP) in 2004, with its MC-FP part II self-assessment modules (SAMs) as the focus of the first year’s activities.
Methods: The SAMs use materials and resources provided at the ABFM’s website (www.theabfm.org). As of April 2005, 7000 Diplomates had successfully completed SAMs in essential hypertension (N = 2351) and type 2 diabetes mellitus (N = 4648). Participants completed categorical modified Likert scale evaluations to receive continuing education credit, and many offered unstructured free-text comments regarding the clinical simulation component. These free-text comments were entered into the AnSWR qualitative analysis program from the Centers for Disease Control and Prevention. Text coding was performed by 2 authors (MDH, DJI). As no inferential analyses or comparisons were anticipated, the authors conducted no studies of inter-rater consistency. Results are reported as means (SD) and medians for continuous data, and as frequencies for count data.
Results: Likert-scale ratings indicated generally favorable responses (predominantly 5 to 6 on a 6-point scale) to the hypertension and diabetes SAMs. In addition, over half (ie, 55% for hypertension and 54% for diabetes participants) of the respondents indicated that the experience would lead to changes in their practices. Navigation and system operation issues predominated in the free-text comments offered for the diabetes and hypertension simulations.
Conclusion: The MC-FP SAMs received generally favorable ratings in the program’s first year. The SAMs underwent a number of modifications and improvements during the first year, largely in response to feedback and suggestions from ABFM Diplomates.
Since its inception (as the American Board of Family Practice, ABFP), the American Board of Family Medicine (ABFM) has embraced ongoing continuing education requirements and periodic recertification for its Diplomates. The ABFM was for many years the only member of the American Board of Medical Specialties (ABMS) to require recertification. The creators of the ABFP selected 7 years as the time limit for its certificates. Although no data existed at the time of the Board’s creation to recommend a particular recertification interval, subsequent experience demonstrated predictable declines in Diplomate cognitive examination performance over the 7-year time frame.1
In response to concerns about the intermittent nature of recertification and public discussion about perceived health care quality problems,2 the ABMS embraced continuous Maintenance of Certification (MOC) in 2000 to replace the periodic recertification concept.3 ABMS member boards agreed to submit implementation plans for MOC by July 1, 2003. The ABFM developed a plan and created the materials and processes to implement Maintenance of Certification for Family Physicians (MC-FP) by January 1, 2004.
The MOC process to which all the ABMS member boards agreed includes 4 components: demonstration of professionalism, commitment to life-long learning, demonstration of cognitive expertise, and evaluation of performance in practice. Although these components comprise new evaluation dimensions for many boards, they represent only a modification and enrichment of processes that ABFM has used since 1969. This report will focus on the second component, commitment to life-long learning.
Whereas the Board has always had a continuing education requirement (which remains in the MC-FP process), participants in MC-FP must demonstrate commitment to life-long learning by completing annually a self-assessment module (SAM) that provides in-depth assessment of current knowledge in specific content areas (beginning in 2007, Diplomates will have the option of extending their certification to 10 years duration; those who take this option will need to complete only 2 SAMs in each 3-year stage of the process).4 The SAM consists of 2 parts: a 60 item knowledge assessment, with direct links to supporting references and critiques, and a clinical simulation that focuses on patient management (at the time of this writing, the Board has created SAMs for hypertension, type 2 diabetes mellitus, asthma, coronary artery disease, major depression, and heart failure). The SAM knowledge assessment includes several different item types: multiple choice single best answer, one or more correct answers, and uncued fill-in-the-blank styles. The 60 items are grouped in competency areas (eg, pharmacologic management of diabetes), and Diplomates must accomplish 80% correct in each competency area to complete the assessment successfully. Once the Diplomate has completed the knowledge assessment, she then proceeds to the clinical simulation (ClinSim). Both components are completed over the Internet via the ABFM website at http://www.theabfm.org. The Diplomates may download the knowledge assessment materials to work off-line at their leisure and subsequently submit their results through the website.
The remainder of this report will focus on Diplomate evaluations and feedback regarding the SAMs completed during MC-FP‘s first year.
Methods
Diplomates became eligible to participate in MC-FP in 2004 by certifying or recertifying in calendar year 2003. SAMs available for 2004 included type 2 diabetes mellitus and essential hypertension. As of April 14, 2005, 4648 Diplomates completed the diabetes SAM, and 2351 fulfilled the hypertension SAM requirements. On completion of both SAM components, the Diplomates executed an evaluation that included several dimensions: (1) a 6-point scale assay of relevance, currency, usefulness, and overall assessment of the module (1 = lowest possible, 6 = highest possible); (2) a query regarding whether or not the SAM experience would lead to changes in practice; (3) an estimate of time spent conducting the knowledge assessment; and (4) an estimate of time spent completing the clinical simulation component. Diplomates submitted 4047 evaluations for the diabetes SAM (87%), and 2317 (98%) for the hypertension SAM. In addition, the evaluation process provided Diplomates with the opportunity to submit free-text comments regarding their experience with the clinical simulation component. The Board received 1160 responses regarding the hypertension simulation, and 2229 comments for the diabetes simulation. Two of the authors (MDH for hypertension, DJI for diabetes) conducted a qualitative analysis of these comments along 5 dimensions: perceived expectations for performance, navigation, natural language parsing, system operation, and overall assessment. The analysis was performed using the AnSWR program from the Centers for Disease Control and Prevention (available free of charge at http://www.cdc.gov/hiv/software/answr.htm).5 The authors (MDH, DJI) did not conduct analyses of inter-rater reliability (see Limitations).
Results in the First Year
By April 2005, over 7000 Diplomates who entered the MC-FP process in 2004 (ie, either certified or recertified in 2003) had completed both SAM components. Two thirds of the Diplomates chose the diabetes module, and the remainder used the hypertension module.
Most users rated both the hypertension and diabetes modules at levels 5 and 6 across all 4 dimensions (Figures 1 and 2, respectively.) Furthermore, a majority of participants in both modules indicated that the experience would lead to changes in their practices (55% and 54% for the hypertension and diabetes SAMs, respectively).
Diplomates reported needing a mean of 9.96 hours (SD 5.87, median = 10 hours) to complete the hypertension knowledge assessment, and 11.4 hours (SD 7.15, median = 10 hours) for diabetes. The hypertension simulation took an average of 2.4 hours (SD 2.54, median = 2 hours) to complete; diabetes took slightly longer at 3.12 hours (SD 3.23, median = 2 hours).
Figure 3 displays the distribution of comments for the hypertension simulation in 5 major categories. Figure 4 displays similar information for the diabetes simulation. Most of the comments related to navigation issues (eg, "Had trouble finding the fundoscopic examination"), followed by observations about system operation (eg, "The system kept hanging up") and Diplomates’ overall assessment of the simulation experience. Although the analyses were performed independently by 2 of the authors, the figures exhibit substantially similar distributions. The free-text comments included a number of suggestions that have led to significant modifications of the simulator interface and operation (eg, additional "help" features, elimination of lengthy image downloads, and more informative consultation reports).
Limitations
Several limitations apply to this report. The SAMs underwent significant modification over the course of the year, but the authors have made no attempt to stratify the evaluation materials as a function of when Diplomates actually participated. Evaluations and comments from early in the year might therefore not directly correlate with those provided later.
The reports of change in practice as a result of SAM participation represent self-reports. The SAM process does not include outcomes assessments that could corroborate these reports. Data from MC-FP part IV, which became available in 2005, may provide the substrate for such corroboration in future reports.
The qualitative analyses of the diabetes and hypertension clinical simulation feedback were conducted by 2 of the authors (MDH, DJI) independently, without an assessment of inter-rater reliability. Differences in the summaries of the comments for these 2 domains could therefore represent inter-rater, rather than real, variation. However, as seen in Figures 3 and 4, the outcomes of these analyses appear quite similar, suggesting that inter-rater variation didn’t substantially affect the results.
Discussion
Recent reports have highlighted potential quality issues in the American health care system.2,6 In addition, evidence suggests that physician practice performance declines over time,7 as does cognitive expertise (at least as measured by proctored standardized examinations).1 These concerns have stimulated the medical specialty certifying boards to re-examine their certification processes to focus on ongoing cognitive development and quality improvement.3
The first year (2004) of MC-FP focused on introduction of the part II SAMs. The evaluations provided by users during this period were overall quite positive. The Diplomates additionally provided very substantive and helpful suggestions that have served to improve the process. For example, the Board has streamlined the process for accessing reference materials, provided earlier access to critiques in the knowledge assessment, and made a number of changes to the simulations that have improved performance for users who access the simulator via dial-up connections.
The ABFM does not yet have direct evidence that the MC-FP Part II process as presently organized will improve health care quality. However, the literature suggests that cognitive performance (the focus of the SAMs) does relate to patient outcomes that matter.8–11
The Future of Family Medicine report identified information management as a key skill for the family physician of the 21st century.12 In addition, the report advocates the integration of evidence-based clinical guidelines into a new model of care. The ABFM created the MC-FP process and the SAMs to facilitate these goals and plans to add at least 2 new SAMs per year. The process remains dynamic and the Board anticipates that feedback and suggestions provided by the Diplomates will continue to enhance these products for all users
Notes
Conflict of interest: none declared.
Received for publication September 2, 2005. Revision received April 7, 2006. Accepted for publication April 17, 2006.
References
Leigh T. Performance of Family Practice Diplomates on successive mandatory recertification examinations. Acad Med 1993; 68: 912–9.
Kohn L, Corrigan J, Donaldson M. To err is human: building a safer health system. Washington (DC): National Academy Press; 1999.
Statement on Commitment to Maintenance of Certification. 2000 [cited 4/5/05]. Available from: http://www.abms.org/downloads/publications/3-approved%20initiatives%20for%20MOC.pdf.
MC-FP Program Enhancements. 2006 [accessed 2006 March 28]; Description of the 10 year option for Maintenance of Certification]. Available from: https://www.theabfm.org/MOC/enhancements.aspx.
Analysis software for Word-based records. 2004, Centers for Disease Control and Prevention: Atlanta, GA. Available from: http://www.cdc.gov/hiv/software/answr.htm.
Committee on Quality of Health Care in America. Crossing the Quality Chasm. A New Health System for the 21st Century. Washington (DC): National Academy Press; 2001.
Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med 2005; 142: 260–73.
Norcini JJ, Kimball HR, Lipner RS. Certification and specialization: do they matter in the outcome of acute myocardial infarction? Acad Med 2000; 75: 1193–8.
Norcini JJ, Lipner RS. The relationship between the nature of practice and performance on a cognitive examination. Acad Med 2000; 75(10 Suppl): S68–70.
Norton PG, Dunn EV, Soberman L. What factors affect quality of care? Using the Peer Assessment Program in Ontario family practices. Can Fam Physician 1997; 43: 1739–44.
Norcini JJ, Lipner RS, Kimball HR. Certifying examination performance and patient outcomes following acute myocardial infarction. Med Educ 2002; 36: 853–9.
Committee, F.o.F.M.P.L., The future of family medicine: a collaborative project of the family medicine community. Ann Fam Med 2004; 2 Suppl 1: S3–32.(Michael D. Hagen, MD, Dou)