A "how to" guide for developing a publishable Scholarship of Teaching project
http://www.100md.com
《生理教育》
Abstract
In order for the efforts of scholarly teaching to be recognized, the work must become public and presented to peers for review. Scholarship of Teaching is not only improving instruction and learning but also methodically assessing whether specific teaching interventions have had the desired effect. In this paper, the author presents a step-by-step guide for how to develop a Scholarship of Teaching project that is well thought out and worthy of publication. Factors to consider before performing such pedagogical research include developing a clear research hypothesis for the classroom, reviewing the background literature, obtaining Institutional Review Board (Human Subjects) approval, and determining which methods of assessment may be used. This "how to" guide discusses how to handle all of these factors and prepare the data for publication and introduces the reader to references related to the Scholarship of Teaching and learning as well as educational research and theory.
Key words: learning assessment; educational research; classroom research; science education
Introduction
IN 1990, Boyer (2) noted that teaching must be assessed in order for the Scholarship of Teaching to be viewed equal to research. Thus, in order for curricular innovations and examinations into teaching to receive the recognition they deserve, these endeavors should be presented publicly and/or published in peer-reviewed journals. Glassick et al. (8) took Boyer's discussion a step further by emphasizing that documented Scholarship of Teaching should have the following standards: 1) clear goals, 2) adequate preparation, 3) appropriate methods, 4) significant results, 5) effective presentation, and 6) reflective critique. Shulman (23) coined the phrase "teaching as community property" and used this phrase as a call for educators to make their efforts visible and share their findings both within and across the disciplines.
An excellent teacher is not a scholar of teaching unless he/she participates in classroom assessment and research and makes his/her teaching public in the form of oral or written presentation (9, 12). Yet a scholar of teaching can be an individual in any disciplinary field, not just the field of education. A scholar of teaching is well informed in both the content knowledge of his/her discipline as well as the cross-disciplinary pedagogical knowledge that allows him/her to effectively teach in the discipline (24, 20).
In the National Research Council's text Scientific Research in Education (17), the authors noted that there are six principles of inquiry that should be followed in any educational research:
Pose significant questions that can be investigated empirically.
Link research to relevant theory.
Use methods that permit direct investigation of the question.
Provide a coherent and explicit chain of reasoning.
Replicate and generalize across studies.
Disclose research to encourage professional scrutiny and critique.
In other words, educational research should be held accountable to the same levels of rigor and replication that traditional scientific studies must follow.
While many of us are comfortable in publishing in our specific research areas, few may be comfortable publishing in the realm of Scholarship of Teaching. How does one become familiar with the background literature? Where would one present educational research findings? What journals are appropriate for such articles? The "how to" guide presented here is meant to be a step-by-step instruction manual to help direct you in the inception, implementation, completion, and publication of a Scholarship of Teaching/classroom research project. Please note that there are many ways to perform educational research and that the guide presented here is merely one means to an end.
The How to Guide for Developing a Publishable Scholarship of Teaching Project
Step 1: develop a clear research question or hypothesis.
Hutchins and Shulman (9) emphasized that Scholarship of Teaching requires the faculty (educational researcher or classroom researcher) to frame and investigate questions related to student teaching. On a related note, Ebert-May et al. (5) stated that scientists should approach pedagogy as they approach traditional science research. Thus the first step in any Scholarship of Teaching project, which is the same step in a traditional scientific research project, is to develop a clear research question or hypothesis.
While most scientists are familiar with how to form such a research question in his/her own scientific discipline, developing an educational research hypothesis sometimes is more difficult. In general, a clear research question for a Scholarship of Teaching project should have the following characteristics. First, it should concisely and clearly describe the setting for the classroom research (how large or small is the class, is it an undergraduate or graduate class, etc.). Second, one should avoid using ambiguous and vague terms to describe the hypothesis. Finally, the educational researcher should define realistic and achievable objectives that can be examined using appropriate methods of assessment. Compare the following "vague" research question to the more "clearly defined" question below:
Vague research question: What is the optimum number of homework assignments to give to an introductory level anatomy and physiology class?
Clearly defined research question: Do students in Anatomy A200 who complete weekly anatomy identification homework assignments achieve higher final exam scores than Anatomy A200 sections whose students do NOT have homework assignments?
The vague research question uses ambiguous terms and presents a question that cannot be answered well with any standard research methods. Another weakness is that the question does not provide any information about the criteria for judging "optimum:" would this be final grades, student satisfaction in the class, or long-term retention of the anatomy material? In contrast, the more clearly defined research question is better scripted and indicates the research will be used with specific materials and methods. There will be an "experimental" group (e.g., the A200 class sections that have the weekly homework assignments), and they will be compared with the "control" group (the A200 class sections that do not have assigned homework). The method of comparison is the score on the final exam.
Other examples of more clearly defined research questions are cited in O'Loughlin (18) and listed here:
Will the use of interactive learning activities in Anatomy A215 lecture increase student engagement in lecture?
Will the use of interactive learning activities improve class performance measures, such as mean lecture exam performances and final grades?
These research questions state the teaching intervention, which is the use of the interactive learning activities. The assessment criteria include student engagement, mean lecture exam performances, and final grades. Although not explicitly stated in the research question, the experimental classes were those that utilized the interactive learning activities and the control classes were previous classes that did not utilize the activities.
Some suggestions about how you can frame your own research questions for a specific course you teach, or even just a particular topic you teach in that course, include:
Is there a particularly effective method you've used to increase student learning, and can you document its effectiveness?
What more would you like to know about the effects of your course design and teaching on student learning?
Are there challenges you have encountered, such as poor classroom attendance, classroom incivility, or poor class performance, that you would like to improve?
While educational research and laboratory bench-based research should follow the same basic methods and protocols, educational research is inherently more "messy." Unlike laboratory bench-based research, the variables (i.e., the students, classroom, etc.) cannot be "controlled" as in a laboratory setting. Unlike cells in a petri dish, students are affected by many factors outside of the classroom. Does this mean that educational research should be abandoned? Of course not! What this does mean is that the methods of assessment need to be as rigorous as possible, and the researcher must acknowledge that there are some factors over which he/she has no control. Conversely, the educational researcher is at an advantage in that he/she is able to question the students about the perceived effectiveness of a teaching intervention.
Step 2: review thebackground literature.
Review of the literature is essential in classroom research, just as a review of background literature is essential in traditional scientific research. This literature review allows the educational researcher to become more knowledgeable on the research topic, see if someone has done similar research before, and what their results were. In addition, one can potentially develop contacts and perhaps collaborators about larger research issues.
Most researchers feel comfortable with the background literature in their specific research field but feel lost with Scholarship of Teaching literature. Here are some suggestions to get started. First, see the list of references in this article for some notable Scholarship of Teaching texts that should be a part of any Scholar of Teaching personal library. For example, Angelo and Cross (1) offer a variety of formative assessment strategies, called Classroom Assessment Techniques (CATs). They provide clear instructions as to how to modify each CAT for each classroom, the time involved in implementing a CAT, and the time involved in examining the data collected. Cross and Steadman (3) take CATs a step further and discuss how to implement them in classroom research. A basic understanding of the science of learning and multiple examples of effective learning environments are given in the text How People Learn: Brain, Mind, Experience and School (15). This text provides an excellent overview of both discipline-specific and interdisciplinary studies in how individuals learn and become experts in the field.
There are many excellent references that specifically address educational research in the sciences. The Handbook of Research on Science Teaching and Learning (7) is a collection of articles written by science educators on topics such as the uses of technology in science education, the affective dimension of science learning, and how laboratory instruction complements learning in the sciences. Each paper comes with an extensive citation list. The National Research Council's Scientific Research in Education (17) describes features essential in education research, how to develop clear and testable research questions, and what methods should be used to explore such questions. A followup to this work, Advancing Scientific Research in Education (14), goes into greater detail about what is meant by "rigor" in educational research and how educational research may be used for professional development. In addition, an extensive bibliography about Students' and Teachers' Conceptions and Science Education (4) may be found online.
Another helpful assistant for the background literature search is the university librarian. Meet with a librarian at your institution and ask for their guidance in conducting a literature search. They will be more than happy to assist you and can provide you with the tips to perform effective literature searches. They also can help you use electronic databases, such as the Educational Resources Information Center (ERIC) or Education Abstracts Full Text, which abstracts and indexes materials in education. Some of these databases will assist you in the appropriate selection of key words to use. For example,the current on-line version of the ERIC database has a tab for its Thesaurus, which includes a list of education-related terms that may be used for key words.
When you find some articles that are applicable to your educational/classroom research question, examine their reference lists and review the articles they cited. Note that more and more articles can now be found online, so you may be able to do most of your literature search at your own desk. Once you have a better understanding and knowledge of the prior research, you may move on to step 3.
Step 3: determine what methods of assessment will be used.
Ebert-May et al. (5) emphasized that "science faculty who want to improve instructional strategies need to design appropriate methods for assessing and analyzing classroom data to determine the effectiveness of their approaches to learning." Classroom research should 1) use methods appropriate to the research goals or questions and 2) effectively apply those methods (8). For example, if a classroom researcher wanted to examine how students' critical thinking evolved in the class, it would be inappropriate to use multiple-choice exam scores as the method of assessment. Most multiple-choice exam questions test for content or application knowledge and rarely test an individual's critical thinking skills. A more appropriate measure of critical thinking may involve student portfolios where they document their understanding of a concept over the semester or perhaps the use of concept mapping (1, 3).
Assessment measures fall into one of three categories: 1) affective measures, 2) process measures, or 3) performance measures. Affective measures are measures that relate to emotions or feelings. These include Likert scores and comments on instructor evaluations, a midsemester evaluation of a course, or a survey of how students liked an activity. Process measures examine to what extent a teaching intervention was used. In O'Loughlin (18), some examples of process measures included the percentage of the class that participated in an interactive learning activity and the number of web "hits" the instructional webpage received. Both types of information gave insight as to how many students were accessing and using the teaching intervention in question. Finally, performance measures document a course's, students', or faculty's performance. Performance measures include comparison of item analyses of exam questions, mean lecture exam performances, comparison of writing samples throughout the semester, withdrawal rates, and evaluation of student portfolios or journals. For example, Michael et al. (13) examined undergraduate students' misconceptions about respiratory physiology by administering questionnaires at the beginning and end of the semester and comparing frequencies of correct answers.
Note that each of the above measures can examine quantitative items (like exam scores and percentage of the class participating) and/or qualitative items (like analysis of a student's writing, reflection of a student's perceptions of the material, etc.). Both qualitative and quantitative data provide depth and richness to a Scholarship of Teaching project, as long as both types of data are used effectively and appropriately. There are multiple references that give examples of how to develop precise assessments that utilize both quantitative and qualitative data (16, 19).
As you prepare a new Scholarship of Teaching project, it is beneficial first to list the methods of assessment of student performance that you have used in the past (e.g., exams, essays, and group projects). Are there assessments you have used consistently over the years that could be used to compare student performance among semesters? Have you kept records of the grades for these assessments and/or kept the actual projects themselves? Can you think of new assessments you could start using to answer your classroom research questions? You may already have some comparative educational data you need for your new project, just waiting for you in your file cabinet.
Step 4: obtain Human Subjects approval from your Institutional Review Board before startingyour data collection.
The Federal Policy for the Protection of Human Subjects, known as the Common Rule and published in 1991, represents the latest Federal regulations for protection of human subjects (25). Educational research projects typically involve human subjects (students) and so such projects must be reviewed by a Human Subjects Committee or Institutional Review Board (IRB). Your institution's IRB is probably located with other campus administrative offices, such as the Dean of Faculties or Sponsored Research Services.
Most universities have their own subregulations to follow with respect to Human Subjects and the IRB. Thus IRB procedures at different institutions may vary, and it is imperative you follow the guidelines of your own institution. As an example, at Indiana University, there are three types of reviews: 1) exempt research reviews (the research is considered "exempt" from a full committee review), 2) expedited reviews (if there is only minimal risk to the participants), and 3) full committee reviews (typically performed if there is greater than minimal risk and if other criteria are met).
A common misconception of educational researchers is that they mistakenly believe all educational research is considered exempt, and so they would not have to obtain IRB approval. All educational research is not automatically considered exempt. If a student is a minor, can be identified in any way, or if a student cannot "opt out" of a particular activity, then the research is not considered exempt under most IRB guidelines (specific guidelines from institution to institution may vary) (11). For example, if you give extra credit for students that participate in your educational study, then those that choose not to participate in the study are penalized (by not receiving the extra credit), and so an IRB may consider this a form of coercion for participation.
Information about what constitutes exempt research at Indiana University is detailed elsewhere (10). Again, keep in mind that Indiana University's criteria for exempt research may vary from that of your own institution. In general, the exempt guidelines state that the research should be conducted in commonly accepted educational settings, involving normal educational practices, and that the information is collected in such a way that the human subject cannot be identified. For example, means and standard deviations of grades from previous classes possibly may be used, as long as the individuals cannot be identified from the data. Note that educational data that could identify the individual (i.e., names and identification numbers on grades and papers) is not considered exempt according to these guidelines. Approval for an expedited or full committee Human Subjects review may be needed or the educational data may need to have personal identifiers stripped. Your institution's IRB will notify you as to what avenue to choose.
Another misconception is that educational researchers think they don't have to contact their IRB if they feel their educational research falls under exempt guidelines. It is up to the IRB, and not the researcher, to determine what constitutes exempt research.
The educational researcher should always check with the IRB first. Some IRBs want to have documentation about all exempt research. If your research does fall under exempt guidelines, usually all you have to do is fill out an exempt form with your IRB and keep it on file.
Does this mean that any time you do something innovative in the classroom, you have to contact IRB? Not necessarily. As a general rule, if you are planning on presenting or publishing your educational research, you must contact your IRB. If you are doing something innovative merely to improve the class, and have no desire (now or in the future) to present your results, then you do not have to contact your IRB. If you are unsure if you will present this data in the future, then contact the IRB. Again, remember that specific guidelines at each institution vary, so it is best to check with your institution's IRB to learn the specifics.
It is good practice to always notify your students that you are conducting classroom research, regardless of whether or not the study is considered exempt by IRB. This may be verbal notification, written notification in the form of a study information sheet, or an informed consent sheet. Note that the study information sheet is not a consent form; it merely notifies the students of the study. Depending on the study and the IRB, you may or may not need to get written consent from your students.
Further information about how Indiana University examines student research and how it affects our Scholarship of Teaching and Learning initiative may be found on the Scholarship of Teaching and Learning website (11a).
Step 5: perform your research and collect your classroom data.
Now you are ready to perform your educational/classroom research. Keep in mind you may have to modify your practices as the semester progresses, as you have a clearer idea of what data and assessment methods can best answer your research hypotheses.
As any good researcher knows, you must have data to back up (or refute) your research hypotheses. In classroom research, what constitutes "data?" Data may consist of exam scores, student surveys, the percentage of students participating in an activity, assignments of any kind, and/or instructor evaluations. Review step 3 (determining methods of assessment) in this guide, and you see that assessment methods can provide you with a variety of useful data. As a general rule, save everything you can that could be used to assess your classroom research. Remember that assessment data from previous years' classes possibly may be used for your control or comparative data as well, as long as the data cannot identify a specific individual and falls under the guidelines listed in IRB requirements.
Step 6:analyze your data.
If possible, it is best to analyze educational data after the semester has ended and grades have been recorded. Thus both students and the IRB will know that assignment of grades was not influenced by students' participation (or lack thereof) in the educational study.
Because educational/classroom research data may be different than traditional scientific research data, some researchers are unsure as to how to analyze their data. Quantitative data may be analyzed a variety of ways (1, 16, 19). One could compare means and standard deviations of exam grades over a semester or semesters to determine if a teaching intervention affected exam score outcome. One could calculate percentages of students participating, students withdrawing from the class, or examine overall grades. Measurements such as means, standard deviations, and frequencies are easy to calculate and can provide a good overview of class performance. The limitation of such measurements is that they cannot provide detailed information about why a class may have done well or poorly or what other factors may have had a role in class performance. Analysis of variance tests could be performed on data (such as exam scores) to determine if there are statistically significant differences among groups. Note, however, that lack of statistical significance does not necessarily mean lack of an effect on the class intervention. Rather, it merely means any differences did not meet a standardized level of statistical significance. More advanced statistics, such as regression analysis, can examine the relationship of multiple variables (such as a class intervention, students grade point average, number of prerequisites taken, etc.) and describe the relation among the variables in the form of an equation. A regression line is calculated, and the null hypothesis is rejected if the value of the slope is not statistically different from zero. Regression statistics may be more difficult for the novice statistician to use and results may not be as easily understood, but regression data possibly may yield more rich data than other simpler statistical methods.
Qualitative data can be a bit more challenging to analyze. Some ways that qualitative data may be examined are as follows:
Compare survey comments from semester to semester: did impressions change?
Did the complexity of student writing increase over the semester?
Were there greater frequencies of higher-order learning questions during the semester where you performed a teaching intervention?
If you are not sure how to analyze your specific data, contact an individual affiliated with your instructional support services office, Education department faculty, or another colleague. In addition, refer to the references listed in this article regarding assessment and data analysis (16, 19). As Shulman (22) and Eisenhart and DeHaan (6) noted, individuals in educational research must become comfortable collaborating with individuals from other disciplines, because it takes experts from many disciplines to develop and analyze a thoughtful educational research project.
Step 7: present your findings to your colleagues.
As mentioned earlier, a scholar of teaching not only performs educational/classroom research but also makes that research public (2, 8), either in the form of an oral presentation, a workshop, and/or a publication. You may want to orally present your findings before you prepare your classroom research article, so you can receive preliminary feedback about your study.
Where do you present? You may want to consider your own departmental colloquium series first, as a "minimal risk" way of receiving preliminary feedback. Anatomy and physiology organizations have expanded their educational research components of their annual meetings. For example, the Human Anatomy and Physiology Society (http://www.hapsweb.org), American Physiological Society (http://www.the-aps.org), American Association of Anatomists (http://www.anatomy.org), and Federation of American Societies for Experimental Biology (Experimental Biology meeting) (http://www.faseb.org/meetings/) all have significant educational research and teaching components to their annual meetings. In addition to presenting your own data, you also can hear about Scholarship of Teaching research that your anatomy and physiology colleagues are performing throughout the world.
There also are several organizations that have meetings examining the Scholarship of Teaching and educational research across all disciplines. For example, you may want to consider presenting at the American Association of Higher Education (http://www.aahe.org) or the Lilly Conference on College and University Teaching (http://www.units.muohio.edu/lillycon/). There also are regional Lilly Conferences (see http://www.iats.com/index.html for a listing) that may be close to home.
Step 9: prepare the Scholarship of Teaching article.
After presenting your findings at a meeting and receiving some feedback, you now are ready to prepare the Scholarship of Teaching article. Before you begin, you should determine which journal(s) would be suitable for your article. Refer to Table 1 and examine the table of contents of the journals listed. Is there a particular journal whose studies best "fit" with yours? For example, if you've done an educational study about misconceptions among physiology undergraduates in a physiology lab, you do not want to publish in a journal that primarily focuses on elementary and secondary school education. A journal that examines undergraduate and even postgraduate studies would be more appropriate. Once a journal is selected, review the guidelines for author submission. Nothing is more disheartening than spending time preparing an article and then discovering the format and citation style is completely wrong.
As you prepare your article, rereview your background literature. Examine how others developed their research papers; this will help you determine the framework for your own. Make sure you cite relevant background literature (of which you are already familiar, because you completed step 2 in this guide). Incorporate the feedback you received when you presented your findings at meetings.
As with any traditional research paper, you should give yourself a deadline to finish the paper and stick to the deadline. The educational research paper is a scholarly work and should be completed in a timely fashion, just like any other scholarly work. Before you submit, ask a colleague (or an instructional consultant) to review and critique your paper. This activity may help save you some revision time later in the process.
Be aware that some educational research journals publish on a much slower time schedule than traditional scientific research publications. It may take months or even a year or longer before the paper is accepted for publication. Also be prepared to perform revisions on the paper after it is sent out for peer review. This process is similar to that for traditional scientific research journals.
Step 10: repeat steps 1–9 for other innovative teaching ideas you have.
"Classroom research is ongoing and cumulative intellectual inquiry by classroom teachers into the nature of teaching and learning in their own classrooms. Inquiry into a question about how students learn typically leads to new questions and thus to continual investigation through classroom research." (3)
Your original educational/classroom research will lead you to explore other educational research questions and teaching innovations. Continue to explore these issues and use this guide to help you with the steps. Continue to present your research and make your findings public, as you become a scholar of teaching.
Acknowledgments
A workshop related to this "how to" guide was first given at the 16th Annual Human Anatomy and Physiology Society conference. The author thanks the Indiana University Scholarship of Teaching and Learning initiative for providing guidance in developing her own Scholarship of Teaching projects, which also led to the development of this paper.
Received for publication May 16, 2005. Accepted for publication December 22, 2005.
REFERENCES
Angelo TA and Cross KP. Classroom Assessment Techniques: a Handbook for College Teachers (2nd ed.). San Francisco, CA: Jossey-Bass, 1993.
Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching, 1990.
Cross KP and Steadman MI. Classroom Research: Implementing the Scholarship of Teaching. San Francisco, CA: Jossey-Bass, 1996.
Duit R. Bibliography: Students' and Teachers' Conceptions and Science Education (Online). http://www.ipn.uni-kiel.de/aktuell/stcse/stcse.html
Ebert-May D, Batzli J, and Lim H. Disciplinary research strategies for assessment of learning. Bioscience 53: 1221–1228, 2003.
Eisenhart M and DeHaan R. Doctoral preparation of scientifically based education researchers. Educational Researcher 34: 3–13, 2005.
Gabel DL (editor). Handbook of Research on Science Teaching and Learning. New York: Simon and Shuster Macmillian, 1993.
Glassick CE, Huber MT, and Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey-Bass, 1997.
Hutchins P and Shulman L. The scholarship of teaching: new elaborations, new developments. Change: 11–15, 1999.
Indiana University. Research at Indiana University. Exempt Research Checklist
Indiana University. Research at Indiana University. Students as Subjects
Indiana University. Scholarship of Teaching & Learning at Indiana University Bloomington. Resources Regarding the Protection of Human Subjects in SOTL Research
Kreber C. Conceptualizing the scholarship of teaching and identifying unresolved issues: the framework for this volume. In: Scholarship Revisited: Perspectives on the Scholarship of Teaching. New Directions for Teaching and Learning, edited by Kreber C. San Francisco, CA: Jossey-Bass, 2001, no. 86, p. 1–18.
Michael J, Richardson D, Rovick A, Modell H, Bruce D, Horwitz B, Hudson M, Silverthorn D, Whitescarver S, and Williams S. Undergraduate students' misconceptions about respiratory physiology. Adv Physiol Educ 22: 127–135, 1999.
National Research Council. Advancing Scientific Research in Education. Washington, DC: National Academy Press, 2005.
National Research Council. How People Learn: Brain, Mind, Experience and School. Washington, DC: National Academy Press, 2000.
National Research Council. Knowing What Students Know: the Science and Design of Educational Assessment. Washington, DC: National Academy Press, 2003.
National Research Council. Scientific Research in Education. Washington, DC: National Academy Press, 2002.
O'Loughlin VD. Assessing the effects of using interactive learning activities in a large science lecture class. J Excel Coll Teaching 13: 29–42, 2002.
Palomba CA and Banta TW. Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. San Francisco, CA: Jossey-Bass, 1999.
Paulsen MB. The relation between research and the scholarship of teaching. In: Scholarship Revisited: Perspectives on the Scholarship of Teaching. New Directions for Teaching and Learning, edited by Kreber C. San Francisco, CA: Jossey-Bass, no. 86, 2001, p. 19–29.
Shulman L. Professing educational scholarship. In: Issues in Education Research: Problems and Possibilities, edited by Lagemann E and Shulman L. San Francisco, CA: Jossey-Bass, 1999, p. 159–165.
Shulman L. Teaching as community property: putting an end to pedagogical solitude. In: Teaching as Community Property: Essays on Higher Education, edited by Shulman L. San Francisco, CA: Jossey-Bass, 2004, p.139–144.
Shulman L. Those who understand: knowledge growth in teaching. Educational Researcher 15: 4–14, 1986.
United States Department of Health and Human Services. Code of Federal Regulations. Title 45. Public Welfare. Department of Health and Human Services. Part 46. Protection of Human Subjects(Valerie Dean O'Loughlin)
In order for the efforts of scholarly teaching to be recognized, the work must become public and presented to peers for review. Scholarship of Teaching is not only improving instruction and learning but also methodically assessing whether specific teaching interventions have had the desired effect. In this paper, the author presents a step-by-step guide for how to develop a Scholarship of Teaching project that is well thought out and worthy of publication. Factors to consider before performing such pedagogical research include developing a clear research hypothesis for the classroom, reviewing the background literature, obtaining Institutional Review Board (Human Subjects) approval, and determining which methods of assessment may be used. This "how to" guide discusses how to handle all of these factors and prepare the data for publication and introduces the reader to references related to the Scholarship of Teaching and learning as well as educational research and theory.
Key words: learning assessment; educational research; classroom research; science education
Introduction
IN 1990, Boyer (2) noted that teaching must be assessed in order for the Scholarship of Teaching to be viewed equal to research. Thus, in order for curricular innovations and examinations into teaching to receive the recognition they deserve, these endeavors should be presented publicly and/or published in peer-reviewed journals. Glassick et al. (8) took Boyer's discussion a step further by emphasizing that documented Scholarship of Teaching should have the following standards: 1) clear goals, 2) adequate preparation, 3) appropriate methods, 4) significant results, 5) effective presentation, and 6) reflective critique. Shulman (23) coined the phrase "teaching as community property" and used this phrase as a call for educators to make their efforts visible and share their findings both within and across the disciplines.
An excellent teacher is not a scholar of teaching unless he/she participates in classroom assessment and research and makes his/her teaching public in the form of oral or written presentation (9, 12). Yet a scholar of teaching can be an individual in any disciplinary field, not just the field of education. A scholar of teaching is well informed in both the content knowledge of his/her discipline as well as the cross-disciplinary pedagogical knowledge that allows him/her to effectively teach in the discipline (24, 20).
In the National Research Council's text Scientific Research in Education (17), the authors noted that there are six principles of inquiry that should be followed in any educational research:
Pose significant questions that can be investigated empirically.
Link research to relevant theory.
Use methods that permit direct investigation of the question.
Provide a coherent and explicit chain of reasoning.
Replicate and generalize across studies.
Disclose research to encourage professional scrutiny and critique.
In other words, educational research should be held accountable to the same levels of rigor and replication that traditional scientific studies must follow.
While many of us are comfortable in publishing in our specific research areas, few may be comfortable publishing in the realm of Scholarship of Teaching. How does one become familiar with the background literature? Where would one present educational research findings? What journals are appropriate for such articles? The "how to" guide presented here is meant to be a step-by-step instruction manual to help direct you in the inception, implementation, completion, and publication of a Scholarship of Teaching/classroom research project. Please note that there are many ways to perform educational research and that the guide presented here is merely one means to an end.
The How to Guide for Developing a Publishable Scholarship of Teaching Project
Step 1: develop a clear research question or hypothesis.
Hutchins and Shulman (9) emphasized that Scholarship of Teaching requires the faculty (educational researcher or classroom researcher) to frame and investigate questions related to student teaching. On a related note, Ebert-May et al. (5) stated that scientists should approach pedagogy as they approach traditional science research. Thus the first step in any Scholarship of Teaching project, which is the same step in a traditional scientific research project, is to develop a clear research question or hypothesis.
While most scientists are familiar with how to form such a research question in his/her own scientific discipline, developing an educational research hypothesis sometimes is more difficult. In general, a clear research question for a Scholarship of Teaching project should have the following characteristics. First, it should concisely and clearly describe the setting for the classroom research (how large or small is the class, is it an undergraduate or graduate class, etc.). Second, one should avoid using ambiguous and vague terms to describe the hypothesis. Finally, the educational researcher should define realistic and achievable objectives that can be examined using appropriate methods of assessment. Compare the following "vague" research question to the more "clearly defined" question below:
Vague research question: What is the optimum number of homework assignments to give to an introductory level anatomy and physiology class?
Clearly defined research question: Do students in Anatomy A200 who complete weekly anatomy identification homework assignments achieve higher final exam scores than Anatomy A200 sections whose students do NOT have homework assignments?
The vague research question uses ambiguous terms and presents a question that cannot be answered well with any standard research methods. Another weakness is that the question does not provide any information about the criteria for judging "optimum:" would this be final grades, student satisfaction in the class, or long-term retention of the anatomy material? In contrast, the more clearly defined research question is better scripted and indicates the research will be used with specific materials and methods. There will be an "experimental" group (e.g., the A200 class sections that have the weekly homework assignments), and they will be compared with the "control" group (the A200 class sections that do not have assigned homework). The method of comparison is the score on the final exam.
Other examples of more clearly defined research questions are cited in O'Loughlin (18) and listed here:
Will the use of interactive learning activities in Anatomy A215 lecture increase student engagement in lecture?
Will the use of interactive learning activities improve class performance measures, such as mean lecture exam performances and final grades?
These research questions state the teaching intervention, which is the use of the interactive learning activities. The assessment criteria include student engagement, mean lecture exam performances, and final grades. Although not explicitly stated in the research question, the experimental classes were those that utilized the interactive learning activities and the control classes were previous classes that did not utilize the activities.
Some suggestions about how you can frame your own research questions for a specific course you teach, or even just a particular topic you teach in that course, include:
Is there a particularly effective method you've used to increase student learning, and can you document its effectiveness?
What more would you like to know about the effects of your course design and teaching on student learning?
Are there challenges you have encountered, such as poor classroom attendance, classroom incivility, or poor class performance, that you would like to improve?
While educational research and laboratory bench-based research should follow the same basic methods and protocols, educational research is inherently more "messy." Unlike laboratory bench-based research, the variables (i.e., the students, classroom, etc.) cannot be "controlled" as in a laboratory setting. Unlike cells in a petri dish, students are affected by many factors outside of the classroom. Does this mean that educational research should be abandoned? Of course not! What this does mean is that the methods of assessment need to be as rigorous as possible, and the researcher must acknowledge that there are some factors over which he/she has no control. Conversely, the educational researcher is at an advantage in that he/she is able to question the students about the perceived effectiveness of a teaching intervention.
Step 2: review thebackground literature.
Review of the literature is essential in classroom research, just as a review of background literature is essential in traditional scientific research. This literature review allows the educational researcher to become more knowledgeable on the research topic, see if someone has done similar research before, and what their results were. In addition, one can potentially develop contacts and perhaps collaborators about larger research issues.
Most researchers feel comfortable with the background literature in their specific research field but feel lost with Scholarship of Teaching literature. Here are some suggestions to get started. First, see the list of references in this article for some notable Scholarship of Teaching texts that should be a part of any Scholar of Teaching personal library. For example, Angelo and Cross (1) offer a variety of formative assessment strategies, called Classroom Assessment Techniques (CATs). They provide clear instructions as to how to modify each CAT for each classroom, the time involved in implementing a CAT, and the time involved in examining the data collected. Cross and Steadman (3) take CATs a step further and discuss how to implement them in classroom research. A basic understanding of the science of learning and multiple examples of effective learning environments are given in the text How People Learn: Brain, Mind, Experience and School (15). This text provides an excellent overview of both discipline-specific and interdisciplinary studies in how individuals learn and become experts in the field.
There are many excellent references that specifically address educational research in the sciences. The Handbook of Research on Science Teaching and Learning (7) is a collection of articles written by science educators on topics such as the uses of technology in science education, the affective dimension of science learning, and how laboratory instruction complements learning in the sciences. Each paper comes with an extensive citation list. The National Research Council's Scientific Research in Education (17) describes features essential in education research, how to develop clear and testable research questions, and what methods should be used to explore such questions. A followup to this work, Advancing Scientific Research in Education (14), goes into greater detail about what is meant by "rigor" in educational research and how educational research may be used for professional development. In addition, an extensive bibliography about Students' and Teachers' Conceptions and Science Education (4) may be found online.
Another helpful assistant for the background literature search is the university librarian. Meet with a librarian at your institution and ask for their guidance in conducting a literature search. They will be more than happy to assist you and can provide you with the tips to perform effective literature searches. They also can help you use electronic databases, such as the Educational Resources Information Center (ERIC) or Education Abstracts Full Text, which abstracts and indexes materials in education. Some of these databases will assist you in the appropriate selection of key words to use. For example,the current on-line version of the ERIC database has a tab for its Thesaurus, which includes a list of education-related terms that may be used for key words.
When you find some articles that are applicable to your educational/classroom research question, examine their reference lists and review the articles they cited. Note that more and more articles can now be found online, so you may be able to do most of your literature search at your own desk. Once you have a better understanding and knowledge of the prior research, you may move on to step 3.
Step 3: determine what methods of assessment will be used.
Ebert-May et al. (5) emphasized that "science faculty who want to improve instructional strategies need to design appropriate methods for assessing and analyzing classroom data to determine the effectiveness of their approaches to learning." Classroom research should 1) use methods appropriate to the research goals or questions and 2) effectively apply those methods (8). For example, if a classroom researcher wanted to examine how students' critical thinking evolved in the class, it would be inappropriate to use multiple-choice exam scores as the method of assessment. Most multiple-choice exam questions test for content or application knowledge and rarely test an individual's critical thinking skills. A more appropriate measure of critical thinking may involve student portfolios where they document their understanding of a concept over the semester or perhaps the use of concept mapping (1, 3).
Assessment measures fall into one of three categories: 1) affective measures, 2) process measures, or 3) performance measures. Affective measures are measures that relate to emotions or feelings. These include Likert scores and comments on instructor evaluations, a midsemester evaluation of a course, or a survey of how students liked an activity. Process measures examine to what extent a teaching intervention was used. In O'Loughlin (18), some examples of process measures included the percentage of the class that participated in an interactive learning activity and the number of web "hits" the instructional webpage received. Both types of information gave insight as to how many students were accessing and using the teaching intervention in question. Finally, performance measures document a course's, students', or faculty's performance. Performance measures include comparison of item analyses of exam questions, mean lecture exam performances, comparison of writing samples throughout the semester, withdrawal rates, and evaluation of student portfolios or journals. For example, Michael et al. (13) examined undergraduate students' misconceptions about respiratory physiology by administering questionnaires at the beginning and end of the semester and comparing frequencies of correct answers.
Note that each of the above measures can examine quantitative items (like exam scores and percentage of the class participating) and/or qualitative items (like analysis of a student's writing, reflection of a student's perceptions of the material, etc.). Both qualitative and quantitative data provide depth and richness to a Scholarship of Teaching project, as long as both types of data are used effectively and appropriately. There are multiple references that give examples of how to develop precise assessments that utilize both quantitative and qualitative data (16, 19).
As you prepare a new Scholarship of Teaching project, it is beneficial first to list the methods of assessment of student performance that you have used in the past (e.g., exams, essays, and group projects). Are there assessments you have used consistently over the years that could be used to compare student performance among semesters? Have you kept records of the grades for these assessments and/or kept the actual projects themselves? Can you think of new assessments you could start using to answer your classroom research questions? You may already have some comparative educational data you need for your new project, just waiting for you in your file cabinet.
Step 4: obtain Human Subjects approval from your Institutional Review Board before startingyour data collection.
The Federal Policy for the Protection of Human Subjects, known as the Common Rule and published in 1991, represents the latest Federal regulations for protection of human subjects (25). Educational research projects typically involve human subjects (students) and so such projects must be reviewed by a Human Subjects Committee or Institutional Review Board (IRB). Your institution's IRB is probably located with other campus administrative offices, such as the Dean of Faculties or Sponsored Research Services.
Most universities have their own subregulations to follow with respect to Human Subjects and the IRB. Thus IRB procedures at different institutions may vary, and it is imperative you follow the guidelines of your own institution. As an example, at Indiana University, there are three types of reviews: 1) exempt research reviews (the research is considered "exempt" from a full committee review), 2) expedited reviews (if there is only minimal risk to the participants), and 3) full committee reviews (typically performed if there is greater than minimal risk and if other criteria are met).
A common misconception of educational researchers is that they mistakenly believe all educational research is considered exempt, and so they would not have to obtain IRB approval. All educational research is not automatically considered exempt. If a student is a minor, can be identified in any way, or if a student cannot "opt out" of a particular activity, then the research is not considered exempt under most IRB guidelines (specific guidelines from institution to institution may vary) (11). For example, if you give extra credit for students that participate in your educational study, then those that choose not to participate in the study are penalized (by not receiving the extra credit), and so an IRB may consider this a form of coercion for participation.
Information about what constitutes exempt research at Indiana University is detailed elsewhere (10). Again, keep in mind that Indiana University's criteria for exempt research may vary from that of your own institution. In general, the exempt guidelines state that the research should be conducted in commonly accepted educational settings, involving normal educational practices, and that the information is collected in such a way that the human subject cannot be identified. For example, means and standard deviations of grades from previous classes possibly may be used, as long as the individuals cannot be identified from the data. Note that educational data that could identify the individual (i.e., names and identification numbers on grades and papers) is not considered exempt according to these guidelines. Approval for an expedited or full committee Human Subjects review may be needed or the educational data may need to have personal identifiers stripped. Your institution's IRB will notify you as to what avenue to choose.
Another misconception is that educational researchers think they don't have to contact their IRB if they feel their educational research falls under exempt guidelines. It is up to the IRB, and not the researcher, to determine what constitutes exempt research.
The educational researcher should always check with the IRB first. Some IRBs want to have documentation about all exempt research. If your research does fall under exempt guidelines, usually all you have to do is fill out an exempt form with your IRB and keep it on file.
Does this mean that any time you do something innovative in the classroom, you have to contact IRB? Not necessarily. As a general rule, if you are planning on presenting or publishing your educational research, you must contact your IRB. If you are doing something innovative merely to improve the class, and have no desire (now or in the future) to present your results, then you do not have to contact your IRB. If you are unsure if you will present this data in the future, then contact the IRB. Again, remember that specific guidelines at each institution vary, so it is best to check with your institution's IRB to learn the specifics.
It is good practice to always notify your students that you are conducting classroom research, regardless of whether or not the study is considered exempt by IRB. This may be verbal notification, written notification in the form of a study information sheet, or an informed consent sheet. Note that the study information sheet is not a consent form; it merely notifies the students of the study. Depending on the study and the IRB, you may or may not need to get written consent from your students.
Further information about how Indiana University examines student research and how it affects our Scholarship of Teaching and Learning initiative may be found on the Scholarship of Teaching and Learning website (11a).
Step 5: perform your research and collect your classroom data.
Now you are ready to perform your educational/classroom research. Keep in mind you may have to modify your practices as the semester progresses, as you have a clearer idea of what data and assessment methods can best answer your research hypotheses.
As any good researcher knows, you must have data to back up (or refute) your research hypotheses. In classroom research, what constitutes "data?" Data may consist of exam scores, student surveys, the percentage of students participating in an activity, assignments of any kind, and/or instructor evaluations. Review step 3 (determining methods of assessment) in this guide, and you see that assessment methods can provide you with a variety of useful data. As a general rule, save everything you can that could be used to assess your classroom research. Remember that assessment data from previous years' classes possibly may be used for your control or comparative data as well, as long as the data cannot identify a specific individual and falls under the guidelines listed in IRB requirements.
Step 6:analyze your data.
If possible, it is best to analyze educational data after the semester has ended and grades have been recorded. Thus both students and the IRB will know that assignment of grades was not influenced by students' participation (or lack thereof) in the educational study.
Because educational/classroom research data may be different than traditional scientific research data, some researchers are unsure as to how to analyze their data. Quantitative data may be analyzed a variety of ways (1, 16, 19). One could compare means and standard deviations of exam grades over a semester or semesters to determine if a teaching intervention affected exam score outcome. One could calculate percentages of students participating, students withdrawing from the class, or examine overall grades. Measurements such as means, standard deviations, and frequencies are easy to calculate and can provide a good overview of class performance. The limitation of such measurements is that they cannot provide detailed information about why a class may have done well or poorly or what other factors may have had a role in class performance. Analysis of variance tests could be performed on data (such as exam scores) to determine if there are statistically significant differences among groups. Note, however, that lack of statistical significance does not necessarily mean lack of an effect on the class intervention. Rather, it merely means any differences did not meet a standardized level of statistical significance. More advanced statistics, such as regression analysis, can examine the relationship of multiple variables (such as a class intervention, students grade point average, number of prerequisites taken, etc.) and describe the relation among the variables in the form of an equation. A regression line is calculated, and the null hypothesis is rejected if the value of the slope is not statistically different from zero. Regression statistics may be more difficult for the novice statistician to use and results may not be as easily understood, but regression data possibly may yield more rich data than other simpler statistical methods.
Qualitative data can be a bit more challenging to analyze. Some ways that qualitative data may be examined are as follows:
Compare survey comments from semester to semester: did impressions change?
Did the complexity of student writing increase over the semester?
Were there greater frequencies of higher-order learning questions during the semester where you performed a teaching intervention?
If you are not sure how to analyze your specific data, contact an individual affiliated with your instructional support services office, Education department faculty, or another colleague. In addition, refer to the references listed in this article regarding assessment and data analysis (16, 19). As Shulman (22) and Eisenhart and DeHaan (6) noted, individuals in educational research must become comfortable collaborating with individuals from other disciplines, because it takes experts from many disciplines to develop and analyze a thoughtful educational research project.
Step 7: present your findings to your colleagues.
As mentioned earlier, a scholar of teaching not only performs educational/classroom research but also makes that research public (2, 8), either in the form of an oral presentation, a workshop, and/or a publication. You may want to orally present your findings before you prepare your classroom research article, so you can receive preliminary feedback about your study.
Where do you present? You may want to consider your own departmental colloquium series first, as a "minimal risk" way of receiving preliminary feedback. Anatomy and physiology organizations have expanded their educational research components of their annual meetings. For example, the Human Anatomy and Physiology Society (http://www.hapsweb.org), American Physiological Society (http://www.the-aps.org), American Association of Anatomists (http://www.anatomy.org), and Federation of American Societies for Experimental Biology (Experimental Biology meeting) (http://www.faseb.org/meetings/) all have significant educational research and teaching components to their annual meetings. In addition to presenting your own data, you also can hear about Scholarship of Teaching research that your anatomy and physiology colleagues are performing throughout the world.
There also are several organizations that have meetings examining the Scholarship of Teaching and educational research across all disciplines. For example, you may want to consider presenting at the American Association of Higher Education (http://www.aahe.org) or the Lilly Conference on College and University Teaching (http://www.units.muohio.edu/lillycon/). There also are regional Lilly Conferences (see http://www.iats.com/index.html for a listing) that may be close to home.
Step 9: prepare the Scholarship of Teaching article.
After presenting your findings at a meeting and receiving some feedback, you now are ready to prepare the Scholarship of Teaching article. Before you begin, you should determine which journal(s) would be suitable for your article. Refer to Table 1 and examine the table of contents of the journals listed. Is there a particular journal whose studies best "fit" with yours? For example, if you've done an educational study about misconceptions among physiology undergraduates in a physiology lab, you do not want to publish in a journal that primarily focuses on elementary and secondary school education. A journal that examines undergraduate and even postgraduate studies would be more appropriate. Once a journal is selected, review the guidelines for author submission. Nothing is more disheartening than spending time preparing an article and then discovering the format and citation style is completely wrong.
As you prepare your article, rereview your background literature. Examine how others developed their research papers; this will help you determine the framework for your own. Make sure you cite relevant background literature (of which you are already familiar, because you completed step 2 in this guide). Incorporate the feedback you received when you presented your findings at meetings.
As with any traditional research paper, you should give yourself a deadline to finish the paper and stick to the deadline. The educational research paper is a scholarly work and should be completed in a timely fashion, just like any other scholarly work. Before you submit, ask a colleague (or an instructional consultant) to review and critique your paper. This activity may help save you some revision time later in the process.
Be aware that some educational research journals publish on a much slower time schedule than traditional scientific research publications. It may take months or even a year or longer before the paper is accepted for publication. Also be prepared to perform revisions on the paper after it is sent out for peer review. This process is similar to that for traditional scientific research journals.
Step 10: repeat steps 1–9 for other innovative teaching ideas you have.
"Classroom research is ongoing and cumulative intellectual inquiry by classroom teachers into the nature of teaching and learning in their own classrooms. Inquiry into a question about how students learn typically leads to new questions and thus to continual investigation through classroom research." (3)
Your original educational/classroom research will lead you to explore other educational research questions and teaching innovations. Continue to explore these issues and use this guide to help you with the steps. Continue to present your research and make your findings public, as you become a scholar of teaching.
Acknowledgments
A workshop related to this "how to" guide was first given at the 16th Annual Human Anatomy and Physiology Society conference. The author thanks the Indiana University Scholarship of Teaching and Learning initiative for providing guidance in developing her own Scholarship of Teaching projects, which also led to the development of this paper.
Received for publication May 16, 2005. Accepted for publication December 22, 2005.
REFERENCES
Angelo TA and Cross KP. Classroom Assessment Techniques: a Handbook for College Teachers (2nd ed.). San Francisco, CA: Jossey-Bass, 1993.
Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching, 1990.
Cross KP and Steadman MI. Classroom Research: Implementing the Scholarship of Teaching. San Francisco, CA: Jossey-Bass, 1996.
Duit R. Bibliography: Students' and Teachers' Conceptions and Science Education (Online). http://www.ipn.uni-kiel.de/aktuell/stcse/stcse.html
Ebert-May D, Batzli J, and Lim H. Disciplinary research strategies for assessment of learning. Bioscience 53: 1221–1228, 2003.
Eisenhart M and DeHaan R. Doctoral preparation of scientifically based education researchers. Educational Researcher 34: 3–13, 2005.
Gabel DL (editor). Handbook of Research on Science Teaching and Learning. New York: Simon and Shuster Macmillian, 1993.
Glassick CE, Huber MT, and Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. San Francisco, CA: Jossey-Bass, 1997.
Hutchins P and Shulman L. The scholarship of teaching: new elaborations, new developments. Change: 11–15, 1999.
Indiana University. Research at Indiana University. Exempt Research Checklist
Indiana University. Research at Indiana University. Students as Subjects
Indiana University. Scholarship of Teaching & Learning at Indiana University Bloomington. Resources Regarding the Protection of Human Subjects in SOTL Research
Kreber C. Conceptualizing the scholarship of teaching and identifying unresolved issues: the framework for this volume. In: Scholarship Revisited: Perspectives on the Scholarship of Teaching. New Directions for Teaching and Learning, edited by Kreber C. San Francisco, CA: Jossey-Bass, 2001, no. 86, p. 1–18.
Michael J, Richardson D, Rovick A, Modell H, Bruce D, Horwitz B, Hudson M, Silverthorn D, Whitescarver S, and Williams S. Undergraduate students' misconceptions about respiratory physiology. Adv Physiol Educ 22: 127–135, 1999.
National Research Council. Advancing Scientific Research in Education. Washington, DC: National Academy Press, 2005.
National Research Council. How People Learn: Brain, Mind, Experience and School. Washington, DC: National Academy Press, 2000.
National Research Council. Knowing What Students Know: the Science and Design of Educational Assessment. Washington, DC: National Academy Press, 2003.
National Research Council. Scientific Research in Education. Washington, DC: National Academy Press, 2002.
O'Loughlin VD. Assessing the effects of using interactive learning activities in a large science lecture class. J Excel Coll Teaching 13: 29–42, 2002.
Palomba CA and Banta TW. Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. San Francisco, CA: Jossey-Bass, 1999.
Paulsen MB. The relation between research and the scholarship of teaching. In: Scholarship Revisited: Perspectives on the Scholarship of Teaching. New Directions for Teaching and Learning, edited by Kreber C. San Francisco, CA: Jossey-Bass, no. 86, 2001, p. 19–29.
Shulman L. Professing educational scholarship. In: Issues in Education Research: Problems and Possibilities, edited by Lagemann E and Shulman L. San Francisco, CA: Jossey-Bass, 1999, p. 159–165.
Shulman L. Teaching as community property: putting an end to pedagogical solitude. In: Teaching as Community Property: Essays on Higher Education, edited by Shulman L. San Francisco, CA: Jossey-Bass, 2004, p.139–144.
Shulman L. Those who understand: knowledge growth in teaching. Educational Researcher 15: 4–14, 1986.
United States Department of Health and Human Services. Code of Federal Regulations. Title 45. Public Welfare. Department of Health and Human Services. Part 46. Protection of Human Subjects(Valerie Dean O'Loughlin)