Vitalizing the evaluation of curricular implementation: a framework for attending to the “how and whys” of curriculum evolution

As curricular reforms are implemented, there is often urgency among scholars to swiftly evaluate curricular outcomes and establish whether desired impacts have been realized. Consequently, many evaluative studies focus on summative program outcomes without accompanying evaluations of implementation. This runs the risk of Type III errors, whereby outcome evaluations rest on unverified assumptions about the appropriate implementation of prescribed curricular activities. Such errors challenge the usefulness of the evaluative studies, casting doubt on accumulated knowledge about curricular innovations, and posing problems for educational systems working to mobilize scarce resources. Unfortunately, however, there is long-standing inattention to the evaluation of implementation in health professions education (HPE). To address this, we propose an accessible framework that provides substantive guidance for evaluative research on implementation of curricular innovations. The Prescribed-Intended-Enacted-Sustainable (PIES) framework that is articulated in this paper, introduces new concepts to HPE—with a view to facilitating more nuanced examination of the evolution of curricula as they are implemented. Critically, the framework is theoretically grounded, integrating evaluation and implementation science as well as education theory. It outlines when, how, and why evaluators need to direct attention to curricular implementation, providing guidance on how programs can map out meaningful evaluative research agendas. Ultimately, this work is intended to support evaluators and educators, seeking to design evaluation studies that provide more faithful, useful representations of the intricacies of curricular change implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Buy Now

Price includes VAT (France)

Instant access to the full article PDF.

Rent this article via DeepDyve

Similar content being viewed by others

Exploring critical components of an integrated STEM curriculum: an application of the innovation implementation framework

Undergraduate medical education programme renewal: a longitudinal context, input, process and product evaluation study

Evaluation Paradigms

Explore related subjects

References

• Albarqouni, L., Glasziou, P., & Hoffmann, T. (2018). Completeness of the reporting of evidence-based practice educational interventions: A review. Medical Education,52(2), 161–170. https://doi.org/10.1111/medu.13410ArticleGoogle Scholar
• Alkin, M. C., & King, J. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation,38(3), 434–450. https://doi.org/10.1177/1098214017717015ArticleGoogle Scholar
• Baker, L., Shing, L. K., Wright, S., Mylopoulos, M., Kulasegaram, K., & Ng, S. (2019). Aligning and applying the paradigms and practices of education. Academic Medicine: Journal of the Association of American Medical Colleges,94(7), 1060–1060. ArticleGoogle Scholar
• Basch, C. E., Sliepcevich, E. M., Gold, R. S., Duncan, D. F., & Kolbe, L. J. (1985). Avoiding type III errors in health education program evaluations: A case study. Health Education Behaviour.,12(3), 315–331. https://doi.org/10.1177/109019818501200311ArticleGoogle Scholar
• Biggs, J. (1985). The role of metalearning in study processes. British Journal of Educational Psychology,55, 185–212. ArticleGoogle Scholar
• Bismack, A. S., Arias, A. M., Davis, E. A., & Palincsar, A. S. (2014). Connecting curriculum materials and teachers: Elementary science teachers’ enactment of a reform-based curricular unit. Journal of Science Teacher Education,25(4), 489–512. https://doi.org/10.1007/s10972-013-9372-xArticleGoogle Scholar
• Boyd, V. A., Whitehead, C. R., Thille, P., Ginsburg, S., Brydges, R., & Kuper, A. (2018). Competency-based medical education: The discourse of infallibility. Medical Education,52(1), 45–57. https://doi.org/10.1111/medu.13467ArticleGoogle Scholar
• Brown, C., & Zhang, D. (2017). Accounting for discrepancies in teachers’ attitudes towards evidence use and actual instances of evidence use in schools. Cambridge Journal of Education.,47(2), 277–295. https://doi.org/10.1080/0305764X.2016.1158784ArticleGoogle Scholar
• Brydges, R., Tran, J., Goffi, A., Lee, C., & Miller, D. M. M. (2020). Resident learning trajectories in the workplace: A self-regulated learning analysis. Medical Education. https://doi.org/10.1111/medu.14288ArticleGoogle Scholar
• Burkhauser, M. A., & Lesaux, N. K. (2017). Exercising a bounded autonomy: Novice and experienced teachers’ adaptations to curriculum materials in an age of accountability. Journal of Curriculum Studies,49(3), 291–312. https://doi.org/10.1080/00220272.2015.1088065ArticleGoogle Scholar
• Castro Superfine, A., Marshall, A. M., & Kelso, C. (2015). Fidelity of implementation: Bringing written curriculum materials into the equation. Curriculum Journal,26(1), 164–191. https://doi.org/10.1080/09585176.2014.990910ArticleGoogle Scholar
• Caverzagie, K. J., Nousiainen, M. T., Ferguson, P. C., et al. (2017). Overarching challenges to the implementation of competency-based medical education. Medical Teacher,39(6), 588–593. https://doi.org/10.1080/0142159X.2017.1315075ArticleGoogle Scholar
• Cianciolo, A. T., & Regehr, G. (2019). Learning theory and educational intervention: Producing meaningful evidence of impact through layered analysis. Academic Medicine,94(6), 789–794. https://doi.org/10.1097/ACM.0000000000002591ArticleGoogle Scholar
• Cofrancesco, J., Wright, S. M., Vohr, E., & Ziegelstein, R. C. (2017). Creating an “education shark tank” to encourage and support educational scholarship and innovation. Academic Medicine,92(11), 1578–1582. https://doi.org/10.1097/ACM.0000000000001715ArticleGoogle Scholar
• Cummings, A., Lund, S., Campling, N., May, C., Richardson, A., & Myall, M. (2017). Implementing communication and decision-making interventions directed at goals of care: a theory-led scoping review. BMJ Open, 7(10), https://doi.org/10.1136/bmjopen-2017-017056. ArticleGoogle Scholar
• Daneman, D., & Benatar, S. (2019). Dynamic tensions following new pedagogy in undergraduate medical education. Academic Medicine. https://doi.org/10.1097/acm.0000000000002795ArticleGoogle Scholar
• Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation. Results from a field experiment. Evaluation and Program Planning.,3(4), 269–276. https://doi.org/10.1016/0149-7189(80)90042-7ArticleGoogle Scholar
• Drake, C., & Sherin, M. G. (2006). Practicing change: Curriculum adaptation and teacher narrative in the context of mathematics education reform. Curriculum Inquiry,36(2), 153–187. https://doi.org/10.1111/j.1467-873X.2006.00351.xArticleGoogle Scholar
• Gehrke, N. J., Knapp, M. S., & Sirotnik, K. A. (1992). In search of the school curriculum. Review of Research in Education,18, 51–110. Google Scholar
• Glasgow, R. E., Harden, S. M., Gaglio, B., et al. (2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2019.00064ArticleGoogle Scholar
• Goldhaber, D., Holden, K. L., & Grout, C. (2019). Errors in administrative education data: A cautionary tale. Educational Researcher,48(3), 179–182. https://doi.org/10.3102/0013189X19837598ArticleGoogle Scholar
• Hafferty, F. W. (1998). Beyond curriculum reform: Confronting medicine’s hidden curriculum. Academic Medicine,73, 403–407. ArticleGoogle Scholar
• Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children,79(2), 181–193. https://doi.org/10.1177/001440291307900204ArticleGoogle Scholar
• Hawe, P., Shiell, A., & Riley, T. (2004). Complex interventions: How “out of control” can a randomised controlled trial be? BMJ,328(7455), 1561. https://doi.org/10.1136/bmj.328.7455.1561ArticleGoogle Scholar
• Hawick, L., Cleland, J., & Kitto, S. (2017). Getting off the carousel: Exploring the wicked problem of curriculum reform. Perspectives Medical Education,6(5), 337–343. https://doi.org/10.1007/s40037-017-0371-zArticleGoogle Scholar
• Hawick, L., Kitto, S., & Cleland, J. (2016). Curriculum reform: The more things change, the more they stay the same? Perspectives on Medical Education.,5(1), 5–7. https://doi.org/10.1007/s40037-016-0252-xArticleGoogle Scholar
• Hawkins, R. E., Welcher, C. M., Holmboe, E. S., et al. (2015). Implementation of competency-based medical education: Are we addressing the concerns and challenges? Medical Education,49(11), 1086–1102. https://doi.org/10.1111/medu.12831ArticleGoogle Scholar
• Hoffmann, T. C., Glasziou, P. P., Boutron, I., et al. (2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ,348(March), 1–12. https://doi.org/10.1136/bmj.g1687ArticleGoogle Scholar
• Holmboe, E. S., Sherbino, J., Englander, R., Snell, L., & Frank, J. R. (2017). A call to action: The controversy of and rationale for competency-based medical education. Medical Teacher,39(6), 574–581. https://doi.org/10.1080/0142159X.2017.1315067ArticleGoogle Scholar
• Horsley, T., & Regehr, G. (2018a). When are two interventions the same ? Implications for reporting guidelines in education. Medical Education,52, 141–143. https://doi.org/10.1111/medu.13496ArticleGoogle Scholar
• Horsley, T., & Regehr, G. (2018b). When are two interventions the same? Implications for reporting guidelines in education. Medical Education,52(2), 141–143. https://doi.org/10.1111/medu.13496ArticleGoogle Scholar
• Hudson, J. N., Farmer, E. A., Weston, K. M., & Bushnell, J. A. (2015). Using a framework to implement large-scale innovation in medical education with the intent of achieving sustainability. BMC Medical Education. https://doi.org/10.1186/s12909-014-0282-1ArticleGoogle Scholar
• Hulme, J. A., & Winstone, N. E. (2017). Do no harm Risk aversion versus risk management in the context of pedagogic frailty. Knowledge Management and E-Learning.,9(3), 261–274. https://doi.org/10.34105/j.kmel.2017.09.016ArticleGoogle Scholar
• Irby, D. M., & Wilkerson, L. A. (2003). Educational innovations in academic medicine and environmental trends. Journal of General Internal Medicine,18(5), 370–376. https://doi.org/10.1046/j.1525-1497.2003.21049.xArticleGoogle Scholar
• Jackson, N. (2004). Developing the concept of metalearning. Innovations in Education and Teaching International,41(4), 391–403. https://doi.org/10.1080/1470329042000276995ArticleGoogle Scholar
• Keiser, J. M., & Lambdin, D. V. (1996). The clock is ticking: Time constraint issues in mathematics teaching reform. Journal of Education Research,90(1), 23–31. https://doi.org/10.1080/00220671.1996.9944440ArticleGoogle Scholar
• Leonard, S. N., Fitzgerald, R. N., Riordan, G., et al. (2017). Using developmental evaluation as a design thinking tool for curriculum innovation in professional higher education curriculum innovation in professional higher education. Higher Education Research and Development,2016, 4360. https://doi.org/10.1080/07294360.2015.1087386ArticleGoogle Scholar
• Leong, Y. H., & Chick, H. L. (2011). Time pressure and instructional choices when teaching mathematics. Mathematics Education Research Journal,23(3), 347–362. https://doi.org/10.1007/s13394-011-0019-yArticleGoogle Scholar
• Lucas, U., & Meyer, J. H. F. (2004). Supporting student awareness: Understanding student preconceptions of their subject matter within introductory courses. Innovations in Education and Teaching International,41(4), 459–471. https://doi.org/10.1080/1470329042000277039ArticleGoogle Scholar
• Maltenfort, M. (2015). Type I, type II, and occasionally type III. Journal of Spinal Disorders and Techniques,28(5), 189. https://doi.org/10.1097/BSD.0000000000000285ArticleGoogle Scholar
• Mansour, N. (2009). Science teachers’ beliefs and practices: Issues, implications and research agenda. International Journal of Environmental and Science Education.,4(1), 25–48. Google Scholar
• Manzone, J., Regehr, G., Garbedian, S., & Brydges, R. (2019). Assigning medical students learning goals: Do they do it, and what happens when they don’t? Teaching and Learning in Medicine,31(5), 528–535. https://doi.org/10.1080/10401334.2019.1600520ArticleGoogle Scholar
• Martimianakis, M. A., & Hafferty, F. W. (2016). Exploring the interstitial space between the ideal and the practised: Humanism and the hidden curriculum of system reform. Medical Education,50(3), 278–280. https://doi.org/10.1111/medu.12982ArticleGoogle Scholar
• McLaughlin, M., & Mitra, D. (2001). Theory-based change and change-based theory: Going deeper going broader. Journal of Educational Change.,2(4), 301–323. https://doi.org/10.1023/A:1014616908334ArticleGoogle Scholar
• Meinema, J. G., Buwalda, N., van Etten-Jamaludin, F., Visser, M. R., & van Nynke, D. (2019). Intervention descriptions in medical education: What can be improved? Academic medicine a systematic review and checklist. Academic Medicine. https://doi.org/10.1097/ACM.0000000000002428ArticleGoogle Scholar
• Montague, S. (2019). Does your implementation fit your theory of change? Canadian Journal of Program Evaluation.,33(3), 316–335. https://doi.org/10.3138/cjpe.53008ArticleGoogle Scholar
• Nelson, J., & Campbell, C. (2017). Evidence-informed practice in education: Meanings and applications. Educational Researcher,59(2), 127–135. https://doi.org/10.1080/00131881.2017.1314115ArticleGoogle Scholar
• Nousiainen, M. T., Caverzagie, K. J., Ferguson, P. C., & Frank, J. R. (2017). Implementing competency-based medical education: What changes in curricular structure and processes are needed? Medical Teacher,39(6), 594–598. https://doi.org/10.1080/0142159X.2017.1315077ArticleGoogle Scholar
• O’Sullivan, P. S. (2019). What questions guide investing in our faculty? Academic Medicine,94, S11–S13. https://doi.org/10.1097/ACM.0000000000002910ArticleGoogle Scholar
• Onyura, B. (2020). Useful to whom? Evaluation utilization theory and boundaries for program evaluation scope. Medical Education. https://doi.org/10.1111/medu.14281ArticleGoogle Scholar
• Onyura, B., Baker, L., Cameron, B., Friesen, F., & Leslie, K. (2016). Evidence for curricular and instructional design approaches in undergraduate medical education: An umbrella review. Medical Teacher,38(2), 150–161. https://doi.org/10.3109/0142159X.2015.1009019ArticleGoogle Scholar
• Patterson, F., & Zibarras, L. D. (2017). Selecting for creativity and innovation potential: Implications for practice in healthcare education. Adv Heal Sci Educ.,22(2), 417–428. https://doi.org/10.1007/s10459-016-9731-4ArticleGoogle Scholar
• Patton, M. Q. (1997). Utilization-Focused Evaluation: The New Century Text (3rd ed.). Thousand Oaks, CA: Sage. Google Scholar
• Petty, E., & Golden, R. N. (2017). Embracing innovation in medical education. Wisconsin Medical Journal,116(3), 179–180. Google Scholar
• Phillips, A. C., Lewis, L. K., McEvoy, M. P., et al. (2016). Development and validation of the guideline for reporting evidence-based practice educational interventions and teaching (GREET). BMC Medical Education,16(1), 1–10. https://doi.org/10.1186/s12909-016-0759-1ArticleGoogle Scholar
• Pintó, R. (2005). Introducing curriculum innovations in science: Identifying teachers’ transformations and the design of related teacher education. Science & Education,89(1), 1–12. https://doi.org/10.1002/sce.20039ArticleGoogle Scholar
• Pleger, L., & Sager, F. (2016). Betterment, undermining, support and distortion : A heuristic model for the analysis of pressure on evaluators. Evaluation and Program Planning,69, 166–172. ArticleGoogle Scholar
• Proctor, E., Silmere, H., Raghavan, R., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7. ArticleGoogle Scholar
• Remillard, J. T. (2005). Examining key concepts in research on teachers’ use of mathematics curricula. Review of Educational Research,75(2), 211–246. https://doi.org/10.3102/00346543075002211ArticleGoogle Scholar
• Scanlon JW, Horst P, Nay JN, Schmidt RE, Waller JD. Evaluability assessment: Avoiding type III and IV errors. In: Evaluation Management: A Sourcebook of Readings. University of California Libraries; 1977:264–282.
• Scheirer, M. A., & Dearing, J. W. (2011). An agenda for research on the sustainability of public health programs. American Journal of Public Health,101(11), 2059–2067. https://doi.org/10.2105/AJPH.2011.300193ArticleGoogle Scholar
• Scriven, M. (1972). Prose and cons about goal-free evalution. Journal Education Evaluation.,3(4), 1–7. Google Scholar
• Shadle, S. E., Marker, A., & Earl, B. (2017). Faculty drivers and barriers: laying the groundwork for undergraduate STEM education reform in academic departments. International Journal of STEM Education. https://doi.org/10.1186/s40594-017-0062-7ArticleGoogle Scholar
• Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning far the sustainability of community-based health programs: Conceptual frameworks and future directions for research, practice and policy. Health Education Research,13(1), 87–108. https://doi.org/10.1093/her/13.1.87ArticleGoogle Scholar
• Sherin, M. G., & Drake, C. (2009). Curriculum strategy framework: Investigating patterns in teachers’ use of a reform-based elementary mathematics curriculum. Journal of Curriculum Studies,41(4), 467–500. https://doi.org/10.1080/00220270802696115ArticleGoogle Scholar
• Smagorinsky, P., Lakly, A., & Johnson, T. (2002). Acquiescence, accommodation, and resistance in learning to teach within a prescribed curriculum - Pr. English Education,34(3), 187–213. https://doi.org/10.2307/40173127ArticleGoogle Scholar
• Sridharan, S., & Nakaima, A. (2019). Till time (and poor planning) do us part: Programs as dynamic systems—Incorporating planning of sustainability into theories of change. Canadian Journal of Program Evaluation.,33(3), 375–394. https://doi.org/10.3138/cjpe.53055ArticleGoogle Scholar
• Stevens, C. D. (2018). Repeal and replace? A note of caution for medical school curriculum reformers. Academic Medicine,93(10), 1425–1427. https://doi.org/10.1097/ACM.0000000000002219ArticleGoogle Scholar
• Tannenbaum, D. (2012). Rebuttal: will the Triple C curriculum produce better family physicians? Yes. Canadian Family Physician.,58(10), e538. Google Scholar
• Van Melle, E., Frank, J. R., Holmboe, E. S., Dagnone, D., Stockley, D., & Sherbino, J. (2019). A core components framework for evaluating implementation of competency-based medical education programs. Academic Medicine,94(7), 1002–1009. https://doi.org/10.1097/ACM.0000000000002743ArticleGoogle Scholar
• Varpio, L., Bell, R., Hollingworth, G., et al. (2012). Is transferring an educational innovation actually a process of transformation. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-011-9313-4ArticleGoogle Scholar
• Wallace, C. S., & Kang, N. H. (2004). An investigation of experienced secondary science teachers’ beliefs about inquiry: An examination of competing belief sets. Journal of Research in Science Teaching,41(9), 936–960. https://doi.org/10.1002/tea.20032ArticleGoogle Scholar
• Whitehead, C. R. (2012). Will the triple C curriculum produce better family physicians? No. Canadian Family Physician.,58(10), 1071–1073. Google Scholar
• Whitehead, C. R., Hodges, B. D., & Austin, Z. (2013). Captive on a carousel: Discourses of “new” in medical education 1910–2010. Adv Heal Sci Educ.,18(4), 755–768. https://doi.org/10.1007/s10459-012-9414-8ArticleGoogle Scholar
• Zimmerman, B., & Schunk, D. H. (2004). Self-regulating intellectual processes and outcomes: A social cognitive perspective. In D. Y. Dai & R. J. Steinberg (Eds.), Motivation, Emotion and Cognition: Integrative Perspectives on Intellectual Functioning and Development (pp. 323–349). Lawrence Erlbaum. Google Scholar

Author information

Authors and Affiliations

1. Centre for Faculty Development, Faculty of Medicine, University of Toronto at Unity Health Toronto, Toronto, Canada Betty Onyura
2. Department of Family and Community Medicine, University of Toronto, Toronto, Canada Betty Onyura, Elliot Lass, Jana Lazor & Laura Zuccaro
3. Institute for Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada Betty Onyura & Elliot Lass
4. Postgraduate Medical Education, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, Canada Deena M. Hamza
5. MD Program, Temerty Faculty of Medicine, University of Toronto, Toronto, Canada Jana Lazor

1. Betty Onyura