Program Evaluation: The systematic use of methods to collect and analyze information to determine whether a program is fulfilling its mission(s) and achieving its goal(s).
Goals of Program Evaluation
- Evaluate and continuously improve every component of 3RC and the curriculum as a whole
- Maintain full LCME accredited status
Approach to Program Evaluation: The Plan for PittMed
The PittMed Program Evaluation uses a tiered system to evaluate 3RC:
- Tier 1: Individual Course Level
- Tier 2: Segment Level (Foundations, Clerkship, Bridges, Threads)
- Tier 3: Whole Curriculum Level
Data sources:
- Course (including clerkship) evaluations
- Faculty evaluations
- “Feedback Friday” summary notes
- Formative student assessments
- Summative student assessments
- School survey
- Graduate questionnaire
- USMLE reports
- NBME reports
- Match list
- Curriculum Map
Structure of Program Evaluation:
| Table 8.3-1 | Role in Curriculum | ||||||
| For each of the listed tasks, indicate the role1 of the individual(s)/group(s) listed below (D, E, R, Rec, A). If an individual/group does not have a role in a task, leave the cell blank. | ||||||
|
Task
|
Course/Clerkship Directors & Faculty
|
CAO/Associate Dean for Medical Education
|
Office of Medical Education Staff
|
Curriculum Committee
|
Curriculum Committee Subcommittees
|
Program Evaluation
|
|
Educational Program Objectives
|
|
Design
|
|
Review, Take Action
|
Evaluate, Recommend
|
|
|
Course/Clerkship Learning Objectives
|
Design
|
|
|
Review, Take Action
|
Evaluate, Recommend
|
|
|
Course/Clerkship Content & Instructional Methods
|
Design
|
|
|
Review, Take Action
|
Evaluate, Recommend
|
|
|
Course/Clerkship Quality & Outcomes
|
|
|
Design, Review
|
Review, Take Action
|
Evaluate, Recommend
|
Evaluate, Recommend
|
|
Faculty/Resident Teaching
|
Design, Evaluate, Take Action
|
Recommend
|
Review
|
|
|
|
|
Curriculum Content, including horizontal & vertical integration
|
|
Design
|
|
Review, Take Action
|
Evaluate, Recommend
|
|
|
The Outcomes of the Curriculum Phases
|
|
Design
|
|
|
Review, Recommend, Take Action
|
Evaluate, Recommend
|
|
The Outcomes of the Curriculum as a Whole
|
|
Design
|
|
|
Review, Recommend, Take Action
|
Evaluate, Recommend
|
1Definitions:
(D) Design/develop = Develop/create the product or process that is the basis of the task (e.g., the educational program objectives, the plan and tools for course evaluation)
(E) Evaluate = Carry out a process to collect data/information on quality/outcome
(R) Review = Receive and consider the results of an evaluation of the product or process and/or of its outcomes
(Rec) Recommend = Propose an action related to the process or product based on a review or evaluation
(A) Approve/Take Action = Have final responsibility for an action related to the product or process
PE Team
Faculty Director of Program Evaluation
Allison Serra, MD, MPH
serraae@upmc.edu
Director of Program Evaluation, CQI, and Accreditation
Greg Null, MA
grn18@pitt.edu
Executive Director, Office of Medical Education
Katie Maietta, MPPM
kmaietta@medschool.pitt.edu
Project Analyst
Michelle Sergent, MPH, MSL
mls124@pitt.edu
Activities and Responsibilities of the Program Evaluation Team:
Tier 1 (Course level) activities:
- Design, review, distribute, analyze course and faculty evaluations
- Facilitate “Feedback Friday” conversations among students, faculty and staff to reflect on 3RC (we use the the After Action Review Framework)
- Solicit and track “on the fly” curricular changes from content leaders in response to “Feedback Friday” conversations
- Assemble a “compendium” of outcomes for each course and distribute to students, Curriculum Committee Executive Subcommittee
- Membership on Foundations Course Design Subcommittee
- Membership on Clerkships/Bridges Subcommittees
Tier 2 (Segment level) activities:
- Design segment level review template for CC reports
- Collect and transmit data to curriculum committee/subcommittees and deans as needed
- Membership on Curriculum Continuous Quality Improvement (CCQI) Subcommittee
Tier 3 (whole curriculum level) activities
- Respond to requests for consultation on issues related to curricular quality/effectiveness
- Membership on Curriculum Continuous Quality Improvement (CCQI) Subcommittee
CQI Greenhouse
The Program Evaluation office is actively seeking medical students who wish to develop CQI projects and hone CQI skills. Medical students who have an idea for a CQI project or wish to learn more about accreditation, program evaluation or CQI skills should contact Greg at grn18@pitt.edu or stop in the OMED offices (S585).
News from the Greenhouse
Lauren Stumpp (Class of 2025) poster abstract was accepted at the 2024 Innovations in Medical Education conference held in April 2024. Her poster was titled “Who Isn’t Sweating Step 1? A Student Led After Action Review of School Provided Step 1 Resources.”
Abigail Yohannes (Class of 2027) poster abstract was accepted at 2024 AAMC Learn Serve Lead conference to be held in November 2024 in Atlanta, GA. Her poster is titled “Feedback at the Speed of…Lunch?”
Julie Bulova (Class of 2026) poster abstract was accepted at 2024 AAMC Organization on Student Representatives National Meeting in April 2024. Her poster was titled “Flex Weeks: An Innovation in Student Wellness and Career Exploration.”
Lauren Stumpp and Abigail Yohannes submitted a workshop abstract to the Medical College of Wisconsin’s Innovation in Health Care Research conference. Their workshop is entitled “Rx for Change: Medical Students Drive Quality Improvement Using the After Action Review.”