Prompt Pattern Engineering for Test Question Mapping Using AI
Curriculum mapping and assessment alignment are essential for accreditation and continuous improvement, yet they are often time-intensive, subjective, and difficult to scale. Faculty frequently face the challenge of mapping hundreds of exam and quiz questions to learning objectives consistently.
Dr. Joey Mattingly, Associate Professor and Vice Chair for Research in the College of Pharmacy, Department of Pharmacotherapy, led a study exploring whether AI could help solve this problem. Dr. Jennifer Babin, Associate Professor and Vice Chair of Teaching for the Department, served as first author, and Dr. Hanna Raber, Associate Professor and Assistant Dean of Curriculum for the College, contributed expertise in curriculum design and assessment.
The team evaluated whether a large language model (ChatGPT) could reliably align assessment items with course-level learning objectives using structured prompt engineering and pattern-based prompting.
The process involved asking ChatGPT to classify each question according to predefined learning outcomes and then comparing its classifications to expert human ratings. The results showed moderate to strong agreement, demonstrating that AI can support faculty in assessment design and quality assurance.
Why It Matters
This AI-assisted approach substantially reduces the time required for assessment review and provides a reproducible method for curriculum analysis. Benefits include:
- Faster curriculum audits
- More consistent alignment judgments across faculty
- Potential for real-time feedback to instructors developing new assessments
The findings highlight how AI can enhance instructional design while preserving faculty oversight and judgment. This project represents one of the earliest empirical evaluations at the University of Utah of how large language models can support instructional assessment processes in health professions education.
As Dr. Mattingly notes, the structured prompting framework developed in this study is openly shareable and may serve as a model for other colleges exploring responsible and practical AI integration.
For more information, you can reach out to Joey Mattingly at joey.mattingly@utah.edu.