Assessment Strategies
Performance objectives are the exit behaviors from the course and should be evaluated and scored as such. Discrete, developing skills should be introduced and assessed with a stepped and scaffolded approach to increase fluency and learner confidence (Van Merriënboer & Kirschner, 2017).
Reliability
1. Reliability refers to the reproducibility of the measurement. Think about students taking an imaginary parallel test or answering an imaginary parallel assessment question; would students get the same scores?
Reliability depends on two major factors:
Those two factors interact—the higher-discrimination your items are, the fewer items you need. A high-discrimination item means students who did better on the test overall were much more likely to get that item correct. A low-discrimination item could be one in which most students of any type got the answer right, or wrong, or one in which there is no relationship between answering this question right and doing well on the test overall.
For these modules, with only a few assessment questions it is very important that they be high-discrimination questions! So among other things we need to make sure that these questions are not too easy or too hard.
Reliability refers to the reproducibility of the measurement. Think about students taking an imaginary parallel test or answering an imaginary parallel assessment question; would students get the same scores?
Reliability depends on two major factors:
- having enough test items (a long enough test) and
- having high-quality discriminating items.
Those two factors interact—the higher-discrimination your items are, the fewer items you need. A high-discrimination item means students who did better on the test overall were much more likely to get that item correct. A low-discrimination item could be one in which most students of any type got the answer right, or wrong, or one in which there is no relationship between answering this question right and doing well on the test overall.
For these modules, with only a few assessment questions it is very important that they be high-discrimination questions! So among other things we need to make sure that these questions are not too easy or too hard.
Reliability refers to the reproducibility of the measurement. Think about students taking an imaginary parallel test or answering an imaginary parallel assessment question; would students get the same scores?
Validity
2. Validity asks does the test measure what it purports to?
For the modules, what this means is, you want to make sure the assessment questions address the objectives you listed at the beginning of the module, and avoid being about minor details. Think about students being asked the question – Evaluate the projected path and velocity of Haley’s comet compared to the projected path and speed of visitors from Disney’s entrance to the entrance of Space Mountain.
For the modules, what this means is, you want to make sure the assessment questions address the objectives you listed at the beginning of the module, and avoid being about minor details. Think about students being asked the question – Evaluate the projected path and velocity of Haley’s comet compared to the projected path and speed of visitors from Disney’s entrance to the entrance of Space Mountain.
Level of Knowledge Tested
3. Level of knowledge tested – ranked using Bloom’s Taxonomy
Questions should be designed to evaluate learners at the Analyze and Apply levels of evaluation. If the intention is to decrease learner test anxiety, then a limited number of lower level questions (at the Understand level) may be appropriate at the beginning of the evaluation.
Questions should be designed to evaluate learners at the Analyze and Apply levels of evaluation. If the intention is to decrease learner test anxiety, then a limited number of lower level questions (at the Understand level) may be appropriate at the beginning of the evaluation.
Best Practices for Validity of Responses
4. Best Practices for Validity of Responses
Avoid using:
All of these forms of true-false questions have multiple validity problems: the best students are those who can think of exceptions to your supposed 100% true or false statement; they can often be answered by recognition/ low Bloom’s skills or with partial understanding; and they have low reliability with a large role for guessing.
Avoid using:
- True/False Questions
- “Choose all correct answers”
- Complex multiple choice i.e. “Choose A and C not B…”
- Double negatives – “Which of these are not the reason the turtles are not reproducing?”
All of these forms of true-false questions have multiple validity problems: the best students are those who can think of exceptions to your supposed 100% true or false statement; they can often be answered by recognition/ low Bloom’s skills or with partial understanding; and they have low reliability with a large role for guessing.
Suggested Assessment Strategies at Bloom’s Taxonomy Level
Remember
Vocabulary Drills and Strategies
Provide in-class and out-of-class strategies for vocabulary acquisition.
Provide in-class and out-of-class strategies for vocabulary acquisition.
- Vocabulary Instruction for Faculty
- Kahoot – free in-class or homework anonymous quizzing (Faculty use)
- Quizlet – free electronic flash card and customizable study set (Student use)
Understand
Concept Maps
Model and then provide opportunities to students to demonstrate their knowledge of relationships between topic and course elements. The concept map is a learner’s representation of the meaning or frameworks specific to a domain of knowledge within the stated context (Novak,1990). Learners can begin with a blank piece of paper or a variety of web tools.
Model and then provide opportunities to students to demonstrate their knowledge of relationships between topic and course elements. The concept map is a learner’s representation of the meaning or frameworks specific to a domain of knowledge within the stated context (Novak,1990). Learners can begin with a blank piece of paper or a variety of web tools.
Apply
Lab Experiments/Mathematical Calculations
Scaffold and increase complexity of problem sets. Combine paired and individual problem-solving. Consider assigning homework sets and allowing students a few minutes to discuss and edit them as an entrance ticket in class before collecting them.
Scaffold and increase complexity of problem sets. Combine paired and individual problem-solving. Consider assigning homework sets and allowing students a few minutes to discuss and edit them as an entrance ticket in class before collecting them.
Analyze
Case Studies/ Compare and Contrast Outcomes
Provide ill-structured, multi-dimensional case studies for students to deconstruct, using existing knowledge and course resources. Allow them to present their findings to another group for feedback and any needed edits prior to submission for faculty review
Provide ill-structured, multi-dimensional case studies for students to deconstruct, using existing knowledge and course resources. Allow them to present their findings to another group for feedback and any needed edits prior to submission for faculty review
Evaluate
Meaningful Reflection Components
Direct learners to address each of these questions when responding to a reflection prompt.
Direct learners to address each of these questions when responding to a reflection prompt.
- What did I learn?
- How, specifically, did I learn it?
- Why does this learning matter, or why is it significant?
- In what ways will I use this learning; or what goals shall I set in accordance with what I have learned in order to improve myself, the quality of my learning, or the quality of my future experiences or service? (Ash & Clayton, 2004, p. 144)
Create
Research Paper/Lab Report/Web Page/Video/Presentation
Provide opportunity for learners to document their synthesis of course topics.
Stations
Variety of individual assessment strategies for each level objective - offered at one time for small groups to rotate through.
Multiple Choice Assessments
Provide ample low-stakes opportunities for students to practice completing assessments that mimic formal, high-stakes assessments. Use Kahoot, online quizzes or paper entrance or exit quizzes of no more than 5 questions.
Review the language in each question to remove double-negatives. Develop questions that align with measured objectives Create an assignment for students to develop multiple-choice questions In low-stakes use, consider including a confidence rating for each question as well as the traditional multiple choice questions
Provide opportunity for learners to document their synthesis of course topics.
Stations
Variety of individual assessment strategies for each level objective - offered at one time for small groups to rotate through.
Multiple Choice Assessments
Provide ample low-stakes opportunities for students to practice completing assessments that mimic formal, high-stakes assessments. Use Kahoot, online quizzes or paper entrance or exit quizzes of no more than 5 questions.
Review the language in each question to remove double-negatives. Develop questions that align with measured objectives Create an assignment for students to develop multiple-choice questions In low-stakes use, consider including a confidence rating for each question as well as the traditional multiple choice questions
Bibliography and Resources
- Allen, D., & Tanner, K. (2005). Infusing active learning into the large-enrollment biology class: seven strategies, from the simple to complex. Cell biology education, 4(4), 262-268.
- Ash, S. L., & Clayton, P. H. (2004). The articulated learning: An approach to guided reflection and assessment. Innovative Higher Education, 29(2), 137-154.
- Nicol, D. (2007). E‐assessment by design: Using multiple‐choice tests to good effect. Journal of Further and Higher Education, 31(1), 53-64.
- Nicol, D. J. & Macfarlane-Dick, D. (2006) Formative assessment and self-regulated learning: a model and seven principles of good feedback practice, Studies in Higher Education, 31(2), 198–218.
- Novak, J. D. (1990). Concept maps and Vee diagrams: Two metacognitive tools to facilitate meaningful learning. Instructional Science, 19(1), 29-52.
- Towns, M. H. (2014). Guide to developing high-quality, reliable, and valid multiple-choice assessments. Journal of Chemical Education, 91(9), 1426-1431.
- Van Merriënboer, J. J., & Kirschner, P. A. (2017). Ten steps to complex learning: A systematic approach to four-component instructional design. Routledge.