【优质】ValidityandValidationMethods.ppt

Validityand Validation Methods Workshop Flow The construct of MKT Gain familiarity with the construct of MKT Examine available MKT instruments in the field Assessment Design Gain familiarity with the Evidence-Centered Design approach Begin to design a framework for your own assessment Assessment Development Begin to create your own assessment items in line with your framework Assessment Validation Learn basic tools for how to refine and validate an assessment Plan next steps for using assessments Assessment Development Process Validity: The Cardinal Virtue of Assessment The degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment. -- Mislevy, Steinberg, and Almond, 2003 Validation is a process of accumulating evidence to provide a scientifically sound validity argument to support the intended interpretation of test scores -- Standards for Educational and Psychological Testing (AERA / APA / NCME, 1999) Assessment Reliability Steps of item Validation Expert Panel Review(Formative) Are the items aligned with… The test specifications? Content covered in the curriculum? State or national standards? Is the complexity level aligned with intended use (e.g., target population, grade-level)? Are the item’s prompts and rubrics aligned? 2. Feasibility of Items (Think-Alouds) Does the item make sense to the teacher? Does the item elicit the cognitive processes intended? Can the item be completed in the available time? Can respondents use the diagrams, charts, tables as intended? Is the language clear? Are there differences in approaches by experts and novices (or teachers exposed or not to the relevant instruction)? SimCalc Example:Think-Alouds Conducting Think-Alouds Sample N: You learn the most in the first 3-6 Who Experts and Novices Low, Medium, and High Achievers Varying in proficiency in English Data capture and analysis Data can be extremely ric