Item Analysis - Outline.ppt

* B. Item Difficulty Measure P Difficulty determined by item and population tested P(i) = # got item correct # taking test B. Item Difficulty Measure P P = .50 is best P = 0 or P = 1 – such items do not distinguish ability levels C. Discrimination Index D Extreme groups method U = # getting item correct in ‘top’ group L = # getting item correct in ‘bottom’ group nU = # in upper group nL = # in lower group D = U – L nU nL D. Item Total Correlation Good item High correlation People who get item correct have high score on the test People who get item wrong have low score on the test Poor item Low correlation: look at wording – may be testing reading skill 5. Item Response Theory (IRT) Item characteristic curves Problems with IRT Adaptive testing using computers A. Item characteristic curves Most important idea: Item Characteristic Curves (ICCs) One curve for each test item X axis: test-taker ability (given by test score) Y axis: probability of choosing an answer Test Score Probability of correct response Item 1 Item 2 Item 3 % of people writing the test who got item correct Ability Each curve shows how probability of getting item correct changes with ability A. Item Characteristic Curves Slope: how quickly the curve rises. indicates how well item discriminates among persons of differing abilities like P(i) in Classical Test Theory but sample-invariant A. ICCs are sample invariant P(i) = # got item correct # taking test P integrates item and group tested: 10 10 20 50 ICCs separate item difficulty and test-taker ability and present both visually in one graph – that is, you can see item difficulty independently of test-taker ability B. Problems with IRT Obtaining stable estimates of IRT parameters requires rather large samples Computationally complex IRT model assumes that the trait being measured is one-dimensional. It may not be. C. Adaptive Testing Using Computers computer selects harder or easier questions as test-taker gets eac