By Stuart Singer, The Teacher Leader
The key to creating an illusion is to distract the viewer. Draw the attention away from the sleight of hand and the audience will believe they have seen magic. Apparently this technique is now a key element when dealing with the analysis of standardized test results. The plan appears to be to use some new and often outrageous assertion to distract the public and “abracadabra” many of the problems inherent in the No Child Left Behind (NCLB) and end-of-course standardized testing magically disappear.
The latest form of deception is an idea being floated by educational leaders in Virginia. They are considering a proclamation announcing that a “pass advanced” on the state’s Standards of Learning (SOL) exams is an indication of college preparation. The word “advanced” in most contexts indicates a significant level of accomplishment; in this case, however, it should not be confused with readiness for post high school study.
It is actually more of a numbers game
At first glance a “pass advanced” might appear to be a significant achievement. It requires a score of 500 or more on a test scaled to go from 200-600 (400 is required for passing). But as in other forms of magic, these numbers are an illusion. Like many other end-of-course exams being used for NCLB, the SOL is a four-option multiple-choice test with no penalties for wrong answers. Consequently the actual range of scores is not nearly as large. The laws of probability decree anyone answering 50 such questions would start with 12 or 13 correct responses simply by random guessing. In 2011 a passing score of 400 on the Algebra 1 exam required 23 correct answers. As a result of that scale, every student begins with a score of at least 340. Thus the real possible range is 340-600. Suddenly a tally of 500 does not seem quite so “advanced”.
Forty-four correct answers will earn a student a 500. Even if this were an exam with open-ended questions and penalties for wrong responses, mastery of only 88% of the curriculum is hardly college-level work. But with a multiple-choice, no penalty format, 44 accurate responses represent much less. A few quick calculations reveal that if a student can answer 42 questions, probability will produce the missing two from the remaining eight. Now the mastery level is down to 84%–a “B-“on most grading scales. Even those numbers are a bit skewed. If an individual can eliminate one or two potential answers in a question the likelihood of a successful “guess” increases exponentially.
Adding to these misperceptions is the limited nature of such forms of questions. They cannot require multiple-step responses or demand a true demonstration of mastery of the most complex or intricate aspects of a subject. They can only ask questions that have reasonably simple answers. It quickly becomes clear that based on almost any analysis, a pass “advanced” on these tests is not a predictor of college success.
To get quality, you need quality
As Mel Riddile discussed in a previous post, tests made on the cheap are susceptible to both cheating and inflated results. If Virginia and other states want to administer tests that are indicators of future educational success, they will need to move away from the current easy-to-grade and inexpensive formats and invest in exams that will accurately measure a student’s mastery of a class. Until then, any claim of academic prowess based on the results is nothing more than an illusion and distraction.