Is CAT punishing incorrect answers far more harshly?

Every year, after CAT results, we have students complaining about the scoring. Generally, it is just considered overreaction by students. However, this time there have been some funny patterns emerging from the scores. There are two key trends that we have seen

1. A few good students have not just underperformed, their scores have fallen off a cliff
2. Students with very high attempts have seen unusually low scores

Anecdotally, I have seen the following scores/numbers that I am not able to completely wind my head around with the current assumptions that we have made.

1. I had two students more or less equally good in quants. One froze in the exam and attempted 6 questions and ended with a score of 86th percentile. The other probably pushed himself too hard in the exam and attempted 19 questions to end with 78th percentile.

2. Another good student attempted almost equal number of questions in Quant and Verbal, and got 90+ percentile in one and less than 60th percentile in the other

3. We have heard multiple stories about someone getting 50th percentile+ with zero attempts in both sections.

All of these lead me to believe that one assumption we have made with the computation of the raw score is probably incorrect. We have all believed that a correct answer fetches a student +3, and an incorrect one fetches -1.  And then these scores are normalized to factor in level of difficulty.

My suspicion is that the IIMs have created a framework that punishes inaccuracy far more harshly than this. 

On a +3/-1 system someone who attempts 19 and gets 6 wrong would get 39 - 6 = 33 marks. Someone who attempts 6 and gets all 6 correct would get 18. Now, this giant difference in raw scores is unlikely to get bridged in the normalization process.

However, if the IIMs said each student gets -1 for the first incorrect answer in the section, -2 for the second one, -3 for the third and so on, then the person with 19 attempts and 6 wrong could get 39 - 21  = 18 which is same as the score of someone who got 6 out 6 correct.

Alternatively, the IIMs could compute the score, normalize it, scale it to 225 and finally multiply by accuracy level to get the final score. Under this assumption, the final score of the person scoring 13 correct and 6 wrong would be their normalized score * 13/19, which is a more than 30% drop in score. Whereas the one scoring 6 out of 6 would retain his entire score.

I have just given two outlines that punish inaccuracy beyond the +3/-1 framework. I am sure we can create many more. My guess is that the IIMs want to say that 10 out of 10 is way better than 14 out of 18. From a student perspective, the earlier you can wind your head around that the better.

In my view, the most important message sitting inside the scores this time around is this - Do not chase attempts, agonize over accuracy. This is the mantra most trainers have been screaming for many years now. The relevance of this mantra just shot up now. So, all of you who are preparing for CAT 2014, get into the habit of getting every attempt correct from now on. If you are unsure, leave the question. If a question seems too easy, have another read to verify. Do not live in this speed-world where attempting 10 questions in 20 minutes and getting 8 correct is considered better than attempting 6 and getting ALL of them correct.

Finally, a disclaimer. All of this is more of less just conjecture. We can never be sure of the method the IIMs use for getting the normalized scores for CAT. This is just an attempt to interpret based on anecdotal evidence. Research based on statistical evidence is often wrong, so take anything based on anecdotal evidence with a pinch of salt.

Best wishes for all students who are appearing for their interviews. I look forward to meeting all 2iim students at the GD/PI workshop.

The above piece is from Rajesh Balasubramanian, a director at 2iim who trains student for CAT and takes CAT every year.