CAT 2014 - Attempts and Percentiles

Let us list and bust a few of the myths regarding CAT attempts and percentiles

1. Number of attempts and percentiles have a strong correlation
This is of course true. But the correlation is extremely overstated. Every year, there are lots and lots of students who attempt 65% of the total questions and end up with less than 85th percentile. The far deeper correlation is likely to be between accuracy and percentile. The number of attempts is merely the most visible metric and therefore the most discussed. Unfortunately, quite a few students chase magic attempts numbers and end up shooting themselves in the foot. Come results date, we will hear a lot of students cribbing that the scoring, normalization, etc were incorrect.

In CAT 2013, a few of our students secured more than 90th percentile in a section having attempted only ~8 questions in the section. If you had read any of the forums, you would believe that guys who attempted less than 15 in a section should basically throw in the towel.

2. All students have 85% accuracy
The only thing the attempts and percentiles tell us is that there are a LOT of people who have accuracy rate of less than 50%. They get more than half the questions wrong. The CAT is a high quality exam. Pausing for thought and worrying about detail pays off in this exam. Guys who have cracked this exam will tell you that being in a frenzy never helps.

3. If you have attempted less than 65 questions in CAT 2014 you have no chance
At least half the students who are going to join IIM ABC are going to be guys who attempted less than 60 questions. The vast majority of guys who attempt 65+ questions are going to end up with less than 95th percentile. There are lots of students who appear for CAT every year, attempt lots of questions, and consistently get between 85th and 90th percentile. Its a shame that they do not change their strategy.

So, where does this leave us.
If you have attempted fewer questions but are sure about most of them. the score might surprise you to the upside.

As far as percentiles go, I would guess that they would be in the following range
Overall attempts ~ 40  - ~90th percentile
Overall attempts ~ 45  - ~94th percentile
Overall attempts ~ 50  - ~98th percentile
Overall attempts ~ 55  - ~99th percentile
Overall attempts ~ 60  - ~99.4 percentile
Overall attempts ~ 65  - ~99.7 percentile
Overall attempts ~ 70  - ~99.9 percentile
Overall attempts ~ 75  - Higher than this

Two caveats
1. These are at-best informed guesses. At worst, they are random numbers in a sequence.
2. All of these numbers are based on the assumption that at worst 2-3 questions are wrong. I do not understand the term "around 70% accuracy". Any paper where you believe you will have around 70% accuracy, you are basically leaving to the machinations of fate. If you think around 3 out of every 10 could be wrong, then very easily 5 out of 10 could be wrong.

Note that the density of scores is usually very low right at the top. Between 100th percentile and 99th percentile, the difference in number of attempts (correct answers) will be around 20 (perhaps even more). It is fairly crowded in the 90th-98th percentile range. With the same number of attempts, 4-5 more correct answers might determine the difference between a middling 89th percentile and sure-fire 96th percentile. This is why we ask students to take gazillion mock CATs and fine-tune everything.

Those students still waiting to take the exam, the one suggestion I would make strongly is to not get carried away by all the attempts numbers you are hearing. An enormous number of students live for the kick of saying that they attempted a lot of questions on D-day. A great proportion of this "huge number of attempts" gang end up having attempted 10 questions too many for their own good.

Mock CATs, Mock CAT percentiles, predictions etc

I am going to start with a series of quotes.

There are two ways of lying. One, not telling the truth; the other, making up Statistics. 

There are lies, damn lies, and statistics.

Statistics are used much like a drunk uses a lamppost: for support, not illumination.

Replace Statistics with Percentile analytics and this would fit in perfectly in the context of CAT. No matter how many fancy charts are shown, it is the responsibility of the students to not over-interpret the results. When you take a mock CAT, the review happens on three planks -

1) Learn better methods by looking at the solutions
2) Think about how you could have improved your attempts by selecting better questions.
3) Evaluate your percentile.

Of the three, the least useful is no 3 (probably also the most harmful). Because it is the least reliable. Almost all mock CAT providers use look-up tables to give percentile scores. Gone are the days where a mock CAT provider could get 10,000 people to beat themselves into a frenzy on a given Sunday to take a mock CAT at 100 god-forsaken locations in the Country.

These days, proctored CATs probably have a sample size in the range of 1% of CAT takers. These are likely to be the most serious, most experienced, most-driven of the CAT takers. So, the test providers "adjust" the look-up table to factor this in. In layman terms, this boils down to - they look at the actual numbers, feed in some other numbers, and pat themselves on the back.

So, the percentiles are fairly unpredictable. And contrary to what all the service providers claim, there is no way to back-verify that database. The only ones who will call a coaching institute and tell them that they scored a decent percentile are the ones who got a decent percentile. So, the reverse-verification process is bogus. So, no one actually knows whether any percentile number has a serious correlation with the actual numbers. There is way more anecdotal evidence to say that the percentile numbers are erratic, than to say that they are accurate.

I recently heard that there was a percentile predictor that predicts percentiles based on numbers you feed in based on the scores you have got on other exams (some of my students called me to give me an update on their predicted results.). I almost fell from my chair laughing. Students who believe that there is a scientific basis for this kind of predictor should re-examine their brains. This level of gullibility cannot be good for the body.

The thought process behind the predictor is as follows - Take three not scientifically proven indicators with no established proof from back-verification, define arbitrary weightages based on your genius to these, come up with a fourth metric and call it a prediction. And supposedly students are flocking to this predictor engine. No wonder soothsaying, tarot card reading are all flourishing industries in our Country. If smart, ambitious people can be made to believe there is such a thing called a percentile predictor, god save the masses.

Anxiety does strange things to the mind. Fretting and fuming about this exam makes you do all kinds of over-analysis. The mind needs something to cling on to, to feel that you are on the right track. Ergo, predictor.

Think of all the guys who have completed their MBAs from IIM A, B, C and think about how each of them would react if you said "percentile predictor" in their midst. I will give you a hint. I spoke to a bunch of them and the only common factor was laughter.

So, what should we do?

I am going to say something that is difficult to implement. The last 2-3 weeks, you must focus on three things

1. Chuck the percentiles. Seriously, chuck them. Forget about them. Banish them.

2. Focus on still being curious and not getting sucked into this pressure vortex. The first 2-3 mocks that you would have taken, you would have enjoyed grappling with the questions. Slowly, the pressure takes over and you stop having fun in the mocks. You are too worried about the score (and that damned percentile). The ones who crack this exam are the ones who are likely to look at a question and go "Wow. Super question. I think I will enjoy solving this". If that joie de vivre is gone, you should rekindle it.

3. Mentally prepare for a paper that is far simpler than your mocks. In our Country, the service providers have a feeling that to create a great mock CAT, you must make it very tough. So, they create phantom tough questions. CAT does not do that. Lot of students panic in a paper because it is way easier than all the mocks they have done. They have difficulty adjusting to the fact that they can attempt 34 questions in Quant in CAT, whereas they never managed more than 22 in any mock. Finding a paper that is easier than mocks can disorient you. You end up shooting for too less, end up adjusting too late, and end up with a just-miss scoreline.

We at 2IIM have resisted every urge to make our mocks tougher and fought tooth-and-nail to keep it representative. This is why good students can sometimes attempt 75-80 questions in our mocks. Remember, the very best ones can and will attempt that many in actual CAT. So, it is very important to keep in mind that the level of difficulty will not be outlandish. If you are really good, but have never attempted more than 60 questions overall in a CAT paper, how will you adjust on the day of the exam to have a crack at 84 questions?

Try your best to have some fun in the paper. be very zen and forget the percentile, and go in thinking "this exam is going to be easy". Best wishes for CAT.

CAT - In defence of the CAT

As CAT 2010 draws to a close, we will soon be hearing a lot of feedback on the quality and consistency of the exam, the technology, organization, etc. It is about the same time that we will hear people saying that the CAT is a pointless exam, an exam that tests unnecessary skill sets, and an exam that is needlessly too difficult. People will tout examples of various calculation-intensive questions friends of friends of theirs saw in the paper and feel vindicated about their claim that the CAT is a very computation-heavy random exam designed to test unnecessary things. You will probably also hear the inevitable "Why is it necessary to do 24.5 * 35.4 in 10 seconds to be a good manager".

Most of the people who say this know little to nothing about the CAT, and at any rate have not spent enough time either thinking about the paper or evaluating it. They are happy to accept the established stereotype and help it along.

My colleagues and I have been taking the CAT almost every year and can state unequivocally that the quality of the paper has improved with time. There are 3 key myths that people have been spreading about the CAT

1. It is a calculation-intensive exam: On an average, about 5 out of 40 questions in Quant + DI are of this type, and one can skip this and still score 100th percentile in CAT. CAT was calculation-intensive in the late 90's, it has tested application heavily in the past 10 or so years.

2. CAT does not test relevant skills to be a manager - also stated as "Why should someone know Set theory to be a good manager?": At some level, CAT is an entrance exam designed to test basic intelligence. Any academic performance is a signal that the student can potentially do well. So, the CAT is looking for a signal of intelligence. Around two lakh students of diverse backgrounds take this exam every year. One needs to find something very basic to use it as a proxy for intelligence. Testing numerical ability, problem solving ability and comprehension ability are probably the best proxies available. What would you rather test - general knowledge, science, subject-knowledge? Anything else pales in comparison. Also, remember, CAT is not an entrance exam to be a manager. It is an entrance exam to get admission into a school that will train you to become a manager.

3. Some questions in CAT are too difficult: This is a problem that has come about because of a specific characteristic of India - one of large numbers. CAT needs to make a distinction between the average and the good. But if this were the only distinction that CAT needed to make, a consistent and simple paper would suffice. But the CAT also needs to make the distinction between good and really good and really good and exceptional. Remember, they need to devise a mechanism to distinguish the top 0.2 percent within the top 1 percent. This suggests the need for creating "men and boys" questions. Questions that demarcate the exceptional from the merely very good. Any paper where the really good students can attempt 55+ out of 60 students within 135 minutes clearly indicates that this is not too difficult. If the paper is peppered with simple questions, the exam just tests just speed, and not understanding. A slightly tougher paper that requires a high level of application is required for getting the top 0.2% from a sea of applicants.

Most people will agree that it does not require a genius to crack CAT. It just requires loads of application, good decision-making ability and adequate preparation. Sounds like a good test for finding managerial talent to me :)

Now, finally to the point that irks people the most, why should calculation-speed be a factor at all? When the whole globe does not really rate calculation-speed, why does India (CAT) cling to this notion? The answer again, is straightforward. India still believes it is a critical skill set. And the CAT is trying to find the best proxy to select smart Indian kids. Why should the CAT not test what India finds important. If anything, the CAT has been doing a lot to de-emphasise this feature. Over the past 10 years, the number of number-intensive questions have fallen sharply, and rightly so. India is moving on, and the CAT is setting the pace. CAT cannot move on all on its own, then they will end up getting all the wrong candidates.

So, there goes my defence of the CAT. I am sure the IIMs wont care much about giving their viewpoint. So, here is my version of how they might have done it.

Having said all this, some things irk me a lot. The complete lack of consistency in the paper is an issue that CAT needs to address. More importantly, the CAT needs to be error-free. Especially now that they have multiple sessions. The CAT was not error-free this year. Far from it. And god knows how they treat the wrong questions. I cannot think of a single fair way of treating error-filled papers. Hope they improve on this.

CAT scores count for no more than 20%

Thanks to the RTI Act, IIM B has made it public that your CAT scores do not have a weightage of more than a fifth when admission decisions are made by the institute.

IIM B always had placed a premium on academic prowess that spanned much more than the 3 to 24 months preparation that students do to crack CAT. They did and still continue to place a lot of importance to a student's long term performance starting from Class 10. So, if there is anyone out there in class X and has a long term aspiration of cracking CAT, then make sure that you score well in your board exams as they matter as much as your CAT scores will 5 to 7 years from now.

For those, who have crossed the 10th, 12th or graduation bridges (especially for those amongst us who crossed the bridge not so colorfully), hopes of graduating from an IIM need not come to an end. IIM Calcutta, at least till the recent past did not place much emphasis on your previous academic record. In fact, the Admissions Committee Chairperson has gone on record a couple of years back that CAT score is sacrosanct and a high CAT percentile is certain to get you a GDPI call from IIM Calcutta.

I can vouchsafe for that claim. I have had classmates who had scraped through their graduation. A gentleman with a 5 CG from IIT Bombay was a classmate of mine. Another gentleman who is presently the CEO of a fast growing IT company barely managed 61% in 12 standard board exam (Tamilnadu State Board) and graduated with a 59% as a BA Sociology graduate.

The trend continued much after I graduated from IIM C. A gentleman three years my junior graduated from IIMC. Before joining IIMC, he graduated from Calcutta University with a B. Com degree. He had an aggregate of a little over 45% in his graduation. Very recently, a student of mine with a 6.5 CG from BITS Pilani made it to the IIMC. However, at IIMC he made a vow to be among the top 20 and made it to the top 18 and is presently working for a world renowned Business Consultant.

So, for all those who cannot go back and correct your scores at school and college level, there are still few IIMs left that give adequate credit to one's CAT scores. A 99.7 + percentile is likely to land you calls from a few of the IIMs, IIMC including, I hope. This is probably your only chance to redeem past sins should you aim to seek an admit from one of the IIMs.

I wish you all success in your CAT endeavors.