CAT is going to become an on-the-fly testing exam - What does this mean and what are the implications?

CAT has seen quite a few changes recently. It turned online in 2009, it turned online without glitches in 2010 (could not resist the dig even though I think the CAT is a great exam), became a partial linear on-the-fly exam in 2011 and could soon become a round-the-year exam. Now, what does all this mean, and what are the implications?

What does this mean? 

A fixed form method of testing is a method of testing where all applicants write the same exam, get to see the same set of questions and are given percentile-based scores after the scores are computed. An on-the-fly exam is a paper where different candidates are given different questions but overall percentiles are calculated post normalizing the scores for varying levels of difficulty (on a per-question basis). A completely on-the-fly exam would probably be one where questions are dynamically selected from a database based on some dynamically changing parameter(s). This changing parameter is usually the candidate's performance in the exam up to that point. This is popularly known as the adaptive testing model.

Why this shift from fixed-form testing to on-the-fly testing?

To start with, the fixed form testing is simpler. One paper for all applicants ensures fair testing, removes randomness, and provides an ideal benchmark. But it has certain constraints - two very important ones stand out.

1. The most important is the fact that it is difficult to design an exam that tests the varying levels of capability of a large applicant pool. Because it is a single instance exam, it can become too tough for a section of the audience so there is no real metric to distinguish between 60th percentile and 20th percentile. On the other hand, it might be too easy for the top quartile, so there is not much to choose between someone who scored 90th percentile and 99th percentile barring a few details here and there. Now, CAT is under pressure to make a distinction between top 0.1 percent and top 1 percent, top 1 percent and top 3 percent, top 3 percent and top 10 percent and so on. A single instance exam can never give you that kind of fine-tuning.

2. The exam happens only once a year. The logistical challenge is significant. More importantly, students find it difficult to take/prepare for this exam.

The on-the-fly test removes both these constraints (at least partly). 

Sod all this technical discussion, what does this do to my preparation?

This is the key factor. This change in testing algorithm alters a few key aspects of preparation, especially practice test-taking. Some conventional ideas get turned on their head.

1. Selecting the easy questions and getting them might not take you all the way: One needs to select the easy questions and get them right, there is no doubt about this. But, very often you will find that this alone is not enough (especially to step up from the 96-97th percentile range). Questions from any CAT paper can be broken into three bunches -

Easy - About 12-14 questions from each section
Do-able but not bleeding obvious - 8-10 from each section
Tough ones, either conceptually or because they are time consuming - 6-8 ones

Because of the differential marking built into the system, any student is going to get rewarded more for getting a few of the tougher ones correct. So, every easy question needs to be nailed. But beyond this, students need to get a majority of the middling ones and at least 2 of the tough ones. Otherwise you will be knocking on the neither-here-nor-there 96th percentile range. It is important to learn the tougher topics from first principles. So, one needs to be either conceptually sound so that you can hit the tricky ones, or frightfully quick so that you can hit the time-consuming ones. Just skimming and doing all simple questions well will just take you close (but no cigar)

2. Need to focus on how well you are selecting questions in your mock CAT series. However, as a candidate you must not only focus on how many easy ones you missed, but also think about how many of the trickier ones you should equip yourself to tackle.

Your score is going to depend not only on how many questions you attempted, but also how you did on the tougher ones. Simply skipping the tough questions in each topic will make sure that you never cross a certain threshold as far as your score goes.

What should one do?

Preparation plan does not change much. Take many mock CATs. Whenever you are analyzing a mock CAT, see if there are some questions that you should have attempted but could not. Revisit the theory to re-equip yourself. Post every mock CAT, you should revisit the topics to plug the gaps.


What should one not do?

Please do not take mock CAT scores seriously. Mock CAT percentiles are one of the most misleading statistics in all of CAT preparation. Take plenty of mock CATs to get a feel for the exam. But please do not take the percentiles seriously. There are too many students who consistently score 99th percentile in one or the other mock CAT series, but end up with a 95th percentile in the overall scheme of things. There is no great mock CAT series provider out there in the market who has managed to capture all the moving parts. Take mock CATs from more than one provider. Get the experience. But do not take the scores seriously.

Product plug alert!!. You can get details of the mock CAT series of 2iim here.

Most people would be starting their CAT preparation at about this time. We often get asked whether the time is sufficient. I will give a further blog post on this, but the short answer is "Yes". Most definitely Yes.