Moving CFA exams to computer based testing?

CFA General
#21

I have taken a number of certification exams, mostly from FINRA in the US. All of them have been on computer at Prometric. Prometric isn’t the only provider for these tests, but its big, with a huge number of choices of location.

Given the various FINRA exams I’ve already taken, I don’t really like the idea of the CFA exams being given there. Here’s why:

  1. The other exams are pretty much stuck with a slowly changing large bank of exam questions. The questions are categorized and rated for difficulty. For a particular exam there might be 2000 questions to select from and you get 100. These exams have to be kept current.

  2. I don’t have much problem with Prometric/FINRA security regarding things like a ringer taking a test for someone else, bringing notes, or nonpermitted calculator. If the CFA society was doing this worldwide, there are countries with poor facilities and security for such exams.

  3. If the Institute uses a question bank that slowly changes, they are vulnerable to somebody getting most or all of the questions. Imagine having to dump all the questions and write new ones. No tests would be administered for a while. Because the bank would contain many more questions than a single exam, it would be harder to replace.

  4. Ransom. Yeah, that ought to scare the hell out of the Institute. Imagine someone asking for money to not publish or sell questions. Have the exams on computer in 100+ countries and this is a real possibility.

#22

The real answer to this one is to tell the person to screw off and begin writing new questions. A reserve bank of prior experimental questions on a different, secure server would also allow for testing to continue for a period of time. It may take longer for scores to release, but that’s just to validate these newer items that just passed the experimental stage.

Any thoughts on paying off a ransom-seeking individual are silly because nothing stops them from releasing information at a later time or making copies of what they might have. Lives aren’t at stake, so it’s better to tell them to buzz off and wait for legal consequences.

I agree that there would be a need for heightened security, but there are plenty of high stakes computer administered tests that handle this well.

I also believe that potential security issues stem from the fact that candidates can take a level as many times as needed to pass the exam. Unlimited attempts per level also makes the designation easier to achieve since the CBOK changes so slowly. Placing a limit on the number of attempts per level, not on average, would increase the challenge faced in obtaining the designation as well as reduce the risk of people taking a level multiple times for the sole purpose of memorizing a few exam questions to report back to a testing company. It wouldn’t be terribly hard for an examinee to run to the car and write something down a few times during a section if their one purpose is to obtain 6-7 questions per 120.

Overall, I don’t think it would be a bad idea to move the exam to computer based administration for L1 and L2. Security is probably no greater a concern if this is implemented properly and the exams are proctored by well known centers like Prometric. I don’t think this would necessitate the exam being administered at-will throughout the year as some people have suggested, nor do I think it necessarily devalues the designation. The grades won’t necessarily come back much quicker, though. Most of the time used for exams like this is on quality control and determining the pass line. Score reporting is a thorough process and can take many weeks for exams administered in this format.

#23

Ticker, do you think Prometric and its foreign equivalents could handle 100,000 candidates in one day? Remember, they have hundreds of other types of exams they administer.

#24

No, but that’s part of the logistics behind implementing it correctly. Possibly X different forms for those who test on any of the X possible testing days (if they move to scaled scores where a certain scaled score is passing, they can do away with “forms” and rely on a random question set of previously vetted questions). I’m also sure that with the exam fees as they are, they could probably reserve several prometric centers for the time needed in a day. Part of those 100,000 candidates can be weeded out by putting a per level cap on the number of attempts you get, and the per day number of candidates can be alleviated by having different levels test on different days (also, L3 wouldn’t be at these prometrics).

Assuming that the current testing setup would be appropriate for a new testing environment and delivery would increase the chances that it doesn’t work. There are other exams that have large numbers of test takers annually, which is obviously only possible because they allow testing on multiple days of the year (and they don’t have to reserve an entire center as I suggested…more testing days would eliminate this). You can bet the exam integrity and security is at least at the level of the CFA exams.

Again, CFAI should cap the number of attempts per level, which would reduce the burden and increase the meaning of the credential. Multiple testing days during the year with the option (or requirement) for different levels to test on different days would work. They could still put a mandatory 6 month gap between when any one test taker sits for an exam at L1 and 12 months for L2/L3. Just an idea.

Edited: I mistakenly put a 6 month gap between attempts at L2, when I meant 12, to keep it consistent with the current framework.

#25

That ain’t work efficiently for level III/AM session. Those who are slow in typing will suffer. Besides, writing some formulas through keyboard might need a special trainer!

#26

I took CFP at Prometric and it was like going through a strip search at a NSA lobby… had to empty pockets, turn inside out, then do cartwheels to shake out any aids, then roll up sleeves and lift shirt to check for ink, and finally checked ears for electronic listening devices… really thorough search, much more secure than CFA checks.

#27

lol yeah

#28

Did you also get the metal detector wand, fingerprinting, photo ID check, and signature requirement each time you entered or exited the testing room? I’m forgetting a few, but I’ve had this experience a few times, although it might depend what test you’re taking. I don’t think people give Prometric enough credit. They run a secure operation in my opinion.

#29

See that’s the thing - some testing centers are secure like this, whereas others are the complete opposite. No uniformity. When I took the CAIA exams on a computer, security was a joke. I could have walked in there with a freggin textbook. I guarantee in foreign countries and I’m sure even in the US it can get even worse than this, or even have guys pay off the security dudes at the testing centers.

This is an awful idea. I’ll say it again. Cheapens the charter to the point where its like getting your MBA - anyone can get one now, it’s nothing special.

#30

Was your experience at a Prometric center? If employees at your testing center weren’t following security protocols that should be reported. Also, I would imagine different testing companies have different procedures, but also different examining bodies request different security measures.

Aside from that, I don’t think the issues you’ve described are eliminated by the paper and pencil testing currently employed. In fact, I think it’s more likely that a “rent-a-proctor” could be paid off more easily than someone who works full time at a test center. If the former gets caught, it’s usually only their second source of income that’s ruined, whereas the latter is screwing with their main source of income. Further, it’s more difficult to police a giant ballroom with hundreds upon hundreds of candidates test without video surveillance. The Prometric type centers seem to be much more tightly regulated, in my experience, and they have video recordings of each cubicle.

What about computer based testing, necessarily, cheapens the charter? I don’t think anyone is arguing that you can take it any day of the week nor that you can take it from home.

Personally, I think one of the worst ways to cheapen the charter is to allow unlimited attempts at any level. It’s clear this is a money making scheme masquerading as sympathy for “a bad exam day”…again, and again, andddd again…I think this also artificially depresses the pass rate on the exams. I knew of several people “just winging it” because the company paid for it unconditionally (but capped the number of attempts they would sponsor per level).

I think the quickest way to beef up the value of the charter is to cut the fat and limit attempts per level to two attempts and, if you fail L1, you need to wait the full year for your next attempt (it’s by design for the other levels). This would weed out many people who are just winging it, possibly increase candidate preparedness, and cut loose a bunch of people who aren’t passing.

#31

Yes, forgot to mention all that, also had to put all belongings in a locker… It was in San Jose, CA… Also video cameras all over the place… very secure, but it was kind of noisy with people coming and going taking different tests with different ending times.

​​

#32

It would be great if the CFAI could move to computerized testing. After all, it’s good enough for the CPA exams (which, unlike their CFA counterparts, are actually a license to practice). And the exams I took at Prometric were a much less unpleasant and more secure experience than spending my time with the amateur CFA proctors on their once-a-year power trip (yes, my passport has been fingered enough now, thank you). But the CFAI is basically a money-making machine and I doubt it will ultimately want to share its circa $300 million annual exam fee income with any third party.

#33

Received survey request from CFAI this morning and it’s all about potential revolution of Level I exam in the future, about exam delivery format, time needed for exam result release, frequency of the exam in a year etc. I chose paper delivery and exam to be arranged as most infrequent as I like the feeling of sitting in a large hall once a year with so many candidates attending a top class exam together. I prefer the traditional paper test although I can adapt well with computer-based test (did CFA mock exams online and also took my last CPA exam using computer).

However, if the delivery format were to change to computer-based and the frequency of Level I test increased to more than twice a year, I would doubt the value of the qualification like Newsuper mentioned, CFAI should just keep the frequency of the exam and control the quality of exam questions. Anyway we’ll see.

#34

I just got the same survey. I always chose the option that said “wait 6 months between tests” regardless of the other parameters because anything less would be a sham, IMO.

#35

Bringing back an old post, but this article should give those who want the CFA exam to be electronic something to think about:

http://www.afr.com/news/policy/education/online-exam-for-medicos-ends-in-disaster-and-unleashes-a-twitter-storm-20180220-h0wdaf

I’ve put the article text below:

Online exam for medicos ends in disaster and unleashes a Twitter storm

It couldn’t have happened to a harsher group of critics. Twelve hundred trainee physicians and pediatricians will have to resit a major exam after an online testing process failed halfway through.

The Royal Australasian College of Physicians has been inundated with complaints since the online exam collapsed on Monday.

The trainee doctors were taking the so-called RACP Divisional Written Exam in Adult Medicine, a two-part test being trialled online for the first time.

When they went to log on for the second half many found their passwords wouldn’t work.

The online contractor, Pearson Vue, declined to comment.

As a result, RACP president Catherine Yelland said the whole group would have to resit the exam, which will be done on paper next time, not online.

Among reactions on Twitter one trainee doctor said: “The flow-on effects to the trainees and hospitals is unprecedented both emotionally and financially.”

Another wrote: “Seriously poor form by one of the largest medical colleges in Australia.”

Rescheduling the exam means a delay of several months for some trainees to take up permanent jobs in hospitals.

“We’re very sorry this has happened,” a contrite Dr Yelland said. "These are students working at registrar level at hospitals. This is the exam they do before specialising.

"Some time ago, because it’s so complicated, so big and happens at exam sites simultaneously all over the country, we decided it was time to shift from paper to online. We felt we really needed to move on with available technology.

“There will be some reflection on this. Our trainees are very hard working and they’re now very upset.”

Online testing is increasingly popular for industry bodies and government agencies, and is increasingly outsourced to specialist digital testing companies. Apart from Pearson Vue, a new entrant in the online testing market, Janison, raised $10 million in public listing in December, indicating investors are behind the shift to contracted, electronic exams. Read more: http://www.afr.com/news/policy/education/online-exam-for-medicos-ends-in-disaster-and-unleashes-a-twitter-storm-20180220-h0wdaf#ixzz57doDphom Follow us: @FinancialReview on Twitter | financialreview on Facebook

#36

I know this is an old thread but I’d share my sentiments anyway.

I don’t mind if the format is changed as I don’t have to sit for the exams anymore :wink: but I would be really pissed off if CFAI decides to administer multiple tests in a year and allow candidates to take or resit the exams more frequently than it is now. They have to keep the frequency of the exam consistent e.g. as soon as a candidate finds out that s/he fails the L2 exam s/he will have to wait 10-12 months before resitting for the exam or take the L3 exam if s/he passes just like the way it is right now. If CFAI allows a candidate who fails the exam to retake again in just a couple of months I think that would drastically cheapen the designation.

#37

Imagine you come back after lunch to write the second exam, and half the students can’t log on to the exam. So CFAI invalidates the exam and tells everyone to come back in 6 months to resit the exam. How would you feel?

#38

I’m bumping this. Did anyone else see the article by Bloomberg today about the CFA moving to computer based? It was on my LinkedIn page as “CFA was mentioned”. Read it quickly and when I went back to reread I couldn’t find it. Can’t even find it on “The Google”. Legit feel like I’m losing my mind. Someone tell me they saw this too…

#39

Found it. Was business insider

https://www.businessinsider.com/cfa-exam-online-testing-2018-11?r=UK&IR=T

i was way off! Knew it started with a B, though

#LloydChristmas

#40

A feel a computer-based exam is a lot better given certain measures.

When I took the GMAT, there were specific locations you could go and the security was pretty tight. You had to get a fingerprint done, then a handprint, then they went through your IDs. After you put all your stuff in a designated locker, before going into the exam room, they patted your pockets and checked everything you took in the room. I just took a water bottle and two pens, but they actually tore the paper off the water bottle and opened my pens up to make sure I hadn’t hid papers inside the pen (apparently people have done that before as well). Then they made me roll up my sleeves before they finally let me in. Additionally they had cameras over each computer and each room had only 10 computers.

If CFA was to do something like that, where they partnered with only certain locations and then administered proper security, the exam would still be as secure as the paper and pen one right now.

Additionally, they could potentially make the exam run over 2 or 3 days where they have modules (like someone said above) so that topics are tested in detail. They don’t need to get rid of the current format or even the current hardness.

In fact if anything a computer-based test might actually make the CFA more rigorous. If they made it an adaptable test, then it would become even more like the GMAT where the questions adjust based on how you answered the previous one and then instead of just a pass or fail, you actually get a score. Only if you’re above a certain score are you allowed to move on to the next level. If you’re below it, you need to write it again.

This would make the exam more accessible as well. I don’t mean more people will write it, but those who do choose to write it, don’t have to fly 3 hours or drive for 6 hours to get to the nearest test center.

This could be implemented at least for Level I and maybe Level 2. I mean these tests are basically multiple choice only and so a computer-based test would be more rigorous and adaptable and really just make it that much more meaningful.