“Over 50,000 exams delivered in a global pandemic. It wasn’t perfect, but we did our best”

I think for students it was hugely stressful. When lockdown started, quite a few in medium and smaller employers were furloughed, so they had the uncertainty of whether they would have a job at the end of it. They had all the uncertainty of the pandemic, the worry of whether they were going to fall ill, and all the lockdowns and stress the whole population had. Then they had the additional layer of uncertainty and anxiety around whether they would be able to continue their examinations – because there was no certainty about that for several months.

When lockdown started in the UK, we were fortunate in that we had just completed the March exams. Some employers were asking at the time whether we would cancel them, but I’m glad we took the decision to go ahead. The next exams were due to take place in June, which seemed a long way off, but students start studying for the June exams at the end of March or beginning of April, so they needed some clarity. I took the decision early, by the end of March, that we would cancel the June Professional Level exams, and postpone the July Advanced Level to late August.

The second thing we did was immediately start looking at the possibility of remote invigilation. The first six ACA and the ICAEW CFAB exams, which are on demand, can be sat anywhere at any time, and are delivered by Pearson VUE, which already had a remote invigilation solution in place. So with some minor IT adjustments at our end, and once we had approval from the Financial Reporting Council (FRC), we were able to roll that out very quickly. We were running our ACA Certificate Level and ICAEW CFAB exams with remote invigilation from the beginning of May, which was very, very good.

Yes, that was a huge challenge. We had to bolt a third-party invigilation provider onto our existing exam system, and that required a huge amount of IT time and investment. Our exam software is almost five years old and, although it had served us well, we were about to move to a new platform, so there was also a risk that our current provider wouldn’t be in a position to deliver our requirements on the existing platform.

This solution was only temporary until we moved to the new platform, and inevitably when you have two different systems talking to each other there can be problems. Going forward with the new exam software, launching in March, we have an integrated solution. We’ve just had approval from the regulator for this new way of delivering exams – so remote invigilation is very much part of our long-term strategy for all ACA and ICAEW CFAB exams.

The easiest way to run it would have been to say there is just one solution – remote invigilation – and we could focus all our efforts on delivering that. But we talked to students and employers, and they were clear they wanted the option of both, and government guidelines at that time would allow that to happen. That complicated things, because the booking systems for exams sat by remote invigilation and in centres had to remain separate.

It was difficult to forecast how many students were going to opt for remote invigilation and how many for centres, too. We thought the balance would be about 50/50 – and then in reality 70% of students chose centres. Suddenly we had to try and find extra in-centre capacity – not only was our existing capacity dramatically reduced by social distancing, but we also had to find space for those students who were due to sit the cancelled June session.

The September exam session turned out to be the biggest we’ve ever held, double the number of exams we usually expect. Since May last year, we’ve had students sit more than 50,000 exams. And just to give you some context, the size of the logistics team delivering these exams is five! And those five people are having to work really fast under huge stress and strain: making sure bookings are in, and transferring hundreds of students to remote invigilation or different centres with very little notice when a centre was suddenly closed because of lockdown.

Each country where we run exams was also at a different stage of the pandemic, so we had to monitor the situation in each region and adapt the solutions to local restrictions.

Under normal circumstances we would of course have increased the size of the team to help deal with all the new challenges, but the situation and the time pressures made that difficult – especially as we were all working from home ourselves. But luckily we were able to draft in help from other teams within ICAEW.

Yes, unfortunately. We compressed the booking window to one week to enable us to deliver in the shortened timescales. On the first day of booking for the September Professional Level exams, the system collapsed due to the volume on the site. We worked on it overnight but it was still unable to cope the following day. So we delayed the opening until the following week, brought in a new queuing system and increased capacity. We tested and tested until we were happy with the performance – and then on the relaunch day our payment provider’s systems crashed worldwide.

Quite understandably, the students were incandescent. This was the final straw as far as they were concerned: they’ve had everything thrown at them and now they can’t even book their exams. That was definitely one of the lowest points for us at ICAEW. But the improvements we implemented ensured the rest of the booking windows ran smoothly. 

Then there were some problems with the exams themselves, particularly on the first day of the Advanced Level in August, for a small number of students. I can completely understand the frustration: if I’m a student at an exam centre and something goes wrong, this is my career at stake. This is me getting to become a chartered accountant, and at best I’m going to be delayed. At worst I might fail this exam, which means I might lose my job. So we totally appreciate the pressure the students are under.

If students do feel that their exam performance has been adversely affected, we have a process in place called ‘special consideration’. This ensures any issues that may have affected their performance are taken into consideration. It also ensures we capture any problems with the exam centres or remote invigilation and can rectify them as quickly as possible. CABA also provides a lot of support, guidance and information to help students.

You can quite understand it but yes, the volume of complaints has increased. And of course my team wants to respond to them all as quickly as possible, because every student’s experience is important to us. But it creates a tsunami of work, and that has been very intense. I’m just the tip of the iceberg – for the people in the engine room it has been really upsetting, because we’ve run exams for many, many years without issue.

Everyone expects the exams to run perfectly – and they’re right to expect that. We can hand on heart say that we’ve done our best under the circumstances, and the vast majority of students have been able to successfully progress with their studies and stayed safe, which was our key objective.

Keeping everyone informed in a constantly changing landscape has been challenging. We had ongoing conversations with stakeholders, and we created a Coronavirus qualifications hub online that we could update when things changed. We responded directly on social media as quickly as possible, and we ran a student webinar to explain the impact of the pandemic on the exams.

I think my team has done phenomenally well in the circumstances. We lost that one sitting in June, but all the other exams have run. For over 95% of students, it has gone pretty well: they’ve sat an exam, got the result on time and are able to go on with their careers. In this situation, where everything has changed, that’s a phenomenal achievement. Of course, for the other 5%, that’s still an awful lot of students who haven’t had the experience we hoped for. We want to learn from that and continue to improve this through 2021 and beyond.

Yes, it’s taken us a lot of work to get to the point of launching our new exam platform and integrating the data analytics software, which will be happening from March for Audit and Assurance, and July for Corporate Reporting. During the December exams we ran live pilots in the UK and Cyprus using the new platform. The plan eventually is to integrate taxation and accounting software too, but we’re doing it incrementally.

With the new exam software and digital Learning Materials, we are world-leading. Usually I prefer to follow closely behind someone else so that they make the mistakes! But this time we are leading the world; no other institute is as advanced as us. That’s a slightly scary position to be in, but it’s brilliant too. We’ve gone out and tested it with hundreds and hundreds of students, and they’ve all been very positive.

I’m proud of my team. I’m proud of the fact that we have kept the examinations going, and that we have kept the development of the next exam system on track. More than 50,000 exams delivered in eight months in 41 countries is no mean feat. I dialled into a Teams meeting on the last exam of the year in December, and the fact that everyone was still smiling shows me that, despite the year we had, we have retained that great team spirit at ICAEW. We made it through.

Find out more about the changes we have implemented here.