From pens down to results
What happens after your exam paper is handed in? Richard Eckersley, senior assessment manager at ICAEW, reveals how your exams are set and scored
Designing exams is an all-year-round cycle
Did you know that it takes between nine and 12 months to produce a written exam paper? Or that examination development is a continuous exam cycle for ICAEW staff and examiners? In fact, examiners work on as many as three exams at any one time.
Step by step: How paper-based exams are set
- First the topic is discussed in an iterative process within the team for that particular paper.
- It is then moderated to ensure technical accuracy. This stage is completed by high-calibre, newly-qualified volunteer students and an independent technical expert.
- Next, it is ‘worked’ by examiners to ensure questions are testing the skills that chartered accountants need to demonstrate.
- The paper is then reviewed at an all-day exam review board by a group of senior examiners representing all areas of the profession. The board is chaired by the senior moderator.
- Finally, the paper is proof read by the exam development team at ICAEW.
- Each stage is then approved by the senior moderator.
Advanced Level exams go through even more development
The exam-writing process is different for the Advanced Level papers, with additional iterations of these quality assurance mechanisms. The examiners come from a variety of disciplines and also work as a development team and informal review board prior to the more formal review processes, which are applied to all papers.
So the paper has been written, what next?
Every paper undergoes at least three trial sittings by moderators; a combination of technical experts and student volunteers who sit the paper under exam conditions. During this time the papers are proofed and revised to ensure they meet our standards before they’re ready for you to sit.
Where does your exam paper go next?
With the exception of the Case Study, all exam papers are marked remotely on computers. Once you’ve completed the exam, your script will be sent to a scanning house, input into our e-marking system, then accessed by our human markers.
Who are the markers?
They are all experienced experts in their field, as well as being trained in the marking process by the examiners who write the papers and the marking schemes. All candidate names and hall details are removed before they are entered into the system.
How we mark your paper
First the exam team reads and marks a range of scripts chosen at random. They then hold a team meeting, which generally lasts all day. The markers will have familiarised themselves with each question and marked sample scripts. The exam team checks the sample marking and discusses the scripts and the mark scheme in detail with markers. Once markers have shown an accurate application of the mark scheme, they will be approved for ‘live’ marking.
Markers complete the marking at home using the e-marking system. They keep in regular contact with the examiner (and each other) to ensure consistency. The e-marking system also assigns them ‘seeding’ scripts, which have been pre-marked by an examiner.
All markers mark the seeding scripts, and these marks are compared with the examiner’s mark to confirm the consistency of marking across the team. Borderline scripts are always re-marked by one of the examiners for additional assurance. Accuracy and reliability are crucial to the process, as are the tight deadlines to ensure results are ready for final approval by the Assessment Committee at ICAEW. It is responsible for oversight of all qualifications and assessments, and comprises leading academics, senior ICAEW staff and members, and senior representatives from employers and tuition providers.
What about Case Study scripts?
These are delivered to the residential marking venue where we check and sort them ready for marker training, which takes place over a weekend. Initially, the supervisors (highly-experienced markers, who will each lead a team of six markers) meet with the examiners and familiarise themselves with the marking scheme. They all read sample scripts and refine the scheme after further discussion.
What if I’m borderline?
Scripts around the borderline are marked twice or even three times by different members of the team. As all marks given by each marker are referenced to where those points were gained in the script, a second marker can identify areas for improvement and consider carefully the previous marker’s decisions, (with the supervisor’s help where needed). The senior examiners then reread scripts close to the borderline for an additional check and come to an agreement on whether or not they are passes.
This article originally appeared in the January 2014 edition of VITAL.