Sample sizes and caps: how much is enough?

A number of audit methodology providers have recently removed or repositioned the sample cap (‘the cap’) and this is causing some audit firms to recalibrate their sample sizes and question the apparent considerably increased amount of testing required. 

Sampling theory

It is well established in statistical theory that, for certain populations, once you go beyond a certain sample size the level of additional confidence gained is incrementally small, in part supporting the use of a sampling cap. Herein lies the problem. For this element of sampling theory to apply typically requires a large homogenous population. It is not always the case – other than in entities at the larger end of the SME spectrum – that populations being tested display these attributes. In short, if you don’t have a statistically valid population, you are dealing with a judgemental sampling approach and ISA 530 has good guidance on this.

Not all firms use a cap, but some do and in my mind the appeal is a conundrum. When we book holidays or buy goods online many of us scour rating sites for customer reviews; often, we dismiss products with a small number of reviews in favour of those with higher review volumes. We seem to value higher sample sizes in our personal lives, so why not in audit?

Some firms are embracing this changed thinking and I am increasingly being asked: “Do we now have to select more items for testing?”. My answer is typically in three parts: possibly yes, not always and, are you asking the right question? Not to be deliberately unhelpful, but in an effort to get to the best audit testing approach. 

Focussing on an appropriate sample size is, I believe, symptomatic of a historic audit response of defaulting to a fully substantive audit approach underpinned by tests of detail. This was one of the reasons why ISA 315 was recently revised and, in this regard, is the audit firm’s friend. 

Embracing ISA 315 plus ISA 530

Leveraging the combined guidance of ISA 315 and ISA 530, my advice to firms and audit teams is:

• build a good understanding of the audited entity’s business model, the marketplace in which it operates and sector norms;
• form your own rationalised and evidence-based expectations, perform an incisive planning analytical review that allows you to really target your audit resource; 
• consider audit risk on a gross basis and understand the spectrum of risk;
• don’t underestimate the value that flows from the requirement to understand and evaluate the entity’s information technology (IT) and control environment (including IT general controls); and
• based on all of the considerations above, do not default to ‘traditional’ substantive tests of detail. Do have the confidence to consider:

- performing controls testing;

- using data analytics;

- using other computer assisted audit techniques; and

- performing other substantive analytical procedures.

Although there are certain areas where detailed testing is required or needed, using one or more of these techniques can significantly reduce sample sizes for tests of detail and provide robust and cost-effective assurance. 

As accessibility and affordability of data analytics solutions are improving, being able to test 100% of the data and then focus on outliers is becoming a more robust and appealing way forward. The power of data analytics is multiplied further when linked to the comparison of the data sets available through open banking. Now is the time to embrace change.

Tests of detail

If, despite all of the above, you do still find yourself with the need to perform traditional tests of detail, do:

• consider the level of overlapping assurance, if any, from testing performed on other relevant financial statement areas;
• make sure you have a good understanding of the size and nature of the population to be tested; and
• if it is not homogenous, do stratify and separately test items above performance or specific materiality, to leave a residual population.

If the residual population is above performance materiality, do develop a clear substantive testing strategy and clearly define what constitutes a failure and the required response. Be very thoughtful of attributing test failures to ‘isolated events’ and finding reasons to not extend the sample size, ensuring that when this can be justified, the rationale is clearly documented on the audit file. If either the overall or residual population is large and homogeneous, then the sample size may well be higher than what you may have tested previously.

The acid test

Ultimately, and as clearly stated in ISA 530, the determination of sample sizes and a testing approach is a matter of professional judgement (which needs to be clearly evidenced). Sampling calculators and templates are absolutely a significant help. However, for me the acid test is the stand back approach in ISA 315. Does the testing strategy and any resulting sample size make sense and, should the need arise, would you be comfortable defending that?

What’s going on with sample sizes in third party audit software? John Selwood offers some insights.

Rhodri Whitlock, Assurance and Advisory Consultant,HPL Associates