ICAEW’s Code of Ethics includes five fundamental principles and the conceptual framework which requires professional accountants to identify potential threats to compliance with the principles.

ICAEW members must evaluate these threats and to address them by either eliminating them entirely or reducing them to an acceptable level.

Dr Sam De Silva, Partner and Co-Head of the Commercial Practice Group at law firm CMS and a member of ICAEW’s Business Law Committee, believes that ethics can be translated into enforceable promises. This can be achieved, he says, by incorporating the fundamental principles into practical procurement and contract controls.

“Using this approach, the fundamental principles in the ICAEW Code of Ethics drive procurement processes,” he explains. ”The processes subsequently become enforceable clauses in the purchase contract. The result is AI you can explain, defend and adjust when needed.”

When procuring AI, buyers should aim for:

• solutions that can be justified to their stakeholders and clients,
• outputs that can be explained, and
• vendors that can be held to account. 

Applying the fundamental principles

Integrity

“Contractually, you can embed this principle by requiring the supplier to make clear, testable statements, and by giving yourself remedies if those statements prove untrue,” explains De Silva.

These could include:

• inserting clauses requiring clear, testable vendor statements on model provenance, data rights, limits and dependencies; and
• building in duties to notify and correct when facts change, supported by audit and verification rights. 

According to De Silva, it is crucial to link truth to consequences. This could include service credits, price adjustments or the termination of the contract for misrepresentation.

Objectivity

It is well known that AI models may be trained on biased data which may impact on the outputs produced. De Silva suggests including contractual terms which require the supplier to show evidence that its solution has undergone a bias-and-fairness testing programme and which refer to the results in the contract.

He advises that it is important that buyers insists on a documented fairness testing programme, with methods, datasets, metrics and remediation.

In addition, this should be tested with the buyer’s own representative data so that the model performance reflects the buyer’s population, not vendor benchmarks.

De Silva advises buyers to:

• identify and stipulate model-evaluation criteria,
• require explainability, and
• keep a human in the loop with clear escalation paths.

Professional competence

On the principle of professional competence, De Silva advises: “Ensure the supplier provides details in the contract on their governance framework.  They should also include how they classify model risk, manage change, respond to incidents and apply responsible AI principles.”

In addition, buyers should consider governance controls, not just software, but also model risk classification. This includes ensuring that the contract includes provisions for change control, incident response and documented responsible AI policies.

It is important for buyers to anchor vendor commitments to recognised standards and sound SDLC (software development life cycle) practices to keep models safe and reliable.

Crucially De Silva advises buyers to insist that the performance of AI systems is measurable. The contract must include provision for continuous monitoring and reserves step in, or suspension rights, when controls fail.

Confidentiality

De Silva notes that is vital to clarify data boundaries within procurement contracts and to ensure effective control over the organisation’s data with unambiguous terms.

Buyers should limit data use to service requirements under the contract and, in particular, should explicitly prohibit training of general models without specific consent.

Contract terms should mandate minimisation and controlled data retention, with tight limits on storing prompts, outputs and logs.

In addition, contract protections should extend through the supply chain and require robust security, encryption and tenant segregation.

Professional behaviour

De Silva stresses the importance of ensuring that the contract requires compliance with current and evolving laws and guidance. This requirement should be backed by a regulatory change mechanism.

The contract should also:

• define unacceptable uses,
• disclose known harmful failure modes, and
• support impact assessments for high risk uses. 

It is also important that the parties provide for a future responsible relationship exit. The contract should, therefore, provide for data export, provision of model records and include provisions for the secure deletion or return of data.

Applying the conceptual framework to procurement

1. Identifying threats

De Silva advises buyers to surface foreseeable risks early. These included matters, such as: hallucinations, bias, potential IP claims, model drift, security vulnerabilities and dependencies.

In particular, the contract should require the provision of a vendor risk register which maps these risks to specific controls and residual exposure.

It would also be prudent to perform a cross functional review to identify threats like self interest, familiarity and time pressure.

2. Evaluating threats

De Silva advises that risks should be scored and evidence about severity and likelihood should be based on evidence gathered at the piloting stage, rather than vendor assurances.

It is important to document any potential trade offs and compensating controls so that decisions on how to address threats are transparent and defensible.

De Silva advises that buyers should make the “go live” date contingent on passing defined evaluation criteria, not just hitting dates.

3. Addressing threats

De Silva notes that it is important to set defined mitigation actions into contract terms. Examples include bias testing warranties, audit rights and explainability deliverables.

Specific terms should be included which limit automated decision making and require human oversight, and which require change freezes during sensitive periods.

A good tip is to align incentives to ethical outcomes by tying service credits to model performance and control effectiveness.