Answers can be corrected. Actions must be answered for.

2024 BCCRT 149

A British Columbia tribunal found the airline owed a duty of care for what its website chatbot told a passenger: a company must “take reasonable care to ensure their representations are accurate and not misleading” (para 26), and ordered compensation.

The tribunal characterised Air Canada's submission as suggesting, in effect, that the chatbot was responsible for its own actions, called that suggestion remarkable, and rejected it.

A chatbot only spoke. Agents act. The same principle, applied to systems that move money and delete records, is a different order of exposure.

The question that matters is not whether the model is good. It is what stands between a proposed action and its execution, and whether the organisation can show what stood there.

That the answer today is usually “nothing”. That regulators are already asking for evidence to the contrary. And that safety engineering settled the pattern for this problem decades ago.

logging and deployer oversight

Regulation (EU) 2024/1689. Article 12(1) requires that high-risk AI systems technically allow “the automatic recording of events (logs)” over the system’s lifetime, and Article 12(2) ties that logging to the deployer’s own operational monitoring.

Article 26 puts duties on the deployer directly: human oversight assigned to named people with competence and authority (Art. 26(2)), monitoring in operation with a duty to suspend where risk appears (Art. 26(5)), and retention of the automatically generated logs for at least six months (Art. 26(6)).

The timetable was amended by Regulation (EU) 2026/1744, in force 27 July 2026: high-risk obligations for standalone Annex III systems now apply from 2 December 2027, and for Annex I product-embedded systems from 2 August 2028.

Many summaries still cite the superseded 2026 dates. Read the amended Article 113.

responsibility does not outsource

Regulation (EU) 2022/2554, applying since 17 January 2025. Article 28(1)(a) is blunt: a financial entity using ICT services shall “at all times, remain fully responsible” for discharging its obligations under the regulation.

Whatever an AI vendor does or fails to do, the burden of showing control stays with the deploying institution, which is why Article 30(3) writes access, inspection and audit rights into every contract supporting a critical or important function.

outcomes, not intentions

FCA policy statement PS22/9 (27 July 2022) made Principle 12: firms must act to deliver good outcomes for retail consumers, structured as four outcomes — products and services, price and value, consumer understanding, consumer support.

The Duty has applied to open products since 31 July 2023 and to closed books since 31 July 2024, and the FCA’s follow-on supervisory work centres on outcomes monitoring. An outcome an AI produced is still an outcome the firm must stand behind.

model risk, including AI

The PRA’s supervisory statement on model risk management, effective 17 May 2024, current version published 23 April 2026. It sets five principles: model identification and risk classification; governance; development, implementation and use; independent validation; and risk mitigants.

It applies to UK-incorporated banks, building societies and PRA-designated investment firms holding internal model approval. Its sub-principles reach the use of AI and machine learning in modelling explicitly, and allocate responsibility for the framework to a named senior manager. It states supervisory expectations, not statute — but expectations a supervisor examines against.

One way to watch an agent is to ask another model whether the action looks right. That yields an opinion, often a good one. But it is a sample from a distribution: ask twice and you may get two different answers.

Ask again after an incident, in front of an auditor, and you cannot reproduce the judgement your system made at the time.

Its failure mode is a confession too. A sampling layer that times out must choose between blocking work and waving the action through, and throughput usually decides.

A deterministic check is a different object. Same input, same verdict, every time. Run the check again next month, or next year, on the recorded input, and it answers the same: the judgement is reproducible, not merely asserted.

And the failure mode is stated in advance, in writing. Fail-open assumes the unexamined action is safe. We fail closed: when the governance layer cannot decide, the action is refused, not waved through.

An action is admissible when it has been examined and admitted before it executes, or it is refused there, not flagged afterwards.

A refused action is recorded with the reason it was refused.

A record is admissible when it survives scrutiny it was not designed around.

Every governed action is sealed in a signed, tamper-evident record, and the evidence a period produces can be handed to a third party and verified with Admissible absent.

This website is held to the same standard as the platform.

Demo Company Limited is registered in England and Wales, company number 00000000.

Every factual sentence about our capabilities is tied to a named verifier, and a sentence that fails verification is removed at build.