Introducing an AI tool into a learning, health or community programme raises the same safeguarding and data protection questions as any new way of working with learners, patients or community members, just applied to a new channel. This checklist sets out what to work through before you start, with links to the relevant UK regulator guidance. It's written to be used whichever AI tool you're considering, including ours.
This is general guidance to help you plan, not legal advice. Follow your own organisation's policies, and take your own advice where you need to.
Start with a data protection impact assessment
Under UK GDPR, a data protection impact assessment (DPIA) is required where processing is likely to result in a high risk to people's rights, and the Information Commissioner's Office is explicit that new or innovative technology is one of the triggers for carrying one out (ICO: when do we need to do a DPIA?). Introducing an AI tool that processes personal data from learners or service users is a reasonable trigger to complete or update one, and the ICO's wider guidance on AI and data protection (ICO: guidance on AI and data protection) is worth reading before you assess a specific product.
Involve your data protection lead or officer early rather than at the end. A DPIA done properly asks what could go wrong for the individual, not only for the organisation, and it should be revisited if the tool's use changes, for example if it moves from one class to the whole provision.
Questions to put to any AI vendor
- What personal data does the tool collect, where is it stored, and for how long?
- Will the vendor support your DPIA and answer specific data protection questions, rather than pointing to a generic policy?
- Who can access conversation data, and is it used to train models beyond your own deployment?
- What happens to the data if you end the contract?
- Is there a data processing agreement, and does it name any sub-processors involved?
- Can you export or delete an individual's data on request?
Age and consent
Confirm the ages of the people who will use the tool, and check whether any of them are under 18. Programmes working with under-18s need a clear position on consent, on how the tool is introduced to them, and on what a parent, carer or guardian is told. If your organisation has an existing policy for introducing new digital tools to under-18s, apply it here rather than treating AI as a special case, and make sure whoever holds that policy has actually seen and signed off this specific tool.
Content rules and escalation routes
Agree, in writing, what topics the assistant will and won't discuss, and what it should do when a conversation moves into safeguarding territory, such as disclosure of harm, abuse or a medical emergency. There should be a clear, simple escalation route from “the tool spotted something concerning” to a named person in your organisation, tested before launch rather than assumed. Health-related tools in particular should be explicit that they give general information, not a diagnosis, and should say so to the user.
Running and evaluating a pilot
Before wider rollout, agree a small, scoped pilot: one class, one cohort or one site, with success measures agreed up front rather than decided afterwards. Build in a fixed review point to look at what happened, including anything that went wrong, not only what went well. A pilot that can't show you anything concrete about safeguarding, data handling or usefulness to your learners hasn't done its job, whatever the vendor's claims.
Ask specifically, at the review point: did any conversation raise a safeguarding concern, and was it escalated the way you agreed? Did the vendor answer your data protection questions promptly and in plain language? Would the people who actually used it, learners, patients or staff, choose to keep using it? Those three questions tell you more than a usage count on its own.
A short pre-launch checklist
- DPIA completed or updated, specific to this tool
- Vendor's data handling and retention confirmed in writing
- Under-18 use identified and a consent position agreed
- Content and safeguarding escalation rules written down and tested
- Pilot scope and success measures agreed and dated
- A named person responsible for reviewing the pilot
- Data processing agreement reviewed by whoever holds that responsibility in your organisation
- Under-18 policy owner has signed off, if relevant