As Artificial Intelligence (AI) becomes embedded in clinical workflows, attention is shifting from capability to reliability. Even well-performing systems can introduce subtle inaccuracies, from fabricated details to missing information, which can impact clinical confidence and patient safety if not properly managed. Understanding and managing these risks is key to ensuring AI supports, rather than compromises, high-quality care.

Across healthcare, where patient safety is paramount, even small discrepancies in clinical documentation can have meaningful consequences. To address this, we have conducted rigorous testing within our AI solutions in healthcare to minimise the risk of hallucinations and omissions, supporting clinicians to spend more time with patients while ensuring they remain in the loop at every stage.

What are AI Hallucinations in Healthcare

AI hallucinations within healthcare settings can be described as content or wording generated by an AI solution that appears in the summary but was not present in the original consultation.

What to Look For:

  • Diagnoses, symptoms, treatments, or decisions that were never discussed in the consultation.
  • Added phrases that imply findings or conclusions not supported by the transcript or conversation.
  • Commentary that is out of context.

Examples of potential AI Hallucinations:

See below for some examples of potential AI hallucinations that could affect clinicians, patients, and families.

Hallucinated Diagnosis

The summary says “patient diagnosed with asthma,” but asthma was never mentioned in the transcript or consultation.

Hallucinated Treatment

The summary includes “started on antibiotics” even though no such treatment was discussed.

Hallucinated Decision

The summary states “referral to cardiology planned” when no referral to cardiology was made during the consultation.

AI Omissions in Healthcare

Omissions made by AI can be defined as clinically relevant information from the consultation that is missing in the summary.

What to Look For:

  • Key symptoms, findings, or decisions that were discussed but not reflected.
  • Missing context that could affect clinical understanding, which could include physical or visual symptoms.

Examples of potential AI Omissions:

See below some examples of AI omissions which could affect clinicians, patients and families.

Omitted Symptom

The transcript mentions “Patient reports chest pain,” but this is not included in the summary.

Omitted Finding

The transcript includes “Blood pressure was elevated”, but this is missing from the summary.

Omitted Decision

The transcript says “Plan to monitor blood glucose weekly,” but it is not reflected in the summary.

Any instance where the AI language model removes or fails to include essential information should be considered an omission. These omissions can be harmful without appropriate human oversight.

The Importance of Quality Control and Minimising Risk

Although infrequent, hallucinations and omissions can undermine a clinician’s trust in AI tools. At System C, we conduct rigorous testing as part of our development process, with a clear focus on safety, accuracy, and real-world clinical use. AI risk management is important, and this testing reflects realistic consultations and outcomes, providing clinicians with confidence in the outputs.

During our tests, we ensured that the AI solution had specific rules for content generation to ensure safe use and good practice. For example, prompts outlined that the AI must:

  • Use English (UK)
  • Not introduce irrelevant or atypical content.
  • Keep expansion realistic: brief clarifications, paraphrasing, simple explanations, recaps, and routine next steps (e.g., investigations, prescriptions, follow-up).
  • Not add new diagnoses or extended small talk purely to lengthen the transcript.

Despite the prompts, it is important to remember that AI is designed to support clinical practice, reducing administrative burden and improving efficiency, not to replace clinical judgement. The responsibility for reviewing and validating outputs must always remain with the clinician.

The Importance of Human-In-The-Loop

We believe that to truly mitigate risks and enhance reliability, all AI-generated content completed during an assessment or consultation must undergo review by a practitioner or clinician prior to finalisation.

This approach ensures that oversight remains central to the documentation process, allowing clinicians to verify accuracy, identify potential errors, and maintain patient safety before any communications or actions are finalised.

The Role of Technology Providers

We have a responsibility as a trusted AI partner to develop safe and responsible AI solutions in healthcare. Safeguarding clinicians and patients is of the highest priority, and we pride ourselves on our research and development techniques to minimise omissions and hallucinations.

Keeping the human in the loop remains the highest priority; this will support the future of healthcare technology by giving clinicians greater control to deliver the best possible care to patients.

Next Steps

Discover our Electronic Patient Record integrated solution, CareFlow Ambient AI – Outpatients, that turns clinic conversations into structured letters, outcomes and tasks in real time.

Explore our extensive collection of blogs to stay informed, inspired and ahead of the curve. Whether you are seeking expert advice, industry trends or thought-provoking perspectives, we have something for everyone.

Please get in touch with our friendly team of experts today. We’re here to assist you with any questions or concerns you may have.