Clinical Safety Audits

🎉 Pleased to share our paper published in Nature Portfolio digital medicine. 🥳 We’ve developed a comprehensive framework called CREOLA (short for Clinical Review Of Large Language Models (LLMs) and AI). This framework is pioneered at TORTUS, taking a safety-first, science approach to LLMs in healthcare. 🔹 Key Components of the CREOLA Framework -Error Taxonomy -Clinical Safety Assessment -Iterative Experimental Structure 🔹 Error Taxonomy Hallucinations: instances of text in clinical documents unsupported by the transcript of the clinical encounter Omissions: Clinically important text in the encounter that was not included in the clinical documentation 🔹 Clinical Safety Assessment: Our innovation incorporates accepted clinical hazard identification principles (based on NHS DCB0129 standards) to evaluate the potential harm of errors: We categorise errors as either ‘major’ or ‘minor’, where major errors can have downstream impact on the diagnosis or the management of the patient if left uncorrected.  This is further assessed as a risk matrix comprising of: Risk severity (1 (minor) to 5 (catastrophic)) compared with Likelihood assessment (very low to very high) 🔹 Iterative Experimental Structure We share a methodical approach to compare different prompts, models, and workflows. Label errors, consolidate review, evaluate clinical safety (and then make further adjustments and re-evaluate if necessary). ----------Method-------------- To demonstrate how to apply CREOLA to any LLM / AVT, we used GPT-4 (early 2024) as a case study here. 🔹 We conduct one of the largest manual evaluations of LLM-generated clinical notes to date, analyzing 49,590 transcript sentences and 12,999 clinical note sentences across 18 experimental configurations. 🔹 Transcripts-clinical note pairs are broken down to a sentence level and annotated for errors by clinicians. ----------Results-------------- 🔹 Of 12,999 sentences in 450 clinical notes, 191 sentences had hallucinations (1.47%), of which 84 sentences (44%) were major. Of the 49,590 sentences from our consultation transcripts, 1712 sentences were omitted (3.45%), of which 286 (16.7%) of which were classified as major and 1426 (83.3%) as minor. 🔹 Hallucination types Fabrication (43%) - completely invented information Negation (30%) - contradicting clinical facts Contextual (17%) - mixing unrelated topics Causality (10%) - speculating on causes without evidence 🔹 Hallucinations, while less common than omissions, carry significantly more clinical risk. Negation hallucinations were the most concerning 🔹 we CAN reduce or even abolish hallucinations and omissions by making prompt or model changes. In one experiment with GPT4 - We reduced incidence of major hallucinations by 75%, major omissions by 58%, and minor omissions by 35% through prompt iteration Links in comments Ellie Asgari Nina Montaña Brown Magda Dubois Saleh Khalil Jasmine Balloch Dr Dom Pimenta M.D.

🔹Surprisingly Rich FDA Case Study : 🧰 #HERO : #BIMO Observation #3 : Delayed Reporting of an Adverse Event of Clinical Interest🔹 From June 22 through July 16, 2020, FDA inspected Site 2004, a Colorado urology research site participating in the #HERO trial. 📘 HERO was a Phase 3 study comparing relugolix with leuprolide in patients with advanced prostate cancer. NDA 214621, Study MVT-601-3201, NCT03085095. Sponsor / Applicant: Myovant Sciences GmbH. FDA selected three participating clinical investigators and the study sponsor for inspection. Site 2004 was one of the inspected clinical sites and received Form FDA 483. Among the findings documented in Form FDA 483 was BIMO Observation #3. FDA reported that a subject receiving relugolix had ALT 134 U/L at Week 49, with a laboratory reference range of 6-43 U/L. Under Section 7.5 of the protocol, an ALT elevation of ≥3× ULN met the protocol definition of an Adverse Event of Clinical Interest (#AECI) and was required to be reported to the sponsor within 24 hours. According to FDA, this reporting requirement was not met. The event was reported approximately 8 weeks later. For a clinical research site, this observation extends beyond one laboratory value. AECI reporting requirements are protocol-defined safety controls. They are intended to ensure that potentially meaningful safety signals reach the sponsor rapidly enough to support ongoing safety surveillance and medical review. According to the FDA inspection record, the Site 2004 Principal Investigator acknowledged the observations and provided #CAPA plans. For Observation #3, FDA noted that the delayed AECI reporting was related to a change in Study Coordinator and lack of awareness of the reporting requirement. The Site 2004 Principal Investigator stated that he followed the subject’s hepatic test abnormality until it returned to normal. FDA also described additional corrective actions, including research staff training, Transition of Study Checklist, Implementation of QC Checklist, Roles and Responsibilities of the Research Staff, and additional training on Following the Protocol. This is the practical regulatory lesson. The issue was not the ALT value itself. The issue was inadequate oversight of a protocol-defined safety reporting control. In clinical operations, safety oversight depends not only on identifying a signal, but also on ensuring that the signal is escalated to the sponsor within the timeline required by the protocol. That is why this FDA case is useful. It shows how a missed AECI reporting requirement can become an inspectional observation when protocol awareness, staff transition, QC checks, and safety escalation do not function as intended. #GxPEssentials #FDAInspection #ClinicalDevelopment #ResearchAndDevelopment #Biotech #Biopharma #ClinicalTrials #ClinicalResearch #ClinicalOperations #DrugSafety #AdverseEvents #AdverseEventOfClinicalInterest #SafetyReporting #InspectionReadiness #FloorSoldiersStrategy🌸🌿

Stillness in Audits: Why Calm Observers Find the Real Gaps In my experience, the loudest voice in an audit is rarely the one with the clearest view. The best auditors and the best-prepared teams are often the calmest ones in the room. Stillness creates space to: ✅ See beyond the documents into what’s happening ✅ Hear what’s not being spoken ✅It’s about noticing what’s truly going on behind the scenes instead of simply rushing to meet requirements When you apply the three principles, see beyond the documents, hear what is not being said, and notice what’s unfolding, it can look like this: - Design documents: Check if the design still reflects the original purpose and patient needs, not only for the sake of the latest version on file. - Risk assessment files: Pay attention to risks that are glossed over or quietly avoided in discussions. - GSPRs: Evaluate whether the evidence clearly demonstrates compliance rather than simply providing a reference. - Checklists: Understand if the actions listed are genuinely being followed in practice. - DHF: See if updates reflect actual design changes or if they were added later to make the file look complete. - Recently approved products: Make sure the marketing and instructions are consistent with what was cleared. - Products with recent changes: Confirm that the impact on safety, performance, and approvals was considered in full, not merely recorded for the sake of formality. Calm observation often reveals the issues that truly matter for patient safety, trust, and long-term compliance. There’s a saying I like from the Tao: "muddy water clears up if you just let it be". In MedTech, that reminds me not to rush or panic when facing audits. Taking a moment to breathe and really looking can help us find the real issues, the ones that matter most for keeping patients safe and building trust. Here’s something to think about: When an audit comes, am I scrambling to protect myself or staying grounded enough to see what needs attention? #MedTechCompliance #MedicalDeviceRegulations #MedTechQuality #LeadershipWithIntegrity #QualityCulture #LeadWithPurpose #MindfulLeadership #IntentionMatters #PurposeInBusiness #Elexes

Compliance Wednesday FDA recently issued a Warning Letter to a Clinical Investigator, underscoring critical lapses in clinical trial conduct, emphasizing the paramount importance of adhering to investigational protocols to safeguard participant rights and data integrity. Key Findings: • Protocol Deviations: Enrollment of ineligible subjects and failure to conduct required in-person visits compromised study validity and participant safety. • Inadequate Documentation: Missing informed consent forms and incomplete records highlight deficiencies in maintaining essential study documents. • Delayed Reporting: Failure to promptly report adverse events impedes timely risk assessment and necessary interventions. This case serves as a reminder of the ethical and regulatory obligations inherent in clinical research. Strict adherence to study protocols, meticulous documentation, and timely reporting are of critical importance to ensure participant safety and data credibility. Upholding these standards is not just about regulatory compliance but is fundamental to advancing medical knowledge and protecting those who volunteer for research. #FDA #GCP #ClinicalResearch #GCPCompliance #PatientSafety #RegulatoryAffairs #FDAWarningLetter

Key Aspects of Internal Audit as per NABH: Scope: Internal audits in NABH-accredited healthcare facilities cover a wide range of areas, including: Quality management system Patient safety Infection control Human resources Infrastructure Clinical services Objectives: To assess compliance with NABH standards and other relevant regulations To identify gaps and non-conformities To evaluate the effectiveness of risk management systems To recommend improvements for continuous quality improvement To prepare the organization for external NABH assessments Frequency: Internal audits are typically conducted at regular intervals, such as quarterly or semi-annually, to ensure ongoing monitoring and compliance. Methodology: Internal audits follow a systematic approach, often involving: Planning: Defining the scope, objectives, and resources for the audit Review: Examining documents, records, and processes Observations: Observing activities and practices on-site Interviews: Talking to staff and stakeholders Findings: Documenting audit findings, including non-conformities and recommendations Corrective Actions: Non-conformities identified during internal audits must be addressed with appropriate corrective actions. These actions should be documented and verified to ensure their effectiveness. Continuous Improvement: The findings of internal audits are used to drive continuous improvement initiatives within the organization. This may involve updating policies, procedures, or training programs to address identified weaknesses. Benefits of Internal Audits: Improved Quality of Care: By identifying and addressing gaps, internal audits help ensure that healthcare organizations provide high-quality care to patients. Enhanced Patient Safety: Internal audits can help identify and mitigate risks to patient safety, such as medication errors or infections. Regulatory Compliance: Adherence to NABH standards through regular internal audits can help healthcare organizations avoid regulatory penalties and maintain accreditation. Cost Reduction: By identifying inefficiencies and waste, internal audits can help healthcare organizations reduce costs and improve operational efficiency. By conducting regular and effective internal audits, healthcare organizations in India can demonstrate their commitment to quality, safety, and compliance with NABH standards. #shalby #NABH #JCI #AACI #PatientSafety #Healthcare #Excellence #AkashTiwari