A system achieving 95% accuracy against a doctor’s 90% might not necessarily perform better in clinical settings. | Photo Credit: Reuters Hospitals worldwide are ill-equipped to assess whether artificial intelligence (AI) systems deployed in their facilities actually improve patient outcomes, raising questions about the clinical and economic value of healthcare AI, Emmanuel Bacry, Scientific Director of France’s Health Data Hub, said at the Bengaluru Tech Summit.“There are maybe six or seven hospitals in the whole world today that are able to evaluate medical devices deployed in their system,” Mr. Bacry said. Hospitals struggle to access and integrate patient information to track outcomes after discharge, including recovery rates and mortality, making it difficult to establish the actual impact of AI-based medical technologies.“In hospitals, we don’t know how to compute that,” he said, referring to the return on investment of such technologies. “So imagine now, if they adopt AI.”Unintended consequencesMr. Bacry cited an alert system based on rules-based AI, or an expert system, designed to warn intensive care unit (ICU) staff about severe kidney failure. Despite functioning effectively, the system was associated with a doubling of deaths in two of the six US hospitals where it was deployed in 2022, he said. He did not identify the study.While researchers had offered hypotheses, the precise cause remained unknown. Mr. Bacry suggested that the absence of a tested response protocol among hospital personnel could have contributed to the outcome.“We just replaced the alert done by a human with the alert done by the medical device. And that changes the whole thing,” he said, underlining the importance of integrating AI into clinical workflows rather than treating it as a standalone technology.He also cautioned against evaluating diagnostic AI tools solely through laboratory accuracy scores or comparisons with doctors. A system achieving 95% accuracy against a doctor’s 90% might not necessarily perform better in clinical settings. “When you deploy in real life, you can forget about these scores,” he said.The black-box dilemmaMr. Bacry flagged another concern: AI systems can identify patterns that even their developers cannot explain. Citing research, he said one model predicted ethnicity from chest X-rays with approximately 85% accuracy, while another inferred sex from eye images with 95% accuracy.Such hidden correlations could influence medical decisions without clinicians understanding the underlying basis, potentially introducing unintended bias into diagnosis and treatment.Western bias in healthcare datasetsIn a separate interaction with The Hindu, Mr. Bacry warned that healthcare AI development was heavily skewed towards Western populations. By his estimate, nearly 70% of global healthcare AI training data came from male, white and Western populations.“We are developing an Occidental AI and even narrow types of Occidental AI,” he said.He advocated smaller, locally tailored AI models trained on representative population data, arguing that deploying large, general-purpose models without adaptation could have disastrous consequences, particularly in developing countries.Mr. Bacry heads the Health Data Hub, established under French law in 2019 to facilitate secondary use of health data for research. He is also a research director at France’s National Centre for Scientific Research (CNRS) at Université Paris Dauphine–PSL. Published - November 20, 2026 04:19 pm IST
Healthcare AI: hospitals lack capacity to measure real-world patient outcomes, warns French expert
Full Article
Original Source
Read the full article at Thehindu →KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.