Artificial intelligence

When the University of Illinois Hospital and Health Sciences System was testing an artificial intelligence-backed tool that drafts responses to messages, a patient misspelled the name of a medication, Karl Kochendorfer, chief health information officer, recalled during a panel at the HLTH conference in October 2024.

The mistake led the AI to give side effects for a drug the patient wasn’t using when a nurse forgot to double-check the response.

Ultimately, it wasn’t a huge issue — they just needed to call the patient or send another message to issue a correction, he said. But it could have had serious implications for the tool.

“It almost killed the pilot. [...] And it happened on day one,” he said.

As healthcare grapples with how to safely implement AI, investors and health systems are first seeing promise adopting tools that automate administrative and back-office work, which could make a dent in provider burnout and pose fewer risks to patient care, experts said at the HLTH conference.

But the pressure is on to adopt the tech. Proponents argue AI could help solve healthcare’s significant workforce challenges: The nation faces a shortage of more than 100,000 critical healthcare workers in 2028, as the overall population ages and needs more care, according to a report by consultancy Mercer.

While AI could be transformative, the sector has to move with caution as it implements emerging tools, experts say. The stakes are high, as policymakers and experts have raised concerns about accuracy, bias and security. 

Implementing AI in healthcare is complex, a lesson the industry should take from some predictive tools that were previously deployed, said Rohan Ramakrishna, co-founder and chief medical officer at health information app Roon, during a panel at HLTH.

“I think one of the things we’ve learned is that you have to be exceedingly careful applying AI solutions in healthcare settings,” he said.

How AI could help a ‘simple mismatch between supply and demand’

AI could help alleviate one of the biggest problems in healthcare: a growing number of older patients with more complex, chronic conditions and fewer providers who can help them, Daniel Yang, vice president of AI and emerging technologies at Kaiser Permanente, said during a panel discussion. 

As older Americans need more care, millennials, the largest generation in the U.S., want a more on-demand consumer experience, he added. But that will be difficult to achieve with supply constraints. It takes years to train a new doctor, and fewer physicians means more burnout, care delays and higher costs.

“It’s not even about AI, it’s really about what I think ails healthcare in general,” Yang said. “What we’re seeing is a simple mismatch between supply and demand.”

AI could augment clinicians’ workflows, potentially helping them offer better care. Yang said an algorithm developed by researchers at the Permanente Medical Group helps save 500 lives each year by flagging patients who were at risk of clinical decompensation, or when their condition worsens.

The technology could also be a way to lessen burnout and improve retention among clinicians by lessening the time they spend on administrative work such as note-taking. Providers have long reported they spend hours on work in their electronic records, often to the detriment of patient care. 

Christopher Wixon, a vascular surgeon at the Savannah Vascular Institute, said he was close to leaving medicine as the industry shifted to electronic health records. Collecting information while trying to listen to patients was a challenge, and it was easy to miss out on non-verbal cues when he had to focus on a laptop screen.

But ambient documentation, typically where an AI tool records conversations between clinicians and patients and then drafts a note, has changed the game, he said. 

“It’s better for me because it does save you time,” Wixon said during a panel. “But at the end of the day, it’s better for the patient because they feel like they’re being heard. It truly has been a transformative experience for me.”

Investors, health systems focus on administrative burden

Given providers’ heavy administrative workloads and worries about errors or bias in models that are more involved with clinical decision making, products that solve administrative concerns are some of the top priorities for AI adoption.

Some investors are more interested in automating those operational tasks too.

“We’re going to continue to focus on the unsexy back-office automation that’s going to alleviate the workforce burden, truly, and nothing related to clinical as it relates to AI,” said Payal Agrawal Divakaran, partner at .406 Ventures.

Administrative AI is also bringing in more venture capital dollars, according to a report published by Silicon Valley Bank. Up until October 2024, administrative AI companies had raised $2.4 billion, compared with $1.8 billion for clinical AI, likely because of lower regulatory and institutional hurdles, especially for decision support tools. 

“Where we’re seeing all that action is on administrative tasks, but also on pre-authorization, more low-value tasks,” Megan Scheffel, head of credit solutions for life science and healthcare banking at SVB, said in an interview. “You could take your office staff and move them to higher value projects.”

A lot of opportunity exists now to use large language models as a drafting tool, including for notes or a nurse handoff document, Greg Corrado, distinguished scientist and head of health AI at Google Research, said during a panel. 

The oversight is built in because providers will have to review the output before finalizing. It’s also easier to evaluate the quality by asking users about their experiences or checking how many revisions they had to make, he said. 

But you still have to be methodical when evaluating operational or administrative tools, testing with the health system’s local patient data, said Todd Schwarzinger, partner at Cleveland Clinic Ventures, in an interview. Cleveland Clinic’s governance structure also focuses on questions about data used in the tool, how that information is protected and whether the product is safe and positively impacts patient care. 

Administrative or operational products, like ambient scribes or revenue cycle management tools, are a safer place to start.

“You’re not running the risk of making clinical decisions, right?” he said. “That level of trust is not there. I think it’s going to take time.”

Preparing for AI deployment

Though AI could show promise at alleviating provider burnout, health systems face challenges to get providers on board, set up pilots, establish governance policies and roll the products out. 

In one example shared at HLTH, St. Louis-based provider BJC decided to use vendor data to identify clinicians who took many days to sign off on documentation or wrote long notes for an ambient note-taking pilot, said Michele Thomas, associate chief clinical information officer for ambulatory at BJC Healthcare and chief medical information officer at BJC Medical Group. 

“I think we had one person out of 20 respond. So right away we’re thinking the people who need it the most aren’t interested in that,” she said during a panel.  

They decided to switch course, inviting anyone interested to join the pilot — prompting a quick response, Thomas said.

Still, it’s important to think through which providers should participate in testing, as some doctors didn’t understand the requirements to participating in a pilot, such as the need to spend time sending feedback or managing frustrations with a new product. 

Health systems should also consider what outcomes they want when adopting AI tools. A lot of physicians didn’t see much time savings because they spent a lot of time editing their notes — not correcting them — to make notes in-tune with their writing style, Thomas said. In contrast, advanced practice providers were quicker to sign off on the ambient notes after reviewing. 

“You really have to decide what your ROI is. Are you looking for a financial savings? Are you looking for time savings? Are you looking for hard numbers that justify this technology? Or is it softer than that? Are you looking for patient satisfaction?” she said.

Cybersecurity — already a challenge for the healthcare sector more generally — is also key to AI deployment.

Organizations should think through the same questions they’d ask any system that uses protected health information, Melanie Fontes Rainer, director of the HHS’ Office for Civil Rights, said in an interview. 

If they enter into a relationship with a developer, do they have a business associate agreement in place? Are they thinking about data deletion policies for information stored in the cloud? Are they considering who needs to have access to the data?

“I think there’s certainly a balance to walk here, but it requires us all to be responsible and think about how we’re using this information and how it might harm our systems, our patients, and how we can take proactive steps to protect it,” she said.

Lightning-fast evolution in artificial intelligence and growing adoption of the models is giving rise to concerns that AI could exacerbate existing divides between technological haves and have-nots in the healthcare industry.

But all types of providers — regardless of location or finances — can adopt AI, experts said in March 2025 at the HIMSS conference in Las Vegas.

It’s just a question of knowing how.

Rural hospitals, smaller systems and health clinics without a massive IT infrastructure should reach out to AI companies to discuss potential partnerships, said Graham Walker, co-director of advanced development at Kaiser Permanente’s medical group, during a panel.

Providers can get access to AI products in return for use of their facility as a testing ground, Walker said. That’s valuable for AI developers, who want to train and test their models and prove that they actually work in the real world.

“A lot of the startups and smaller companies want some access to health systems. It’s much harder to get into large health systems than a solitary hospital. And so you can very well imagine a world where there’s some mutual benefit there,” Walker said.

Smaller providers can also band together to research and share information on tools that work for them, said Brenton Hill, head of operations at the Coalition for Health AI, a health AI standards consortium. They can explore joint purchasing arrangements, or benefit by knowing which AI products have already been validated by other hospitals with more resources.

“The more people that you can get to buy into a concept, come together and do it, the better price you’re going to get, the more leverage you’re going to have,” Hill said. “We have 1000-plus hospitals that need a solution. Can we identify what they need, prioritize that and then partner with a vendor and come together and do this so it’s mutually beneficial?”

And smaller providers almost benefit from their lack of existing partnerships with tech companies, said Tanay Tandon, the CEO of provider automation company Commure. Without a relationship with a vendor, no one is nudging them toward a specific product or fix. Instead, they’re unencumbered to explore a variety of solutions and find what works best for them.

“Some of the best AI tools out there are free and accessible online,” Tandon said. Because of that,“we’ve seen physicians all over the world in rural clinics get access to ambient documentation,” which listens and records patient visits.

Similarly, price shouldn’t be a barrier for smaller providers much longer, according to Tandon.

The introduction of more low-cost AI models — including DeepSeek, a Chinese model with performance metrics akin to those released by leading U.S. companies, but offered at a fraction of the cost — means these AI tools will likely get cheaper, the CEO said.

Yet it’s important that providers put in the work ahead of time to have a clear view of the problem they’re trying to solve, whether that’s improving billing processes, prior authorizations or patient triage, said Brian Spisak, program director of AI and leadership at Harvard University’s National Preparedness Leadership Initiative. That way, providers can be ready to jump on inexpensive and helpful AI products when they emerge.

As for AI vendors, companies should also be open to exploring deals with smaller providers — and should view those as a partnership, not a sales relationship, given the importance of ameliorating the numerous struggles providers face, experts said.

Current challenges for U.S. providers include labor shortages, inflation driving up the cost of medical supplies and inadequate insurance rates for some patient care. Most of these are trickier for rural facilities to navigate. And such facilities are already used to being left behind when it comes to technological advances — though some research suggests the digital divide may be shrinking, mostly due to improved access to the Internet in rural areas, the cost of technology, insufficient broadband access and poorer-performing devices have proved perniciously difficult to overcome.

As a result — and because of rural regions’ less-than-stellar history with technological innovations putting many Americans out of jobs — the tech industry should treat facilities in rural areas with extreme care and tact, according to Tandon.

“When you’re in Silicon Valley, the approach to technology and to some extent the conversations about technology come from a place of rapid iteration, speed, hope — you know, this is a force for good. When you go out to middle America one of their core interactions with technology was losing the manufacturing industry. Automation — it wrecked the region,” Tandon said. With AI, “we have to get it right.”

Sponsored content

By Microsoft

Healthcare organizations today face immense pressure to deliver high-quality care while managing increased demand, chronic staff shortages, and a growing epidemic of clinician burnout. These challenges impact care quality, patient safety, and financial stability, with many organizations struggling to break even.

AI is already making a difference by automating administrative tasks and uncovering diagnostic clues, but there is much more potential to be realized. This guide explores five essential areas for healthcare organizations to consider when developing a generative AI strategy and evaluating vendors.

1. AI tailored for healthcare workflows

Generic generative AI models offer powerful data analysis and response generation, but they must be fine-tuned for the complex realities of healthcare. Solutions should fit seamlessly into clinicians’ daily workflows to deliver meaningful value and avoid unintended harm. Fine-tuning also reduces the risk of “hallucinations” or inaccurate outputs. Safe, effective AI in healthcare requires training on real clinical tasks.

What to look for:

• Vendors with a proven track record of trusted technology in clinical workflows.
• Solutions that address high-value, low-risk use cases, such as content generation and text summarization.
• A “human in the loop” approach, where AI supports decision-making, but clinicians remain in control.

2. A responsible approach to AI

Handling sensitive patient data and making life-changing decisions demands a responsible approach to AI development and deployment. Ethical guidelines and robust governance must be built into solutions to protect patient data, filter harmful content and prevent misuse.

What to look for:

• Vendors with comprehensive responsible AI practices focused on fairness, reliability, safety, privacy, inclusiveness, transparency and accountability.
• Solutions with best-practice guidelines and built-in safeguards.
• Transparent data usage and enterprise-grade security and privacy.

3. Deployment and optimization expertise

With hundreds of generative AI applications available, organizations must prioritize solutions that can be effectively deployed and deliver rapid, valuable outcomes. Integration with EHRs, PACs and other core systems is essential, as is scalability across large organizations. Change management support from vendors is critical for successful adoption and ongoing optimization.

What to look for:

• Vendors with a history of successful large-scale healthcare deployments and partnerships with leading technology providers.
• Change management and ongoing support to maximize adoption and long-term value.
• The ability to measure and demonstrate clinical and business impact.

4. Enterprise-grade dependability

Innovation in healthcare must not compromise stability. Solutions must offer enterprise-grade availability, reliability, security and compliance to maintain care quality and patient safety. Vendors should keep pace with rapid technological evolution and orchestrate various AI technologies at scale.

What to look for:

• Robust security, governance credentials and stringent availability SLAs, supported by global cloud infrastructure.
• Continuous evaluation and advancement of underlying technologies.
• Flexible solutions that adapt to emerging capabilities across AI and related technologies.

5. Deep healthcare experience

High-impact AI applications are built on a deep understanding of healthcare challenges and priorities, with input from both leadership and frontline teams. Effective innovation requires strong partnerships across the technology stack and direct involvement of practicing clinicians in development.

What to look for:

• Vendors with extensive experience supporting healthcare organizations and partnerships with research institutions, technology providers and clinical practitioners.
• Solutions developed with clinician feedback to address real problems and fit into daily workflows.
• Interoperable capabilities that improve patient care and business performance.

AI will play a transformative role in healthcare’s future. To maximize its benefits and minimize risks, responsible vendors must collaborate closely with healthcare providers and partners. Proven solutions, such as those from Microsoft for Healthcare, are already helping organizations achieve better outcomes through responsible, scalable and innovative AI.

To find out how AI can help you deliver high-impact healthcare outcomes, visit aka.ms/healthcarefuture

©2025 Microsoft Corporation. All rights reserved. This document is provided “as-is.” Information and views expressed in this document, including URL and other Internet website references, may change without notice. You bear the risk of using it. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

Health records companies are racing to weave artificial intelligence into multiple segments of their software, hoping that by streamlining clunky experiences they can attract clients and keep up with competitors doing the same.

Some of the largest EHR vendors, including Epic and Oracle, have announced or accelerated modernization efforts in recent months. The companies say they aim to increase the productivity of a healthcare system facing rising costs — while improving job satisfaction for doctors and nurses, increasing the quality of patient care and advancing the state of the art of medicine.

The shift represents a sea change for a software often cited by providers as a significant source of frustration.

“Doctors are not looking for AI to act as a doctor, to step in their place,” said Leigh Burchell, chair of the Electronic Health Records Association. “They really want to take the things that are administrative burdens, or that take them time after a visit to document, and get help with that. So those are really the areas where our members are most focused.”

Marrying medicine and technology

EHR software allows clinicians to access and update a patient’s complete medical history electronically, including diagnoses, lab results and medications. Healthcare providers in the U.S. were required to adopt EHRs more than one decade ago in order to continue receiving full funds from federal programs like Medicare and Medicaid. Use of the software spiked, with 78% of physicians and 96% of hospitals using an EHR in 2021, according to government data.

However, doctors often express annoyance with EHRs. The systems can be tricky to use, cluttered with pop-ups and screens and requiring significant time for clinicians to enter and find data. Another common criticism is that EHRs were designed more for billing purposes than for clinical decisionmaking. Many doctors say the systems distract from patient care as a result.

Major EHR vendors have long said they’re working to make their products simpler and easier to use. That crusade is being significantly accelerated by recent advances in AI: like generative AI, which can understand and create original text and images; ambient listening, which marries microphones and machine learning to produce transcripts of clinician-patient encounters in real time; and agentic AI, which can complete complex tasks largely without human oversight.

In the past, navigating EHRs required a lot of data entry and “persistent screen after screen navigation,” Girish Navani, the CEO of EHR vendor eClinicalWorks, said. “If you look at the last two years, the advent and the rapid rise of generative AI makes a lot of those paradigms obsolete.”

Epic, the largest EHR vendor in the U.S., has “been working to incorporate in particular generative AI across the software,” said Seth Howard, Epic’s executive vice president of research and development.

The Wisconsin-based company holds more than one-third of the hospital market, trailed by Oracle Health, at 22%, and Meditech, with 13%, according to health analytics company Definitive Healthcare.

More than 3,100 hospitals use Epic, the company says. And about two-thirds of its customer base are live with the vendor’s generative AI products, according to Howard. Those include algorithms that help doctors write notes and summarize patient information for clinicians prior to a visit.

Epic integrated ambient listening software from a startup called Abridge into its workflow in 2024, but is working to make its software compatible with more products that can transcribe patient visits in real time, Howard said. The company is also working to make its documentation tools multimodal, so they can understand videos, images and other types of information beyond text.

Over the past year, Epic has also started building a framework around agentic AI into its software, to see how digital assistants that can summarize information, interact with patients, schedule followups and more can work together, according to Howard.

Overall, Epic has 125 AI use cases either in development or live, he said.

Similarly, in October 2024 Oracle announced that it plans to release a new EHR platform this year that incorporates its clinical AI agent, voice-activated navigation and search capabilities. The clinical agent, which can record and transcribe patient visits and draft a note in the EHR for the clinician’s approval, is currently available for more than 30 medical specialties, the company said.

Bharat Sutariya, Oracle’s chief health officer, called the refresh “nothing short of a complete reinvention of the EHR.”

“We are investing massively to ensure we rapidly innovate and get it right at the same time,” he said.

Meanwhile Meditech and eClinicalWorks, a company that offers cloud-based EHR software currently in use at 70,000 care sites, are pursing a partnership approach.

Meditech has long worked with Google to accelerate clinicians’ access to data and with Microsoft to automate scribing, including deploying Microsoft’s ambient listening and clinical documentation tools into its EHR.

Meditech also plans to include Microsoft’s latest clinical assistant as it rolls out in 2025, according to Rachel Wilkes, Meditech’s corporate lead for generative AI initiatives. A variety of other AI scribes are also available to Meditech’s customers through integrations with the company’s software, she said.

“We want to keep our existing customer base engaged,” Wilkes said. But “yes, we are looking to attract new customers as well by having a platform that is innovative, scalable, and a really good alternative to maybe other competitors that are out there.”

eClinicalWorks has also partnered with Microsoft to use machine learning to match patient records, and with a company called Healow to create an AI medical scribe and an AI contact center.

AI checks and balances

EHR vendors are remaking their software with AI despite dogged concerns about quality, privacy, bias and a lack of federal regulation around the technology.

Generative AI in particular is tricky to monitor. The models are known to hallucinate, or provide factually incorrect or irrelevant answers, or omit information altogether. Their performance can also degrade over time, in a phenomenon called model drift.

Still, records companies say that they, along with their clients, are carefully supervising AI to make sure they’re not making up any information — or leaving anything out.

Leaders with eClinicalWorks, Meditech and Epic said they’ve invested heavily in risk management and have stood up stringent guardrails, including ongoing assessment by clinicians.

There should always be a human in the loop when it comes to ambient documentation and generative AI products, said Meditech’s Wilkes.

Meditech also includes disclaimer language on any records created by AI, sharing that it was created by an algorithm and reviewed by medical staff. The company is working to put more formal citations into the workflow to clarify what’s generated and where the model pulled relevant information from, Wilkes said.

Epic applies the same principles to ensuring AI quality as it does to any other software to make sure it works well and isn’t causing harm, according to Howard.

But “the techniques you might use to understand, ‘is it working well?’ are different,” Howard said.

Still, despite concerns with the models, hospital customers are full steam ahead on adoption, vendors say. According to a survey from HIMSS conducted in the fall, 86% of respondents already leverage AI in their medical organizations, mostly for administrative tasks, though there’s growing interest in more clinical use cases.

“Our customers are very warm to it. More warm over the past 12 months,” said Wilkes. “We’ve gone from doing a lot of education about the impact, to — now the question we’re getting is, ‘When? When can I get it? How much is it going to cost? What’s the plan?’”

“There’s always going to be the early adopters, the mid adopters and the lagging adopters of technology. It’s happened with smartphones, it happened with color TV, it’s happened with every product,” said eClinicalWorks’ Navani. “All I can tell you today is we are seeing incredible multiples of customers who you would call early adopters.”

But “this is not just being done because modern is new and it’s good,” Navani continued. “It’s got a clear efficiency benefit written all over it. That’s driving the change.”

Tech companies are trying to get healthcare companies to adopt artificial intelligence tools, hoping they can pull in revenue by appealing to the industry’s need to reduce costs and tackle clinician burden.

Google is one such company. The search and cloud computing giant has unveiled a number of health AI products in the past few years, including a large language model called Med-PaLM trained specifically on medical data, generative AI products for healthcare organizations and a platform to help companies create their own AI agents.

Google also offers a tool that allows clinicians to search for and answer questions about information in patient notes and other medical documentation. That product, called Vertex AI Search for Healthcare, has quickly ramped up after launching in March 2024. Google announced that Vertex AI Search was multimodal earlier in March 2025, meaning it can now understand images as well as text, allowing the tool to scrub charts and scans for relevant information.

Google has inked deals with major EHR vendors and leading health systems to integrate its AI into their workflows. But not everyone is gung ho about AI, especially as the models become more evolved and the potential for mistakes grows.

Concerns include hallucinations, when AI makes up a response; omissions, when AI leaves important information out; and model drift, when AI becomes less reliable over time. Meanwhile, fledgling oversight and a lack of regulation are also hampering adoption as the industry confronts weighty questions of accuracy, privacy and bias.

Healthcare Dive sat down with Aashima Gupta, the head of healthcare for Google Cloud, to chat about Google’s work in health AI and the future of the technology — including how AI is evolving from a task-based helper to a collaborator for clinicians, uncertainty from the Trump administration and why she’s excited about agentic AI.

HEALTHCARE DIVE: Vertex AI Search for healthcare can now understand images. Why was this a necessary update?

AASHIMA GUPTA: Healthcare information is scattered in different forms and types. During an annual diabetes foot exam, for example, when a physician is looking for ulcers, they mark on a diagram of a foot where it’s callous, pre-ulcer and ulcer, using different symbols. Healthcare is full of examinations like this. Now, with multimodal, you’re able to see and contextualize that diagram of a foot and extract that information, including any potential ulcers, and put it into the medical record automatically. That saves clinicians time, because they don’t need to interpret this themselves.

Healthcare, as an industry, has a lot of paperwork. We talk about burnout. These are the type of innovations we want to add to Search to reduce it. Last year, we said Search was ‘semantic’, meaning it knows what people mean when they say ‘diabetes’ or ‘A1C’ — it knows clinical concepts and how they’re related. Now we’re taking that and applying that to forms in an exam room that have different pictures. So our results are much more accurate and helpful.

What other inputs might Google want to add to the search-and-answer tool — sound?

You’re right. We will continue to add different modalities here. Sound and videos, as an example.

What else is Google working on in healthcare that you’re excited about?

We are very excited about agentic AI. The last few years have been a lot about generative AI, which is task-based — ‘Give me a discharge summary. Give me a nurse handoff. Write me a referral.’

AI agents are a leap forward, because they can think multiple steps ahead. They can plan unique steps for the goal in mind.

Imagine this in healthcare workflows. Let’s say I want to figure out, in my revenue cycle, where there’s variability by payer type, market, for a certain CPT code — that’s a multistep process. And that’s what agentic AI offers. Imagine hundreds of agents helping nurses and physicians do their jobs.

Given the number of companies offering AI agents, what’s Google’s elevator pitch?

We believe there’s a need for centralized coordination and management, and that’s the platform we provide, and we will give agents the tools.

So if a company is building their first-party agent, they have three things: An ability to use [Google’s family of large language models] Gemini, our search functionality based on clinical knowledge and an orchestration layer.

We believe that people have choice. Some people they really want to build themselves. Some want a partner. How do you organize all of that? That’s where we want to be.

Google doesn’t publicly share the number of healthcare organizations using its AI. How would you characterize uptake?

We have relationships with Basset Health, Highmark Health, Mayo Clinic, HCA — customers using generative AI to streamline prior authorization submissions, close screening gaps, enhance radiology workflows and more.

Healthcare used to be a laggard in technology adoption. But this AI is different — this is actually going into the back office, the revenue cycle, claims, to tackle burnout. You are seeing this move much, much faster. You are seeing generative AI pilots move from experimentation to scaling.

As for agentic AI, I’d say healthcare companies are definitely adapting to it. But it’s a newer space. 

Generative AI is becoming more integrated into healthcare workflows, despite the technology’s tendency to make mistakes. Does Google track error rates for its AI products?

Not only us — customers are building guardrails. At HCA, for example, we worked with them on a very robust evaluation framework, not just to catch errors but also — Is this model reliable? Is it hallucinating? Did it omit something? So we provide those tools for an evaluation framework, and for all of this there’s a human in the loop capturing the feedback, and that feedback loop then makes the model more effective. That’s what gives me comfort. That’s best practices being deployed.

But no data around accuracy that you’ll share? Is it accurate enough that you’re comfortable with increasing adoption?

Our health systems have put humans in the loop, and there’s change management around that. Technology by itself is not enough. You need robust change management, and to have a human in the loop and make sure they’re evaluating this. These processes are in place before the AI is brought in.

There are checks like grounding, or citing where the AI pulled information from. That helps combat hallucination. For omission, when the AI leaves something out, this is where we’re seeing evaluation frameworks. We’ll say, ‘Okay, did this result omit something pertinent?’ And eventually, after that feedback loop goes a few times with human oversight and the model learns, the next situation is better.

Also, use cases matter. This is why AI is being used more in administrative tasks.

The Biden administration was working to create a federal governance framework for health AI, but Trump has canceled those efforts and is instead focusing on deregulation. What are your thoughts on that approach?

At Google, we have our own AI principles and processes. But while self-regulation is good, we don’t believe that it’s sufficient. We believe AI is too important to not be regulated. And I will stop at that.

Do you think we’ll see anything from the Trump administration around oversight?

It’s hard to say at this point. I think we are trying to figure out how we best work with them, share our best practices with them. Too early to say. I think the entire healthcare community is waiting for that.

Executives from America’s largest technology companies descended upon the HLTH conference in 2024 with promises to lead the next round of innovations in healthcare artificial intelligence.

AI products, offered by companies like Google, Microsoft, Amazon, GE Healthcare and Nvidia, vow to help systems solve a range of problems from reducing documentation time to optimizing operation room schedules. Tech executives say they’ve moved past earlier iterations of AI tools — which tended to be single point solutions — to offer platform solutions, which can be tailored to the needs of an individual health system.

“We are truly at an inflection point,” said Sally Frank, worldwide lead for health and life sciences at Microsoft.

But while tech companies look to sell their products to health systems and providers, executives are split on how to offer the new technology to the industry responsibly — and whether clinicians are equally ready for the new tools.

“Any evaluation that we do at the foundation model level is only a first draft,” said Greg Corrado, senior director at Google Research and co-founder of the Google Brain Team. “It does need to be pioneered by healthcare systems that are willing and able to do the research on the ground, and not every health system can do that.”

Tech companies emphasize a platform approach

Technology firms pitch themselves as partners for healthcare organizations in AI development, offering deep expertise in product development and testing.

However, tech executives at HLTH caveated that they rarely tell health systems how they should apply AI.

“Google isn't a healthcare provider, and we don't want to become a healthcare provider,” said Corrado.

The executive said use cases for AI should come directly from health systems. Technology companies are like semiconductor chip manufacturers — they build a resource intensive, valuable raw material — in this case, a large language model — and the healthcare industry applies it, Corrado said.

“The technologies that we develop should be things that allow healthcare organizations to conceive of and build their own futures in this space,” he said.

Google has worked with health systems, including the Cleveland Clinic and Community Health Systems, to pilot technology ranging from a healthcare-specific cloud services platform to a documentation tool that searches the electronic health record.

Other tech companies like Microsoft have rolled out similar partnerships that encourage health systems to tailor their own AI solutions.

It’s a promise that blows the door open past “out-of-the-box” tools solving “straightforward use cases,” said Kees Hertogh, vice president of health and life sciences product marketing at Microsoft. 

In October 2024, Microsoft said it would make it easier for health systems to build their own AI tools directly. The company is also leveraging generative AI to help health systems organize their unstructured data, as well as imaging and medical imaging data, to enable clients to “build their own co-pilots, their own AI agents,” said Hertogh.

Health systems are likely to prefer such tools above earlier offerings, according to Bill Fera, a principal consultant with Deloitte. 

“I think hyperscalers and platform players are going to take over the space as people become more comfortable with doing their own builds and understanding how to do that,” Fera said in an interview. “There will be a move away from applications to our own platforms.”

A need for robust testing triggers questions about access

In healthcare, tech companies say they’re currently interested in helping health systems manage information overload. The companies’ ability to wrangle large amounts of data in written forms, as well as images, could help cut down the time it takes providers to review notes or schedule patients for critical operations, executives say.

But different AI applications require varying levels of human supervision. 

For now, tech companies at HLTH agree all healthcare AI should have a human supervisor. Some companies have focused first on administrative-focused AI tools because they require less oversight. At GE Healthcare, for example, a scheduling tool might require less supervision than one used to assist in cancer treatment.

“These are the places where we think that we can bring AI in really, really fast because it's not directly tied to clinical decision support,” said Abu Mirza, general manager and global SVP of digital products at GE Healthcare. “It's not directly tied to making a decision about someone's surgery and other stuff. This is where AI can really get started.”

However, when AI is close to a clinical decision point there must be even more robust testing, said Google’s Corrado. 

Significant attention has been paid to the problem of AI “hallucinations” — when a model produces an answer that can’t be backed up by source text.

But equal attention ought to be paid to omissions, Corrado argued. Omissions occur when AI fails to cite relevant information in its answer. In some ways, omissions are harder to test for because they require a full review of often lengthy medical records. 

However, not every health system has the resources or the know-how to robustly test cutting-edge technology, according to Microsoft and Google executives.

Microsoft has connected health systems seeking to use AI, so that more advanced systems can provide smaller systems with resources and knowledge ahead of deployment, according to Hertogh. The consortium has over 15 hospitals and health systems, including Providence, Advocate Health, Boston Children’s Hospital, Cleveland Clinic, Mercy and Mount Sinai Health System.

Microsoft offers voluntary testing guidelines for health systems that use its AI products.

“We provide them with guidance and tech so you can create visibility and transparency into the system... The relationship is a bit of like, ‘Hey, this is how we're approaching building, this is how we're thinking about responsible AI,’” Hertogh said.

Microsoft also benefits from the information exchange, according to the executive. While Microsoft offers its take on how to implement AI, the exchange is a two-way street. Engaging with leaders at some of the nation’s best health systems helps promote trust in Microsoft’s offerings and establish credibility, he said.

This broad collaborative spirit is relatively new, according to Hertogh. He thinks it could help smaller providers access new tech and act as an “accelerant” for broader adoption of generative AI across health systems. 

Google, however, doesn’t just suggest health systems test their AI — the company all but requires it.

Corrado said clinical feedback it integral to Google’s testing process, so to date, the company has only partnered with health systems that have some “sophistication” around AI and shared values about model testing.

“I think health systems that want to buy an off-the-shelf solution — they should probably wait. They should wait for things to mature, for things to stabilize,” the researcher said. “I don't think there are any evaluation frameworks that are sufficiently mature in any [healthcare] context where you can go and just deploy the tech. ... There's nothing that is that iron clad. You have to put it in the environment, evaluate how it's really going to be used on the ground and make adjustments there.”

In an escalating technology arms race, providers are beginning to adopt the same technology they say insurers are using to deny claims in masse — artificial intelligence — to combat claims disputes. 

Insurers so far have the upper hand in the race to implement AI into the claims review process, panelists at HLTH said. Some of the nation’s largest insurers, including UnitedHealth, Humana and Cigna, already leverage algorithmic decision tools when deciding what care to cover. 

In October 2024, nonprofit Blue Shield of California said it will also test claims automation technology in early 2025, through a partnership with Salesforce. The insurer billed the tool as way to alleviate provider and patient frustration with the prior authorization process, saying it would cut down decision time from weeks or days to seconds.

Payers argue the tools — which have mostly been applied to the prior authorization process— allow them to quickly flag services that do not align with coverage criteria and plan terms.

However, providers’ experience with current tools — supported by investigations and reports from lawmakers — suggest that when AI enters the equation, more claims are denied.

Providence CFO Greg Hoffman told Healthcare Dive that his nonprofit health system noticed underpayments or initial denials increased by more than 50% over a two-year period when payers began to more heavily adopt AI tools to review claims.

The surge forced Providence to increase its number of human “touches per claim” by over 50% because physicians had to submit additional documents, the executive said.

A Senate subcommittee report alleged three of the nation’s largest Medicare Advantage insurers — UnitedHealthcare, Humana and CVS — use predictive technology to systematically deny patients access to post-acute care in order to increase their profits.

The largest offender was UnitedHealth, which increased its post-acute services denial rate from 8.7% in 2019 to 22.7% in 2022, alongside the roll out of its predictive tool, NaviHealth-backed nH Predict.

In some sense, payers are forcing an “AI arms race,” said Jeffrey Cribbs, a distinguished vice president analyst on consultancy Gartner’s healthcare team.

“We will see more efficient coding, more efficient submission of authorization claims and then resubmission of claims,” he said. “And then on the other side, we will see more efficient extraction of exceptions and things that might be cause for denial of those claims.”

Providers say they’re currently behind payers in that race — though they’re dedicated to catching up.

During a HLTH panel on provider and payers’ use of AI, Sara Vaezy, EVP and chief strategy and digital officer at Providence said providers lag payers in developing large language models, but are likely to have sophisticated options soon. 

The executive urged providers to form coalitions to share data and develop standards around AI in the claims process to better compete in the space.

Once providers catch up, experts believe AI could ultimately help them in prior authorizations.

“It’s still early innings, but I think the technology is actually going to go a long way to leveling that playing field, from a provider's perspective,” said Amit Phull, chief physician experience officer for online networking service Doximity, during the panel.

He said the tech will ultimately offer providers “a leg up” in claims disputes and cut down time it takes to complete claims documentation.

Bill Fera, a principal consultant at Deloitte, agreed.

“Plans have been building for longer. They have the the upper hand right now. Providers are newer to thinking this way and building these tools,” Fera said during an interview. But he said AI will be an equalizer, because it will help providers quickly go through massive amounts of data to find a “truth” about whether the patient ought to qualify for a service under their plan’s terms.

“Policies will now be available to a provider. I wouldn't go to read a policy in full, but ... now we're surfacing all that knowledge. We're taking the mystery away,” Fera said. “And there is a fact base. There's a core piece of information that can be interrogated. It's just now, it can be interrogated very quickly.”

Still, health systems are contending with payers that are ready to apply AI right now, and many don’t have the AI know-how to build competing models today.

Providence decided to outsource its revenue cycle management to compete in the short-term, Hoffman said. The nonprofit entered a 10-year partnership with R1, a revenue cycle management company, in January 2024, in part to combat rising claims denials associated with payers’ increased use of AI tools.

“As we began to look at our technology roadmap for us to respond quickly and so forth, we realized that it would take us two to three years to build the technology roadmap to be more efficient in this area,” Hoffman said. “And that's where we've now leaned into our partnership with R1 to leverage equivalent technologies.”

As hype grows around artificial intelligence in healthcare, a seemingly paradoxical question is setting off faint warning bells in the minds of skeptics and proponents alike: Could AI tools meant to reduce clinician burden actually increase it?

It’s too soon to say, given the technology’s nascency. But the answer — which has big implications for the future adoption of AI in the industry — depends in part on how intensely clinicians have to fact-check models, along with financial incentives in healthcare’s majority fee-for-service payment infrastructure, according to experts.

All told, AI products could add up to weaker time savings than developers promise, and any time saved could quickly be filled up by additional patient visits as hospitals and medical groups hustle to increase earnings.

“That’s absolutely a worry,” said Graham Walker, the co-director for advanced development at Kaiser Permanente’s medical group, during a panel at the HIMSS healthcare conference in Las Vegas. “The easiest way to get more revenue out of your healthcare system is by telling the doctors and [nurse practitioners] and pharmacists to go faster and see maybe just one more patient.”

‘The tools aren’t perfect’

The goal of many AI tools in healthcare is to reduce administrative friction by helping with rote tasks like documenting patient visits, looking up medical information or filing paperwork. Such tasks contribute to clinician burnout, which is a major problem in the industry and can lead physicians to exit the field.

About one-third of doctors responding to an American Medical Association survey in 2023 said they were interested in or planning to leave their jobs in the next few years. Many cited work overload as the reason: According to some research, healthcare workers spend the majority of their week working on administrative tasks.

Enter AI. Algorithms have been in use by healthcare companies for decades, but excitement around the tech has reached a fever pitch in the past few years with advancements like generative AI, which can create original text, and AI agents, which can perform tasks without human oversight.

A number of companies are folding the technology into sleekly-packaged products, touting time-saving metrics that may almost seem too good to be true to clinicians sick of after-hours ‘pajama time’ spent on documentation.

Abridge, a startup that uses ambient listening and generative AI to automate clinical notetaking, and Nabla, another ambient AI assistant, both claim to save providers about two hours each day.

Software giant Oracle has woven AI into its health records platform for providers, including a clinical AI assistant. Physicians using that tool see a 30% decrease in documentation time, according to the company.

Meanwhile, Microsoft says its AI documentation product saves doctors five minutes on average per patient visit.

“The capabilities that AI scribes bring to the practice of care are remarkable,” said Rohit Chandra, the chief digital officer at the Cleveland Clinic, during a panel. (The academic medical system is currently rolling out an AI documentation software from Ambience across its provider network.)

Documentation products tackle an acute pain point for doctors, while being relatively simple to use, easy to implement and safe, because they’re supervised by clinicians, Chandra said.

However, AI performance metrics such as time saved should be taken with a grain of salt, according to experts.

Clinicians still need to review the AI’s output to catch mistakes. That process, known as ‘human in the loop’, is key to ensure accuracy and build trust in AI, experts say.

Letting AI tools operate unsupervised could result in medical records containing made-up symptoms, or missing information that could be key to a patient’s health.

“That’s where I think we can get into trouble. The technology is going to get to the point where, at least in my opinion, I don’t think we’re going to need to worry as much. But as we think about early adoption, we do have to be cognizant that it is going to make mistakes,” Brenton Hill, head of operations at health AI standards group the Coalition of Health AI, said during a panel.

But deputizing clinicians to police what AI generates can cut significantly into time savings or, in some cases, erase them altogether.

“The tools aren’t perfect,” said Deborah Edberg, a family medicine physician with CVS-owned primary care chain Oak Street Health, during a panel. “We use AI to do our documentation and I do spend quite a bit of time going back and editing ... It can be a bit of a burden to make sure that what is recorded is accurate.”

To date, there have been no extensive, independent reviews of AI scribes in healthcare. But one recent study published in JAMA Network Open found Microsoft’s ambient scribe received mixed feedback from users.

“A recurring theme was the need for substantial editing and proofreading of the AI-generated notes, which sometimes offset the time saved,” the researchers wrote.

Time savings may not reach clinicians

Fee-for-service incentives could also prove a problem. The payment structure, which underpins the majority of healthcare spending in the U.S., rewards providers based on the volume of visits, tests and procedures they perform.

Meanwhile, despite a recovery in operating margins since the COVID-19 pandemic, costs are continuing to rise for hospital operators.

As a result, hospitals and other physician owners are incentivized to stuff their clinicians’ schedules as tightly as possible, to bring in as much revenue as possible. Any time freed up for doctors by AI might be filled with another visit, experts say.

“It could happen. And some organizations want to do that. And from our perspective, that’s an organizational decision,” Rachel Wilkes, the corporate lead for generative AI initiatives at EHR vendor Meditech, said. “If they want to schedule more patients with time savings, they can do that.”

Things that improve the productivity of the healthcare system — like reducing patient no-shows — often result in a heavier visit load for physicians, said Seth Howard, executive vice president of research and development at Epic.

“The conversation between the health system and the doctors need to be, ‘What do we do with those time savings?’ Does it go back into helping the doctor have some flexibility during the day? ... Or for doctors to see more patients?” Howard said.

Vendors are aware that the draw of AI products for hospitals includes the potential to increase revenue. 

Along with saving five minutes per encounter, Microsoft’s ambient AI frees up “13 additional appointment slots per provider, per month,” according to marketing materials.

The need to bolster flagging finances is one reason hospitals are ravenous for the tools, experts say. Demand has been skyrocketing, with the CEO of one company using AI to automate provider workflows likening it to “a fairy tale.”

“Our ambient product went from zero appointments a year two years ago, to a couple hundred thousand last year, to on track for 10 million this year,” said Tanay Tandon, the CEO of Commure, during a panel. “That sort of expansion and usage of AI in healthcare is incredibly exciting. That’s no longer your nerdy technical doctors playing around with it in the corner. This is meaningful use.”

Experts say like with any other technology, there are pros and cons when it comes to AI scribes, assistants and similar tools. In addition, other metrics besides time saved are important to physicians, like reducing their mental load by simplifying tasks or improving their job satisfaction by allowing them to focus on clinical care.

And anecdotal feedback from clinicians has been great, according to tech giants, AI scribes and the EHR vendors weaving the tech into their workflows.

Epic, for example, doesn’t have an average figure for how much time its own ambient documentation tool saves. But according to Howard, “physicians say, very commonly, I could never go back to how I did documentation before.”

Since the launch of ChatGPT in 2022, technology companies have been racing to bring generative AI-powered tools to the healthcare market. However providers have faced a quandary about what — and whether — to buy. 

As Google, Amazon, Microsoft and OpenAI rapidly expand their suite of artificial intelligence offerings, providers say they don’t know how to compare the efficacy of products or determine which tool might best meet their specific needs.

A group of health systems, led by Boston-based Mass General Brigham, is hoping to solve that problem. 

In November 2024, the academic medical center launched the Healthcare AI Challenge Collaborative, which will allow participating clinicians to test the latest AI offerings in simulated clinical settings. Clinicians will pit models against each other in head-to-head competition and produce public rankings of the commercial tools by the end of 2024.

Participating health systems say that the chance to directly compare AI products is overdue.

Despite the rapid proliferation of AI in healthcare, the industry has been slow to agree upon how to assess quality. Industry groups have taken stabs at rolling out evaluation frameworks, however, guidelines remain in draft form. 

Without standardized evaluation metrics, it’s difficult to compare between even the most similar tools, said Richard Bruce, associate professor of radiology and the vice chair of informatics at the University of Wisconsin School of Medicine and Public Health.

“Are there any [common] metrics that directly compare them? Currently, to the best of my knowledge, other than surveys of users and anecdotes, tools are not directly benchmarked against each other,” he said. “There’s no easy way to get an apples to apples comparison.”

So far, Emory Healthcare, the radiology departments at the University of Wisconsin School of Medicine and Public Health and the University of Washington School of Medicine, and industry group the American College of Radiology are participating in the collaborative. MGB said they plan to expand the program.

The health systems will initially test nine models, according to a MGB spokesperson, including products from Microsoft, Google, Amazon Web Services, OpenAI and Harrison.AI. 

The clinicians will assess the models on draft report generation, key findings, differential diagnosis and other factors, according to MGB. 

The metrics for evaluating the models are “evolving,” Bruce said, and may depend on the tool’s clinical use case. For example, while model accuracy will always be highly weighted, there are some instances —like when the model is used to produce a text report — where readability might be more important.

“Some of those are going to be very much on a subjective quality,” Bruce said. “Like, do I feel that the style in which this text is presented is more easily readable or amenable to patients?”

Ultimately, the health systems will create a “leaderboard” of tools, according to Dushyant Sahani, professor and chair of the department of radiology at the University of Washington.

The leaderboard will be used both to provide feedback to technology companies and to help health systems shop for tech, according to MGB.

Health systems that do not directly participate in the challenge may be able to use the rankings to decide what tools to purchase, according to Sahani — which he views as a win for health equity.

In the race to implement AI, experts have raised concerns that smaller-resourced providers, who may not have time to research new tools, might be left behind.

“Health systems can use the transparent rankings to inform decision-making and establish benchmarking standards,” Sahani said. “Insights and best practices from the consortium can be adopted by non-participating health systems.”

Google and Microsoft declined to comment on this article.

The Peterson Health Technology Institute launched a task force in January 2025 aimed at studying the business impact of artificial intelligence on health systems. 

Implementing AI tools can be a challenge, as providers need to be wary of safety, accuracy and equity concerns that could come with the emerging technology. But rolling out safe products isn’t the only question for health systems implementing AI: They also need to figure out how to measure success.

The institute, which has previously worked to evaluate other digital health tools, is pulling together health system leaders to discuss how they measure financial, efficiency and workforce effects of AI products.

Caroline Pearson, executive director of PHTI, joined Healthcare Dive to discuss the task force, including the group’s first focus areas.

HEALTHCARE DIVE: Can you tell me a little bit more about what the task force is and how it will work?

CAROLINE PEARSON: So we formed a task force because we really wanted to understand how some specific AI solutions were being adopted in health systems, and how they were thinking about measuring the impact of those solutions.

We started with a pretty narrow focus on AI tools for documentation — AI scribes — and revenue cycle management, which are two of the fastest areas of AI adoption that are currently happening in health systems in this country. We wanted to talk with executives of those health systems to really understand how they’re thinking about implementing those AI solutions and what the business impacts of that implementation are, from both a financial perspective, efficiency perspective and a workforce perspective.

Why do you think this task force is needed for health systems?

Health system leadership are working to both adopt technology quickly and struggling to figure out how to measure that and think about its business implications as they go. The leadership has been really excited to share best practices with one another and learn how other systems are doing this, and also to think about what kinds of data and metrics they want to be measuring for the future.

So each of the systems has made their own business case about how they’re thinking about these initial purchases on the documentation side. But over the next several years, how will they continue to be able to measure the impact of those solutions on their delivery systems? And what kinds of data are they going to need out of the AI tools in order to do so effectively?

You mentioned you wanted to focus on documentation and revenue cycle products. Is that because they’re popular, or are there any other reasons why you decided to hone in on those two areas?

I think that documentation is going to be the fastest adoption of health technology that we ever see. It feels like they’ve gone from zero to very widely adopted in less than two years. And so part of it was really driven by that growth. 

But both rev cycle management and documentation are places that are really at the business heart of how a health system operates, right? It really affects productivity, patient throughput, as well as patient experience and provider experience. So they were very good places to start, because they're just brass tacks issues around how health systems operate. 

So it’s a way of analyzing, “Sure, you’ve adopted this documentation assistant. But is this actually saving you time? Is that allowing you to see more patients? Is it reducing provider turnover?”

Yeah, exactly. Some systems are looking to increase the number of patients that they can see in a given clinic. Others are really focused on provider burnout and trying to reduce the “pajama time” that they’re requiring of doctors to do documentation, and then ultimately reducing turnover.

And yet, other systems are really thinking about that patient experience and how to make sure that every clinical interaction is giving the doctors enough time to spend face-to-face with their patients. All of those are valuable outcomes, but systems want to be clear-eyed about what their expectations are for each of those and how they will know whether they’re accomplishing their goals.

Health systems will likely still focus on containing costs this year. Do you think that might also push them to consider eventually dropping some of these AI tools if they’re not worth it in the long term?

I don't know. My sense is that these tools, the documentation tools, are quite sticky, and once providers get used to using them — and patients feel comfortable with them — they will be hard to remove from the exam room. 

However, we’re also seeing those tools become commoditized more rapidly. So the price may come down as that market is quite competitive. And then the question is, what other solutions are those companies selling into the health system? And how do we begin to measure the value of those investments? 

So, eventually those companies might start adding to their offerings, and health systems might be more interested in saying, “I’ll pick up this AI documentation company that might also offer me something else.”

Yes, and we’re seeing that happen very quickly, a sort of spreading of these AI solutions within systems. And rev cycle is the sort of first and most obvious place that AI solutions grow out of documentation. 

Once you’ve captured the entire patient encounter, you have all of the data to convert that into billing. And so it may be that the price of the AI scribe becomes pretty efficient, but the price attached to all the revenue that you generate off of that encounter is much more valuable. Thinking about how you measure those other lines of business is important.

Healthcare companies struggling to adopt artificial intelligence amid a lack of federal standards are unlikely to get any help from the Trump administration, placing the burden of responsible implementation squarely on the industry’s shoulders.

That mandate is becoming increasingly difficult as the technology grows more complex, experts said during the HIMSS healthcare conference in Las Vegas.

“One thing that’s clear is that this administration is not going to regulate AI. Good or bad, take that for what it is,” Tanay Tandon, the CEO of provider automation company Commure, said during a panel.

President Donald Trump’s hands-off approach to AI governance means hospitals turning to the tools to save money and give overworked clinicians some relief will likely be operating in a regulatory gray area for at least the next four years.

The president says his goal is to free up U.S. developers to innovate. But that comes with downsides for tech developers and healthcare companies desperate for guardrails given AI’s proclivity to make mistakes, degrade over time and exacerbate existing bias.

Not to mention, a lack of national standards might actually hamper AI development and adoption, according to some experts.

“When there isn’t a federal framework, it can just absolutely cause all kinds of problems,” said Leigh Burchell, the chair of the Electronic Health Records Association. “We all just want to know what our rules are. And then we can comply.”

Biden versus Trump on health AI

To date, a handful of federal agencies, including the HHS’ technology office, the CMS and the Food and Drug Administration, have published targeted rules around the use and quality of AI in healthcare. But neither Congress nor the executive branch have zeroed in on a comprehensive framework to regulate the models — despite some progress during the Biden administration, when an HHS task force was working to build a unified regulatory structure.

That task force unveiled a strategic plan in January 2025 — just 10 days before Trump’s inauguration. However, Trump nixed the blueprint in one of his first executive orders.

Meanwhile, federal employees working on AI oversight, including at the FDA, have been caught up in the Trump administration’s purge of the government’s workforce. Amid the turmoil, the future of the HHS office that oversees AI policy remains unclear.

As a result, what little momentum there was in Washington to create a concrete strategy for overseeing health AI appears to have stalled out, at least for now. In its place, Trump has announced the Stargate Project, a $500 billion investment deal with private companies to prioritize AI development and maintain U.S. supremacy in the space — a high-stakes bet that was immediately complicated by the release of DeepSeek, a high-performing and inexpensive open source model from China.

The Trump administration did issue a request for information in early February 2025 to get public input on a potential national AI action plan. However, the plan’s wording makes clear the administration’s priorities: to “sustain and enhance America’s AI dominance, and to ensure that unnecessarily burdensome requirements do not hamper private sector AI innovation.”

The revocation of the Biden-era AI plan was largely symbolic, as agencies hadn’t yet gotten around to imposing any requirements on developers or users.

But in “the current administration — the brakes have come off and the accelerator has come down,” said Brian Spisak, program director of AI and leadership at Harvard University's National Preparedness Leadership Initiative, at HIMSS. “There’s a lot of responsibility to leadership of health systems to find the optimal balance between innovation and speed and safety and tradition.”

A technological sea change

That responsibility — which also rests on AI developers creating the models, software vendors weaving them into health records and the clinicians using them — is not negligible.

Currently, even the most futuristic AI at healthcare institutions is being used to automate administration, and only peripherally touches patient care. But that appears to be changing: There’s growing interest among medical organizations in more clinical use cases for AI, like tailoring treatment plans or helping clinicians arrive at a diagnosis, according to a survey from HIMSS.

Many of those use cases involve generative AI, which can create original text and images. But such models are known to hallucinate, or provide answers that are factually incorrect or irrelevant. AI can leave important information out, an error known as omission. Models can also drift, a term for when an AI’s performance changes or degrades over time.

Given the increasing prevalence of AI in pulling data from EHR systems, transcribing doctor-patient visits and more, errors like these could interfere with clinicians’ ability to care for patients, experts say.

Meanwhile, the technology is advancing at a stunning rate. In 2024, the healthcare industry was just beginning to come to terms with governance for generative AI. But now, the conversation has already moved onto AI agents, which can complete complex tasks largely unsupervised by humans.

Commure’s Tanay equated the current moment to when America moved from kerosene to electricity at the end of the 19th century.

“The way that we did things six months ago is completely irrelevant,” he said.

Because of the breakneck speed of development, federal standards from any administration would likely need to be flexible, experts say. 

The Biden administration’s HHS task force suggested the government could create guidelines around testing and piloting tools, along with some support for adoption. However, it shied away from a prescriptive approach.

That’s in line with the wish list from many stakeholders. Numerous executives at tech companies and hospital systems said any federal standards should be stratified by the level of risk an AI model imposes — for example, stricter oversight for algorithms that help doctors diagnose diseases, and looser restrictions for algorithms that help hospital staff allocate patients to beds.

“We have to weigh the balance between underregulation, which can potentially increase risk, and overregulation, which will stunt innovation,” said Anthony Chang, the chief intelligence and innovation officer at the Children’s Hospital of Orange County in California, during a panel.

“The current administration is more likely to be on the underregulation side. So we have to be careful as a profession that we don’t allow that to happen,” Chang said.

States, industry groups filling the void

The lack of a playbook from Washington has left hospitals and medical groups scrambling to build their own internal controls amid a patchwork of state laws and voluntary standards released by industry groups.

States including Colorado, Utah and California have already enacted legislation establishing disclaimer requirements for AI systems. More states are considering similar laws: The Electronic Health Records Association is tracking 150 different state bills related to health AI, according to Burchell.

“There is a massive explosion of bills,” Burchell said.

But differing standards could stop health AI developers and software companies from rolling out products in specific states, potentially disadvantaging patients depending on where they live. More risk-averse software companies may avoid AI, or certain states, altogether, she added.

“Legislation of a lot of sizes and shapes at the state level is a risk to us, because it means we have to do all kinds of different development. We would rather develop one system that can be used broadly and accepted across the country,” Burchell said.

Health AI standards groups are also stepping into the gap left by the federal government. The groups — often made up of leading hospitals, digital health companies and tech giants — include the Health AI Partnership, an AI learning network for the industry, and the Coalition for Health AI, which recently launched an AI registry for hospitals.

”I think we’ll probably see more of these non-government organizations like Health AI Partnership and some of those emerge as maybe our north stars today, as here’s some leadership in the space,” said Rachel Wilkes, the corporate lead for generative AI initiatives at EHR vendor Meditech.

But standards from industry consortia hold little weight without the heft of the federal government behind them, experts say. Historically, voluntary standards aren’t particularly effective.

“There’s room for people to act in their own best interest, whatever that may be, without any sort of federal framework,” Wilkes said.

‘We still don’t quite know how to deal with it’

EHR vendors and hospital operators say they’re building up rigorous internal standards for AI tools, including validation and frequent auditing.

“Government oversight has its place, but I do think the way that clinical practice evolves tends to be more driven by what’s happening at a health system,” said Seth Howard, the executive vice president of research and development for Epic, the largest EHR company in the U.S.

In interviews, executives with Epic, Oracle, Meditech and eClinicalWorks said they’re making AI available to doctors with rigorous oversight, including back-end accuracy checks and ongoing monitoring.

However, tech leaders stressed that it’s also the hospital and the clinicians’ responsibility to make sure everything is working as planned.

“We are working in an industry that deals with human life. It cannot be trivial. It cannot be under-exaggerated on what guardrails and checks and balances and what discussions need to happen,” said Girish Navani, the CEO of eClinicalWorks.

Technology behemoths betting heavily on AI feel similarly. Google, for example, has worked with for-profit hospital giant HCA on an evaluation framework to catch any errors generated by its AI models and ensure their reliability, according to Aashima Gupta, the head of healthcare for Google Cloud.

“We provide those tools for an evaluation framework, and for all of this there’s a human in the loop capturing the feedback, and that feedback loop then makes the model more effective,” Gupta said. “That’s what gives me comfort.”

Though some in the private sector say they’ve got governance handled, modern AI is incredibly hard to oversee, according to AI engineers. The main advantage of generative AI — its creativity — also introduces subjectivity, making grading its outputs complicated.

For example, if two clinicians are tasked to summarize a patient’s medical history based on clinical notes, their results could be incredibly different, albeit still accurate. It’s the same with generative AI, experts say: How do you measure quality in a standardized way when there’s that degree of variability?

“AI governance is still very much a maturing process,” said Harvard’s Spisak.

Hospital executives say they’re tackling oversight carefully. But some research suggests governance systems for simpler predictive AI models already aren’t rigorous enough. Hospitals with explicit procedures for the use and assessment of AI tools are still struggling to identify and mitigate problems, according to a study published in 2024 in the New England Journal of Medicine.

Even some of the most well-resourced and tech-savvy systems are struggling.

The Cleveland Clinic has an AI governing body that includes stakeholders from across the academic medical center, according to Rohit Chandra, the Cleveland Clinic’s chief digital officer. The task force oversees AI’s impact on the organization’s patients and workers while ensuring clinical safety and discussing thorny questions of privacy, legality and bias.

But “I don’t think we’ve figured it all out,” Chandra said during a panel. “The term hallucination, for one, has just shown up in the last two, three years. And we still don’t quite know how to deal with it.”

Hospitals should try to make specific people accountable for the performance of the tools, as part of a larger governing body that includes executives, lawyers, doctors and nurses, according to Brenton Hill, the head of operations at standards group CHAI.

Hospitals need to decide how to monitor AI effectively and report that information, which can depend on the products they have in place. They also need to consider what resources the AI will use and develop appropriate data use agreements with AI vendors, Hill said during a panel.

But “there’s not one silver bullet governance structure that you can put out there that will solve all your problems,” Hill said.

‘A pipe dream’

Though a road map from federal regulators would be helpful, stakeholders working to integrate AI tools into healthcare said they’re not holding their breath.

“While self-regulation is good, we don’t believe that’s sufficient. We believe AI is too important not to be regulated,” said Google’s Gupta.

But when asked what she expects from the Trump administration, Gupta was noncommittal. “It’s hard to say at this point. We are trying to figure out how we best work with them, share our best practices with them ... Too early to say. I think the entire healthcare community is waiting for that,” Gupta said.

Other experts said the Trump administration is a wake-up call for hospital executives who believed Washington could assume responsibility for overseeing AI.

Instead, the mandate should be on everyone touching the technology to ensure AI algorithms — given their shifting nature and innate opacity — perform as they’re designed, especially in an industry where any mistakes could impact patient health.

“The easy answer is, if this regulatory body says it’s safe, then I can trust it. I think people were hoping that might happen for AI,” said Aaron Neinstein, the chief medical officer of agentic AI company Notable. “I think that was a pipe dream.”