Artificial intelligence

Technology developers, electronic health records vendors and hospitals are grappling with how to govern artificial intelligence as futuristic algorithms become increasingly pervasive in the operations and delivery of patient care.

Growing AI adoption, coupled with a lack of comprehensive supervision from the federal government, is leading some to worry the industry may have too much latitude in building and implementing the technologies, experts said at the HIMSS conference in Orlando, Florida, in March.

Hospitals and developers say they’re being careful, creating a forest of internal controls and standards groups to ensure AI tools — including those based on generative AI — are accurate and unbiased. But those may not be enough, given algorithms are becoming more complex, hampering oversight.

“With modern AI systems, the most transparent you can be is here are the mechanics how we trained it, here’s what we measure about its behavior. What happens in the middle is a black box. Nobody knows. It’s a mystery,” said Heather Lane, senior architect in health software company Athenahealth’s data science team, in an interview.

Washington is currently working to create a strategy to oversee AI, with a new HHS task force kicking off work in January.

With the threat of regulations looming, technology companies building AI and the hospitals benefiting from it are warning Washington not to enact policies that could smother the future of AI, given its potential to lower hospital costs, ameliorate doctor burnout and improve patient care.

“There’s an alignment between health systems and the government — they’re looking at us to put safety in place,” said Robert Garrett, CEO of New Jersey hospital operator Hackensack Meridian Health, during a panel on AI. “What we don’t want to do is stifle the innovation.”

‘New horizons’

Over the past year, technology giants like Google and Microsoft launched a flurry of AI tools for healthcare organizations and partnered with major hospital chains to explore new uses. They’re also linking with large electronic health records vendors like Epic and Meditech to integrate their algorithms directly into clinicians’ workflows.

As a result, hospitals interested in exploring AI have never had more options. That’s true both of tried-and-tested predictive AI, which analyzes data to make predictions — what’s increasingly being thought of as “traditional” AI — and the more futuristic generative AI, which can produce original content.

Generative AI roared into the public sphere late 2022, when OpenAI launched ChatGPT, a human-like chatbot built on a large language model called GPT.

Much of the subsequent debate around generative AI in healthcare has centered around if and when the technology could replace physicians. If that happens at all, it’s far off, experts say.

Healthcare organizations using generative AI today are mostly focusing on low-risk, high-reward use cases in administration, like summarizing records, improving nurse handoffs between shifts or transcribing doctor-patient conversations.

California system Stanford Health Care is experimenting with new software from EHR vendors that automatically drafts clinician replies to patient notes, said Gary Fritz, Stanford’s applications chief, during a panel.

Vanderbilt University Medical Center in Nashville, Tennessee, even created its own internal GPT-backed chatbot after discovering some of its clinicians were accessing ChatGPT onsite, according to Yaa Kumah-Crystal, VUMC’s clinical director of health IT.

Mayo Clinic recently put out an internal call for proposals for projects using generative AI and received over 300 responses, Cris Ross, CIO of the Mayo Clinic, said in an interview. The academic medical system has since narrowed it down to about 10.

Generative AI “is so exciting to us because so much of our data is language-based and/or multimodal,” Ross said. “There are new horizons that have opened because of the power of the language tools.”

Internal control efforts ramp up

As AI adoption accelerates, so do worries about responsible AI. The technology is far from perfect: Generative AI is known to hallucinate, or provide answers that are factually incorrect or irrelevant. Model drift, when an AI’s performance changes or degrades over time, is also an issue.

The technology also raises larger questions of privacy, ethics and accountability. Bias is an especially acute worry in healthcare, an industry struggling to quash historical inequities in care delivery and outcomes. If an algorithm is trained on biased data or applied in bias ways, it will perpetuate those biases, according to experts.

EHR vendors and hospitals at the forefront of AI adoption say they’ve created robust internal controls to keep their models in check, including validation and frequent auditing.

Meditech, a major vendor for hospitals that’s partnered with Google on AI, extensively tests its models on its own datasets to ensure information is protected and outputs are reliable, said COO Helen Waters in an interview.

“We do a lot on the validation process before we ever release anything to a customer,” Waters said.

Epic, the largest EHR vendor in the U.S., has taken similar steps. The performance of all its AI models are constantly tracked by monitoring software once they’re in a customer’s EHR, said Seth Hain, Epic’s senior vice president of R&D, in an interview.

Health systems are also wary. Hospital executives said they’re standing up governance committees to oversee AI pilots, training employees on the technology and continuously monitoring any AI tools.

Highmark Health, an integrated health system based in Pittsburgh, created a committee to oversee generative AI use cases as they emerge, said Brian Lucotch, president of Highmark’s tech subsidiary enGen, in an interview. The system has also formed groups working on reengineering Highmark’s business processes and training its workforce around AI, Lucotch said.

Along with oversight and intake processes for traditional AI, nonprofit giant Providence has stood up specific governance structures around generative AI, Sara Vaezy, Providence’s chief strategy and digital officer, told Healthcare Dive via email.

The seven-state system is also building infrastructure to test, validate and monitor AI at every phase of its deployment, from development to maintenance, Vaezy said.

Similarly, academic medical system Duke Health has stood up an “algorithmic-based clinical decision support oversight process,” said Michael Gao, a data scientist at the Duke Institute for Health Innovation, during a panel on AI adoption strategies.

The process begins after an AI project is initiated, and monitors models for fairness and reliability, including any signs of drift or bias, Gao said.

VUMC also has a process for the intake and vetting of models, along with ensuring their outputs remain applicable to its patient population, said Peter Embí, VUMC’s SVP for research and innovation, during a panel on operationalizing AI.

But “one of the things we know, when we deploy these tools in practice more so than just about any other tool we’ve dealt with in the past, things are going to change,” Embí said. “And when those things change they can have very significant impacts that can be very detrimental.”

Ever more challenging to police

Some AI ethicists and computer scientists are warning existing governance may not be up to the task of overseeing increasingly complex models.

It was clear how traditional predictive AI worked and obvious when it diverged from its purpose, said Athenahealth’s Lane.

But “for generative AI, it’s much harder to get a hold of what we call ground truth — a clear right answer,” Lane said.

For example, it’s clear if an AI meant to standardize insurance information gets an output wrong because you can refer back to the original dataset, Lane said. But for generative AI tasked with summarizing a patient’s entire medical history based on unstandardized clinical notes, “there’s a lot more variability,” Lane said. “No two clinicians would write the same note in exactly the same way. And generative AI isn’t going to write it the same way twice in a row either.”

As a result, evaluating these tools is incredibly difficult, according to Michael Pencina, chief data scientist at Duke Health.

The notion of explainability, or knowing how an AI system reached a particular decision, “goes out the window,” Pencina said during a panel on operationalizing AI. “It works, but we don’t really know why.”

And when it comes to quantitatively analyzing the performance of large language models, “there are really no standards,” Pencina said. “How do you measure — a written note — how good it is?”

These concerns are exacerbated by research finding governance systems for existing AL models may not be rigorous enough. Even well-resourced facilities with explicit procedures for the use and assessment of predictive tools are struggling to identify and mitigate problems, according to a study published January in NEJM.

When the functioning of an AI affects patient care, gaps in governance can become a huge issue — even with more knowable traditional models.

California nonprofit giant Sutter Health at one point used a predictive AI tool that flagged potential cases of pneumothorax, when air collects outside the lung but inside the chest, for radiologists, said Jason Wiesner, Sutter’s chief radiologist, during a panel on AI in healthcare delivery.

The tool was implemented into radiologists’ imaging system before false positives and negatives were sorted out, causing clinicians to see pneumothorax warnings during patient visits despite no other evidence of the serious condition.

The algorithm was pulled after three-and-a-half weeks. But “there was some time there when we could have done some patient harm,” Wiesner said.

Washington moving slowly

AI’s rapid advancement and implementation is occurring in a regulatory environment multiple experts described as “patchwork.”

A handful of federal agencies, including the CMS, the Office of the National Coordinator and the Food and Drug Administration, have issued targeted regulations around AI use and quality. Some states have also passed health AI legislation. However, Washington has yet to create a concrete strategy for overseeing health AI.

That’s expected to change this year, as a new HHS task force works on building a regulatory structure to assess AI before it goes to market and monitor its performance and quality after it does.

The task force has “made significant progress” on an overall strategic outline, along with deliverables due by the end of April, including a plan for AI quality and assurance, Syed Mohiuddin, counselor to the deputy secretary of HHS, told Healthcare Dive via email. Mohiuddin is co-leading the task force along with Micky Tripathi, the head of the ONC.

In the meantime, AI developers and healthcare providers say the solution is for the private sector to come together to share best practices and create guidelines for themselves.

Resulting standards groups have published blueprints for trustworthy AI adoption and plan to operationalize those standards in practice, like inventorying what algorithms are being used by members and tracking outcomes.

In an interview, David Rhew, Microsoft’s chief medical officer and VP of healthcare, stressed that such networks are public-private partnerships and that the federal government has ongoing input into the practices of private companies today.

But implementation challenges are “not going to be figured out by any one company or even the government. It’s going to be figured out by organizations that are doing it,” Rhew said.

Some executives suggest there may not be a bigger role for the government to play in overseeing AI, given the lightning speed of technological advancement.

​​​“AI technology is advancing so rapidly — can the regulations even keep up? I don’t think so. Governance should emanate from the healthcare sector itself, and we can work hand in glove with regulators,” said Hackensack’s Garrett.

Some experts have concerns about developers being left wholly to their own devices, given the massive profit motivation in racing AI applications to market. Demand for the services is causing revenue and profits for technology giants to snowball and their valuations to soar.

“It would be really best if everyone acted ethically and responsibly without any regulation or coalition, but that’s unlikely,” said Mayo’s Ross.

Private sector efforts will “play an essential role” in setting standards for AI in healthcare, Jeff Smith, ONC’s deputy director of certification and testing, told Healthcare Dive over email. But “participation in these governance groups will not replace the need for regulation.”

Balancing regulation with innovation

AI developers said they’re open to regulation, but Washington needs to be careful it doesn’t enact so much oversight it stifles new advances. 

Past regulations around EHR technology, for example, has rankled vendors as slowing down product launches. Tech developers don’t want more of the same for AI, especially given how quickly the technology evolves.

“I can tell you that I don’t want it to be too heavy. But we don’t want it to be too late. There has to be some form of a litmus test that says we can produce enough information to validate our assumptions, to validate our models, to tell you why you should trust them,” said Meditech’s Waters.

The government could be helpful when it comes to continuous monitoring of AI tools, said Nasim Afsar, Oracle chief health officer, in an interview.

For example, regulators could check the output of algorithms, after the private sector does its due diligence during development and implementation.

”How do we make sure that at multiple gates the algorithms are doing what they need to do? So first the developing gate, then at the implementation gate in the healthcare delivery system, and then there needs to be a continuous monitoring piece,” said Afsar. “From a policy standpoint, what could that look like?”

Regulators should become more involved when AI begins to be used in higher-risk use cases, like as a diagnostic agent, according to Aashima Gupta, Google Cloud’s global director of healthcare strategy and solutions.

“This has moved so fast, just in a year, and we’re talking use cases. There’s a lot of discussion, and now we’re talking actual use cases of actual impact. That absolutely needs to be regulated,” Gupta said in an interview.

Other experts said requiring more transparency from AI developers would go far in making their products more trustworthy.

That’s especially as a small number of gargantuan companies are actually responsible for building large language models, and “the rest of us are a little bit at their mercy,” said Athenahealth’s Lane.

“What I would love to see from them is much more detailed quantitative information on the behavior of the system with respect to correctness, with respect to omissions, not just on control test sets the way they tend to publish in white papers but in relevant real-world conditions,” Lane said. “If anybody is in a position to coerce the large tech companies about this, it would be the government.”

The difficult road ahead

Though AI stakeholders are split on what they’d like to see from Washington, they agree on one thing: Regulators seeking to erect guardrails around AI are facing a Sisyphean endeavor, tasked with overseeing a technology that’s constantly changing and already in the hands of doctors.

Washington will have to struggle with a number of open questions: regulating different types of AI technologies, accounting for different workflows across organizations, making sure AI is available for all patients, ensuring accuracy and not exacerbating disparities, said Epic’s Hain.

“It’s a challenging path to weave,” Hain said.

Despite the growth of private sector-led standards groups, the HHS will have to weigh in on those questions soon. Regulators are aware of the tightrope they tread.

“It is critical that we ensure the fair, appropriate, valid, effective, and safe use” of AI in healthcare, ONC’s Smith said. “Balancing this critical need while enabling innovation to flourish is a primary objective of ONC’s policies.”

Still, companies with a stake in the AI game are waiting to see what emerges from Washington — whether any regulations will fill gaps not covered by standards organizations while leaving the private sector free to innovate, or require onerous testing and reporting requirements that could slow down the breakneck speed of AI advancement.

“I hope they will be more permissive than restrictive until proven that we need more regulation, because these technologies are going to save lives. And we really don’t want to slow them down,” said Mayo’s Ross. “We want to go as fast as possible but no faster than is safe.”

The buzz around AI has increased as the ChatGPT model takes the internet by storm. It’s no different in healthcare. The ChatGPT AI tool has passed the U.S. medical licensing exam, authored a number of scientific papers and is being used to appeal insurance denials, hinting at real-world applications for the algorithms.

However, actual adoption of AI-based tools in the healthcare industry is low, despite research suggesting benefits of the tech.

A new paper from McKinsey and Harvard researchers estimates that broader adoption of AI could lead to savings between 5% and 10% in healthcare spending, or roughly $200 billion to $360 billion a year. The estimates are based on AI use cases employing current technologies that are attainable within the next five years, without sacrificing quality or access.

That’s a big “if,” as AI uptake has been limited in the industry due to a lack of trust among patients and doctors, heterogeneous data and misaligned incentives, the researchers said.

But broader adoption of AI is likely in the near future and should also have a slew of non-financial benefits like better healthcare quality, increased care access and better patient and doctor satisfaction, according to the paper.

For hospitals, the cost savings come mostly from improved clinical operations, quality and safety — like optimizing operating rooms, or detecting adverse events. The benefits are similar for physician groups, which could leverage AI for continuity of care, like referral management.

Health insurers would see savings from use cases that improve claims management, like automating prior authorization, along with healthcare and provider relationship management, including preventing readmissions and provider directory management.

Based on the AI-driven use cases, private payers could save roughly 7% to 9% of their total costs, amounting to $80 billion to $110 billion in annual savings within the next five years. Physician groups could save 3% to 8% of costs, amounting to between $20 billion and $60 billion in savings.

Meanwhile, hospitals could see savings between 4% to 11%, or between $60 billion and $120 billion each year, the report estimates.

Despite the potential benefits of AI in healthcare and rising funding in the space, the use of AI by doctors for clinical cases is still hit-or-miss. A recent study published in JAMA found a “paucity of robust evidence” to support claims that AI could enhance clinical outcomes.

Despite that, the Food and Drug Administration has been accelerating approvals of medical artificial intelligence tools, authorizing more than 520 devices as of November. And, experts believe 2023 could be an inflection point for adoption as more evidence around AI’s efficacy in real-world settings emerge.

Sponsored content

By Nuance

As long-time users of Dragon Medical One, Fisher-Titus Medical Center was accustomed to using speech recognition technology to ease the burden of clinical documentation. But as they looked to deepen their Nuance footprint with DAX, their administration was unsure how it would fit into their existing workflows. Since deploying Nuance DAX, Fisher-Titus has reaped the benefits of using the two solutions in tandem to optimize productivity, enhance the patient-provider experience, improve the quality of documentation and drive better outcomes.  

“I’m using both Dragon Medical One and DAX throughout the day interchangeably. It’s sometimes easier to use Dragon Medical for one chart and DAX for another, and I can preferentially pick which one I’m using for which visit. I know which one I’m grabbing my mic for, and which one I’m grabbing my computer for.” — Samuel Ross, MD, Fisher-Titus Medical Center

Dragon Medical One and DAX: a perfect marriage

One of the primary benefits that clinicians at Fisher-Titus have realized by using Dragon Medical One and DAX together is enhanced efficiency. Samuel Ross, MD, offers insight into his streamlined workday, saying “I come in about 30 minutes early, because that’s about all it takes for me to review my notes and prep my charts for the day. But the moment I’m finished with my last patient I’m done in the office. I maybe stay an extra 5 minutes to wrap up my DAX recording and send in my orders. And then I go home. I’m not spending hours and hours going back and having to document.” He notes that Dragon Medical One is particularly helpful in editing his notes post-encounter, allowing him to complete his documentation in a timely manner. Glenn Trippe, MD and CMIO, reinforces the time savings his organization has seen across providers, saying “easily, one to two hours a day is saved on average and that is really significant.” 

The reduced time pressure, distraction and cognitive overload of their physicians has enhanced the patient experience as well. “My patients like it because I can actually have a conversation with them. Patients find this much more personal and less institutionalized,” remarks Dr. Ross.  

Moving from cookie-cutter to custom-made documentation

Prior to deploying DAX, many providers at Fisher-Titus relied on templates. Dr. Trippe shares the impact that DAX has had in moving beyond ‘cookie-cutter’ Case study Nuance Dragon Medical One Dragon Ambient eXperience templates, saying “What you get from a templated note is a very sterile note. Yes, they had all the information, but they really didn’t tell the story as well as we can. Now, with these tools we really can get a true patient story into the note.” Dr. Ross adds that DAX has also been invaluable in catching the small but important details that may have otherwise been overlooked or omitted from the patient record—an important component in improving documentation accuracy. “Being able to just pull up DAX and use Dragon Medical One at the same time to adjust my notes has exponentially improved my documentation,” remarks Dr. Ross. He shares that while he often uses DAX to document during the patient encounter, he values the flexibility that he now has in choosing a tool that is suited to the circumstances of each encounter. “I have more freedom in how I need to do something, because not every patient is the same.”  

Using technology to retain and recruit physicians

Unsurprisingly, the reduction in after-hours work has had a positive impact on physician morale. Dr. Trippe notes that beyond just improving physician satisfaction, Nuance’s technology has been pivotal in reinvigorating physicians at the cusp of quitting. “DAX has salvaged at least 2 positions, if not more, that were ready to quit because the loads were just too much. DAX was able to offload them sufficiently so that they can continue on in our organization successfully.” Physician retention is a significant financial consideration, he adds, noting that the minimum cost of replacing a physician is approximately $250,000. 

Similarly, Dr. Trippe shares that the organization’s investment in these sophisticated technologies has been advantageous for recruiting. “We present ourselves as a forward-thinking, technologically rich organization. We use DAX and Dragon Medical One to help with recruiting, and our technological expertise has positively contributed to the decisions of recruits.”  

A future-proof platform  

With strong outcomes and utilization, Fisher-Titus looks to continue expanding on their successes with Nuance. “There’s a strong commitment from our institution. Our senior leadership has seen the benefits of this program, so they are strongly committed to being involved with this for the future.”  

Learn more about Dragon Medical One and Dragon Ambient eXperience. 

Details are emerging on a new HHS task force faced with a monumental task: creating a regulatory structure to oversee utilization of artificial intelligence in healthcare.

An executive order signed by President Joe Biden in October directed the HHS to create a comprehensive plan for assessing AI before it goes to market, and monitoring performance and quality once the technology is actually in use. The executive order gave the task force 12 months to deliver after it starts work. 

That clock is ticking. And it’s a tall order: Though individual agencies have promulgated rules around AI in the past, regulators are faced with putting guardrails around ever more advanced models, including some that automatically learn and evolve in real time without human feedback.

The task force is thinking about how to create appropriate safety programs and strategies to manage those technologies, according to Greg Singleton, chief artificial intelligence officer at the HHS.

“The real-time learning environments — we’re going to have to come up with and develop some sort of assurance, monitoring, risk-management practices around — just to kind of put a buffer around those, so we’re comfortable with them. And a lot of good work is going into that,” Singleton said during a panel on AI policy at the HIMSS conference in Orlando, Florida, in March.

Task force makeup

The HHS’ task force is composed of senior members of the administration — mostly the heads of the agencies making up HHS: the CMS, the Food and Drug Administration, the Office of the National Coordinator, the National Institutes of Health and the Centers for Disease Control and Prevention, according to Singleton and another HHS source.

Beneath those leaders, the task force has specific working groups around core issues in AI: drugs and devices, research and discovery, critical infrastructure, biosecurity, public health, healthcare and human services, internal operations, and ethics and responsibility, the HHS AI head said.

Those working groups raise questions and bring suggestions to senior leadership on the task force, which meet monthly and provide guidance on how to proceed.

The task force also plans to engage with the private sector to get their perspective on where it should focus, Singleton said. In the absence of a comprehensive federal strategy, AI developers and their technology and hospital partners have been standing up governance groups to self-police their utilization of AI.

“I’m expecting we’ll have a series of listening sessions and engagement opportunities, perhaps some request for comment, a number of communications coming out,” Singleton said. “There will be more details to come.”

U.S. lags in regulations

Despite numerous bills that have bubbled up in Congress, the U.S. has taken a largely hands-off approach to AI oversight so far, even as the technology becomes increasingly pervasive in the public sphere and a number of industries, including healthcare.

Tech giants like Google and Microsoft are notching partnerships with major hospital systems to use their reams of patient data to train and deploy AI models. Use cases for the technology run the gamut from streamlining nurse handoffs between shifts to improving contouring for radiotherapy planning to nudging at-risk patients when they’re overdue for a checkup.

The U.S.’ inaction stands in sharp contrast to the European Union, which on Wednesday approved the AI Act — one of the first laws in the world to erect comprehensive strictures around AI. The AI Act divides the technology into levels of risk ranging from “unacceptable” to “low hazard,” which determine how much scrutiny an AI application will face.

There are concerns the law — which also includes steep fines for noncompliance — will stifle AI progress in the bloc. 

Regulators and legislators in Washington have said they don’t want to inadvertently tamp down innovation in the private sector as the U.S. — the home of most of the heaviest hitters in AI development, like Microsoft, Alphabet and Meta — continues to jockey with China for AI supremacy.

Building on agency authorities

Still, federal agencies have taken some steps they could build on to oversee AI in the healthcare space, Singleton said.

The CMS is exploring how the authorities already granted to it by Congress around the administration of patient benefits and reimbursements could be extrapolated to regulate AI, Singleton said. There’s some precedent: The agency finalized a rule last spring prohibiting Medicare Advantage plans from using AI to make bulk determinations on coverage decisions.

Similarly, the HHS Office for Civil Rights is looking at how Section 1557 of the Affordable Care Act, which prohibits discrimination in the administration of care, could apply to AI governance, according to the HHS AI lead.

In addition, the ONC, which oversees U.S. health IT, finalized a rule late last year requiring government-certified electronic health records using algorithms for clinical decision support to disclose how the AI is maintained and updated. The FDA also has an approval process in place for medical devices using AI. Regulators have cleared almost 700 to date.

Approving AI-based medical devices is entirely different than approving models that constantly evolve, such as generative AI, Singleton said. Still, the technology isn’t entirely new for regulators.

“Those are just examples of how we’re approaching it, looking at what we have in our stable and where that aligns with our values,” Singleton said. “Is it the end result answer? Does it give the answer for what AI should be in 10 years? No.”

Artificial intelligence is already used widely in healthcare, but generative AI is a big — and somewhat risky — step forward due to rapid deployment of the technology paired with a lack of regulation and oversight, industry experts said at the HIMSS conference in Chicago in April 2023.

As a result, tough ethical questions are dogging generative AI like GPT-4, OpenAI’s latest large language model that backs the premium version of the wildly popular chatbot ChatGPT. Still, generative AI has huge potential to help improve U.S. medical care, proponents say.

One of the biggest companies in the space, Microsoft, has been working with GPT-4 developer OpenAI over the last eight months to understand the technology’s implications for healthcare. Generative AI is showing early success in a number of use cases, including simplifying explanation of benefits notices and writing prior authorization request forms, Peter Lee, corporate vice president of Microsoft Healthcare, said in a keynote panel .

Physicians can give the AI prompts to help them interpret novel medical cases or rewrite a patient conversation in a standardized format, Lee said.

Microsoft recently unveiled plans to embed generative AI into clinical software from Epic, the biggest hospital EHR vendor in the U.S.

In April 2023, Microsoft and Epic went live with the first sites integrating GPT into EHR workflows to automatically draft replies to patient messages. The companies also are bringing generative AI tools to Epic’s hospital database, allowing laypeople to ask the AI general questions instead of needing a data scientist to query specific data.

Epic is investing a significant amount of R&D resources into generative AI, Seth Howard, Epic’s vice president of R&D, said in an interview.

The health IT company also is researching the use of generative AI to summarize a patient’s medical history, and translate patient-facing materials between different languages and reading levels, to improve health literacy.

“That’s kind of a big category of work,” Howard said. “There are many use cases we’ll put out over the next year.”

Google also is making its own large language model, called Med-PaLM 2, available to a select group of customers to explore use cases. Med-PaLM was specifically trained on medical data, allowing it to sift through and make sense of massive amounts of healthcare information. Still, it has room for improvement in the complexity of queries it can perform and meeting product excellence, according to Google’s AI team.

Med-PaLM won’t be used in patient-facing settings, Google Cloud head Aashima Gupta said in an interview. Instead, hospitals could use the AI to analyze data to help diagnose complex diseases, fill out records or as a concierge for patient portals.

One notable use case for generative AI is streamlining medical notetaking, experts say. Physicians can spend up to six hours a day logging notes in their EHR, which can result in cutting back on time with patients and contribute to burnout.

Leading clinical documentation companies are integrating generative AI technology that they say improve their products’ accuracy and speed.

Nuance, a documentation company owned by Microsoft, said last month that it had integrated GPT-4 into its clinical notetaking software. This summer, providers using existing Nuance documentation products can apply to beta test the application, called DAX Express.

Ten to 15 clients summing 300 to 500 physicians will test the product in private preview in mid-June, before it’s generally available in early fall, said Diana Nole, general manager at Nuance.

Suki, a documentation company that’s partnered with Google, launched its generative AI-powered “Gen 2” voice assistant earlier this month.

Now, Suki can generate a clinical note ambiently from a conversation and fill in the note automatically, CEO Punit Soni told Healthcare Dive. Closer to the end of the year, doctors will also be able ask the AI questions and give it commands, such as plotting a patient’s A1C levels over the last three months, Soni said.

Gen 2 currently is being piloted with “very few” customers, according to the CEO. Suki plans to make it generally available later this year.

”The interest is unbelievable,” Soni said. “If you could have somebody to just do all your notes for you — wouldn’t that be awesome?”

Weighty risks

To be sure, generative AI makes mistakes, raising pernicious ethical questions around accuracy, equity and accountability. That means the medical community needs to have a robust ongoing role in deciding whether, when and how this type of AI is used, experts said at HIMSS.

One issue with AI is the black box problem: How can a result be trusted if it’s unclear how the AI got there?

It’s easy to be fooled into thinking that this type of technology is able to reason, but users have to remember generative AI doesn’t engage in deliberation — just predicts the next set of words that makes the most sense, said medical ethicist Reid Blackman.

“Maybe we’re okay with magic that works. That’s one option. The other option is to say okay, if you’re making a cancer diagnosis, I need to absolutely understand the reasons why you’re making that diagnosis… The problem is that GPT doesn’t give reasons. It gives what looks like reasons,” Blackman said on a panel.

Generative AI also has issues with accuracy. Large language models are known to hallucinate, or provide answers that are factually incorrect or irrelevant. GPT-4 “still has many known limitations that we are working to address, such as social biases, hallucinations, and adversarial prompts,” OpenAI’s website reads.

Bias is another key worry with AI, especially in healthcare. If an algorithm is trained on biased data, or its results are applied in biased ways, it will reflect and perpetuate those biases, experts said.

Accountability also remains a concern. Since generative AI is such a new field, it’s unclear which stakeholder — the owner? the user? the developer? — is at fault for any negative consequences stemming from its use, according to Kay Firth-Butterfield, CEO of the Centre for Trustworthy Technology.

“Who do you sue when something goes wrong? Is there somebody to sue?” Firth-Butterfield asked during the keynote panel.

More than 27,000 stakeholders have signed an open letter asking AI labs to immediately pause training AI systems more powerful than GPT-4 for at least six months. The letter argues that industry should use the moratorium to develop shared safety protocols and work with policymakers to develop AI governance systems.

“You shouldn’t be exploring, engaging in artificial intelligence, without having some understanding of responsible AI issues,” said Firth-Butterfield, who signed the letter. “We are hurtling into the future without actually taking a step back and designing it for ourselves.”

‘Mysterious,’ ‘awe-inspiring,’ ‘frustrating’

AI companies argue many of these problems can be ameliorated with careful human oversight. Organizations should be explicitly aware of the dataset a model was trained on and when precisely it should be used, experts said.

Microsoft’s Lee said while questions about generative AI are important, especially those around accountability, the black box problem might not be as troubling as some say.

“We’ve been unable to provide its incapable [of reason],” Lee said. ”It’s a mysterious and awe-inspiring and frustrating research roadblock that we’ve been unable to prove that. I have to admit, as a scientist, that this black box issue, at some point of development starting with GPT-4, actually might not really exist.”

And there are techniques to ensure generative AI gives a reliable output — a field called prompt engineering, where scientists figure out how to query the AI in a way that produces trustworthy results, Epic’s Howard said.

Both Suki and Nuance declined to share the accuracy rates of their generative AI-based notetakers, but said accuracy was high enough that they’re comfortable instituting them in a real-world setting.

Soni said Suki could publish its Gen 2 accuracy rates in a few months. And Nuance has been working with AI for years, Nole said.

“The level of accuracy we were already producing is advanced. We know how it works. We know how it acts. Now, we’re just utilizing GPT to kind of raise the level,” Nole said.

As AI has advanced, some models have outperformed doctors on diagnostic tasks, creating concerns that machines could one day replace clinicians. Experts at HIMSS agreed that day is nowhere close, even with generative AI.

Med-PaLM’s ability to pass a medical exam is “in no way indicative of medical knowledge or competence,” Alan Karthikesalingam, Google’s health AI research lead, said in an interview.

“There are a lot of material safety, bias and ethical concerns that need to be attended to very carefully,”  Karthikesalingam said. But replacement? “I think by and large that on its own is hopefully not a huge concern to us physicians. The positioning of these systems is very much as tools.”

Suki and Nuance stressed that their AI-created notes allow clinicians to edit and approve what is finalized in the EHR. Generative AI is an instrument for doctors, not a substitute for them, AI executives said.

Still, “anyone who can make predictions on what this epoch will bring is a fool,” Soni said. “What I can tell you is that in the foreseeable future, I see AI being a very strong and serious assistant that makes a lot of our lives better.”

Google Cloud launched a tool in March backed by generative artificial intelligence that allows clinicians to search for information across patient notes, scanned documents and other clinical data.

That tool, called Vertex AI Search for Healthcare, is now available for Google Cloud customers, the tech giant announced during the HIMSS conference in March.

Along with search, the technology — which can be directly integrated into physicians’ workflows — can also answer questions, like queries about a patient’s medical history.

The tool is one of a flurry of generative AI-backed software over the past year developed by technology behemoths like Google, Microsoft and Amazon, along with healthcare-specific vendors like Oracle Cerner and Epic. Most are targeted toward streamlining clinical notetaking.

A number of hospitals are already using the tools despite concerns about the ethical use of AI, including the potential for errors or exacerbating existing disparities — and a lack of regulation from the federal government.

Last fall, President Joe Biden signed an executive order calling for the responsible use of AI in healthcare, but the HHS has yet to stand up a regulatory structure overseeing the futuristic technology. A department task force created by that executive order has one year to complete a plan.

As the threat of oversight looms, industry is rushing to weave AI products into the operations of hospital and health plan clients.

Google has now unveiled yet another suite of artificial intelligence-backed tools for healthcare aimed at ameliorating administrative burden on clinicians. The technology — which Google made available for early access in October — is an offshoot of Vertex AI Search, a platform that helps developers build search engines for their websites and data stores, that was tuned specifically on medical information.

In a press release on the news, Google touted the technology’s understanding of medical terminology that allows its search and question-answering capabilities to answer complex queries. Vertex AI Search for Healthcare also cites specific data points that generated the answers to give clinicians more transparency into where the data originated, to reduce the risk of hallucinations (a tech term for when AI makes up a response).

Vertex AI Search for Healthcare integrates with another Google tool aimed at making data more interoperable, called Health Data Engine. It also integrates with MedLM, a group of so-called “foundation models” that Google launched in December. MedLM is a portfolio of algorithms trained on a broad set of healthcare data so they can be applied to a variety of use cases.

As a result, “there’s nothing in the market like [Vertex AI Search for Healthcare],” said Aashima Gupta, Google Cloud’s director of healthcare, in an interview at HIMSS.

Vertex AI Search for Healthcare’s integration with MedLM is available in early access for select customers, according to the release.

Highmark Health, an integrated health system in western Pennsylvania, has taken the Google components to centralize and standardize its data, and build a tool around it that suggests next-best actions for health plan members, according to Brian Lucotch, president of Highmark tech subsidiary EnGen.

Google also announced two new capabilities for MedLM on Tuesday: one specific to the classification of chest X-rays and one that can provide a chronological list of patient conditions.

Google plans to add more models in the coming months. Those include a model based on Gemini, a group of large language models Google released in December as a competitor to OpenAI’s ChatGPT, which is backed by Microsoft.

In February, Google paused Gemini’s image generation capabilities after it released historically inaccurate images, causing Google’s stock price to dip.

Artificial intelligence can improve clinicians’ diagnostic accuracy, but biased models make it worse — and explaining the technology’s logic didn’t help providers mitigate potential errors, according to a study published in JAMA.

The research found an AI model improved clinician accuracy when examining accounts of patients hospitalized with acute respiratory failure, and accuracy increased even more when clinicians’ used AI models alongside explanations of the technology. 

But a systematically biased model, or models that consistently misdiagnose some patient subpopulations, decreased their diagnostic accuracy, and explanations of the model’s reasoning didn’t significantly help them improve, according to the study.

Systematically biased models could lead to errors or patient harm, the study notes. For example, a model that’s trained on data where women are often underdiagnosed with heart disease could be inaccurate once it’s deployed. 

Ideally, providers would be able to ignore models if they seem off, and explanations have been offered as a way to help them understand the model’s reasoning before they act on recommendations. Guidance from the Food and Drug Administration has highlighted the value of explanations to improve interpretability, researchers wrote.

But the JAMA study found explanations might not be enough to mitigate harm. 

The research showed hospital physicians, nurse practitioners and physician assistants nine clinical vignettes of patients hospitalized with acute respiratory failure, including symptoms, physical exams, lab results and chest radiographs. 

They were then asked to determine the likelihood of pneumonia, heart failure or chronic obstructive pulmonary disease. Some vignettes included AI models with image-based explanations to lay out its logic. 

Clinician accuracy increased over baseline by 2.9 percentage points when using a standard AI model and by 4.4 percentage points when clinicians were also shown AI model explanations.

But a systematically biased model decreased accuracy by 11.3 percentage points, which only improved a nonsignificant 2.3 percentage points when the model included explanations for its recommendations. 

“Although standard AI models improve diagnostic accuracy, systematically biased AI models reduced diagnostic accuracy, and commonly used image-based AI model explanations did not mitigate this harmful effect,” researchers wrote.

Clinicians’ AI literacy might one challenge preventing them from rooting out biased recommendations, the study’s authors said. Nearly 67% of participants weren’t aware models could be systematically biased.

They also might need more experience and training with the explanations. And using images might be a poor way to describe what the model is doing, and other methods, like text descriptions, might be more helpful, researchers wrote. 

Dive Brief:

• Generative artificial intelligence can be used to pull social determinants of health data, like housing or employment status, from clinician notes to identify patients who need additional support, according to a new study.
• Large language models trained by researchers could identify 93.8% of patients with adverse social determinants of health, while official diagnostic codes include that data in only 2% of cases. 
• The finely tuned models were also less likely than OpenAI’s GPT-4 to change their determination when demographic information like race or gender was added. Algorithmic bias is a major concern for AI use in healthcare, amid fears the technology could worsen health inequities. 

Dive Insight:

Large technology firms like Microsoft, Google, Oracle and Amazon have invested heavily in generative AI over the past year. Many firms have honed in on using the tools, which can create original text and images, to lessen administrative burden for providers, through use cases like automatically documenting conversations between patients and doctors. 

But using large language models to extract social determinants of health data is another area of potential promise for the technology, according to the study published in npj Digital Medicine. 

Social determinants, which include information like access to transportation, stable housing and healthy food, are key to health outcomes. But the data often isn’t systematically organized in health records, making it challenging to pull that information into databases or easily identify patients who may need help.  

“Algorithms that can notice things that doctors may miss in the ever-increasing volume of medical records will be more clinically relevant and therefore more powerful for improving health,” Danielle Bitterman, a study author and faculty member in the Artificial Intelligence in Medicine Program at Mass General Brigham, said in a statement.

Researchers set out to build language models that could extract that data from health records. They reviewed 800 clinician notes from 770 patients with cancer who received radiotherapy, tagging sentences that referenced social determinants like employment status, housing challenges, transportation issues, parental status and social support. 

The researchers then tested the models with an additional 400 notes from patients who had been treated with immunotherapy. 

Their finely tuned models could consistently find references to social determinants, even though sentences could be linguistically complex and few sentences in the notes referenced social issues at all, researchers said. 

The study’s authors also created synthetic clinical notes using ChatGPT, and used that data to compare their models’ performance against the general language model GPT-4.

Though both model types changed their determinations about social determinants when demographic and gender information was added, the discrepancy rate for the finely tuned models was half that of GPT’s.

Fine-tuning language models could be one strategy to lessen the risk of bias, but more research is needed, Bitterman said in a statement. 

The HHS’ Office of the National Coordinator for Health Information Technology finalized a sweeping rule in December aimed at boosting data interoperability and patient access, including a provision to establish transparency requirements for artificial intelligence in health software.

Under the rule, developers of clinical decision support and predictive tools certified by the ONC should enable a limited set of identified users to access information like the value of the intervention, how it should be used, known risks or inappropriate uses, and how the tool is maintained and updated. The requirements will go into effect at the end of next year. 

The final regulation also included a number of information blocking updates and exceptions, including for actors sharing data under the TEFCA framework, which went live in mid-December. 

Artificial intelligence’s role in the healthcare sector has become an increasingly significant issue for the industry, as lawmakers and experts mull the potential benefits to drug discovery, medical documentation and data analysis. 

But some are concerned the rapid rollout of new tools — including generative AI products that aim to reduce the number of healthcare workers’ administrative tasks — could perpetuate inaccuracies or bias. 

The new regulations surrounding health clinical decision support software and predictive tools aim to boost transparency and mitigate risk in the industry, said Steven Posnack, deputy national coordinator for health information technology, during a Wednesday media briefing.

“It's been a long time in the making to have an update to this certification criterion to be reflective of technical evolution in industry, as well as in response to some of the new challenges posed by artificial intelligence and machine learning and other types of large language models and the like,” he said. 

The Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing, or HTI-1, rule also added a number of provisions for interoperability and data sharing. 

The rule adopts an updated standard for interoperability — the United States Core Data for Interoperability Version 3 — starting at the beginning of 2026. The industry is currently using Version 1, and the regulation creates a “predictable step” for the sector to move toward and build new services and connectivity, Posnack said. 

The standard includes updates that expands the data elements and classes that can be exchanged, which the ONC said will help promote equity, reduce disparities and promote public health data interoperability. 

The rule details some exceptions to information blocking regulations too. Actors who don’t fulfill information sharing requests in any matter besides the Trusted Exchange Framework and Common Agreement, or TEFCA, aren’t considered to be information blocking.

TEFCA, which sets a governance framework and technical requirements for nationwide data exchange, went live with five Qualified Health Information Networks this week. It marked a significant milestone in the yearslong effort, HHS leaders said. But the framework needs industry buy-in to succeed.

“We make it a kind of clear set of guidelines and approaches associated with this information blocking exceptions for those actors to be responsive, and to either make that electronic health information accessible, exchangeable or usable through TEFCA, in order to promote its use and provide convenient incentives for TEFCA participation,” Posnack said.

Dive Brief:

• Three-quarters of healthcare providers and pharmaceutical professionals think artificial intelligence-related technologies will be “widespread” in the next three years, according to a new survey from consultancy Berkeley Research Group.
• However, only about 40% of surveyed professionals said their organizations reviewed or planned to review AI regulatory guidance.
• Surveyed providers were more optimistic than pharmaceutical professionals that regulation could provide necessary guardrails for AI use in healthcare. Industry stakeholders have raised concerns that a too-rapid adoption of AI could pose risks to data privacy and exacerbate health inequities.

Dive Insight:

Generative AI has been a target of healthcare investment as companies say the technology, which can generate content like text and images, has the potential to improve diagnostic screenings and decrease the administrative load on clinicians.

AI investment has held steady even as other health markets contracted last year. The market size of healthcare AI is projected to balloon from $28 billion in 2024 to over $180 billion in 2030.

Concerns surrounding the technology have also mounted, prompting the federal government to mull how to best regulate AI in healthcare.

“It is very clear that not enough is being done to protect patients from bias in AI,” Sen. Ron Wyden, D-Ore, said during a Senate finance committee meeting this month. “[...] Congress now has an obligation to ensure the good outcomes from AI set the rules of the road for new innovations in American healthcare.”

Three-quarters of provider respondents surveyed by BRG said they felt “confident” that future regulation and guidance could impose guardrails on AI. Over 80% of providers also said they felt confident in their organizations could comply with potential regulations and said that accuracy, safety, data privacy and HIPAA violations were top concerns. In contrast, 56% of pharmaceutical professionals had confidence in AI regulation.

Surveyed professionals were especially concerned with regulations regarding data privacy, with over 50% of provider respondents saying cybersecurity and data management were their chief concern.

Data privacy and cybersecurity concerns have abounded as the amount of healthcare breaches rise. The federal government is eyeing cybersecurity guidelines for the sector and the Federal Trade Commission levied fines last year against GoodRx and BetterHelp for inappropriately disclosing personal health information.

Still, more than four in ten provider respondents said AI has already been widely accepted and implemented in their organizations, according to the survey. Providers said AI use in diagnostics and imaging, preventative screenings, health predictions and patient safety would significant impact the industry. Administrative tasks, including revenue cycle management and supply chain management, could also be positively impacted.

The survey polled 150 provider and pharmaceutical professionals.

Nuance’s automated clinical documentation product embedded in the Epic electronic health record is now generally available to provider customers, the Microsoft-owned technology company announced in January.

Clinical documentation has become a popular use case for generative artificial intelligence in healthcare, with a number of technology companies like Amazon, Oracle and NextGen Healthcare unveiling their own products last year.

The Dragon Ambient eXperience Copilot records doctor-patient conversations and drafts notes directly in Epic’s mobile app for physician review. Nuance first announced over the summer that the scribe would be integrated into Epic, one of the largest EHR vendors in the country.

More than 150 health systems, hospitals and medical centers are set to deploy the DAX Copilot system embedded with Epic, according to Nuance. Those include Rhode Island hospital operator Lifespan Health and North Carolina academic medical system UNC Health.

Providers have long reported that they spend hours a day on EHR-related administrative tasks, often cutting into their time with patients and into after-work hours. The growing burden of administrative tasks is sometimes linked to provider burnout, a problem that spiked during the COVID-19 pandemic.

Nuance, which was acquired by Microsoft for nearly $20 billion in 2022, said DAX users cut the amount of time spent on clinical documentation by 50%, allowing them to save time per visit and add additional appointments. 

Healthcare executives have shown interest in adopting generative AI tools, especially products that could enhance operational efficiency like documentation, communication with patients or workflow automation, according to a recent survey by Klas Research. 

Though relatively few organizations currently use the tools, more than half surveyed said they were looking to implement or buy them within the next year. 

But some experts have raised concerns about a too-rapid deployment of generative AI technologies in healthcare, citing concerns about the risk of errors and who would be accountable when mistakes occur. Accuracy and reliability topped the list of potential challenges in the Klas survey.