New study suggests AI is starting to outperform doctors, sheds light on growing capabilities

SAN FRANCISCO (KGO) -- A newly published study suggests artificial intelligence systems can outperform doctors in complex medical reasoning tasks, raising new questions about how clinicians, hospitals and patients may rely on the technology in the future.

The research, released Thursday, examines how newer AI "reasoning models" performed not only on test-style questions but on complicated diagnostic scenarios that more closely mirror real-world medical decision-making.

Jonathan Chan, a Stanford doctor and one of the study's authors, said the pace of improvement has been striking.

"And the summary point is these models are getting even more capable and kind of shockingly fast way," Chan said.

RELATED: Can artificial intelligence chatbots outperform human doctors? Here's what new Stanford study found

"That really begs the question, where are the things that uniquely human needs are, and where are things you could trust a computer for? It's not so obvious anymore, and I think it dynamically changes."

Chan emphasized that the study was not intended to answer a simple yes-or-no question about whether human doctors are still needed. Instead, he said, it highlights how difficult medical reasoning can be replicated by AI when cases move beyond straightforward multiple-choice answers.

In the interview, Chan described how doctors often approach a patient with multiple symptoms - such as fever and weight loss - by generating possible explanations, weighing probabilities and explaining their thinking.

He said AI systems have become "pretty good at even answering these complex reasoning questions."

Still, Chan rejected the idea that humans could be removed entirely from clinical decision-making.

"So no, humans cannot be cut out of the loop. And trust is a very loaded word," he said. "But boy, would it be foolish for me and for other people not to be using these things to help us in diagnosis and medical reasoning? It would be like trying to practice without the internet."

He compared AI tools to online medical resources doctors already use, arguing they can serve as a way to double-check conclusions - even if they are not always correct.

"We really should be probably using and double-checking things with AI," Chan said. "Not because it's always right by the way. But boy, are humans not always right either. And the combination could be the more effective way we have the grounded responsibility and working in the loop."

Chan said there are limited areas where AI could soon take on tasks with minimal human involvement, particularly administrative work.

He cited documentation and summarizing hospital notes as examples, noting that doctors' time is better spent with patients than on paperwork.

He added that medication refills are already being handled by automated systems in places such as Utah, a development he said would have seemed unthinkable just a few years ago.

At the same time, Chan acknowledged anxiety within the health care community, pointing to concerns raised by mental health professionals and recent actions in California and Illinois related to AI use in therapy.

"I get it, I totally get it. There's a lot of panic and angst here," he said. "It is scary. We have to hold these things to very high standards because it matters."

The study also has implications for patients' everyday interactions with technology, he said, such as using chatbots to better understand medical summaries or jargon after brief doctor visits.

"I don't think it's that weird to ask questions about your health," Chan said, adding that AI can help explain information doctors may not have time to fully walk through.

On the question of whether AI could eventually replace medical jobs, Chan said some administrative and support roles may change, but he does not foresee dwindling demand for clinicians.

"There's a lot more complexity and responsibility involved," he said of roles such as doctors, nurses, pharmacists and mental health counselors.

"Those roles are going to shift a lot, but there is unlimited and plenty of need. We're going to need people more than ever."

Chan said the challenge ahead will be learning how to harness AI's growing capabilities while managing the risks that come with its use in health care.