As health systems accelerate AI adoption, technology leaders are discovering that some of the biggest risks aren’t obvious during the vendor demo stage.
At Tampa, Fla.-based Moffitt Cancer Center, one of the most revealing lessons came during implementation of a patient-facing generative AI chat agent, according to Beth Lindsay-Wood, senior vice president and chief informatics and technology officer.
“Since we hadn’t done something like that before, it was more complex than we initially realized,” Ms. Lindsay-Wood told Becker’s. “We had to think through everything, including the large language models being used to develop the chat agent.”
In this case, the agent interacted directly with patients, so Moffitt Cancer Center had to clearly define what parameters it could operate within.
“It’s easy to overlook something, and then the agent assumes it’s OK to discuss topics you never intended it to address,” Ms. Lindsay-Wood said.
One use case involved contacting patients prior to radiology procedures — asking questions about claustrophobia, implants and other preparation details. Moffitt Cancer Center already validated everything again when the patient arrived, but the agent helped the organization reach more patients ahead of time and avoid scheduling disruptions.
“Still, we had to determine exactly how those scripts would run and how personal the interactions should become,” Ms. Lindsay-Wood said. “Some AI agents try to build familiarity by referencing birthdays or previous conversations. We decided some of that went too far for us, so we were very prescriptive about what we did and didn’t want the agent to do.”
Since this was the cancer center’s first truly independent patient-facing agent, the organization had to make clear boundaries.
“It [the agent] couldn’t prescribe anything or acknowledge treatment options,” Ms. Lindsay-Wood said. “If a patient said something concerning, the agent needed to escalate appropriately. There’s a tremendous amount of detail involved.”
Moffitt Cancer Center piloted it internally first and during testing the organization realized there were things it hadn’t considered.
“We had to go back to the vendor and address those gaps,” Ms. Lindsay-Wood said. “It took many months of testing before we were comfortable with the outcome.”
Ms. Lindsay-Wood said Moffitt Cancer Center’s vision is to be out front on AI, but not just for AI’s sake.
“We want to improve cancer care, support discovery and cures, and better manage operating expenses as we grow,” she said. “There’s huge demand across areas like pathology and radiology to pilot AI tools because they can genuinely improve patient care. That’s exciting.”
But, Ms. Lindsay-Wood also believes there’s a wave coming for all IT organizations.
“We have to be disciplined about selecting the right solutions — not just chasing whatever is bright and shiny,” she said. “We should leverage the platforms we already have where appropriate and focus on the AI initiatives that align most closely with our mission and strategic goals: improving care delivery and advancing cancer treatment and cures.”
Currently, Moffitt Cancer Center’s chief risk officer is leading AI governance, which has helped the cancer center become more attuned to issues such as the agentic AI example.
“We already perform cybersecurity reviews and technical evaluations, but what we lacked was a structured way to implement, monitor and continuously validate AI systems,” Ms. Lindsay-Wood said. “For example, if a vendor says they have “a human in the loop,” we now ask: On how many claims? How often? How are those reviews actually happening?”
Moffitt Cancer Center is implementing a governance platform that will help it monitor and document all of this more effectively, but right now, much of it is still manual.
“AI governance is now front and center for us, especially as we implement Epic and Workday — both of which include AI capabilities — while also developing our own AI models in research,” Ms. Lindsay-Wood said. “We need governance across all of those environments. You can’t just buy something and put it into production anymore, especially if it directly touches patients.”
At the Becker's Perioperative Summit, taking place September 14–15 in Chicago, perioperative leaders and healthcare executives will focus on improving operating room efficiency, enhancing patient safety, optimizing staffing and driving innovation across surgical services. Apply for complimentary registration now.
