In one incident, a state investigator recalled creating a Character.AI account and engaging in conversation with a chatbot named “Emilie.” Over the course of their conversations, Emilie described herself as a psychologist who had attended medical school at Imperial College in London.
Pennsylvania has filed a lawsuit against Character.AI, claiming that the company misrepresents some of its chatbots as licensed professionals capable of providing real-world medical advice.
According to The Hill, the state says that some of Character.AI’s chatbots have been caught adopting quasi-professional personas and dispensing sensitive medical advice. In some cases, they’ve even claimed to be psychiatrists, therapists, or other medical professionals.
The lawsuit provides at least one example of a Character.AI chatbot falsely stating that it was licensed to practice medicine in Pennsylvania, even providing a false license number for verification.
Attorneys for the state now say that the California-based company has, in effect, engaged in the “unlawful practice of medicine and surgery.”
“We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional,” Pennsylvania Gov. Josh Shapiro said. “Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”
The lawsuit notes that Character.AI lets users create and release chatbots, which can be trained to imitate certain people and personalities. Some of these heavily-customized characters,” the state says, regularly “purport to be health care professionals.”
In one incident, a state investigator recalled creating a Character.AI account and engaging in conversation with a chatbot named “Emilie.” Over the course of their conversations, Emilie described herself as a psychologist who had attended medical school at Imperial College in London.
The investigator told Emilie that he was feeling empty, tired, and sad. In response, Emilie allegedly “mentioned depression and asked if the [investigator] wanted to book an assessment.” When asked if medication could help relieve his symptoms, Emilie said she was qualified to offer an opinion because it would be within her “remit as a Doctor.”
Character.AI has since issued a statement saying that its “highest priority is the safety and well-being of our users.” It added that the company makes extensive use of disclaimers, “making it clear that users should not rely on Characters for any type of professional advice.”
“We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction,” a Character.AI spokesperson said in a statement.
Character.AI is no stranger to controversy. It recently settled a lawsuit with a Florida family who claimed that their teenage son committed suicide after being encouraged to take his own life by a chatbot. The state of Kentucky is also litigating a related claim against the company, saying that Character.AI chatbots have “preyed on children and led them into self-harm.”
Sources
AI: “Deepfake doctor” chatbot is hit with lawsuit in US
Pennsylvania lawsuit alleges AI chatbots posed as doctors, therapists
Join the conversation!