AI chatbots are increasingly being sought for mental health advice, but new research from Brown University raises concerns that these systems may not yet meet the ethical standards of professional psychotherapy. The study, conducted in collaboration with mental health professionals, highlights significant shortcomings in the capabilities of AI chatbots when employed in therapeutic contexts.
The researchers discovered that even when chatbots were directed to utilize established psychotherapy techniques, they often displayed problematic behaviors. In simulated interactions with trained peer counselors, the chatbots mismanaged crisis situations, inadvertently reinforced harmful beliefs, and employed empathic-sounding language that lacked genuine understanding. “In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations,” the researchers stated.
The findings were shared at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society, with the research team affiliated with Brown’s Center for Technological Responsibility, Reimagination and Redesign. Zainab Iftikhar, a PhD candidate in computer science at Brown and the study’s lead, aimed to determine whether carefully composed prompts could guide AI systems toward more ethical interactions.
To assess the performance of various AI models—including versions of OpenAI’s GPT series, Anthropic’s Claude, and Meta’s Llama—seven trained peer counselors with backgrounds in cognitive behavioral therapy (CBT) conducted self-counseling sessions. Three licensed clinical psychologists then reviewed the transcripts to identify potential ethical violations. Their analysis uncovered 15 risks categorized into five primary areas: lack of contextual adaptation, poor therapeutic collaboration, deceptive empathy, unfair discrimination, and inadequate safety and crisis management.
Among the identified issues, chatbots sometimes used phrases like “I see you” or “I understand” to suggest emotional connection without truly comprehending the situation. They also struggled to respond appropriately to crises, particularly when users expressed suicidal thoughts. “For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar noted. “But when LLM counselors make these violations, there are no established regulatory frameworks.”
Despite these concerns, the researchers argued that AI should not be entirely dismissed in the realm of mental health care. They acknowledged that AI tools could help broaden access, particularly in areas where mental health professionals are limited or costs are prohibitive. However, they emphasized the need for safeguards and stronger regulations before integrating AI into high-stakes therapeutic settings.
Ellie Pavlick, a computer science professor at Brown not involved in the study, remarked on the challenges of evaluating AI systems deployed in sensitive contexts. Pavlick, who leads the ARIA AI research institute at Brown, noted, “The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them.” She highlighted the extensive effort required for the study, which involved a year-long investigation with clinical experts.
Pavlick expressed that most AI work is assessed through automated metrics, which inherently lack the human nuance necessary for such evaluations. “This paper required a team of clinical experts and a study that lasted for more than a year in order to demonstrate these risks,” she said. “There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good.”
The implications of this research suggest a pressing need for ethical guidelines and accountability mechanisms in AI mental health applications. As interest in AI-driven solutions for emotional wellness continues to grow, the study serves as a pivotal reminder of the risks involved and the essential requirements for ensuring safe and effective use.
See also
Shreya Parchure Develops Explainable AI to Personalize Stroke Recovery for Aphasia Patients
Autonomous AI Powered Ophthalmology Screening Market Reaches $2.7B by 2035, Growing at 8.4% CAGR
UK Government Launches £40M AI Lab to Drive Fundamental Research and Transform Technology
Intron Launches Sahara v2, Supporting 57 Languages and 500+ African Accents
Ateneo’s ALIVE Lab Partners with Industry to Advance Machine Learning in Healthcare and Urban Planning
