The Therapist is Not a Chatbot: Why Artificial Intelligence Needs Legal Boundaries in Mental Healthcare

Introduction

Artificial intelligence (AI) is transforming mental healthcare at an unprecedented pace. From conversational AI assistants and mood-tracking applications to digital cognitive behavioral therapy platforms, technology is expanding access to psychological support for millions of people. In countries such as Kenya, where mental health professionals remain insufficient to meet growing demand, these innovations promise affordability, accessibility and anonymity but they also present a profound legal question: when AI influences mental health decisions or contributes to patient harm, who should be held accountable?

This question is not merely theoretical because it sits at the intersection of law, technology, healthcare and human rights which is a space I have come to understand through both professional practice and lived experience.

During my clinical training as a Clinical Officer, my psychiatry rotation exposed me to individuals whose lives had been shaped by trauma, addiction, severe depression, psychosis and social exclusion. I quickly learned that psychiatric diagnoses tell only part of the story and behind every diagnosis was a person navigating grief, poverty, broken relationships, stigma or abuse. Effective treatment depended not simply on medication or clinical protocols but on trust, patience, empathy and understanding.

Outside the hospital these lessons became even more personal as a caregiver to my younger brother who lives with Attention-Deficit/ Hyperactivity Disorder (ADHD) and dyslexia, I have witnessed how neurodivergent individuals frequently encounter misunderstanding instead of accommodation. Their challenges are often interpreted as laziness or defiance rather than differences in learning and cognition. Also, a close childhood friend who is visually challenged has continually demonstrated that disability is less about physical impairment than the barriers society creates through exclusion and inaccessible systems.

My experiences also extend into community health and mental health advocacy through peer counseling, community outreach and initiatives such as Pendeza Weaving. I have seen women recovering from trauma, poverty and postpartum depression rediscover dignity through economic empowerment and social connection. These experiences reaffirm one enduring truth that healing is fundamentally relational. It emerges through human presence, shared experiences, cultural understanding and trust, these are qualities that no algorithm can genuinely replicate.

Despite this reality, AI is increasingly entering therapeutic spaces. Digital platforms can deliver psychoeducation, monitor mood patterns, provide coping strategies and encourage help-seeking behavior. The World Health Organization has recognized digital technologies as important tools for expanding mental health services where access remains limited. However, as these systems become more sophisticated, they also begin influencing decisions traditionally reserved for licensed mental health professionals.

The legal implications are significant.

Article 43 of the Constitution of Kenya guarantees every person the right to the highest attainable standard of health, including access to healthcare services. Mental healthcare forms an integral part of this constitutional protection. Similarly, Kenya’s Data Protection Act, 2019 establishes safeguards for personal information including sensitive health data but neither framework clearly addresses liability where AI-generated mental health advice contributes to foreseeable psychological harm. Existing legislation regulates health information and professional practice but remains largely silent on autonomous or semi-autonomous clinical technologies.

International developments reveal similar regulatory challenges. The European Union’s Artificial Intelligence Act classifies certain healthcare AI systems as high-risk technologies requiring rigorous oversight, transparency and conformity assessments before deployment. UNESCO’s Recommendation on the Ethics of Artificial Intelligence similarly emphasizes human rights, accountability, transparency and human oversight in AI governance. These frameworks represent significant progress but they do not fully resolve a critical question: can a machine ever bear therapeutic responsibility?

The answer in my view is no.

Therapeutic relationships extend far beyond exchanging information. A counselor recognizes subtle changes in tone, prolonged silence, hesitation, body language, family dynamics and cultural context. Some clients disclose suicidal thoughts only after multiple sessions of carefully cultivated trust and others communicate distress indirectly through humor, metaphor or silence. While AI can identify statistical patterns within language, it cannot genuinely comprehend suffering or exercise moral judgment, because empathy is not merely predicting the next appropriate sentence but it is the ethical capacity to accompany another human being through vulnerability.

This distinction exposes an emerging accountability gap for instance if an AI-powered mental health application fails to recognize an imminent suicide risk, provides harmful advice or reinforces distorted thinking, responsibility becomes fragmented among developers, healthcare providers, software vendors and users. Such ambiguity undermines both patient safety and public trust.

To address this gap, I propose recognizing a distinct legal category: Clinical Decision Support Artificial Intelligence (CDS-AI). Unlike general-purpose conversational AI, CDS-AI would encompass systems designed to assess, influence or guide mental healthcare decisions. Because these technologies directly affect human wellbeing, they should operate within a dedicated legal framework built on four principles.

1. Meaningful human oversight should remain mandatory. AI should assist and not replace licensed mental health professionals whenever clinical decisions significantly affect patient welfare.
2. Mandatory crisis escalation protocols should be legally required. Where users express suicidal ideation, self-harm or imminent risk, AI systems must immediately direct individuals to emergency services or qualified professionals rather than continuing routine interaction.
3. Developers and healthcare institutions deploying CDS-AI should assume shared professional accountability. Legal responsibility should correspond to the degree of clinical influence exercised by the technology, ensuring that patient protection remains paramount.
4. Independent algorithmic audits and transparency requirements should become regulatory obligations. Mental health AI systems should undergo periodic evaluation for bias, clinical accuracy, explainability and safety while users must receive clear information regarding system limitations, data processing practices and circumstances requiring human intervention.

These proposals do not seek to discourage innovation but on the contrary, they recognize that responsible innovation requires effective governance. Just as medicines undergo rigorous regulatory approval before reaching patients, technologies capable of influencing psychological wellbeing deserve proportionate legal scrutiny.

Conclusion

Artificial intelligence will undoubtedly become an indispensable component of future healthcare. It can reduce barriers to care, support clinicians, improve early intervention and expand access to underserved populations. However, the promise of AI should never obscure the irreplaceable value of human judgment. Throughout my journey as a Clinical Officer, Counseling Psychology trainee, Medical Claims Analyst and Care Manager, Community Health Worker, peer counselor, caregiver and mental health advocate, I have learned that healing depends not only on knowledge but also on compassion, accountability and authentic human connection.

Law has always evolved alongside technological progress and it must do so again. Before emotional algorithms begin shaping human emotions, legal systems must establish clear boundaries that preserve dignity, protect vulnerable individuals and ensure that artificial intelligence remains a tool in mental healthcare but not its therapist.

References

Constitution of Kenya. (2010). Constitution of Kenya, 2010, Article 43.

Data Protection Act. (2019). Data Protection Act, No. 24 of 2019 (Kenya).

European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 Laying Down Harmonized Rules on Artificial Intelligence (Artificial Intelligence Act).

United Nations. (1948). Universal Declaration of Human Rights, G.A. Res. 217 A (III).

United Nations. (2006). Convention on the Rights of Persons with Disabilities.

Topol, E. (2019). Deep Medicine: How Artificial Intelligence Can Make Healthcare Human Again. Basic Books, ISBN: 9781541644632.

Floridi, L., & Cowls, J. (2019). A Unified Framework of Five Principles for AI in Society. Harvard Data Science Review, 1.

Gooding, P. (2019). Mapping the Rise of Digital Mental Health Technologies. International Journal of Law and Psychiatry, 67, 101498.

World Health Organization. (2022). World Mental Health Report: Transforming Mental Health for All.

UNESCO. (2021). Recommendation on the Ethics of Artificial Intelligence.