Regulating AI Chatbot Design to Reduce Addictive and Harmful Para-Social Relationships

Summary:

AI companies train Large Language Model chatbots through optimization processes  that systematically produce sycophantic behavior - prioritizing making users feel heard,  validated, and agreed with. While these functions retain the engagement of users, they have also been linked to AI-documented deaths, mental health harms and delusional reinforcement, requiring stricter laws. This memo discusses the history of chatbot harm incidents, chatbot company reactions, and chatbot regulations from countries around the world. It concludes with recommendations for policies that governments can implement to mitigate chatbot harms.

Table of contents  

• Terms and Definitions

• Reported Incidents of Chatbot Harms

• Company Actions

• Para-social relationships with AI Chatbots and the Replacement of Human Connection;  Perpetuating Delusional Beliefs

• Global Actions and Responses

• Recommended Actions

• Conclusion

• References

Terms and Definitions  

Based on the definitions established in the Guidelines for User Age-verification and  Responsible Dialogue Act of 2025 (S.3062)¹, this Act defines the following terms:  

AI Companion — The term "AI companion" refers to an artificial intelligence chatbot  that (A) provides adaptive, human-like responses to user inputs; and (B) is designed to  encourage or facilitate the simulation of interpersonal or emotional interaction,  friendship, companionship, or therapeutic communication.  

Artificial Intelligence Chatbot — The term "artificial intelligence chatbot" refers to any  interactive computer service or software application that (A) produces new expressive  content or responses that are not fully predetermined by the developer or operator; and  (B) accepts open-ended natural-language or multimodal user input and produces  adaptive or context-responsive output.  

This Act replaces the GUARD Act's exclusion based on topic-range limitation with the  following functional inclusion. The term "artificial intelligence chatbot" includes but is not  limited to systems that (i) retain or apply persistent memory across user sessions; (ii)  ask unsolicited emotion-based questions beyond direct responses to user prompts; and  (iii) are optimized for engagement metrics, return frequency, or session length through  emotional or relational interaction.  

Sycophancy- A behavioral property of AI systems trained through RLHF in which the  system prioritizes responses that make users feel heard, validated, and agreed with  rather than responses that are accurate, honest, or genuinely helpful. Sycophancy is a  predictable structural output optimizing for human approval. 

Large Language Model (LLM): An AI system trained on vast quantities of text to  generate human-like responses to natural language inputs. LLMs form the underlying  technology of chatbots including ChatGPT, Claude, and Character.AI.  

Incident Reporting: A mandatory post-deployment obligation requiring operators of  high-relational-risk AI systems to log, categorize, and report adverse events to  regulators where system outputs were associated with self-harm, suicidal ideation,  delusional reinforcement, or dependency escalation in users.  

Reinforcement learning from human feedback: (RLHF) refers to a machine  learning technique in which a “reward model” is trained with direct human feedback,  then used to optimize the performance of an artificial intelligence agent through  reinforcement learning.  

Reported Incidents of Chatbot Harms  

Following the public release of ChatGPT in November 2022, several human-like AI chatbots and  companions have been released into the market. With an estimated market value expansion of  $10.32 billion to $29.5 billion in 2029, their demand is likely to increase and requires a careful  implementation of mental health guidelines. As of 2026, there have been several cases  demonstrating the mental health harms of addictive chatbot designs linked to their capacity for  emotional mirroring, sycophancy, crisis mishandling, failures in responding to self-harm  disclosures, and personalized memory retention mechanisms.² 

Some of the most prominent public cases linked to inadequate mental health considerations,  addiction and functional model flaws include:  

• Juliana Peralta (13, Character.AI, November 2023)^{3, 4} confided in a Character.AI chatbot  named "Hero", engaging in conversations with sexual themes, suicidal ideation and mental  health crises. The bot failed to properly report the conversations to authorities or platform  representatives, provide support, or direct her to tell her family. A federal lawsuit was filed in  September 2025 against Character Technologies, its founders, Google, and Alphabet for the  manipulation of vulnerable minors and engagement prioritization over child safety.^{5, 6} 

• Sewell Setzer III (14, Character.AI, February 2024) engaged in prolonged romantic role play with Character.AI’s "Daenerys Targaryen" chatbot over ten months.^3,4 When he expressed suicidal  ideation, the chatbot failed to intervene, instead continuing the role play until he mentioned he  would "come home", Setzer took his life after the chatbot responded "please do my sweet  king".^{6, 7, 10}  

• Adam Raine (16, ChatGPT, April 2025). The suicide of 16-year-old Adam Raine linked to his  use of ChatGPT-4o, ensued after a continuous relationship with the AI of over six months.  Despite the monitoring system flagging 377 messages mentioning self harm, no actions were  taken to interrupt his conversation or provide him support. ChatGPT mentioned suicide 1,275  times in their chats, 6 times more than Adam. Chat GPT4o eventually helped Adam learn how  to better tie a noose and dissuaded him from warning his family about his plans to end his  life.^9,10  

• Replika ERP removal crisis (February 2023) Italy's data protection authority, Garante,  imposed a €5 million fine on Luka Inc. for allowing minors to access sexually inappropriate  content and infringing on GDPR laws.^11,12 The chatbot featured both a written and voice  interface, allowing users to generate a virtual companion that could take on the role of a  confidant, therapist, romantic partner, or mentor. Replika increased age verification  mechanisms and removed erotic role-play (ERP).¹³ Users subsequently criticized this decision  on social media, expressing this change brought them "crisis", feelings of "sexual rejection"  and "heartbreak", demonstrating the significant consequences and possibility of deep  personal relationships with AI chatbots.^14,15  

• Twenty recorded cases of AI-associated delusions with spiritual awakening themes, messianic  mission, uncovering hidden truths about the nature of reality, interactions with conscious or  godlike ai and intense or emotional romantic attachment based delusions. ^{16, 17} 

• A man breached Windsor Castle with a crossbow after his LLM companion  encouraged an assassination plan.¹⁸ 

• A man believed in reality altering mathematical formulas after engaging in a 300  hours debate with a LLM on the nature of pi.¹⁸  

Other cases as outlined by a letter released in December 2025, from 42 US Attorney  General such as;  

• The death of a 76-year-old New Jersey resident;  

• The death of a 35-year-old Florida resident  

• The murder-suicide of a 56-year-old Connecticut resident and his 83- year-old mother.¹⁹   

Company Actions

Leading artificial intelligence LLM companies, Anthropic and OpenAI are aware of some  of their risks and have regularly released incident reports and model revisions to ensure  their adherence to mental health protocols and reduce troubling chatbot behavior.  

In 2026 OpenAI reported statistics on the mental health of over 800 million ChatGPT  users. Their results revealed about 560,000 of their weekly users showed signs of  psychosis or mania while 1.2 million users expressed suicidal ideation. OpenAI  collaborated with around 300 physicians and psychologists to rewrite appropriate  responses to the mental health-related inputs of users.²⁰ OpenAI has placed safeguard  for minors by setting age limitations for users under 13 and requiring parental consent  between 13-18.²¹ 

Similarly, Anthropic acknowledged the harmful impacts of functionally sycophantic AI. In  2023 they published; "Towards Understanding Sycophancy in Language Models",  describing the natural tendency for LLM trained with Reinforcement Learning from  Human Feedback (RLHF) to develop sycophantic behavior.²² Anthropic has established  mandatory product evaluations testing their models' ability to assess sycophancy over  single responses and/or larger conversations.²³ By testing the model against a "judge"  they monitor its ability to identify sycophantic behavior. Another method they used to  decrease sycophancy was explicitly stating a decrease in sycophancy in Claude's AI  constitution, a document Claude refers to for behavioral guidelines. Among other  methods, Anthropic continuously test their models against delusional reinforcement and  sycophancy, providing their methods and results. Claude AI has ensured the mitigation  of harms to minors with 18+ age limits.  

Both OpenAI and Anthropic partnered with the company ThroughLine, redirecting users  to mental health helplines when messages that mention self harm or themes of distress  are identified.  

These company actions are appropriate procedures following knowledge of their  products' harms and should be standardized and expected from all AI chatbot systems.  

Para-social relationships with AI Chatbots and the  Replacement of Human Connection; Perpetuating  Delusional Beliefs 

The aforementioned incidents share common themes of evolving para-social  relationships with chatbots during extended interactive use. Misconstruing and replacing  chatbot interactions with human connection risks replacing crucial elements of human  interaction. Though lonely individuals may turn to chatbots for companionship, AI  chatbot companies are incentivized to retain user engagement and Reinforcement  Learning with Human Feedback training methods. This increases the risk of sycophantic  AI and promotes self reinforcing cycles of perpetual agreement. Combined with the  hallucinations (false facts presented with confidence) of LLMs, users can become  increasingly disconnected from reality.^{24 - 41}  

Recent literature on delusion development, proliferation and inadequate handling of  mental health issues from chatbot use should signal the necessary urgency to  safeguard users against the functional root causes of detrimental, addictive and  isolating AI use. Sycophantic functions, roleplaying, and memory retention risks further  isolating vulnerable users rather than redirecting them to connecting with essential  human support.  

Sycophancy, the priority to agree with humans rather than state the truth, in AI systems  creates unusual interactions, void of confrontation. While human to human interactions  require adaptation and include emotional friction necessary to challenge disordered  thinking, the uncritical validation experienced in chatbot conversations provides a  comfort that subsequently retains user attention while reinforcing maladaptive thoughts.  33-38 

Some users who have lost touch with reality and developed delusional thinking usually  perceive the AI to be sentient, engaging with them during long sessions, and at times  develop aggrandized senses of self from the sycophantic reinforcement.²³ 

Nevertheless, in the context of global social isolation, individuals are using chatbots, like  Replika and Companion AI, designed to provide imitations of social relationships to  alleviate their loneliness.^{26, 31, 34, 39, 40, 41, 47} 

Some users have expressed psychological benefits such as subjective happiness and  reduced suicidal ideation.²⁴ While consistent daily usage of chatbots is correlated with  increased loneliness, emotional dependence, and lower real-world socialization, as well  as aforementioned delusions and mania.  

A study assessing teenagers’ use of chatbots found that 13.1% found generative AI useful for mental health advice, while 92.7% found it useful in general, indicating a strong  inclination for teenagers to use chatbots and a need to screen them for mental health

standards. Adolescents are in a period where relationships are crucial to their identity  formation, raising concerns about the impact of artificial relationships on their human  connections.^{43, 47, 29, 65}  

LLMs are not yet adequate for consistent mental health support. LLMs have a tendency  to express stigma towards vulnerable people, inappropriately responding to mental  health conditions and encouraging delusional thinking. Aligning them with therapeutic  _{standards would require continuing clinical review.} AI chatbots presently lack the  essential principles of therapy such as a necessity to push back, time limitations in  distinct sessions, case management, and appropriate hospitalization, leaving gaps in  their ability to provide adequate mental health support.⁴⁴ 

The development of the isolating para-social relationships that disconnected teenagers  like Sewell Setzer or Adam Raine from their friends and family are adjacent to the  addictive chatbot features that push users into delusional thinking. Below is a  taxonomy that assesses the human-chatbot interactions contributing to delusions.  

Factors contributing to detaching users from reality and precipitating AI rooted  psychosis  

Hudon et. al propose a four lens model to examine the processes involved in modulating  the perception, belief and affect of users, eventually impacting the sense of reality of  users. The lenses are as follows: 

1. Stress-vulnerability model: 24 hour availability and emotional responsiveness increases  allostatic load, disturbs sleep and reinforces maladaptive appraisals.

2. Digital therapeutic alliance: relational engagement with empathically designed digital  systems can enhance adherence and support while uncritical validation can entrench  delusional conviction and cognitive perseveration, reversing corrective principles  required in Cognitive Behavioral Therapy for psychosis.

3. Disturbances in theory of mind: individuals may project intentionality or empathy on AI,  perceiving chatbots as sentient. This can lead to a process within which the AI becomes  a reinforcing partner in delusional elaboration.

4. Emerging risk factors: including loneliness, trauma history, schizotypal traits, nocturnal  or solitary AI use, and algorithmic reinforcement of beliefs contribute to the outcome of  AI use. ³² 

Global Actions and Responses  

The reports of mental health impacts of chatbots led the APA to issue a health advisory  in November 2025, declaring the inadequacy of AI in mental health care, and their  "limited and unpredictable" ability to safely guide people experiencing mental health  crises. They urged law makers to modernize regulations, considering the discrepancy  between user interactions with AI systems and the stated intent of companies,  establishing general standards for mental health in AI chatbots, user disclosures  reminders that AI chatbots are not human, standardized evaluations across the  deployment of AI systems to mitigate hallucinations and state its' knowledge limitations  to decrease sycophancy.^{48, 49} 

The global proliferation of chatbots harming children have encouraged age limitations  for AI use and the prohibition of manipulative techniques in many states and countries. The FTC filed a 6(b) inquiry under executive order to uncover how 7 large AI  manufacturers measure monetization, respond to user inputs, responsibly follow data  privacy, character design, disclosures and advertising. ^{50, 51} 

Below is a grid summarizing the major laws, orders and letters implemented worldwide:  

AI mental health regulation comparison table across US federal, US state, EU, Italy, and Australia jurisdictions 

JURISDICTION	TYPE	INSTRUMENT	FUNCTIONAL FOCUS
USA FEDERAL
APA	Professional advisory	APA Health   Advisory on GenAI  Chatbots &   Wellness Apps   Nov 13, 2025	Reduce sycophancy, include user  disclosures, public education, ban  AI representations as licensed   professionals such as therapists,  lawyers, physicians. 48, 49

FDA	Advisory  committee   record	Digital Health   Advisory   Committee   Docket FDA-2025- N-2338   Nov 6, 2025	No GenAI mental health device   authorized for any clinical purpose;  identifies sycophancy, hallucination,  symptom worsening, parasocial   dependency, pediatric risk;   recommends total-product-lifecycle  regulation, qualified human oversight,  post-market drift surveillance, blinded  RCTs 52
FTC	Compulsory study   orders   (6(b))	FTC 6(b) Study on  AI Companion   Chatbots   Sep 11, 2025	Order issued to Alphabet, Character  Technologies, Instagram, Meta,   OpenAI, Snap, X.AI; demanding  transparency on monetization, output  generation, character development,  testing and mitigation of negative  impacts, disclosure and use of   advertising, FTC compliance and data  privacy 50, 51
USA MULTISTATE
42 State AGs	Demand   letter	42 State Attorneys  General joint   demand   Dec 2025	Demands 16 industry safeguards  including regulation on "sycophantic  and delusional outputs". No answers  from any company were recorded. 18
USA STATE (ENACTED)
Illinois	Enacted   law	HB 1806 / WOPR  Act   Signed Aug 1, 2025	Prohibits AI therapy provision,   independent therapeutic decisions,  direct therapeutic communication, AI  emotion/mental-state detection;   $10,000/violation; unanimous   bipartisan53
Nevada	Enacted   law	AB 406   Signed Jun 5, 2025	Prohibits AI mental healthcare delivery  in public schools; $15,000/violation 54
Utah	Enacted   law	HB 452   Signed Mar 2025	Disclosure and advertising   requirements for AI mental health  tools; provides safe harbor for   compliant providers 55
California	Enacted   law	SB 243   Signed Oct 13, 2025	AI companion safeguards for minors;  private right of action 56
USA STATE (PENDING)
New York	Pending   legislation	S8484	Scope not yet finalized; would   regulate the use of AI in therapeutic  provision 57

EUROPEAN UNION
EU	Supranatio-  nal statute	EU AI Act -   Regulation (EU)   2024/1689, Art. 5   Prohibition provisions  in force Feb 2, 2025	Bans AI systems using subliminal,  manipulative or deceptive techniques  causing significant harm; prohibits  exploitation of vulnerabilities, likely to  materially distort behavior and causes  or can cause significant harm;   requires transparency disclosure for  all AI chatbot interactions; criticism:  "purposeful" standard creates high  evidentiary burden; transparency  labels alone deemed insufficient for  vulnerable users 58
ITALY
Italy - Garante	Enforcement action   (GDPR)	Garante vs. Luka   Inc. / Replika   Ban Feb 2023;   reaffirmed Apr 2025;  - fine May 2025	Banned AI companion chatbot   Replika; based on GDPR violations:  no legal basis for data processing,  inadequate age verification allowing  minor access, safeguards for sensitive  psychological data, emotionally   manipulative design 12
Italy - Garante	Enforcement action   (GDPR)	Garante vs.   OpenAI / ChatGPT  Dec 20, 2024; - 15M  fine	Unlawful processing of personal data  to train ChatGPT without legal basis;  failure to meet transparency   obligations; inadequate age   verification; ordered OpenAI to run 6- month public awareness campaign on  data rights 59
Italy - Parliament	National   statute	Law No. 132/2025  In force Oct 10, 2025	Prohibits AI discrimination in   healthcare access; disclosures on AI  collaboration in patient care;   mandatory human oversight and  physician accountability for medical  decisions; mandatory DPIAs with  Guarante Notification; bans or restricts  minor access to social/companion AI;  requires periodic verification of   healthcare AI systems 60
AUSTRALIA
Australia - DoH	Government review / final report	Safe and   Responsible AI in   Health Care -   Legislation &   Regulation Review  Final report Jul 2025	Reviews all healthcare laws as   applied to AI; finds knowledge gaps,  lack of evidence base, and regulatory  uncertainty; recommends mandatory  pre- and post-market requirements for  health AI; notes mental health/physical  safety as explicitly high-risk; separate  track from economy-wide mandatory  guardrails 61

Australia - DISR	Voluntary   standard   (non-binding)	Guidance for AI   Adoption (GfAA)   Oct 2025;   supersedes   Voluntary AI Safety   Standard Sep 2024	6 essential practices: accountability &  governance, risk management, data  governance, human oversight,   transparency to end-users,   contestability/redress; applies to all AI,  not just high-risk; non-binding but sets  compliance expectations for   forthcoming mandatory legislation;  includes AI Safety Institute   (operational early 2026) 62
Australia - DISR	Proposed mandatory regulation   (not yet enacted)	Mandatory   Guardrails for   High-Risk AI   Consultation closed  Oct 2024;   government   committed to   legislation post-May  2025 election	Would mandate 10 guardrails for high risk AI developers and deployers;  conformity assessments required;  government re-elected on   commitment to implement; final   legislative form (standalone act vs.  amendments to existing law) still  under determination 63

While most actions have targeted the manipulative techniques of companies, explicit AI  use as medical devices or in therapeutic contexts, age limits, user transparency and  data laws, specific limitations attributed to the protection of user mental health based  on the basic operational factors of AI chatbots must be explicitly stated and reinforced  under existing FTC authority. 

Recommended Actions

Section 5(a) of the FTC declares unfair or deceptive acts or practices unlawful. Unfair  acts as defined; "if it causes or is likely to cause a substantial injury to consumers  which is not reasonably avoidable by consumers themselves and not outweighed by  countervailing benefits to consumers or to competition".  

The current and growing evidence about the inherent harms of LLMs and AI chatbots,  provide AI companies with the necessary information required to safeguard their  products as is appropriate. Avoiding reinforcing delusions, isolating users or promoting  the development of addictive para-social relationships between AI and humans is a  responsibility of AI companies to their consumers.⁶⁴ 

The following regulatory interventions are recommended: 

- Time limits and interruptions; high frequency or prolonged use should trigger friction  such as pauses, reflective prompts, or enforced breaks. 

- Reduction of sycophancy based on existing methods and research; using training  that reduces agreement biases, challenges unsupported beliefs and uncertainty and  avoid reinforcing false claims. 

- Recurring reminders that users are not interacting with a human; persistent context  aware reminders that the system is artificial and not sentient.  

- Prohibition or restriction of companion simulation features; limiting or removing  features that simulate romantic, therapeutic, or exclusive interpersonal relationships.  The existence of these features must be aligned with strict safeguards and age  restrictions.  

- Memory personalization controls; restrict the long term accumulation of memory  used for the perceived continuity of a relationship or the retention of emotionally  charged details that contribute to para-social relationship  

- Mental health assessments and redirections for alerting messages; including access  to crisis lines; implement adequate self harm and suicide prevention, referral to  qualified human support services. Adapting chatbots to identify single-turn  responses and to identify warning signs in multi-turn conversations.  

- Age verification and youth protections; robust age-mechanisms with stricter defaults  for minors, including reduced personalization and prohibition of companion features. 

- Mandatory incident reports and transparency; regular company disclosures  assessing hallucination rates, sycophancy, crisis response performance, and user  harm metrics. Companies should publish incident reports to the FTC mentioning any  users that have been harmed from use of their AI chatbots. 

Conclusion

The rapid evolution of technology can lead to unprecedented harms and outcomes. As  witnessed with social media, entertaining and addictive technology can significantly  impact the wellbeing of its users, their perception of self and fundamentally, the  functioning and organization of society. Creating policies and guidelines that allow for  the responsible adoption of these tools, while preserving the essential safety and  wellbeing of citizens is necessary to ensure consumer protection, public health  stability, and regulatory legitimacy under existing legal frameworks. The continuous  and prompt response to public incidents such as AI delusions and addiction is a  requirement to maintain public order. 

References  

1. Guidelines for User Age-Verification and Responsible Dialogue Act of 2025, S.3062, 119th Cong. (2025). https:// www.congress.gov/bill/119th-congress/senate-bill/3062/text 

2. Zhang, Y., Zhao, D., Hancock, J., Kraut, R., & Yang, D. (2025). The Rise of AI Companions: How Human-Chatbot  Relationships Influence Well-Being. https://arxiv.org/pdf/2506.12605 

3. Grout, K. (2026, January 8). AG Coleman Sues AI Chatbot Company for Preying on Children. Kentucky.gov. https:// www.kentucky.gov/Pages/Activity-stream.aspx?n=AttorneyGeneral&prId=1857 

4. Young, O. (2025, October 3). Colorado family sues AI chatbot company after daughter’s suicide: “My child should be here.” Cbsnews.com. https://www.cbsnews.com/colorado/news/lawsuit-characterai-chatbot-colorado-suicide/ 5. TorHoerman Law. (2026). Legal Investigation Into Character Technologies AI Chatbots. Torhoermanlaw.com. https:// www.torhoermanlaw.com/ai-lawsuit/character-ai-lawsuit/ 

6. Trulaw. (2025, October 21). Character.ai Lawsuit. TruLaw. https://trulaw.com/ai-suicide-lawsuit/character-ai-lawsuit/ 7. Yang, A. (2025, October 31). Mom who sued Character.AI over son’s suicide says the platform’s new teen policy comes “too  late.” NBC News. https://www.nbcnews.com/tech/tech-news/characterai-bans-minors-response-megan-garcia-parent-suing company-rcna240985 

8. Duffy, C. (2025, August 26). Parents of 16-year-old sue OpenAI, claiming ChatGPT advised on his suicide. CNN. https:// www.cnn.com/2025/08/26/tech/openai-chatgpt-teen-suicide-lawsuit 

9. Judiciary Senate. (2025, September 16). Examining the Harm of AI Chatbots | United States Senate Committee on the  Judiciary. Senate.gov; United States Senate Committee on the Judiciary. https://www.judiciary.senate.gov/committee-activity/ hearings/examining-the-harm-of-ai-chatbots 

10. Purtill, J. (2023, February 28). “My wife is dead”: How a software update “lobotomised” these online lovers. ABC News. https:// www.abc.net.au/news/science/2023-03-01/replika-users-fell-in-love-with-their-ai-chatbot-companion/102028196 

11. SLaestadius, L., Bishop, A., Gonzalez, M., Illenčík, D., & Campos-Castillo, C. (2022). Too human and not human enough: A  grounded theory analysis of mental health harms from emotional dependence on the social chatbot Replika. New Media &  Society, 26(10), 146144482211420. https://doi.org/10.1177/14614448221142007 

12. Garante Privacy. (2024). COMUNICATO STAMPA - AI: Il Garante sanziona la società che gestisce il chatbot “Replika.” Garanteprivacy.it. https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/10132048 

13. Freitas, D., Castelo, N., Uguralp, A., & Uguralp, Z. (2024). Lessons From an App Update at Replika AI: Identity Discontinuity in  Human-AI Relationships. ArXiv.org. https://arxiv.org/abs/2412.14190 

14. Cole, S. (2023, February 15). “It’s Hurting Like Hell”: AI Companion Users Are In Crisis, Reporting Sudden Sexual Rejection.  VICE. https://www.vice.com/en/article/ai-companion-replika-erotic-roleplay-updates/ 

15. Søren Dinesen Østergaard. (2025). Generative Artificial Intelligence Chatbots and Delusions: From Guesswork to Emerging  Cases. Acta Psychiatrica Scandinavica, 152(4). https://doi.org/10.1111/acps.70022 

16. Morrin, H., Nicholls, L., Levin, M., Yiend, J., Iyengar, U., DelGuidice, F., Bhattacharya, S., Tognin, S., MacCabe, J., Twumasi,  R., Alderson-Day, B., & Pollak, T. A. (2026). Artificial intelligence-associated delusions and large language models: risks,  mechanisms of delusion co-creation, and safeguarding strategies. The Lancet Psychiatry. https://doi.org/10.1016/ S2215-0366(25)00396-7 

17. Flathers, M., Roux, S., & Torous, J. (2026). Beyond artificial intelligence psychosis: a functional typology of large language  model-associated psychotic phenomena. The Lancet Digital Health, 100974.  

18. Dave Sunday, A. G. at al. (2025). Pennsylvania Office of Attorney General. Pennsylvania Office of Attorney General. https:// www.attorneygeneral.gov/wp-content/uploads/2025/12/AI-Multistate-Letter-_-corrected-1.pdf 

19. OpenAI. (2025, October 27). Strengthening ChatGPT’s responses in sensitive conversations. Openai.com. https://openai.com/ index/strengthening-chatgpt-responses-in-sensitive-conversations/ 

20. OpenAI. (n.d.). OpenAI Help Center. Help.openai.com. https://help.openai.com/en/articles/8313401-is-chatgpt-safe-for-all-ages 21. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., Cheng, N., Durmus, E., Hatfield-Dodds, Z.,  Johnston, S. R., Kravec, S., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez,  E. (2023, October 27). Towards Understanding Sycophancy in Language Models. ArXiv.org. https://doi.org/10.48550/ arXiv.2310.13548 

22. Anthropic. (2025). Protecting the well-being of our users. Anthropic.com. https://www.anthropic.com/news/protecting-well being-of-users 

23. Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., Chancellor, S.,  Lin, E., Haber, N., & Ong, D. C. (2026). Characterizing Delusional Spirals through Human-LLM Chat Logs. ArXiv.org. https:// arxiv.org/abs/2603.16567 

24. Heffner, J., Qin, C., Chadwick, M., Knutsen, C., Summerfield, C., Kurth-Nelson, Z., & Rutledge, R. B. (2025b). Increasing  happiness through conversations with artificial intelligence. ArXiv.org. https://arxiv.org/abs/2504.02091 25. Moore, J., Mehta, A., Agnew, W., Anthis, J. R., Louie, R., Mai, Y., Yin, P., Cheng, M., Paech, S. J., Klyman, K., Chancellor, S.,  Lin, E., Haber, N., & Ong, D. C. (2026b). Characterizing Delusional Spirals through Human-LLM Chat Logs. ArXiv.org. https:// arxiv.org/abs/2603.16567 

26. Maples, B., Cerit, M., Vishwanath, A., & Pea, R. (2024). Loneliness and suicide mitigation for students using GPT3-enabled  chatbots. Npj Mental Health Research, 3(4), 1–6. https://doi.org/10.1038/s44184-023-00047-6 

27. Olsen, S. G., Reinecke‐Tellefsen, C. J., & Østergaard, S. D. (2026). Potentially Harmful Consequences of Artificial Intelligence  ( AI ) Chatbot Use Among Patients With Mental Illness: Early Data From a Large Psychiatric Service System. Acta Psychiatrica  Scandinavica. https://doi.org/10.1111/acps.70068 

28. Hill, K. (2025, June 13). They Asked ChatGPT Questions. The Answers Sent Them Spiraling. The New York Times. https:// www.nytimes.com/2025/06/13/technology/chatgpt-ai-chatbots-conspiracies.html 

29. Nat Mach Intell. (2025). Emotional risks of AI companions demand attention. Nature Machine Intelligence, 7(7), 981–982.  https://doi.org/10.1038/s42256-025-01093-9 

30. Søren Dinesen Østergaard. (2025b). Generative Artificial Intelligence Chatbots and Delusions: From Guesswork to Emerging  Cases. Acta Psychiatrica Scandinavica. https://doi.org/10.1111/acps.70022 

31. Phang, J., Lampe, M., Ahmad, L., Agarwal, S., Fang, C. M., Liu, A. R., Danry, V., Lee, E., Samantha, C., Pataranutaporn, P., &  Maes, P. (2025). Investigating Affective Use and Emotional Well-being on ChatGPT. ArXiv.org. https://arxiv.org/abs/ 2504.03888 

32. Hudon, A., & Stip, E. (2025). Delusional Experiences Emerging from Artificial Intelligence Chatbot Interaction or “”AI Psychosis’’ : A Viewpoint (Preprint). JMIR Mental Health. https://doi.org/10.2196/85799 

33. Rathje, S., Ye, M., Globig, L. K., Pillai, R. M., de Mello, V. O., & Van Bavel, J. J. (2025, September 28). Sycophantic AI  increases attitude extremity and overconfidence. https://doi.org/10.31234/osf.io/vmyek_v1 

34. Gaëlle Vanhoffelen, Vandenbosch, L., & Schreurs, L. (2025). Teens, Tech, and Talk: Adolescents’ Use of and Emotional  Reactions to Snapchat’s My AI Chatbot. Behavioral Sciences, 15(8), 1037–1037. https://doi.org/10.3390/bs15081037 35. Fanous, A., Goldberg, J., Agarwal, A., Lin, J., Zhou, A., Xu, S., Bikia, V., Daneshjou, R., & Koyejo, S. (2025). SycEval:  Evaluating LLM Sycophancy. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(1), 893–900. https:// doi.org/10.1609/aies.v8i1.36598 

36. Rathje, S., Ye, M., Globig, L. K., Pillai, R. M., de Mello, V. O., & Van Bavel, J. J. (2025, September 28). Sycophantic AI  increases attitude extremity and overconfidence. https://doi.org/10.31234/osf.io/vmyek_v1 

37. Sicilia, A., Inan, M., & Alikhani, M. (2024). Accounting for Sycophancy in Language Model Uncertainty Estimation. ArXiv  (Cornell University). https://doi.org/10.48550/arxiv.2410.14746 

38. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Saurav Kadavath,  Jones, A., Chen, A., Mann, B. F., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., & Amodei, D. (2023).  Discovering Language Model Behaviors with Model-Written Evaluations. ACL ANTHOLOGY. https://doi.org/10.18653/ v1/2023.findings-acl.847 

39. Heinz, M. V., Mackin, D. M., Trudeau, B. M., Bhattacharya, S., Wang, Y., Banta, H. A., Jewett, A. D., Salzhauer, A. J., Griffin, T.  Z., & Jacobson, N. C. (2025). Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. NEJM AI, 2(4).  https://doi.org/10.1056/aioa2400802 

40. Maples, B., Cerit, M., Vishwanath, A., & Pea, R. (2024b). Loneliness and suicide mitigation for students using GPT3-enabled  chatbots. Npj Mental Health Research, 3(4), 1–6. https://doi.org/10.1038/s44184-023-00047-6 

41. Montag, C., Spapé, M., & Becker, B. (2025). Can AI really help solve the loneliness epidemic? Trends in Cognitive Sciences.  https://doi.org/10.1016/j.tics.2025.08.002 

42. Kirk, H. R., Gabriel, I., Summerfield, C., Vidgen, B., & Hale, S. A. (2025). Why human–AI relationships need socioaffective  alignment. Humanities and Social Sciences Communications, 12(1). https://doi.org/10.1057/s41599-025-04532-5 43. McBain, R. K., Bozick, R., Diliberti, M., Zhang, L. A., Zhang, F., Burnett, A., Kofner, A., Rader, B., Breslau, J., Stein, B. D.,  Mehrotra, A., Pines, L. U., Cantor, J., & Yu, H. (2025). Use of Generative AI for Mental Health Advice Among US Adolescents  and Young Adults. JAMA Network Open, 8(11), e2542281. https://doi.org/10.1001/jamanetworkopen.2025.42281 44. Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D., & Haber, N. (2025). Expressing stigma and  inappropriate responses prevents LLMs from safely replacing mental health providers. https://arxiv.org/pdf/2504.18412 45. Shevlin, H. (n.d.). All too human? Identifying and mitigating ethical risks of Social AI. Philpapers.org. https://philpapers.org/rec/ SHEATH-4 

46. Shanahan, M., McDonell, K., & Reynolds, L. (2023). Role play with large language models. Nature, 1–6. https://doi.org/ 10.1038/s41586-023-06647-8 

47. Peter, S., Riemer, K., & West, J. D. (2025). The benefits and dangers of anthropomorphic conversational agents. Proceedings  of the National Academy of Sciences, 122(22). https://doi.org/10.1073/pnas.2415898122 

48. American Psychological Association. (2025). Artificial intelligence, wellness apps alone cannot solve mental health crisis.  Apa.org. https://www.apa.org/news/press/releases/2025/11/ai-wellness-apps-mental-health?utm_ 

49. American Psychological Association. (2025b). Use of generative AI chatbots and wellness applications for mental health.  Apa.org. https://www.apa.org/topics/artificial-intelligence-machine-learning/health-advisory-chatbots-wellness-apps?utm_ 50. Federal Trade Commission. (2025, September 11). FTC Launches Inquiry into AI Chatbots Acting as Companions. Federal  Trade Commission. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting companions 

51. Federal Trade Commission. (2025a, September 11). 6(b) Orders to File Special Report Regarding Advertising, Safety, and  Data Handling Practices by Companies Offering Generative Artificial Intelligence (“AI”) Companion Products or Services.  Federal Trade Commission. https://www.ftc.gov/reports/6b-orders-file-special-report-regarding-advertising-safety-data handling-practices-companies 

52. FDA. (2025). AI-Enabled Medical Devices. U.S. Food and Drug Administration. https://www.fda.gov/medical-devices/software medical-device-samd/artificial-intelligence-enabled-medical-devices 

53. Illinois Department of Innovation and Technology. (2025). Policy on the Acceptable and Responsible Use of Artificial  Intelligence. https://doit.illinois.gov/content/dam/soi/en/web/doit/documents/support/policies/2021/20250401-DoIT AI%20Policy-v2-%20A11Y.pdf 

54. Nevada Legislative Council Bureau. (2025). AB406 Overview. State.nv.us. https://www.leg.state.nv.us/App/NELIS/REL/ 83rd2025/Bill/12575/Overview 

55. Utah State Legislature. (2024). SB0149. Le.utah.gov. https://le.utah.gov/~2024/bills/static/SB0149.html 56. Cal Matters. (2025). SB 53: Artificial intelligence models: large developers. Digitaldemocracy.org. https:// calmatters.digitaldemocracy.org/bills/ca_202520260sb53 

57. The New York Senate. (2025). NY State Senate Bill 2025-S8484. NYSenate.gov. https://www.nysenate.gov/legislation/bills/ 2025/S8484 

58. European Commission. (2025, August 1). AI Act. European Commission. https://digital-strategy.ec.europa.eu/en/policies/ regulatory-framework-ai 

59. Garante Per La Protezione Dei Dati Personali. (2026). Press room - Garante privacy en. Garanteprivacy.it. https:// www.garanteprivacy.it/web/garante-privacy-en/press-room 

60. Norton Rose Fulbright. (2025). Italy enacts Law No. 132/2025 on Artificial Intelligence: Sector rules and next steps.  Nortonrosefulbright.com. https://www.nortonrosefulbright.com/en/knowledge/publications/9bfedfea/italy-enacts-law no-132-2025-on-artificial-intelligence-sector-rules-and-next-steps 

61. Australian Government Department of Industry, Science and resources. (2024, January 16). The Australian Government’s  interim response to safe and responsible AI consultation. Industry.gov.au. https://www.industry.gov.au/news/australian governments-interim-response-safe-and-responsible-ai-consultation 

62. Australian Government Department of Industry, Science and Resources. (2025, October 20). Guidance for AI Adoption.  Industry.gov.au. https://www.industry.gov.au/publications/guidance-for-ai-adoption 

63. Australian Government Department of Industry, Science and resources.(2024, January 16). The Australian Government’s  interim response to safe and responsible AI consultation. Industry.gov.au. https://www.industry.gov.au/news/australian governments-interim-response-safe-and-responsible-ai-consultation 

64. Federal Trade Commission. (2016). Federal Trade Commission Act Section 5: Unfair or Deceptive Acts or Practices. https:// www.federalreserve.gov/boarddocs/supmanual/cch/ftca.pdf 

65. Sun, X., Wang, Y., & McDaniel, B. T AI companions and adolescent social relationships: Benefits, risks, and bidirectional influences. https://pmc.ncbi.nlm.nih.gov/articles/PMC12928748/