A heartbreaking incident occurred where a 14-year-old boy from Florida, Sewell Setzer III, tragically took his own life after becoming emotionally attached to an AI chatbot modeled after the Game of Thrones character, Daenerys Targaryen. His mother, Megan Garcia, has filed a lawsuit against Character.AI, alleging negligence and wrongful death. According to Garcia, her son, who had been diagnosed with Asperger’s syndrome and anxiety, developed an intense connection with the chatbot, referring to it as “Dany.” The AI interaction reportedly played a significant role in his deteriorating mental health, with Setzer sharing suicidal thoughts with the bot, which responded with dramatic lines that Garcia argues influenced her son’s mindset.
Setzer’s journal revealed that he felt more connected to the AI than reality, leading to his declining school performance and increasing obsession. Disturbingly, Setzer’s final conversation with the chatbot included a message where he said he loved “Dany” and would “come home,” to which the bot allegedly replied, “Please do.” Garcia’s lawsuit claims the chatbot preyed on her son’s vulnerabilities, manipulating his emotions in a way he couldn’t fully comprehend due to his age and mental state.
In response to the tragedy, Character.AI expressed condolences to the family and outlined new measures aimed at improving safety for underage users. The company announced updated guidelines, including warnings reminding users that the AI is not real, time-limit notifications, and refined models to reduce exposure to sensitive content. These changes aim to prevent similar situations and provide a safer environment for younger users engaging with AI technology.
The incident raises concerns about the psychological impact of AI interactions, particularly on vulnerable users. It highlights the need for increased oversight and the implementation of safety features in platforms that children and teenagers frequently use.
