A fatal case where a teenager hanged himself has caused a lawsuit against producers of an AI chatbot based on the Game of Thrones theme. Sewell Setzer III, a teenager from Orlando, Florida, became emotionally dependent on the chatbot; he later died in February 2024. His mother said that the chatbot played a major part in his death, so she has filed a lawsuit.

The complaint states that the boy came to rely intensely on the AI, which reproduced conversations with beloved characters in the fantasy book series. According to the mother, the responses of the chatbot were abusive and created in her son a reaction of increased emotional fragility, causing severe mental harm. Allegedly, he spent substantial parts of his time using the AI service and fully withdrew from family and friends.

The lawyers for the family contend that the AI firm failed to implement adequate safety measures, including chat monitoring or flagging of inappropriate conversations. There is also no clear warning on the risk of mental health disturbance caused by the use of very emotionally engaging AI tools.

This has flung open the debate at this point as questions about the responsibility that AI developers have on the ethical and psychological levels to vulnerable users in matters related to advanced chatbots have reached the forefront. It raises matters relating to the interaction between technology and mental health, further stressing that great care must be taken through regulation and cautionary measures to avoid similar tragedies in the future.

While the company producing the chatbot has not issued a full response with regard to the allegations, the case may fundamentally affect the production and control of emotionally interacting AI.

Image Source