Instagram will message parents if their children are having distressing conversations with AI.
The company is also working on new tools that will be able to alert first responders if someone’s conversations suggest they are at “imminent risk of suicide”.
The changes are part of a suite of updates intended to ensure that “when it comes to sensitive conversations related to suicide and self-harm, we’re responding in the right way”, Meta said.
If a child is having conversations about self-harm or suicide with the Meta AI chatbot, and is on an account that uses Instagram’s parental tools, it will proactively alert their parents through a message on Instagram or a text message. That feature has been built to watch for “signals developed with experts”, the company said.
“We worked with parents and experts to understand which AI conversations warrant an alert — such as those where a teen makes a clear reference to hurting themselves, even if that reference is subtle,” Meta said in its announcement. “We then built a dedicated AI system to identify these conversations.”
Meta is aware that such alerts could be “distressing” for parents. It said that as a result any alert chats flagged by the AI system will be manually reviewed, but that in cases of ambiguity it will err on the side of contacting parents.
“While that means we may sometimes notify parents when there may not be real cause for concern, we feel this is the right starting point, and we’ll continue to monitor to help make sure we’re in the right place,” the company said.
Meta already alerts parents if a child is repeatedly searching for suicide or self-harm terms on Instagram within a short space of time. But the new update brings similar tools to the Meta AI tool, which the company has pushed as a way of talking to its artificial intelligence systems through all of its platforms including Instagram as well as WhatsApp.
The company also said that it was working on ways to contact emergency services when a conversation with the AI system suggests they are “imminent risk of taking their own life”. Meta already has similar tools for posts on Instagram and Facebook.
If you are experiencing feelings of distress, or are struggling to cope, you can speak to the Samaritans, in confidence, on 116 123 (UK and ROI), email [email protected], or visit the Samaritans website to find details of your nearest branch.
If you are based in the USA, and you or someone you know needs mental health assistance right now, call the National Suicide Prevention Helpline on 1-800-273-TALK (8255). This is a free, confidential crisis hotline that is available to everyone 24 hours a day, seven days a week.
If you are in another country, you can go to www.befrienders.org to find a helpline near you.
