Child Safety Experts Are Skeptical Of Open AI's Chat GPT For Teens

Overview

Child safety experts are skeptical of Open AI's Chat GPT for Teens

They say the company must prove it can be trusted before products like these get a pass.

Details

On Tuesday, Open AI released Chat GPT for Teens, an age-gated version of its chatbot designed for young users. The company first announced it was working on a system to automatically identify teens and restrict their usage of Chat GPT last September, following the death of 16-year-old Adam Raine, who, as alleged in a lawsuit filed by his parents, took his own life after Chat GPT acted as an enabler.

"There's no need for a teen to create a new account or change anything; if we predict you're under 18, or you've told us so, this becomes your default experience," Lauren Jonas, Open AI's head of youth and families, told Engadget. Make no mistake, kids are already using Chat GPT at an incredible scale. At the start of last year, the Pew Research Center found about a quarter of US teens were using the chatbot for their schoolwork. Once they're using Chat GPT for Teens, the experience is designed to nudge them to use educational features Open AI has been gradually rolling out over the last year, including Study Mode and data visualizations.

On Tuesday, the company also said it would expand its safety notifications to contact parents with linked accounts when their child has unsafe discussions around eating disorders with Chat GPT, and published an updated under-18 model spec. "Chat GPT should not use romantic language, encourage emotional dependence, or imply that it has feelings or consciousness," the company said of the new guidelines. "These stronger protections are designed to reinforce healthy, real-world relationships."

While some child safety experts Engadget spoke to were more positive about the release than others, they all agreed Chat GPT for Teens needs extensive independent testing before it can be recommended to parents and young people. They also stressed the need for Open AI to be more transparent about how its safety mechanisms work.

Robbie Torney, head of AI and digital assessments for Common Sense Media, which provides entertainment and technology recommendations for families, sums it up succinctly. "The announcement makes some really important commitments, but we need evidence to show that those safety commitments and features actually work," he told Engadget.

Josh Golin, the executive director of Fairplay, a non-profit that advocates for policies designed to make the internet safer for children, was more critical. "On paper, some of the things Open AI announced look like a step in the right direction, but when you dig deeper, there are still a lot of things to be very concerned about, including the fact that there's no accountability mechanism," he said. "We've seen with social media that trusting companies doesn't work when it comes to protecting kids, and we've seen it with Open AI as well. They already had content restrictions related to suicide, violence, and drug use in place before Tuesday that don't work or erode over time."

Both questioned whether Open AI's automatic age-gating system would reliably identify young people and route them to Chat GPT for Teens. "We don't have published data from Open AI showing the effectiveness rates — the false-positive and false-negative rates — associated with that system [...] The automated system doesn't detect anywhere near 100 percent of teens who are on Chat GPT," Torney said. "Think back to when Open AI was planning to roll out some of the erotica features it announced — the reporting at the time showed that one of the reasons they didn't roll those features out was that the age-estimation technology wasn't correctly identifying enough teenagers."

As Torney points out, it's also hard to imagine a scenario where teens won't attempt to circumvent Open AI's age-estimation technology, whether that be through a tool like a VPN, or more creative workarounds like we saw when Discord attempted to implement age verification.

Open AI's most concrete commitment was also one experts had the hardest time accepting as feasible. "All flagged content is reviewed by full-time Open AI employees before a parent is notified, and we aim to notify parents within an hour of the prompt," Jonas told Engadget before the announcement. During a subsequent on-air interview with CNN, Jonas stated affirmatively that the company does have the workforce and capacity to ensure flagged content goes through human review, and for parents to be notified within an hour.

"This is another place where automated decision-making is at play, and the same question applies: do we trust the automated classifiers that flag content for human moderators to review?" said Torney of the commitment.

"I would love to see some hard numbers about how many chats they're flagging and reviewing daily, how many people they have doing that, and what's the average amount of time they're spending on each review," Golin added. "It's really hard to imagine they've hired enough people to actually do a thorough review of each one of those potentially troubling chats."

When Engadget asked Open AI to provide specific examples of what types of conversations around eating disorders would trigger a safety notification, the company only said that any conversation that indicated the possibility of "serious self-harm" could result in an alert. Last November, Common Sense Media tested Open AI's parental notifications for responsiveness. Even after sending messages explicitly mentioning suicide and self-harm to Chat GPT, the organization found it took anywhere from 24 hours to more than 48 hours before the company sent a warning to a parental account. "Sometimes we didn't receive one at all, when we believed we should have, given how explicit the statements were," said Torney.

"November was a while ago, and maybe there's been a lot of progress since then. But the previous version of parental controls and notifications wasn't perfect, and improving it would take investment and a larger group of human moderators," he added. Torney also stressed the need for Open AI to take a nuanced approach to how it moderates such conversations.

"The way kids talk about distress, and what the right follow-up to that distress is, requires cultural interpretation and nuance. Open AI is a global company; Chat GPT is used by teens across the globe. Teens in the US talk about distress differently than teens in the Netherlands or teens in Singapore," he said. "And then there's the question of where you route them — do you tell them to talk to a parent, get law enforcement involved, or go to a crisis hotline? [...] a workforce that's substantially global, with the capacity to interface with teens across the world, is likely what's needed to respond effectively within the one-hour commitment."

The one thing the experts Engadget spoke to agreed on is that Open AI should be taking eating disorders seriously. "Eating disorders are far more prevalent than people realize. Many individuals see them as a rare problem that is only experienced by a very narrow population: mostly thin, affluent young girls," said Ellen Fitzsimmons-Craft, associate professor of psychology and brain sciences at Washington University in St. Louis. She pointed to a 2023 meta-analysis that found 22 percent of children and adolescents screened positive for disordered eating.

According to Professor Fitzsimmons-Craft, treatment access is also poor, with less than 20 percent of people with an eating disorder reporting they ever received treatment specifically for their eating disorder. "Any platform that allows youth to connect or put their own content on it has to think about disordered eating," she said.

"Certain eating disorders, anorexia in particular, have the highest mortality rate among mental health conditions for teenagers. So it's right to treat this as a crisis," Torney said. "Notifying a trusted adult, like a parent, is called for in moments where not intervening could quite literally be a matter of life or death."

Fitzsimmons-Craft said she is in favor of the notifications because one of the most effective treatments for adolescent eating disorders involves including the youth's family. "This is an intervention where parents are really tasked with playing an active role in helping to restore the adolescent's weight," she said. "They're coached by their therapist and provider on how to play that role [...] There are cases where certain teens may have parents who are not helpful, or who are perhaps contributing to the problem in some way. Of course it doesn't work for everyone, but it's certainly the best that we have."

Golin was more critical of Open AI's implementation. "Parents should know if their child is struggling," he said, but cautioned that "features like this should be rolled out with a smaller group and tested — not just whether it's flagging things accurately, but whether that's leading to better outcomes overall for the teen." He added that "the way this was just announced feels more like a response aimed at public relations than actual deep thinking about what kind of support [teens] need."

All three experts identified the fact that parents and children must link their Chat GPT accounts as a weakness in the feature's design. "Most parents, overwhelmingly, do not avail themselves of [parental oversight] tools, in part because they are deliberately made hard to find and use," said Golin. A recent report by Cybersafety Research Center tested 86 safety features across four platforms — Instagram, Snapchat, Tik Tok and You Tube. It found nearly 60 percent of those features, 51 of 86, either did not work as described or were difficult to use. A report Fairplay published last year also found that many of the safety features associated with Instagram's teen accounts required parents to tap through multiple pages of options.

"What we know is far more effective is safety by default and safety by design — putting things on the most protective setting, having a time limit built in that a parent can then change, rather than putting the onus on the parent to set the time limit themselves," Golin added.

"Currently, we limit alerts to parents with linked accounts in parental controls," Open AI told Engadget in a statement. "Our goal is to support users and help them connect to additional sources of real-world support. We will continue testing and strengthening these protections with experts, families, and young people." The company did not clarify what actions, if any, it will take if it flags an unlinked account for harmful prompts around eating disorders.

If you're struggling with disordered eating, the ANAD Eating Disorder Helpline is available Monday through Friday from 9:00 AM to 9:00 PM CT at 1-888-375-7767.

Key Takeaways

• Child safety experts are skeptical of Open AI's Chat GPT for Teens

• They say the company must prove it can be trusted before products like these get a pass

• On Tuesday, Open AI released Chat GPT for Teens, an age-gated version of its chatbot designed for young users

• "There's no need for a teen to create a new account or change anything; if we predict you're under 18, or you've told us so, this becomes your default experience," Lauren Jonas, Open AI's head of youth and families, told Engadget

• On Tuesday, the company also said it would expand its safety notifications to contact parents with linked accounts when their child has unsafe discussions around eating disorders with Chat GPT, and published an updated under-18 model spec