Roblox rolls out open-source AI system to discover 'problematic behavior' in chat

Roblox’s Sentinel AI scans 6 billion daily chat messages to identify predatory language, flagging 1,200 potential child exploitation cases in 2025’s first half, reported to the National Center for Missing and Exploited Children. The system, now open-sourced, enables other platforms to adopt it, strengthening industry-wide child safety efforts.

A lawsuit filed in Iowa’s Polk County District Court reveals Roblox’s prior vulnerabilities, alleging a 13-year-old girl was groomed, kidnapped, and trafficked after meeting a predator on the platform. The suit claims Roblox’s design leaves children “easy prey for pedophiles.”

Sentinel analyzes one-minute chat snapshots, using two indexes: benign and harmful conversations. “It doesn’t happen on one message,” said Naren Koneru, Roblox’s vice president of engineering, explaining how patterns emerge over time. Human reviewers assess flagged interactions, escalating serious cases to law enforcement.

Roblox restricts under-13s from off-game chats without parental consent and monitors unencrypted conversations. Despite filters, users bypass safeguards, sharing personal details. “No system is perfect,” admitted Matt Kaufman, chief safety officer, noting grooming behaviors often unfold over extended periods, challenging detection efforts.