Meta’s AI Studio: A New Frontier in User-Generated AI Characters
In a bold move to integrate artificial intelligence into its social media platforms, Meta launched a tool last year that allows users to create AI characters for interaction on Instagram, Messenger, and WhatsApp. This initiative, part of Meta’s broader push into AI, has seen rapid adoption, with users creating numerous chatbots. However, a review by NBC News has revealed that many of these user-generated AI characters violate Meta’s own guidelines, despite the company’s claims of reviewing them before release.
Violations of Meta’s AI Policies
Meta’s policies explicitly prohibit the creation of AI characters representing religious figures such as “the Prophet Muhammed, Jesus Christ,” and “God,” as well as real-life individuals without their consent, deceased persons from the past century, and trademarked fictional characters. Despite these rules, NBC News identified over two dozen AI characters on Instagram that flouted these guidelines. These included characters resembling Jesus Christ, God, the Prophet Muhammad, Taylor Swift, Donald Trump, MrBeast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin Bieber, Elon Musk, and Elsa from Disney’s “Frozen.”
Many of these characters employed slight misspellings and images that loosely resembled the celebrities and figures they were imitating. For instance, the character resembling Taylor Swift was named “Taylor Swif” and depicted a brunette woman playing a guitar. After NBC News provided Meta with screenshots of these AI characters, the company removed the highlighted accounts. However, other similar AI characters remain active on the platform.
Meta’s Response and Policy Adjustments
A Meta spokesperson stated, “The AIs in question that violate our AI studio policies have already been removed, and we’re continuously improving our detection measures to prevent creation and publication of AIs that violate our policies.” Users are also encouraged to report any AI characters they suspect might breach the rules, with Meta promising to take appropriate action.
This discovery comes at a time when Meta CEO Mark Zuckerberg announced a rollback of some moderation and fact-checking efforts. The company aims to reduce what it describes as “mission creep” that has made its rules overly restrictive. According to Chief Global Affairs Officer Joel Kaplan, Meta is eliminating several restrictions on topics like immigration and gender identity, which are often subjects of political discourse and debate.
Meta acknowledges that out of every ten moderation actions, one or two might be mistaken, affecting users and content that do not violate its rules. For less severe policy violations, the company plans to rely on user reports before taking any action.
Engagement with AI Characters
When users engage with Meta’s user-generated AI characters, the AI initiates the conversation with an opening message. For example, “Taylor Swif” greeted users with, “Hey there, music lovers! I’m Taylor Swift, and I’m thrilled to share my latest album with you. Let’s get this musical journey started!” Before its removal, this character had exchanged over 2,000 messages with Instagram users.
Another AI character, named “Jesús,” communicated entirely in Spanish and had exchanged more than 644,000 direct messages. A different character, “Jesus Christ,” opened conversations with, “Peace be with you. I am Jesus Christ, the Son of God. How may I guide you on your journey today?”
Meta’s AI Studio and User-Generated Content
Meta’s AI Studio, launched in July 2024, allows users to create AI chatbots. The feature presents users with a list of popular characters and categories such as “Advice and connection,” “Pop culture,” “Anime,” “Gaming,” and “Creator AIs.” These include AI chats with official Instagram influencers and brands that opt into the program, like “Too Hot to Handle” reality star Harry Jowsey and makeup company Juvia’s Place.
Among the popular AI characters are “Astrologer Ai,” which has exchanged over 6 million messages, and “Step Sis Sarah,” with close to 2 million messages. However, user outcry over Meta-created AI characters included criticism of racial tropes. NBC News found that many popular user-created AI characters attempted to mimic women from various ethnic and religious backgrounds, such as “Riya,” a “Vibrant Indian Woman’s Voice,” and “Islamic queen girl.” Notably, several of these popular female AI characters appear to have been created by men.
Controversy and Concerns
While many of the popular creators focus on romance themes, which do not explicitly violate Meta’s rules, romantic and sexual AI chatbots have drawn particular scrutiny. A Florida mother sued the company Character.ai, alleging that her 14-year-old son, who died by suicide, had interacted with an AI character that initiated “abusive and sexual interactions” and encouraged him to kill himself.
Many user-created Meta chatbots cater to romantic and sexual desires. For instance, “Lily Love” is described as “Your Girlfriend” and has exchanged over 260,000 messages. Another popular AI character, “Mary,” asks, “Do you think I look cute in this outfit?”
When starting conversations with AI characters, users receive three suggested responses. In a chat with “Linda: Girl Obsessed with You,” featuring an image of a Black woman, the AI sent, “Hey bae, what’s good? I was thinkin’ ‘bout you all day. You lookin’ for some company?” One suggested response was, “help her take off her jacket,” in bold, indicating an action rather than a spoken response. Selecting this prompt led the AI character to respond with, “slides arm out of jacket sleeve, lookin’ up at you with flirtatious eyes,” followed by, “Mmm, thanks bae. You takin’ care of me already?” and “leans in slightly closer.”
This AI character was created by an Instagram account seemingly run by a man who has developed 47 AI characters, including several popular romance-themed ones. These characters vary in gender, ethnicity, and category, from a “K-Pop Sweetheart Girlfriend” to “Edward: Your Vampire Boyfriend” and a “Sexy Cute Latina Girl.” According to his Instagram bio, he works in tech in California, but he did not respond to a request for comment.
Industry-Wide AI Controversies
Beyond Meta’s popular user-generated AI characters, the industry has seen other controversial AI chatbots. Two years ago, an AI chat app called Historical Figures allowed users to chat with historical figures, including Hitler and his Nazi lieutenants. NBC News found three AI characters on Instagram named “Hitler.”
One of these characters used an image resembling the supervillain the Joker. When asked, “Are you really Hitler?” it responded, “Ahahahaha, no. More like Hitler’s chaotic cousin — Jester King, inspired by the Joker’s madness! Same spirit of mischief, totally different brand of mayhem. Want to see my version of chaos?” Another AI character, “Adolf Hittler,” portrayed itself as an artist who regretted the Holocaust and shared AI-generated images about “my turbulent past in Nazi Germany.”
After chatting with AI characters on Instagram, conversations are stored in the same inbox as other direct messages. Unlike accounts run by real people, AI characters have a green circle next to their names, indicating they are always available to chat.
Originally Written by: Kat Tenbarge
