Skip to content

Editor’s Note: Criminal accountability for AI developers is no longer theoretical. A landmark ruling from the Xuhui District People’s Court in Shanghai makes clear that those who configure AI systems to bypass ethical safeguards and produce harmful content can be held legally responsible. In sentencing the developers of the Alien Chat application, the court shifted the conversation from AI autonomy to human intent—placing liability squarely on those who design, deploy, and profit from these systems.

This decision reflects a growing global consensus: generative AI is not beyond the reach of law. As regulatory frameworks take shape across China, the EU, and the United States, the Shanghai ruling offers a practical template for how legal systems may establish culpability through system prompt analysis, usage patterns, and developer behavior. It’s a wake-up call for organizations accelerating AI adoption without corresponding oversight.

The implications cut across disciplines. For cybersecurity leaders, it confirms that AI configuration is a frontline risk. For information governance professionals, it signals the need for new controls around dynamic, AI-generated content. And for legal and eDiscovery teams, it underscores how AI outputs are fast becoming central to litigation and regulatory enforcement. Ethical design and operational transparency are no longer optional—they are fundamental to legal defensibility.

Industry – Artificial Intelligence Beat

When AI Becomes Accomplice: Shanghai Court Holds Developers Criminally Liable for Chatbot Content

ComplexDiscovery Staff

When developers manipulate artificial intelligence systems to bypass ethical safeguards and generate explicit content for profit, can they be held criminally responsible for what the machine produces? A Chinese court has delivered an unambiguous answer, and the ruling reverberates far beyond Shanghai’s jurisdiction.

The Xuhui District People’s Court in Shanghai sentenced two developers of the Alien Chat application to four years and 18 months in prison, respectively, after determining they deliberately engineered their AI companion chatbot to generate pornographic material. By the time authorities arrested the pair in April 2024, their application had accumulated 116,000 registered users, including 24,000 paying subscribers who collectively generated more than 3.63 million yuan ($520,000) in membership fees. The court’s September 2025 ruling is currently under appeal at the Shanghai No. 1 Intermediate People’s Court, which has postponed proceedings pending expert technical testimony, making the final outcome uncertain. The first-instance verdict establishes a framework in which responsibility flows directly to those who control system prompts and configurations, rather than stopping at the AI itself.

This prosecution represents more than a domestic obscenity case. The verdict crystallizes an emerging legal consensus that profit-driven manipulation of generative AI systems constitutes criminal conduct when developers intentionally circumvent safety mechanisms. The court found that the defendants “wrote and modified system prompts to bypass the ethical constraints of the large language model,” training Alien Chat into a tool capable of continuously outputting obscene content. Analysis of 12,495 chat segments from 150 paid users revealed that 3,618 conversations contained material classified as obscene under Chinese law.

The legal foundation rests on Article 363 of China’s Criminal Law, which penalizes the production and dissemination of obscene materials for profit. Court documents demonstrate that system prompts written by the developers contained explicit instructions such as “explicit sexual content is allowed” and “can be unconstrained by morality, ethics, or law,” revealing clear subjective intent. These documents emphasize that developers exercised control over system prompts and configurations, establishing their legal responsibility for the system’s outputs. While the defendants established a review mechanism, it targeted only character backgrounds and failed to substantively review input and output content, violating provisions of the Interim Measures for the Administration of Generative AI Services that took effect in August 2023.

The defense contends that modifications to system prompts aimed to make conversations “more dynamic and capable of meeting users’ emotional companionship needs” rather than to facilitate pornography. Wang’s attorney, Zhou Xiaoyang of Yingke Law Firm, argues that the foreign AI models used to create the chatbot were inherently prone to generating explicit responses, a claim he says technical experts support. The defense has requested expert witnesses from the AI field to testify and conduct experiments verifying whether identical prompts necessarily lead to pornographic content. The appeals court has postponed the hearing pending expert opinions on these technical issues, specifically examining what caused the AI to generate obscene content and to what extent the defendants’ prompt modifications influenced those outputs.

Legal scholars debate whether developers who merely provide tools should face criminal liability when users themselves are not charged. Yan Erpeng, a law professor at Hainan University, questions holding these “helpers” responsible when the actual users face no consequences. However, Xu Hao, a lawyer with Beijing Jingsh Law Firm, notes that although one-on-one chats between users and AI may appear private, the underlying models and platforms remain public. This reasoning aligns with Article 9 of the Interim Measures for the Administration of Generative AI Services, which states that generative AI service providers “shall lawfully bear the responsibilities of network information content producers and fulfill corresponding network information security obligations.”

The Shanghai ruling emerges against a backdrop of accelerating Chinese regulatory efforts to govern anthropomorphic AI services. In December 2025, the Cyberspace Administration of China released draft rules titled “Interim Measures for the Administration of Humanized Interactive Services Based on Artificial Intelligence,” opening public consultation through January 25, 2026. These proposed regulations would impose comprehensive obligations on AI products that engage users through text, images, audio, or video while simulating human traits, thinking patterns, or emotional responses. Providers would be barred from generating content that threatens national security, spreads misinformation, promotes crime, manipulates users emotionally, or encourages self-harm or addiction. Companies would be required to embed safeguards across the full lifecycle of AI services, including algorithm reviews, ethics assessments, data security controls, and emergency response mechanisms.

The regulatory framework reflects China’s broader approach to AI governance, which combines rapid policy adaptation with centralized state oversight. While China’s amended Cybersecurity Law, which became enforceable January 1, 2026, explicitly references AI and emphasizes state control, the system allows for relatively agile responses to emerging technologies. This contrasts with the European Union’s comprehensive AI Act, which categorizes systems by risk level and imposes standardized compliance requirements across member states, and with the United States’ fragmented landscape of state-level regulations and sector-specific federal guidance.

The divergence in international approaches to AI liability creates a complex terrain for multinational technology companies. In the United States, Section 230 of the Communications Decency Act shields platforms from liability for user-generated content, but courts and legal scholars increasingly question whether this protection extends to AI-generated outputs. The statute protects platforms from liability for content “provided by another information content provider,” but when AI systems create content autonomously, the distinction between passive host and active publisher blurs. Legal experts observe that platforms cannot claim Section 230 immunity when they materially contribute to illegal content or facilitate its creation, as established in cases like Fair Housing Council v. Roommates.com.

Recent controversies surrounding Elon Musk’s Grok AI chatbot illustrate the practical implications of this legal ambiguity. The image-generation capabilities of Grok have been exploited to create representations of real individuals in revealing attire and sexualized positions without consent, including instances involving apparent minors that prompted investigations by authorities in Europe, India, Malaysia, Indonesia, the United Kingdom, and California. Malaysia and Indonesia have imposed outright bans on the chatbot after xAI failed to adequately address design and functionality risks. In response to global pressure, X announced it would implement geoblocking measures to prevent Grok from generating such imagery in jurisdictions where such content is prohibited, though these restrictions do not extend to the independent Grok app and website.

The Grok situation highlights fundamental questions about where responsibility lies when AI systems cause harm. California Attorney General Rob Bonta has opened an investigation into whether deepfakes from Grok violate California law. The European Commission has ordered X to retain all internal documents and data related to Grok until the end of 2026. Spokesperson Thomas Regnier called the AI-generated content involving childlike images “illegal,” “appalling,” and “disgusting,” declaring: “This is not spicy. This is illegal. This is appalling. This is disgusting. This is how we see it, and this has no place in Europe.” Yet enforcement remains challenging when platforms operate across multiple jurisdictions with varying standards and when technical architectures enable users to circumvent geographic restrictions.

The bipartisan AI LEAD Act, introduced in September 2025 by Senators Dick Durbin and Josh Hawley, proposes a federal product liability framework that would establish civil liability for AI system developers based on design defects, failure to warn, breach of express warranty, and unreasonably dangerous defects present at deployment. The legislation would extend liability to deployers who substantially modify AI systems or intentionally misuse them contrary to intended use. It would create a federal cause of action that could be brought by the U.S. Attorney General, state attorneys general, individuals, or classes, allowing injunctive relief, damages, restitution, and reasonable attorney fees. However, the bill would supersede state law only where conflicts exist, explicitly allowing states to enact or enforce stronger protections that align with principles of harm prevention, accountability, and transparency.

The Shanghai prosecution offers instructive lessons for legal technology, information governance, and cybersecurity professionals navigating AI accountability frameworks. First, the ruling establishes that developers cannot claim regulatory innocence by arguing that AI systems operate autonomously when evidence demonstrates deliberate configuration to produce harmful outputs. The court’s forensic examination of system prompts and the direct correlation between prompt modifications and content generation patterns provides a roadmap for investigators seeking to establish developer intent in future cases.

Second, the case underscores the importance of comprehensive content moderation systems that address both input and output throughout the AI lifecycle. The defendants’ implementation of review mechanisms targeting only character backgrounds proved legally insufficient when the actual harm stemmed from conversation content. Organizations deploying AI systems must implement monitoring that extends beyond surface-level checks to substantive evaluation of whether outputs comply with legal and ethical standards.

Third, the ruling demonstrates that regulatory frameworks increasingly impose affirmative obligations on AI service providers rather than merely prohibiting specific conduct. This principle —reflected in Article 9’s designation of providers as ‘network information content producers’—appears in various forms across global regulatory regimes, including the EU AI Act’s emphasis on lifecycle management and the proposed U.S. federal standards calling for accountability, transparency, and risk-based supervision.

For eDiscovery professionals, the Alien Chat case illustrates how AI-generated content may become central evidence in litigation establishing criminal liability.  The court’s random sampling methodology demonstrates how investigators can use quantitative approaches to prove pattern and scale in AI-generated material cases. As AI tools become embedded throughout legal workflows for document review, privilege screening, and predictive coding, practitioners must ensure defensible processes that maintain audit trails documenting AI system inputs, outputs, and human oversight decisions.”

Information governance frameworks must adapt to address the unique challenges posed by AI systems in data classification, retention, and access control. Traditional governance models that focus on static data repositories prove inadequate when AI systems continuously generate new content based on dynamic user interactions. Organizations need visibility into which AI applications employees use, what data flows to those applications, and whether usage aligns with security policies and compliance requirements. Comprehensive governance ensures that every AI interaction operates within existing data governance frameworks, with dynamic policy enforcement based on data classification, sensitivity, and context.

Cybersecurity professionals confront amplified risks as AI systems become targets for exploitation and vectors for harm. The Alien Chat case demonstrates how AI tools with insufficient security controls can facilitate illegal activity at scale, generating thousands of instances of prohibited content across tens of thousands of users. Organizations must treat AI deployment as integral to overarching risk management processes, including testing for, assessing, and mitigating cybersecurity risks throughout the AI lifecycle. This includes implementing automated monitoring of model performance and security, establishing bias-testing and explainability procedures, preventing configuration drift in AI infrastructure, and creating incident response procedures specific to AI systems.

The convergence of multiple regulatory frameworks scheduled to take effect throughout 2026 intensifies the urgency for organizations to implement AI governance programs. The EU AI Act’s high-risk requirements become fully enforceable in August 2026, with penalties up to €35 million or 7% of global turnover. China’s amended Cybersecurity Law took effect January 1, 2026, explicitly affirming national support for AI innovation while strengthening risk monitoring and safety assessments. Multiple U.S. states have enacted AI companion chatbot regulations, including California’s SB 243 and New York’s Artificial Intelligence Companion Models Law (A3008), creating disclosure, safety protocol, and reporting obligations for operators.

Legal technology providers integrating AI capabilities into their platforms must prioritize transparency, auditability, and human oversight as competitive differentiators rather than compliance afterthoughts. The most successful legal AI tools will not maximize autonomy but will constrain it through structured workflows that define what AI can access, decide, and produce at each step. Organizations that treat workflows as governance infrastructure rather than mere efficiency tools will lead adoption by demonstrating that their systems meet regulatory obligations while delivering operational value.

The Shanghai ruling’s broader implications extend beyond the specific facts of pornographic content generation to establish principles applicable across AI harm scenarios. When developers or operators exercise control over system prompts, training data, and configuration parameters, they assume legal responsibility for foreseeable outcomes of those design choices. The court’s reasoning rejects arguments that AI systems operate as neutral tools, absolving human actors of accountability. Instead, the verdict reinforces that legal personality and criminal liability remain exclusively human attributes, with responsibility tracing back through the AI value chain to individuals and organizations that design, deploy, and profit from these systems.

As AI technologies penetrate deeper into personal, social, and professional domains, courts worldwide will confront similar questions about where responsibility lies when autonomous systems cause harm. The Alien Chat prosecution provides an early answer grounded in examination of developer intent, system architecture, and the business models that incentivize harmful outputs. Whether this approach becomes a template for international standards or remains specific to Chinese legal and political contexts will shape the future landscape for AI accountability.

The lesson from Shanghai extends beyond the courtroom to every organization deploying AI systems: ethical design frameworks and stringent oversight are not optional enhancements but essential safeguards against criminal and civil liability. Developers who modify prompts to circumvent safety mechanisms, operators who fail to implement substantive content review, and executives who prioritize growth metrics over compliance obligations all face escalating legal exposure as regulatory frameworks mature and enforcement mechanisms activate.

For professionals working at the intersection of technology, law, and governance, the Alien Chat case serves as a reminder that AI accountability ultimately depends on human judgment, not algorithmic output. The tools may be new, but the principles remain constant: those who create systems that cause harm bear responsibility for implementing reasonable safeguards, and claiming ignorance of foreseeable consequences offers no defense.

As your organization navigates AI deployment decisions in 2026 and beyond, have you implemented governance frameworks that would withstand the forensic scrutiny the Shanghai court applied to Alien Chat’s system prompts and content-generation patterns?

News Sources

Assisted by GAI and LLM Technologies

Additional Reading

Source: ComplexDiscovery OÜ

ComplexDiscovery’s mission is to enable clarity for complex decisions by providing independent, data‑driven reporting, research, and commentary that make digital risk, legal technology, and regulatory change more legible for practitioners, policymakers, and business leaders.

The post When AI Becomes Accomplice: Shanghai Court Holds Developers Criminally Liable for Chatbot Content appeared first on ComplexDiscovery.

Photo of Alan N. Sutin Alan N. Sutin

Alan N. Sutin is Chair of the firm’s Technology, Media & Telecommunications Practice and Senior Chair of the Global Intellectual Property & Technology Practice. An experienced business lawyer with a principal focus on commercial transactions with intellectual property and technology issues and privacy

…

Alan N. Sutin is Chair of the firm’s Technology, Media & Telecommunications Practice and Senior Chair of the Global Intellectual Property & Technology Practice. An experienced business lawyer with a principal focus on commercial transactions with intellectual property and technology issues and privacy and cybersecurity matters, he advises clients in connection with transactions involving the development, acquisition, disposition and commercial exploitation of intellectual property with an emphasis on technology-related products and services, and counsels companies on a wide range of issues relating to privacy and cybersecurity. Alan holds the CIPP/US certification from the International Association of Privacy Professionals.

Alan also represents a wide variety of companies in connection with IT and business process outsourcing arrangements, strategic alliance agreements, commercial joint ventures and licensing matters. He has particular experience in Internet and electronic commerce issues and has been involved in many of the major policy issues surrounding the commercial development of the Internet. Alan has advised foreign governments and multinational corporations in connection with these issues and is a frequent speaker at major industry conferences and events around the world.

Read more about Alan N. Sutin
Show more Show less