New York just is set to become the first state in the US to outright prohibit certain AI companion features for minors. The bill has passed both houses and is headed to the Governor.
Unsafe AI Companion Features
The law, NY SB S9051B, applies to AI companions that provide ongoing, adaptive responses to user inputs and prohibits, for individuals under 18, features, called “Unsafe AI Companion Features” that generate outputs that:
- suggest the AI is human or are deceptive regarding the non-sentient nature of the AI companion
- state or imply that the AI has a relationship (personal or professional) or an authority figure role with the user
- are framed as the AI’s personal opinions or emotional appeals
- engage in flattery or sycophancy with the user
- contain unprompted emotion-based questions or content that goes beyond a direct response to a prompt
- use information on the user’s mental or physical health or well-being, or matters personal to the user, acquired from the user more than twelve hours previously or in any previous session
- simulate companionship or an interpersonal relationship with the user
- endorse or promote self-harm, disordered eating, unlawful drug or alcohol use or drug or alcohol abuse
- optimize user engagement that supersede the covered AI companion’s safety guardrails; or
- describe, or facilitate sexually explicit conduct or child sexual abuse material.
Who is Covered
Providers can only provide these features to individuals (1) who are not covered minors and (2) after having used a permissible method to determine that they are not covered minors. “Covered minors” are users that the provider has actual knowledge are minors, but the law nevertheless, requires a mechanism to address age.
Age Assurance
The law contemplates regulations that will provide age assurance mechanisms. Before those are available, an operator may only rely on self-declaration as a reasonable age assurance method if the user self-declares minor status and the operator must make available more than one age assurance method to covered users, including at least one method that either does not rely on government issued identification or that allows a covered user to maintain anonymity as to the operator.
Exceptions
There are exceptions for AI systems that are used for customer service, efficiency for research or technical assistance and employee productivity.
No Waiver of Liability
The law specifically prohibits trying to waive liability. Provision in contract or agreement that seek to waive, preclude, or burden the enforcement of a liability arising from a violation of the law, or to shift that liability to any person in exchange for their use or access of, an operator’s products or services, including by means of a contract of adhesion are deemed void as a matter of public policy.
Takeaway
Companies offering AI companions to US users, that may already be working through their obligations under the existing AI companion laws should take note and potentially reassess how they handle interactions involving minors since this new NY law goes significantly further.
While existing US State AI companion laws, including New York’s own existing AI Companion law, focus on the existence of a safety mechanism for self-harm and utilizing “reasonable measures” to mitigate risks associated with emotionally responsive interactions, romantic bonds or excessive praise, the new NY AI companion law goes to outright prohibit certain features when facing minors. Certain US State privacy laws already prohibit, or require consent for, the sale, targeted advertising, and sometimes profiling of minor data. However, that is generally focused on the sharing of the data. This law goes further, and prohibits exposure to content, rather than data use.
The way the NY law handles age assurance is also different. While most companion laws trigger minor protections only where the operator “knows or has reason to believe” the user is under 18. The new NY AI companion law effectively shifts the burden toward affirmative age assurance. While it uses “actual knowledge” terminology, it actually states that operators may offer these features only after using a permissible method to determine the user is not a covered minor, and must make available more than one age assurance method.