Skip to content

Editor’s Note: In our latest Industry News feature, “NYC’s AI Misstep: An Opportunity for Learning in Public Sector AI Deployment,” we delve into the recent controversy surrounding New York City’s attempt to integrate artificial intelligence into government services. The city’s AI-powered chatbot, designed to assist small business owners by providing streamlined guidance, has unfortunately disseminated inaccurate legal advice, leading to a significant outcry from both the technical and legal communities. This incident underscores the complex challenges and responsibilities inherent in deploying AI within the public sector, especially regarding legal implications and public trust.

Industry News – Artificial Intelligence Beat

NYC’s AI Misstep: An Opportunity for Learning in Public Sector AI Deployment

ComplexDiscovery Staff

New York City’s latest foray into technology-driven innovation has become a concern among the technical and legal communities. The city’s AI-powered chatbot, meant to provide streamlined guidance to small business owners, has come under scrutiny for dispensing legal advice that could potentially lead to unlawful actions, stirring a debate on the role of artificial intelligence in government services.

The chatbot, devised as part of the MyCity portal and powered by Microsoft’s Azure AI, was launched last October with the intention of being ‘a one-stop shop for city services and benefits.’ However, multiple reports indicate that it has done exactly the opposite, providing entrepreneurs with information that contradicts current laws. A report by The Markup, co-published with local nonprofit newsrooms Documented and The City, has highlighted these inconsistencies.

For instance, the AI has been reported to tell business owners it’s permissible to operate as cashless establishments, despite the fact that in 2020, the New York City Council prohibited such practices to ensure accessibility for all residents. Moreover, the chatbot has been found telling landlords that they can discriminate based on the source of income and informing employers that they can retain workers’ tips, both of which are inaccurate pieces of advice and practices deemed illegal in New York City.

These revelations have sparked widespread concern. Rosalind Black, the Citywide Housing Director at the legal assistance nonprofit Legal Services NYC, asserted, ‘If this chatbot is not being done in a way that is responsible and accurate, it should be taken down.’ The implications of these inaccuracies are grave, as they not only confuse business owners but also put them at risk of legal liabilities.

Despite the backlash, municipal leaders remain committed to the chatbot. Mayor Eric Adams has publicly supported the continued deployment of the chatbot on the city’s official government website. On Tuesday, Adams acknowledged that while the chatbot produced ‘wrong answers in some areas,’ keeping the chatbot operational allows the city to identify and correct issues, embodying a forward-thinking approach: ‘Only those who are fearful sit down and say, ‘Oh, it is not working the way we want, now we have to run away from it all together.’ I don’t live that way,’ said Adams.

Both Microsoft and the NYC Office of Technology and Innovation, represented by spokesperson Leslie Brown, have committed to improving the AI service. ‘We will continue to focus on upgrading this tool so that we can better support small businesses across the city,’ Brown stated, indicating the pilot program’s ongoing nature. The objective is not just to remedy the current issues but to refine the system for future utility and reliability.

While progress is being made towards betterment, instances such as these raise deeper questions about the accountability of AI in performing critical functions, especially those with legal implications. The incident serves as a reminder of the importance of rigorous testing and transparent governance when integrating AI into public service applications.

As the conversation around the responsibilities of AI continues, other municipalities looking to employ similar technologies can look to New York City’s experiences as a learning opportunity. The pursuit of innovation must be coupled with a strong adherence to accuracy, ethics, and the law. With proper oversight and continuous refinement, AI has the potential to be a powerful ally in public administration, but the road there is proving to be fraught with challenges that demand cautious navigation.

News Sources


Assisted by GAI and LLM Technologies

Additional Reading

Source: ComplexDiscovery OÜ

The post NYC’s AI Misstep: An Opportunity for Learning in Public Sector AI Deployment appeared first on ComplexDiscovery.

Alan N. Sutin

Alan N. Sutin is Chair of the firm’s Technology, Media & Telecommunications Practice and Senior Chair of the Global Intellectual Property & Technology Practice. An experienced business lawyer with a principal focus on commercial transactions with intellectual property and technology issues and privacy

Alan N. Sutin is Chair of the firm’s Technology, Media & Telecommunications Practice and Senior Chair of the Global Intellectual Property & Technology Practice. An experienced business lawyer with a principal focus on commercial transactions with intellectual property and technology issues and privacy and cybersecurity matters, he advises clients in connection with transactions involving the development, acquisition, disposition and commercial exploitation of intellectual property with an emphasis on technology-related products and services, and counsels companies on a wide range of issues relating to privacy and cybersecurity. Alan holds the CIPP/US certification from the International Association of Privacy Professionals.

Alan also represents a wide variety of companies in connection with IT and business process outsourcing arrangements, strategic alliance agreements, commercial joint ventures and licensing matters. He has particular experience in Internet and electronic commerce issues and has been involved in many of the major policy issues surrounding the commercial development of the Internet. Alan has advised foreign governments and multinational corporations in connection with these issues and is a frequent speaker at major industry conferences and events around the world.