In two recent episodes of the podcast I host, Technically Legal, the issue of whether the legal industry needs its own specialized or “purpose-driven” artificial intelligence has come up. Both guests on those episodes believe that legal does, in fact, need its own AI.
Cecilia Ziniti, Co-founder and CEO of GC.ai, says that without specialized legal training, in its current incarnation, AI becomes “dopey.” In the interview, Cecilia pointed out that while lawyers use ChatGPT, they need something more. Her issue with generalized AI tools is their “fatal flaws,” such as the inability to accurately cite specific sources or correctly manipulate document clauses.
In its current state, she compares general AI to a “dopey” intern and emphasizes that legal work isn’t just generating text; it is also about analyzing risk through the use of specific workflows and maintaining attorney-client privilege. She notes that general AI tools may not offer attorney-client privilege protections or a specific UI (user interface) lawyers need.
She told me that “you can use ChatGPT for legal work…but it’s just not really good enough. A lawyer needs to know it terminates for material breach on 60 days’ notice…and by the way, [here it is in] section 8.2.”
Similarly, Anna Guo, founder of AI testing organization LegalBenchmarks.ai, arrives at a similar conclusion. Her testing revealed that while general models are surprisingly accurate, AI specialized for the legal industry wins on utility and risk management.
She told me that her group’s research found that general models (like Gemini and ChatGPT) sometimes beat specialized legal AI tools on raw text accuracy, but that legal-specific tools scored higher on “qualitative” metrics — how helpful and usable the output really is for lawyers.
She also said something very interesting, and when she did, it made complete sense: AI specifically trained on legal data was often more thorough analyzing risk scenarios and went the extra mile. Legal AI flagged unenforceable or illegal clauses that humans missed entirely. The legal AI kept going when the humans stopped.
In our work building out our AI-augmented contract review services, Cecilia’s point about AI becoming dopey has borne out.
By way of background, our app’s workflow is this: During onboarding, our customers enter their organization’s contract “playbook” and preferred exemplar language for common contract clauses. Armed with the playbook, AI takes a first pass at redlining a contract and then someone from our team reviews the redlines (just as a senior attorney would review a junior attorney’s first pass). The human corrects errors, catches misses and returns the contract to our customer.
Initially, when asked to analyze even the most straightforward contracts like non-disclosure agreements, in our app’s AI “mind”, if the contract language did not precisely match our customer’s exemplar language, it would think that it was not consistent with the playbook and replace it–even if the intent of the original language was the same. Moreover, the AI had to be prompted multiple times to complete the contract redlining tasks when a human would not stop until the entire contract was reviewed.
After some trial and error, we got it dialed in, but Cecilia’s point is well taken, without tweaking and prodding, the AI output was subpar to say the least and definitely missed nuance.
But all of these factors point to an existential question about legal AI. As the AI companies continue to improve their models, they will presumably become pretty damn good at analyzing legal questions. If so, how does that change specialized legal AI products?
Whatever AI becomes — the “singularity,” artificial general intelligence or superintelligence that someday equals or surpasses human skill — we are still a long way off. The history of AI development is not linear, and for decades has been a series of fits and starts.
Currently, the frontier AI companies are spending a great deal of resources on reinforcement learning to refine LLM specialization and improve AI’s ability to identify nuance. This is obviously paramount for knowledge work like legal, so the AI models will get better, to be sure. But the question is: how much better?
Regardless, while AI is very, very helpful to legal professionals in its current state, for the foreseeable future, it still needs a little “Human Touch.” And, that being the case, we should use the best tool for the job and put the lawyer in the loop in the best possible position after the tech does its magic. Because that gap still persists, the “purpose-driven” legal AI tools are not a luxury, but an absolute necessity.
The post Is ‘Purpose-Driven’ Legal AI Needed or Will Generalized Tools be Enough? appeared first on Percipient – Legal Services Powered by Technology.