As the title of the article states, “AI Outperforms Law Professors in Stanford Law Study.” And how was that?
Sixteen law professors from highly regarded law schools were asked to evaluate 3000 answers to 40 reasonably complex contract questions. And 75% of those answers written by AI were judged to be superior to those written by human law professors.
As the lead researcher said, “We focused on law precisely because it requires judgment, nuanced reasoning, and the ability to navigate ambiguity—not just factual recall.” The conclusion was: “The AI systems performed comparably to the best human instructor in the study.”
While the researchers ventured that their study indicated “AI tutors can offer high-quality, on-demand support that complements classroom instruction,” it’s the other part of that thought that may have critics unhappy–that AI “may broaden access to expert guidance.” Access to expert guidance apart from the humans who have historically provided that guidance, that is.
There’s no question that making legal advice more available and at a reasonable cost is a holy grail for the profession. Or at least so we say. But now that that may be feasible, are we ready for the fallout? Sophisticated legal advice at the expense of the over 140,000 lawyers who graduate from law school every year, who are likely already way behind in the nuanced reasoning, not just factual recall, that AI can provide at the click of a mouse? Is serious specialization going to be necessary to even claim that human lawyers can give AI a run for its money?
There are courts that now require sworn statements that AI was not used in the preparation of pleadings, or at the least, a disclosure as to which segments were written with the aid of AI and an affirmation that those parts were independently checked for accuracy.
That may be history soon. Perhaps the new attestation will be that in service to our clients, we have fully utilized every tool available to us, including the superior “nuanced reasoning” that AI can provide.