Anna Guo reports on a study comparing the performance of lawyers and AI apps. Key takeaways:
1. Several AI tools matched, and in some cases outperformed, lawyers in producing reliable first drafts.
2. The top AI (Gemini 2.5 Pro) marginally outperformed the top individual human lawyer: 73.3% vs. 70% reliability rate.
3. Specialized legal AI tools surfaced material risks that human lawyers missed entirely.
My assessment:
The conclusions align closely with my subjective impression of comparative performance in several legal domains.
We should expect the performance of human lawyers vs. AI apps to vary depending on the type of domain and the facts. Most likely, contract drafting is a strong point for AI. My guess is that AI apps will also have an advantage in estate planning.
Most important: Teams of lawyers working with AI apps will usually be superior to lawyers or AI apps working alone. It’s what I call the Centaur Approach.