Artificial intelligence is rapidly evolving from a simple drafting assistant into a sophisticated analytical powerhouse. To test the limits of modern AI, 16 professors from 14 prestigious U.S. law schools—including Yale, NYU, and the University of Chicago—formulated 40 intricate contract law questions spanning legal doctrine, case law, and hypothetical scenarios.
The Shocking Results of Blind Testing
In a massive blind evaluation consisting of 2,918 comparisons, professors consistently preferred AI-generated answers over those written by their academic peers. The models proved to be highly coherent, structured, and legally sound.
- 75.92% — Win rate of Google’s Gemini 2.5 Pro against human law instructors.
- 74.75% — Win rate of Google’s NotebookLM in head-to-head matchups.
- 3.41% — Harmfulness rate of Gemini’s answers, compared to 12.06% for human professors.
To determine whether this success was merely due to surface-level writing styles, researchers engineered a set of lexico-syntactic features. The analysis confirmed that the AI’s advantage was rooted in substantive content, including superior recall of doctrine, better handling of hypotheticals, and more nuanced policy discussions.
“The potential benefits of these new technologies as a force multiplier in the practice of law just can’t be ignored. Future employers will expect familiarity with these AI tools.” — John P. Anderson, Dean of Mississippi College School of Law.
The Reality Check: Hallucinations and Risks
While AI excels in controlled academic environments, the legal industry still grapples with real-world integration challenges. Hallucinations remain a critical issue. For instance, the prominent law firm Sullivan & Cromwell recently admitted to a U.S. bankruptcy court that a filing in a high-profile case contained fake citations generated by AI.
Frequently Asked Questions
How did AI perform compared to human law professors?
AI models like Gemini 2.5 Pro and NotebookLM won roughly 75% of blind matchups against human instructors, offering more accurate, structured, and safer answers.
Which AI model performed the best in the study?
In a broader analysis of advanced models, Anthropic’s Claude Opus 4.7 claimed the top spot, followed closely by OpenAI’s ChatGPT 5.4 and Google’s Gemini 2.5 Pro.
