The Stanford Law Study Tells Us More About Who Pays Than Who Wins

A peer-reviewed Stanford Law study published this week finds that AI outperformed law professors in blinded contract-law evaluations roughly 75% of the time. The finding is real and the implications are worth taking seriously. But the conversation that has followed is having the wrong argument — focused almost entirely on whether AI can "replace" professors, and almost not at all on who absorbs the cost if law schools decide to act on it.

What the study actually shows. Professor Julian Nyarko's team at Stanford Law ran nearly 3,000 blinded pairwise matchups, using 40 contract-law questions, with 16 professors from 14 US law schools as the human comparator group. Evaluators did not know which answers came from AI (Google Gemini 2.5 Pro and NotebookLM) and which came from credentialled faculty. The AI won between 75.33% and 75.92% of those matchups depending on scoring method.¹

The harm-flagging result attracted less attention but is arguably more significant. Only 3.53% of AI answers were flagged as potentially harmful to student learning. The figure for professors was 12.06%, roughly 3.4 times higher.¹

The credential is doing different work than the teaching. The most common defence of legal education in response to findings like this is that law schools sell more than knowledge transfer. They sell professional socialisation, network formation, supervised clinical experience, bar exam infrastructure. All of this is true. It is also a partial concession that the teaching function, the actual delivery of doctrinal knowledge and legal reasoning, may not be the core product law schools are selling.

That is a power argument, not a quality argument. The JD degree functions as an accreditation gate: you cannot sit the bar in most US states without one. Law schools have used that gate to sustain tuition models that, at many institutions, now exceed $60,000 per year.² If the pedagogical output of faculty is measurably inferior to a free AI tool in a blinded test of the domain most central to first-year legal curriculum, the gap between what is being charged and what is being delivered becomes harder to paper over.

The study is deliberately narrow: contract law, a doctrinal area with settled rules and a large training corpus. Performance in constitutional litigation, sentencing advocacy, or novel regulatory questions is not tested here and almost certainly varies.¹ NotebookLM is a retrieval and summarisation tool, not a general reasoning model, and grouping it with Gemini 2.5 Pro as a unified "AI" category flatters the finding's generalisability.³ These are real limits. They don't neutralise the finding in the domain tested. They do caution against reading it as a full indictment of legal education.

Who pays is not who was tested. Here is the gap in almost every commentary I have read since Tuesday: the 16 professors who participated in the study are, almost certainly, tenured or tenure-track faculty at research universities. They have job security, institutional standing, and relatively little to fear from a benchmark result. They are also a small minority of the people actually doing the teaching in American law schools.

Adjunct law instructors, legal writing lecturers, contract clinical supervisors, and part-time practitioners who fill course schedules at lower cost — these are the people who are exposed if law schools respond to this study by cutting instructional headcount or substituting AI tools for supplemental teaching. The study evaluated the best-credentialled segment of the teaching workforce. The disruption, if it comes, will fall on the least protected segment.

This is not a hypothetical pattern. It is the pattern. When technology is used to justify headcount reductions in credentialled professions, the reductions concentrate in the precarious parts of the workforce, while the tenured core remains largely intact. The professors who were outperformed in this study are probably safe. The adjuncts who were never evaluated are probably not.

The credential stays. The teaching labour gets cut. The tuition does not go down.

The "hallucination" narrative is losing ground in this domain. The dominant brake on AI adoption in legal contexts has been a version of "AI hallucinates law": it invents citations, mischaracterises holdings, and confidently produces wrong answers in ways that are hard for non-experts to catch. That concern is legitimate in many legal contexts. It is harder to sustain as a blanket objection to contract-law pedagogy after a finding of 3.53% harmful responses versus 12.06% for professors.

One important caveat: the criteria for "harmful" flagging are not publicly specified in the available source material. If evaluators were instructed to flag answers that expressed uncertainty, and professors gave more jurisdictionally hedged answers than AI did, the metric could be partly capturing calibration differences rather than accuracy differences.² That is a genuine methodological question the study needs to answer more clearly.

But legal-AI vendors — Harvey, Legora, Spellbook — do not need methodological certainty to use this finding. They need peer-reviewed institutional cover for sales conversations with resistant general counsels, and Stanford Law has now provided it. The narrow scope of the study (contract-law educational Q&A) will disappear in the slide deck. The headline number will not.

The voice that is absent. Law students were not the evaluators in this study. The people paying $60,000 or more per year for the credential the professors hold, the people whose outcomes the pedagogy is meant to serve, were not consulted about what they want from legal education or how they experience the difference between AI-generated and faculty-generated instruction.¹ This is not a criticism of the study's design for its stated purpose. It is a reminder that "AI answers are preferred by evaluators" is a different question than "AI answers produce better lawyers" or "students prefer AI instruction."

The study measures output quality as judged by unspecified evaluators in a blinded Q&A format. It does not measure learning outcomes over time. It does not measure mentorship. It does not measure what happens to a 1L who receives excellent contract-law explanations from a tool that cannot supervise a negotiation, respond to a client's fear, or write a letter of recommendation.

What follows. The Stanford result will be used, selectively and aggressively, to justify cost-cutting decisions that were already being considered on other grounds. That is not the fault of the study. It is the normal way that research enters institutional decision-making. The people making those decisions are administrators and deans with budget pressures; the people who absorb the consequences are contingent educators and, downstream, students whose tuition does not decrease when a faculty line is cut.

The credential is not going away. The JD is a legal requirement for bar admission; no benchmark result changes that. What is at stake is the labour and pedagogy inside the credential, and the question of whether the people paying for it have any say in how it changes.

So far, no one is asking them.

Footnotes

¶