The most prestigious law school admissions discussion board in the world.
|
 |
 
|
Crazy that the AI Legal Agent benchmarks are amongst the lowest
Poast new message in this thread
Date: July 28th, 2026 1:59 PM Author: gunnerretttired (gunneratttt) STEMmos have always looked down on us, but it turns out automating that stuff is way easier than putting together A DEAL! (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50029944) |
Date: July 28th, 2026 3:28 PM Author: gunnerretttired (gunneratttt) (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030208) |
|
Date: July 28th, 2026 3:32 PM Author: femgooner meta scientist My irl lawyer friends don't believe me when I tell them that it's only a matter of time before enough money is thrown at RL environments to make LLMs almost as good at law as they are at programming
Right now they are already better than most attorneys (most attorneys suck) |
 |
Date: July 28th, 2026 3:41 PM Author: gunnerretttired (gunneratttt) I agree it's better than many attorneys already and that someday it may get to that point you're saying. I'm just saying it's a bit suprising how much of a gap there is between legal performance benchmarks and other things.
https://x.com/i/status/2064394151441863006 |
|
Date: July 28th, 2026 3:48 PM Author: femgooner meta scientist It's because they haven't thrown resources at RL for law specifically. The models are extremely good at programming because there has been an enormous amount of resources sunk into training them for that specifically. It's true that programming is easier (less resource intensive) than law to RL train for, but it still required a lot of effort to get models this good at programming
Eventually the same thing will happen for law. The incentives are there because lawyers are outrageously overpaid for the actual level of expertise and training that they have. They're able to command the excessive compensation they get only because they're a monopoly/cartel
AI displacing lawyers is going to be one of the most positive outcomes from AI because lawyers have such disproportionate power and influence in our society and their interests conflict with those of most people. They are an extremely anti-social force in modern society and responsible for a lot of our society's ills |
|
Date: July 28th, 2026 4:00 PM Author: gunnerretttired (gunneratttt) But that monopoly exists because you cannot provide legal services without a license. The power and influence you're talking about won't change just because the work gets easier. According to you the work is already over overpaid relative to the effort required. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030280) |
|
Date: July 28th, 2026 4:20 PM Author: femgooner meta scientist Law's power in the US comes from it being an unassailable religion first, and a cartel second
AI will do to law in the public consciousness what "covid" and AI have done to medicine in the public consciousness. Exposed industry professionals as largely fraudulent rent-seekers rather than the Experts and Priests they supposedly are
It will happen a lot faster than people think. It's a question of technical capability. Once LLMs can straight up beat almost all human lawyers, the flood of pro se pressure on the legal system will cause huge societal changes. People will not respect The Law anymore. It will become just another scam industry in the public mind |
|
Date: July 28th, 2026 4:02 PM Author: ,.,.,.,.,.,.,.,.,.,..,.,.,...,,.,.,.,.,.,.,..,.,, You seem retarded. What makes you think the scores are normalized axroas fields idiot? (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030285) |
|
Date: July 28th, 2026 4:11 PM Author: gunnerretttired (gunneratttt) Well it can't be normalized because we're talking about different tasks. But it's hard for me to think of a legal task that's so much "harder" than those on Humanity's Last Exam that the difference in correct answers is attributable to something like the legal questions being "harder."
I think the correct answer is that there's much less data to train AIs on for legal work because the nature of the work means much of it is confidential. |
|
Date: July 28th, 2026 4:14 PM Author: ,.,.,.,.,.,.,.,.,.,..,.,.,...,,.,.,.,.,.,.,..,.,, Or it could be that harder than average problems are a bigger part of the legal benchmark's question mix than of the other benchmark. You have no idea. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030308) |
|
Date: July 28th, 2026 4:17 PM Author: gunnerretttired (gunneratttt) Humanity's Last Exam is designed to extremely difficult. It's hard for me to even conceive of how a legal task could be harder than questions on that. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030313) |
|
Date: July 28th, 2026 4:21 PM Author: femgooner meta scientist They are trained to beat humanity's last exam. They aren't trained to be lawyers
It really is that simple. The "benchmarks" are heavily gamed because there's enormous incentive to do so |
|
Date: July 28th, 2026 4:16 PM Author: lex I, for one, am optimistic for the future of this majestic profession. (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030311) |
|
Date: July 28th, 2026 4:23 PM Author: law is ea$y (🧐) I Philosophize, therefore I Poast (http://www.autoadmit.com/thread.php?thread_id=5887079&forum_id=2,#50030324) |