Next step - Lozza the bazza?
An 'AI paralegal' has breezed through the Solicitors Qualifying Exam multiple choice section on its first attempt.
The AI chatbot, called 'Lawrence', got 67 out of the 90 multiple choice sample questions correct for SQE1 on the SRA's website. The 74% score achieved by the bot is higher than the average human pass rate, which has fluctuated between 55% to 65% since the exam was introduced.
Lawrence was created by Lawhive, an online legal services network, and has a LLM; which in this case isn't a Masters of Law, but a Large Language Model (the same technology underpinning applications like Chat-GPT).
Flinn Dolman, co-founder of Lawhive and a human, told RollOnFriday that a potential shortfall of this type of AI is that it can be prone to "hallucination", which means that it says "things that sound right but are complete nonsense". One US lawyer discovered this flaw the hard way.
A multiple choice exam, like SQE1, is a useful way of testing how much a system hallucinates, as the questions require contextual application of knowledge, said Dolman.
The Lawhive co-founder said the team was "quite impressed by the general breadth of areas" that the bot performed at, as "ultimately there was no correlation between the questions he got wrong and any given legal area".
Dolman said Lawrence's performance was "slightly weaker on questions that required a logical chain of interdependent thought", and that it was prone to confusing two related but separate concepts, such as private and public nuisance.
Lawrence also undertook a mock client interview based on the legal analysis section of SQE2, with his answers pitted against those of a human solicitor. To carry out the experiment, Lawhive captured a written transcript for both Lawrence and the human and showed them to a blind panel of reviewers.
The panelists agreed that Lawrence had adequately covered all necessary questions, but that the solicitor demonstrated more client empathy. Although the ability to understand another human's emotions and feelings may not be a strong point for all solicitors.
Dolman said that Lawrence was currently limited to written conversation, but that "long term we are considering whether different modalities could be useful (especially where accessibility is concerned)", like stretching a rubber lawyer's face over a metal skull with unblinking red eyes.
"In the future we want to roll him out so that all of the solicitors we work with can benefit," said Dolman. "Right now we're trialling a more limited version of him with a subgroup of users because it's incredibly important given the potential impact this kind of technology can have that we don't rush headlong in."
"I actually believe the technology is already mature enough to majorly disrupt", he said, "the only unknown quantity is how resistant to change the various players in the legal space will be".
Dolman considered that it might cause a rift between "younger, newer firms that can quickly adopt new tech (AI included) and the older legacy firms that aren't able to move as quickly because of operational debt".
But he did not anticipate that the legal profession would need fewer lawyers, at least in the next 10-20 years or so. "I think it's more likely that the capacity in which legal professionals work will change and that new roles/opportunities will be created," he said. "If I was to draw an analogy to programming, where AI has already had a measurable impact, the demand for skilled professionals is higher than ever."
Last week, RollOnFriday reported that the SRA had warned firms of the risk of using AI, whilst also highlighting its benefits.
Comments
50
9
I think this shows clearly how dumbed-down the SQE is. Law is not a multiple choice practice. You have to identify the choices yourself and then advise which is best / least bad based on analysis and reasonable assumptions. Try AI with a mediation...
24
10
SQE1 being entirely multiple choice does worry me even though it requires good English and skills and is closed book. I don't mind that the LPC is in smallish part multiple choice as it supplements the longer normal questions and is a kind of extra back up to check knowledge. My Finals in the 80s were 100% closed book over 2 weeks in late July examining the whole year's work - no multiple as indeed was my LLB. I use that knowledge all the time now, years later, judgments I read at university, cases etc. it is why the City Consortium is wise to insist on an LLB or PGDL and SQE courses, not just someone mugging up the SQE as a teenager in their bedroom at home after they had gone their level 6 online diploma in knitting studies.
The new system is fraught with problems too - what courses do people pay for - a huge balancing exercise between do I pay for the best course but cannot afford it (if no sponsoring firm) or buy a cheap course that may look bad, do I do my 2 years volunteering in the law centre near home for my QWE or does that mean I will never get a job. It was simpler when there were just solicitors and barristers and when the route was clear (and I write that as someone with 4 lawyer children including 2 trainee solicitor ones (LPC vintage) and also with 3 of their cousins doing or planning tod o the PGDL./SQE so I have looked at both routes recently, never mind hosted in a sense an exam centre for 2020 finals at home (covid), PGDL - 100% online due to covid lockdowns and then the LPC - online exams at home for two (although they could have chosen to go into an exam hall by the time of the LPC).
On AI it made up a complete section number the other day when I tested it on something although it conculsion on the law was the same result as mine so it was not entirely useless. It also founds me a Denning uote but lied and said it was from a case in the 1890s which cannot possibly be right.
It also keeps saying it is normaly the view that XYZ is so even if the thing asked is a definite point and the answer should be it is 100% certain that XYZ is so.
22
5
In praise of AI? Or a thorough demonstration of how useless the SQE is?
Por que no los dos?
22
14
If you're regularly referring to forty year old cases and judgments in your daily practice, you may want to update your knowledge.
23
8
Shouldn't headline be "SQE 1 so easy that even nascent AI bot passes with flying colours"?
38
4
Having sat SQE 1 in Jan 23 (and passed), the questions on the SRA website are significantly easier than those in the actual exam
21
7
More concerning is that 55% is a pass
17
4
Lydia. The “Daddy Pig” (bit of an expert) of legal practice…
27
4
If the exam it passed with a reasonably good grade is "closed book" and the point of AI models is that they are fed massive amounts of data to be taught what to say, surely the accurate comparison is the AI vs a solicitor with all their law books in front of them? It is a bit like saying my hoover can outrun a sprinter, when it is sitting in the boot of my car being driven down the motorway.
27
6
Perhaps Lawrence could be tasked to identify when a lawyer is likely to steal millions from under the SRA's noses as well.
20
4
Lol @ all this shows is how dumbed down the SQE is. What this shows is that legal profession as we know it is going to be royally fvcked when it comes to AI.
26
7
Its been a while since a client asked me a question and then offered me five different options to select as an answer...
23
3
I think that @12:53 might be onto something here.
Perhaps there could be scope for an alternative AI model, which could be fed with real-world imagery and trained to recognise images of Elbows and Arses, respectively. It could be hooked up to a sort of co-pilot program installed on every single member of the SRA and its Board's computers to helpfully let them tell the two apart as they went about their working days?
19
4
it says "things that sound right but are complete nonsense".
I am currently dealing with a number of AIs it appears
18
4
@13:57 - yes, that's my experience too.
Indeed, my clients are usually very clear that there is only one answer that they are interested in hearing, and they have no time for any others.
19
5
New technology will phase out existing professionals, eh? Really?
MS Excel (or its predecessor Lotus 1-2-3) was predicted to reduce necessity for accountants and tax professionals. How's that working out?
New tech = new value chain = new skills required = new jobs we have not yet conceptualized
But please let the mainstream media continue to spruik the "AI = all bad and you should all be scared" storyline ...
16
4
Could have been an ideal candidate for Axiom Ince
16
3
"Although the ability to understand another human's emotions and feelings may not be a strong point for all solicitors."
A classic.
18
2
We use AI tools for notes, disclosure, doc reviews and DD - honestly, without a human being to guide the machine learning, they are just expensive toys.
18
2
Apparently AI cannot reliably identify fire hydrants or bicycles, so why you would place your transaction or litigation in its care I have no idea.