Lawyers v. AI

The days of lawyers whose main tasks are routine contract work seem numbered – the robots are definitely coming for (the boring parts of) our jobs.

Two recent studies tested algorithms trained to evaluate legal contacts against the professional skills and speed of experienced lawyers either directly, or by real world analogy.

It was no contest.

The first study, conducted by legal AI platform LawGeex and professors from three prestigious US lawschools (Stanford, Duke and Southern California), pitted twenty experienced lawyers against an AI. The task was to review five non-disclosure agreements (NDAs) and to identify 30 legal issues.

Checking of non-disclosure agreements and similar documents, being a routine, expensive and high-volume task, is an obvious target for automation. Yet it is also a major income generator for many law firms.

Both the AI, which had been trained with thousands of similar documents, and the group of lawyers had to check five documents (which had not been part of the AI training data set). One of the lawyers who competed against the AI confirmed that the task was very similar to what lawyers do every day.

The human lawyers achieved an average of 85% accuracy rate vs the AI’s 95%.

The human lawyers took on average 92 minutes. The AI 26 seconds!

The fastest lawyer did the job in 51 minutes and the slowest 156 minutes.

In terms of accuracy the highest performing lawyer was on par with the AI but the lowest performing lawyer only achieved a 67% accuracy rate.

Listen to Sandra and Kai discuss AI distrupting lawyers on The Future This Week podcast.

This sort of work, assessing standard legal documents, is the bread and butter of legal services and so this test goes to the core of what many lawyers do day to day – and highlights the existential ‘threat’ facing law firms from AI.

These stunning results were mirrored in a second study by a Dutch AI entrepreneur, where the AI achieved an accuracy of 94.9 reviewing NDAs, but was not specifically pitted against human lawyers.

However Arnoud Engelfriet also highlights the complex process involved in building the AI so it can operate at this level of efficacy. Engelfriet uploaded 300 NDAs, all of which had to be meticulously labelled by hand. His data set contained about 10,000 sentences and to train the AI each sentence had to be marked if it contained something untoward for an NDA.

Moreover Engelfriet cautions the AI is very much a creature of its creator, meaning it will necessarily express the inherent preferences or biases of both its author and training data set. For example early testing of the AI consistently red flagged NDA written under Californian law as too strict, even though some were OK. It turned out the training set had a disproportionate amount of strict Californian NDAs and therefore the AI concluded all NDAs from California were too strict.

It seems to us there are two ways AI will change the business model of legal practice:

Lawyers will use AI to speed up their operations – which in the short term could increase their profit margins as they can continue to charge premium for their services. However competition in the industry will drive prices down and it is more likely law firms will hire fewer lawyers and more data scientists and use their past data to train their own branded AI. So firms might soon start to compete by promoting their own customised AI.

Alternatively, ‘legal’ AI will be an independent resource (free or subscription service) available to other businesses or individuals who will use it to check if a document raises concerns. If the AI raises a red flag a lawyer will then be engaged to provide highly personalised legal knowledge and individual advocacy skills for only a small subset of documents.

So this is the message to law students and junior lawyers: Change is coming. The profession is changing and some employment opportunities might disappear. At the same time, it is a must to be informed about these developments and to consider how the coming AI can be used to make the most of its opportunities.

For the general public – this can be an important development if access to these new technologies provides more equal access to affordable legal services.

You can subscribe to this podcast on iTunes, Spotify, SoundCloud, Stitcher, Libsyn or wherever you get your podcasts. You can follow us online on Flipboard, Twitter or sbi.sydney.edu.au.