Unlearn automation on The Future, This Week

Disclaimer We'd like to advise that the following program may contain real news, occasional philosophy and ideas that may offend some listeners.

Sandra So, Kai, what are we doing this week?

Kai Well, we're still away on holidays.

Sandra We are well into our longitudinal auto ethnographic research on leisure time. So we're not recording a regular episode.

Kai We talk a lot on The Future, This Week about robots and algorithms, machine learning, automation.

Sandra They're all meant to come and make our lives easier, right?

Kai But turns out, they don't. AI is coming to make your work harder.

Sandra But speaking of automation, we do have something to tide you over until season 11 starts up.

Kai And that's our new podcast series, and it's actually very good.

Sandra If we may say so ourselves. It's called The Unlearn Project. And it's our new series about changing common sense.

Kai And it's quite a different show to The Future, This Week, where you know, we sit down and banter every week. This one is about those topics, ideas, concepts, where the world has changed without us even noticing.

Sandra You might have already heard our episode on unlearning computers. We do have another one for you today, another episode from our new series, this time about unlearning automation.

Kai Enjoy.

AUDIO Oxford University predicts nearly half of all jobs will disappear within the next 25 years. For the first time in history, workers will be able to be human. Automating some of the grunt work so that we can be more efficient. robots will steal your job, but that's okay.

Kai That's okay.

Sandra That's what we hear all the time, automation will reduce the amount of work that we have, it will lead to some job losses. But at least what we're going to be left with is of higher quality, the creative, fun, interesting work where we'll get to use our imagination.

Kai But is the common wisdom, right, that underpins automation. And to be fair, like, historically, that has always been the case. If you think about farming, manufacturing, mining. Robots and automation have always had this right, made work less strenuous, less dangerous, repetitive, difficult, really helped with the physically taxing work. Some jobs will go but at least what's left is better.

Sandra Yeah, but is it? Really, actually, is it? Because turns out with AI, this is no longer true. I mean, cognitive automation leaves us worse off. And in many cases with AI, the quality of the work that's left really goes down.

Kai Yeah, what's left is more difficult.

Sandra Secondly, AI makes our work harder.

Kai It's definitely different this time around, and we're going to figure out what's going on and what we can do about it.

Sandra So, on this episode of The Unlearn Project, we'll hear about the effects of automation, and how work at tech companies, for example, can become tedious and unpleasant as employees are left wrestling with only difficult decisions all day long, which can come as a real surprise, like, all of a sudden…

Fred Benenson Suddenly, you know, somebody says, like, "oh, yeah, I guess the robot is getting to approve all the good projects." And then people just gonna sit there, you know, drinking their coffee, and they're like, "that's funny."

Kai We will also hear from rideshare drivers who have a really hard time relating to their algorithmic managers.

Michael You realise, in reality, you're working with an algorithm. And you're trying to find favour with that algorithm, and you realise you don't actually find favour with that algorithm.

Sandra And we'll hear from airline pilots who now require much more training and experience, not less, to fly highly automated planes.

Stuart Beveridge Rather than look at reduced training requirements when introducing new automated systems, but actually more training requirements with automated systems.

Kai And we'll talk about lawyers who no longer do as much of the kind of document-based work that used to be a key training ground for them.

Kimberlee Weatherall It's one of those things that I worry about for junior lawyers, you know, and people who are coming through the profession now.

Sandra So on this episode, we're looking at why automation and artificial intelligence will often make work more difficult, more tedious, stressful, and overall less enjoyable.

Kai Which runs counter to the common wisdom that's held true for physical automation since the industrial revolution.

Sandra We'll look at the impact of what's called cognitive automation and what to do about it. AI is coming to make your job much, much harder.

Sandra I'm Sandra Peter.

Kai I'm Kai Riemer.

Sandra Welcome to The Unlearn Project.

AUDIO Economists are predicting that 47% of all jobs across the world are going to be stolen by artificial intelligence, robots. I really like the idea of automating some of the grunt work so that we can be more efficient. And then as a creative team that sort of will free us up to solve bigger problems. Robots will steal your job, but that's okay. And it can be more than okay, it can be marvellous. We will not need to care about processes or numbers, we will just focus on being.

Sandra So, yeah, automation and AI are everywhere. And that's pretty much how it gets talked about. The robots are coming for our jobs, there will be less work available, but at least we'll get to do all the fun stuff, the creative, imaginative work.

Kai So when we came across this story at Kickstarter a couple of years ago, it left us really puzzled. They were automating clear cut, repetitive parts of people's jobs. But people seem to be left with only the difficult, challenging, often frustrating work.

Sandra We were like, "hang on, wasn't AI going to make all our jobs easier?"

Kai This was so interesting we started researching this and turns out AI really does make our jobs harder.

Sandra The story in this episode is really the story of how we started The Unlearn Project in the first place. And it all kicked off with Kickstarter. And do people know what Kickstarter is?

Kai Probably.

Sandra So just in case,

Kai It's this tech company, right?

Sandra Yeah, this tech company that started back in 2009. And people would come to this crowdfunding platform with wild ideas and then Kickstarter staff would get to decide which projects could ask for money from the public.

Fred Benenson This idea of fans directly supporting creators, Kickstarter was just, to me, it was like this really elegant way of creating culture.

Kai So you actually spoke to someone who was there from the very beginning.

Fred Benenson My name is Fred Benenson and I'm currently starting a company so you could call me the founder and CEO of Breadwinner. And before that I was the VP of Data at Kickstarter. And I was there for six and a half years as the second employee.

Sandra Not only that, but Fred also automated all the stuff at Kickstarter, but we'll come to that. So Kickstarter is this tool for artists, designers, makers, musicians and creative people. And it's just for creative projects.

AUDIO This is a Kickstarter campaign video. So the idea is this. I'm reading a book. Feature films. Stop motion animations. Sci-fi music video album. Help make something strange. Enjoy it. Bach wrote the Goldberg Variations. Cards Against Humanity is a party game for horrible people. After months of collaborating with our community, our team is back with a streamlined everyday carry backpack.

Kai Ha! I actually backed this project, I got this backpack.

Sandra No, not the, not the backpack. We know all about the backpack.

Kai Shame this is a podcast because I would show it to you. It is the best backpack.

Sandra Yeah, I know. But back…

Kai So when I saw this in 2019, I thought this is really cool. This is the perfect backpack and I backed it straight away.

Sandra But back to our story. So Kickstarter has this all or nothing model that allows creators to choose a funding goal and set a number of days to reach that goal. And then people like you, Kai, come and back the project with money.

AUDIO Last piece of the puzzle is going to be you. We cannot do this without you, we can't. Do back this Kickstarter project, you will be cool and everybody will like you.

Kai And again, shame this as a podcast because this is such a cool backpack. You've got this laptop compartment, iPad, a padded compartment for my headphones…

Sandra But back to our story, and back to Fred, who also remembers how the team at Kickstarter were super excited in those early days as fun projects were launching on the website.

Fred Benenson You'd kind of be like holding out for it and excited for it to launch. And especially if you had a personal connection to the creators, you'd really be rooting for them.

Sandra And further recalls how at that time,

Fred Benenson We had some pretty exciting moments, and at that time, the only way to get a Kickstarter project on the site and live was to like, basically write in, can I use Kickstarter? And we would just flag your account and say, okay, you're a creator now. And then people would kind of be off to the races.

Sandra And some of the projects were

Fred Benenson Really surprising, like out of left field style hits that were really amazing and wonderful to watch it really change people's lives, you know.

Sandra People at Kickstarter really had a blast,

Fred Benenson It was kind of a cool job, right? This job was kind of seen as like a bit of an ambassador to the Kickstarter brand.

Sandra And people really got a lot of satisfaction out of their job approving some of these cool projects and so the site grew very quickly.

Fred Benenson Kickstarter was growing incredibly,

Kai Like really, really fast. We looked it up, they went from approving less than 200 projects a month in August 2009 to over 2,000 a month in April 2011. And by the beginning of March 2014, Kickstarter had passed $1 billion in pledges to creative projects.

Sandra And again, shame this is a podcast because here is where we will show you these hockey stick graphs, they started with three projects and went on to 1 billion in pledges.

Kai It's like those charts, every startup dreams about starting flat, and they really showed up.

Sandra But it was also kind of teetering on the edge of falling apart because of the volume. You know, we are getting more and more popular and more and more projects are coming online and it was becoming like taxing for the team. So fast forward a year or two.

Sandra That was when it all exploded.

Fred Benenson I don't think anybody could have ever really predicted how big that company would become.

Sandra These people were there for hours working nights and weekends reviewing projects for the website.

Fred Benenson And that team was getting overwhelmed.

Sandra And Fred told me how holiday weekends were especially difficult for the team as the queue would back up with hundreds of worthy projects.

Fred Benenson And so this is all kind of reaching a boil.

Sandra And so Fred oversaw the development of an automated system to help with the influx of projects.

Fred Benenson And I tried to build an algorithm.

Sandra He built quite a few algorithms actually.

Fred Benenson And I just got obsessed with this problem.

Sandra The idea was to develop an automated system that would consider each project's stated purpose, its creators past record of success and other factors, and fast track the most promising ones.

Kai So it was really straightforward, right? Projects that scored the highest would gain approval to launch without any human intervention.

Sandra Yeah, and it worked!

Fred Benenson And so I built that, like a robot, that was like working on a weekend.

Sandra It worked really well.

Fred Benenson We could probably take 40 or 50% of this team's work off their plate.

Sandra Fred and his team even made sure that the experience was still best for the creators.

Fred Benenson And so we built it so that the robot would never reject people. It would only send them for human review. And it was only allowed to accept projects.

Kai This should have been great, right?

Sandra You'd really think so, right? But turns out, not quite.

Fred Benenson I don't think we totally realised that, that very kind of subtle, like product decision meant that the robot would just pick off all the best projects.

Sandra And it led to a dramatic drop in the average quality of projects that human reviewers would see.

Fred Benenson The effective end result was that the robot was saying yes, to all the good projects, and the human team that was working on it, or had been working kind of in the trenches, looking at all the projects, were solely only seeing the projects that the robot didn't think passed muster.

Kai So what Fred's really talking about is that the robot was picking off all the projects that were the easiest to analyse.

Sandra Yeah, so what was left for human reviewers were the projects with the muddy scores, particularly the ones where the ideas really tested the limits of Kickstarter's guidelines.

Fred Benenson And so it really changed the kind of like, feeling on that team.

Sandra The job was suddenly not as enjoyable. These people were now only seeing,

Fred Benenson Like the difficult edge cases where they're like, "I don't know, this person seems like really fine, but like the project feels really off."

Sandra Or like really dreary projects such as impractically heavy duvets or like questionable anime games, or fundraising for medical issues.

Kai So ones that are outside the guidelines.

Sandra Yeah, remember, they're only funding creative projects. So now there's no more easy wins, projects that the team can easily see take off.

Fred Benenson They were looking at previously, the full spectrum, you know, everyone who came through the front door, and they got to see all the good and all the bad to suddenly they're only looking at like the bottom half, according to a robot.

Sandra All of this really caught people by surprise, like, it caught us by surprise.

Kai It did catch us by surprise, totally.

Fred Benenson Suddenly, you know, somebody says, like, "Oh, yeah, I guess the robot is getting to approve all the good projects." And people just kind of sit there, you know, drinking their coffee, and they're like, "that's funny."

Sandra Because remember, no one is thinking about it because automation was always meant to make our work easier and more enjoyable.

Kai Exactly, like with the physical stuff, farming, manufacturing, mining, robots and automation really help, right? They help with the heavy, the difficult, the dangerous, the physically taxing stuff.

Sandra And we imported this assumption into cognitive automation.

Kai So while we're prepared for "Robots are coming for Phil in accounting."

Sandra Yeah, I remember that headline, that was in the New York Times.

Kai Yeah, but we aren't quite prepared for poor, old Phil to be a whole lot more miserable in the process.

Sandra Again, because the assumption going in is that what is left for us will be better, will be of higher value, creative, imaginative, inspiring.

Kai Yeah, but it's not. It might be higher value, but it's also ambiguous, difficult, often stressful.

Sandra So it turns out, we must unlearn automation, cognitive automation is different. It often does make our work harder. And that's because unlike when we automate physical work, cognitive automation goes for the easy parts.

Kai Yeah, absolutely. Physically strenuous tasks are relatively easy to automate, right? Lifting heavy objects, grinding things down, cutting stuff. While on the other hand, tasks that are relatively easy for people like handling delicate objects and picking flowers or blueberries, they're much harder to automate.

Sandra In cognitive automation, most often, it's the other way around. We can automate easy, routine tasks that we understand well, but algorithms then leave the hard and demanding stuff for the human workers.

Kai Yeah, and doing only the hard and demanding work, no matter how creative, that's not great.

Sandra It's not great.

Kai We were so puzzled by what had happened at Kickstarter that we started looking around at other industries to see if similar things were happening in other places as well.

Sandra And we found evidence of it in different forms, in many industries. Health, the gig economy, the airline industry, even the legal profession.

Kai Yeah, the creation of automated legal services, right, simple things such as drafting a contract, setting up a will, or dispute resolution, have really been shaking up the legal profession. There's been a lot of interest, experimentation, and a lot of promise for automation.

Kimberlee Weatherall To provide better access to justice for populations, where probably a majority of the Australian population would consider that they do not have the money to hire a lawyer where they have disputes.

Sandra That's our colleague from across campus, really across the road, Professor Kim Weatherall.

Kimberlee Weatherall My name, maybe? Kimberlee Weatherall from the University of Sydney Law School, I wear too many hats, so there's too many titles. So Professor of Law, I specialise in questions around the regulation of technology. I'm also a researcher with the ARC Centre of Excellence for Automated Decision Making and Society.

Sandra Kim has a keen interest in and does research on algorithms and automation in the legal profession.

Kimberlee Weatherall I'm speaking from talking to lots of lawyers, talking to lots of students, you know, following the legal tech and the discussion as it's occurring in the literature.

Sandra So on the one hand, automating services at the front-end of the legal profession really has the potential to…

Kimberlee Weatherall To give more people access to justice

Sandra Artificial intelligence or machine learning can provide certain legal services much more readily at a much lower cost, like checking or creating simple legal documents like contracts,

Kai And also quite well, right? I remember a study by academics from Stanford, Duke and Southern California law schools that found AI could actually do it better than humans.

Sandra And much faster, right?

Kai 26 seconds, where humans would, on average, take 92 minutes.

Sandra So on the one hand, AI promises to make accessible simple legal services at scale. On the other hand, automation is also used to do away with a lot of the repetitive and tedious work that lawyers, especially junior ones have to do. Things like…

Kai Like document discovery,

Sandra Yeah, like document discovery,

Kimberlee Weatherall If you're involved in a big litigation, you have to go and look back over all the documents about that dispute. And this can be hundreds of 1000s of pages, millions of pages in a really big piece of litigation. And in the past, this would have been done by the most junior lawyers, you would have quite literally put them in a room with boxes of documents and had them there with their highlighters and their sticky notes, picking out what were the relevant documents that needed to be looked at further by someone higher up the chain.

Sandra And that work is not the most fun job, right?

Kai It's not. It's a bit like at Kickstarter before they automated, having to go through all those projects after a long weekend.

Sandra Yeah, exactly.

Kimberlee Weatherall So as a junior lawyer sitting in a sometimes windowless room for hours and hours and weeks and weeks, just going through documents. A, it's not very interesting, it's really not very interesting work. B, it is easy to get tired, it's easy to get bored, it's easy to get distracted, it's easy to miss things.

Sandra That really doesn't sound like much fun.

Kai And it is a lot of work, right? And that's why many companies are looking at machine learning algorithms for pattern recognition. You can train these algorithms with millions of documents, like case files, or legal briefs to flag the kind of stuff that lawyers might need for a court case, for example.

Sandra So document discovery is at least partially being automated.

Kai And you could argue that this is actually a good thing, right? It will save a lot of time and saves the clients money.

Sandra Didn't JPMorgan already, in 2017, announce that it was using some software called contract intelligence or COIN?

Kai Yeah, COIN, right,

Sandra Which could, in seconds, perform document review tasks that took their legal aids something like 360,000 hours.

Kai A lot of time, so all around a good thing.

Sandra But is it?

Kai Well, well, actually, it creates a big problem for junior lawyers, but also for the profession as a whole.

Kimberlee Weatherall Now, firstly, there's the question of whether you get the job because quite simply, they just don't need as many people to do that work, maybe as you would have expected.

Sandra So the robots are coming for us, some jobs will be lost because of automation.

Kai Yeah, but the work itself is also changing. So Kim points out that, first of all, there'll be a loss in the social aspects of work.

Kimberlee Weatherall If you're interacting with a screen and a program as opposed to a team of people, you do lose something, you know, you lose a degree of camaraderie or collegiality. That used to be a really big part of the job.

Sandra And that's not alright. There's a more important question…

Kimberlee Weatherall How do you develop the level of expertise if you're not just having to read hundreds of cases? If in fact, you know, this program is doing a certain amount of the narrowing of the cases, you're still reading cases, but you're not reading all of them. You're not doing the same degree of work, maybe, that you did before.

Sandra So this discovery work might have been tedious, laborious, and at times boring.

Kai Yeah, going over stacks of dusty law books and case files.

Sandra But this grunt work is typically a key training ground because it exposes first year lawyers to lots and lots of documents and legal material. And as Kim says, There's,

Kimberlee Weatherall No better way to understand the business of your clients than to have to read every document that they produced on a dispute over the course of you know, multiple months. No, no better way to understand how contracts are put together and what goes wrong in contracts than seeing it litigated at the end, you know, seeing how a contract was put together by looking back over the documents. Where did that, where did that term in that contract come from, you sort of look back behind that, and then you work at it. And that's why it's gone horribly wrong, or no better way to understand how to put a contract together than to read hundreds and hundreds of them. This is where you get your expertise. It's one of those things that I worry about for junior lawyers and, you know, and people who are coming through the profession now.

Sandra And that creates a huge problem for career progression, when large chunks of a lawyer's work become automated.

Kai McKinsey actually predicted back in 2019 that about 23% of lawyer's time could be automated.

Sandra And that's a lot, right?

Kai Yeah, and if lots of these tasks fall away, I mean, how do you train junior lawyers, right? How will they learn? As always, with automation, lawyers might be able to concentrate on those higher value tasks, the more complicated stuff.

Sandra But they do have to develop those skills in the first place. And as Kim says, that's done by practicing, by spending the time,

Kimberlee Weatherall And going and reading hundreds of those cases, or reading hundreds of those contracts brings a degree of, I want to say expertise, but I also want to say wisdom, that I think is going to be hard to replicate at a speed or without just those years of thinking, reading, putting stuff together, getting feedback on it from your partner.

Sandra So it's really about the lack of exposure to these many documents.

Kai Yeah, but it's also about missing out on that apprenticeship that you get from working with senior colleagues on a team.

Kimberlee Weatherall Obviously, you do, you learn a lot from other, other team members, and you learn a lot from, you know, the senior associate who might be supervising that particular task.

Sandra And that's really the unexpected, counterintuitive effect of automating work in the legal profession.

Kai So when we automate the tedious stuff, the boring tasks, which is a good idea from an economic inefficiency point of view, we do make it harder to learn certain skills, like building really good arguments from studying loads and loads of documents.

Sandra So again, we need to unlearn the common wisdom that automation will make work easier. In this case, because junior lawyers lose the apprenticeship aspects that have now been automated.

Kai Okay, so we discussed Kickstarter, and the law profession.

Sandra But then we found even more places where automation makes work harder in quite counterintuitive ways.

Keith Chen Uber and other kind of gig economy platforms are basically the first time in history that large swathes of the economy can basically, kind of, you know, tap their smartphone and turn on work. That's a tremendous change and I think something that is potentially revolutionary, and that we don't fully understand the implications of yet.

Sandra That's Keith Chen, and he used to be the head of Economic Research at Uber.

Keith Chen My name is Keith Chen. I'm a behavioural economist and a professor of strategy and economics at UCLA Anderson School of Management.

Sandra And Keith and I spoke at UCLA during his lunch break teaching in the University of Sydney's Global Executive MBA program.

Kai And Keith designed some of the original algorithms, right, that are used to match and manage rides on the Uber platform.

Sandra He also taught monkeys how to use money, but that's a whole other episode.

Kai So if you've been living under a rock, Uber is a ride sharing service, right? Like a taxi service. And you take Ubers to work every day, right?

Sandra Every day, except during COVID lockdowns.

Kai Yeah, and so while Uber is set to disrupt the taxi industry, that's how it's often discussed, it did actually bring big changes to how work is organised.

Sandra But as you can imagine, a platform the scale of Uber can only work with automation. And much like Kickstarter, this story of automation starts with really good intentions.

Keith Chen Uber as a platform only really makes money when drivers make money and a lot of what my team at Uber tried to do was to help drivers make as much money as possible in as short a time as possible, you know, kind of maximise their kind of hourly earnings, hopefully spend more time on the platform and make both themselves and Uber more money.

Sandra So for people who want to earn money by driving, Uber has a really attractive offer.

Michael Choosing to work with Uber in the first place, is about the flexibility that I can just choose my own hours, I can choose 12 minutes or I can choose to work 12 hours.

Kai That is Michael and Michael drives for the platform.

Michael With Uber now for three years.

Sandra And researchers who have spoken to lots more drivers report the same.

Mareike Möhlmann They feel that there's a lot of autonomy, right, so they can log into the app whenever they want to, whenever they please to.

Ola Henfridsson For the workers, I think the main positive aspects relates to the flexibility, the ability to take on extra work on top of some other work.

Kai That's Mareike Möhlmann and Ola Henfridsson who together did an elaborate study on how Uber drivers experienced their work for the platform.

Mareike Möhlmann

My name is Mareike Möhlmann, Assistant Professor at Bentley University.

Ola Henfridsson So my name is Ola Henfridsson, I'm a Professor of Business Technology at Miami Herbert Business School.

Kai So, Mareike, Ola, and the team looked at the effects of what is now known as algorithmic management.

Ola Henfridsson Algorithmic management is really the large scale collection of use of data to develop and improve algorithms that can carry out coordination and control functions.

Sandra So the algorithm that Keith Chen and his team created perform the kind of work that in many businesses is done by managers.

Mareike Möhlmann Uber is really using this app to manage millions of remote drivers.

Sandra And there are two main things that are being automated here.

Mareike Möhlmann On the one hand, Uber is a marketplace, matching drivers to customers and users, just uses algorithms to do that very efficiently. And on the other hand, we find that the algorithms are also controlling drivers very heavily, right? They use algorithms to monitor and control platform workers.

Sandra So for Uber customers like me, it's really important that we find the driver quickly and that we get a good price.

Kai Yeah, and the Uber app takes care of this, right? But it also manages the workers. And it is this algorithmic management, the fact that workers have an algorithm as their boss that creates,

Ola Henfridsson You know, tensions when it comes to the feeling of being autonomous on the one hand, and feeling very highly controlled, on the other hand.

Michael You realise, in reality, you're working with an algorithm, and you're trying to find favour with that algorithm, and you realise you don't actually find favour with that algorithm.

Kai That's Michael, again, the Uber driver,

Michael I'd like to say I'm collaborating with it but I know it's attempting to manage me to make profit for the company.

Kai So in the case of Uber, what has been automated is not the work itself, the driving, but the management, right? What would normally be done by middle managers.

Sandra As with all cognitive automation, it's always partial. So some parts can be automated really well,

Kai Like work allocation.

Sandra Yeah, like finding the right drivers for someone like me who wants a ride. But the other parts can be automated less well, if you think of the app as being your manager, the automated manager can be quite inflexible.

Ola Henfridsson Sometimes it felt very frustrated by the fact that, you know, the the app suggested certain routes, when they felt that they had a better sense of the, of the area themselves.

Sandra What happens at Uber also happens at other gig platforms around the world.

Kishi Pan This is Mr Wei, he drives for Didi in Shanghai, Mr Wei explains how he's at the mercy of the app. That he is working for a machine that allows him very little flexibility in how he drives, especially when the traffic is really bad. There is a real issue when he can’t get to a customer fast enough. Customers are allowed to cancel. But he is the one who gets blamed by the algorithm. And when too many customers cancel his account is blocked for hours. He says the app works for the customer, it is 95% on the customer’s side. Sometimes the only autonomy he has is to switch the app off so that the algorithm can’t tell him what to do anymore.

Mareike Möhlmann So they really feel tightly controlled, they feel that they are really supervised by the platform. The platform is constantly tracking the data, right? So their GPS location, how fast they drive, so they felt very frustrated, right, so they felt supervised.

Sandra And the algorithm will nudge drivers to take on more work.

Michael So it's always on about making money, obviously, for the business. If you drove this route, you may pick up more customers.

Sandra But it can also nudge drivers to make certain rides that they don't really want to make.

Mareike Möhlmann Uber kind of made them behave in a desired way that is beneficial to the company but not to the rider by forcing them to take these rides.

Kai So drivers sometimes have this feeling of not really being in control of how they do their work. And they often don't quite know why their algorithmic manager would make certain decisions.

Alex Veen One of the sort of main things that stood out from our research is that workers experience algorithmic management, they try and make sense of it, but they actually don't fully understand how it operates.

Kai That's our colleague down the hallway, Alex Veen, who has done a lot of research into gig workers in food delivery, such as Uber Eats and Deliveroo.

Alex Veen So my name is Alex Veen from the University of Sydney Business School. I'm a lecturer in the discipline of Work and Organisational Studies.

Kai And Alex and his colleagues have found that gig workers are often frustrated because workers often don't get it.

Alex Veen Workers often don't have a very precise understanding of how these systems operate.

Kai But they will always try to figure out how these algorithms work. Because,

Alex Veen If workers actually had a better understanding of how these systems operated, they could more efficiently engage with them. So some have theories, I need to wait at a particular location, at least 100 meters away from the restaurant because if I'm too close by, congregating with other workers, I might not get picked.

Sandra But it's often impossible for drivers like Michael to figure out how the algorithm makes decisions.

Michael Sometimes I think about, I wonder how these algorithms are working. And wonder if I can find any advantage inside this algorithm just by tapping into it in an algorithmic way, no real success on that.

Kai So workers really struggled because the decisions made by algorithmic managers are often intransparent, drivers don't receive any explanations.

Mareike Möhlmann It was just like a huge black box for the drivers and really hard for them to understand what was going on.

Sandra So algorithms make decisions that sometimes just don't make sense to the workers, and that makes their life harder and more stressful.

Kai Especially when a decision threatens their livelihood, right, when their performance is assessed by the algorithm. And that can have severe consequences as Ola explains.

Ola Henfridsson Where perhaps some problem there is related to Uber, rather than themselves, you know, will still reflect in their customer reviews. And since those reviews are so important, in order not to be banned from the platform, it is a constant worry.

Kai So at the heart of the matter, here are those reviews that drivers or delivery workers receive from their customers.

Sandra And Michael says receiving a bad review and not knowing why can leave some of his fellow drivers quite distressed.

Michael The elderly drivers who have had a low rating, and they will just ponder on that for three, four or five days until they can get past that. So in that sense, they wellbeing's not being served very well.

Sandra And all those individual customer reviews do add up. Taken together, they make up a driver's rating as a score from one to five.

Alex Veen So everyone's familiar with these five star rating systems. And so the inputs in these rating systems determine the priority that these workers get in terms of task allocation, but also to what extent they have ongoing access on these platforms to work opportunities.

Sandra And if a driver's rating falls below a certain threshold, there can be severe consequences.

Alex Veen What we see in certain platforms, that depending on these performance ratings, accounts are deactivated or blocked, workers have to undertake further training.

Kai And so as a driver, what do you do, then when you feel that you've been treated wrongly?

Sandra Normally, you go and take that up with your manager, wouldn't you?

Kai Except your manager is now an algorithm, right? And that algorithm lacks any human judgment, understanding, goodwill, or really the ability to change it's mild.

Kishi Pan Mr Wei says that it is very difficult to appeal the decision that the Didi app makes. He tried to appeal against a penalty and the algorithm just shut him down, he says there is no means to appeal really.

Mareike Möhlmann So in many cases, they actually had the feeling that they were treated like robots that led to what we label as dehumanisation, right? So drivers did not have the opportunity to really talk to a supervisor, something that is quite normal in traditional work environments.

Kai So what we learned from the research that Mareike, Ola and Alex have done is that when management is automated, parts of the job become harder and more stressful.

Sandra Again, our point here is that cognitive automation is always partial, because certain parts of work are just hard to automate.

Kai And as a result, important things go missing, right, such as human understanding or human judgment. Also, algorithmic decisions go mostly unexplained.

Sandra That's not to say it's all bad. Everyone we spoke to emphasised the many good things that gig works offers.

Kai Like taking you to work in the morning.

Alex Veen That's not to say that it's always problematic and that these workers think that it's a terrible job or anything, but it's really that there are certain aspects of this sort of dehumanised algorithmic management that have proven to be quite problematic to workers.

Kai So this research into algorithmic management in gig work provides us with another example of how automation can make life harder for workers.

Sandra And this is really important because these days algorithmic management is making its way into many other places, including office work.

Kai So we've had Kickstarter where the algorithm is now having all the fun. We've had junior lawyers who find it hard to learn their way into the profession, and know gig work where people fail to find favour with their algorithmic managers.

Sandra But we have one more story, which is a really interesting one. And contrary to what people who know Kai might think, it's not here, because planes.

Kai But planes are so cool.

Sandra It's here because it's another counterintuitive one. Everyone knows about autopilots on planes, but automation on planes goes far beyond that.

Kai And you'd think that this kind of automation makes life easier for pilots,

Sandra But does it?

Kai Yeah, but does it?

Shem Malmquist Autopilots themselves are very simple. All it does is it, is you put in a program of where you want it to be and it just looks at the difference and applies that change. So, let's say you want it to hold an altitude, so all it does is it looks at, has a sensor that has what altitude you're at, and then it has a command that it can move the flight controls to try to adjust for that altitude. The autopilot is really a very, very simple device, it's not making any kind of decisions.

Kai Okay, that was Shem.

Shem Malmquist Shem Malmquist, I am an instructor professor at the Florida Institute of Technology, and I am also a Boeing 777 captain.

Kai And Shem is also safety researcher and accident investigator.

Sandra And also an aerobatics instructor with 32 years experience and he's down in Melbourne.

Kai Up in Melbourne actually, because it's Melbourne, Florida.

Sandra Not our Melbourne, Victoria, okay.

Stuart Beveridge Aviation has been dealing with the gradual introduction of automation for many decades. Essentially, since the introduction of jet aircraft, autopilots get more and more complex. And now they have multiple layers of automated systems that the pilot uses to fly an aircraft from A to B.

Kai And that was…

Stuart Beveridge So my name is Stuart Beveridge, I am an airline pilot. I currently fly internationally on the Boeing 747. I am also the Safety and Technical Director of the Australian Federation of Air Pilots.

Sandra And Stuart's also done aviation research in the field of human factors.

Stuart Beveridge So pilots of airliners, our job largely involves managing these multiple layers of automation. And that's often all at once to achieve our goal. In a normal flight, a pilot will intervene and guide automation and automated systems on a regular basis and this constitutes a significant portion of our workload and also of our skill set.

Sandra But today's planes are much more automated than just simple autopilots and that can create surprises that pilots are not familiar with.

Stuart Beveridge We've been seeing more problems with pilots in terms of a mismatch of understanding between what the automated system does and what the pilot intends or needs the aircraft to do.

Kai Like in the case of the Air France flight from Rio to Paris that crashed in 2009.

Stuart Beveridge The Air France 447 accident, where the flight crew were not able to ascertain the issue of the flight deck instrumentation and totally misdiagnosed the issue and essentially flew the aircraft into the ocean without intervening correctly. One key factor was a lack of understanding of what the aircraft was doing at the time.

Kai This was a devastating crash, and there have been other accidents such as the two Boeing 737 Maxes that crashed in 2018 and 19, and also other less severe incidents.

Shem Malmquist Qantas has experienced several. There was the QF72 flight with the A380, then there was also the Airbus A330, there was also a Boeing 777 that went into an uncommanded climb up the west coast of Australia.

Sandra And at this point, most people would be thinking that it must be pilot error that is the cause of most accidents.

Kai Yeah, but not true though.

Sandra Pilot error is not the cause of most accidents.

Kai As Shem points out, it's more accurate to say that pilots sometimes find themselves in those situations that just overwhelmed them.

Shem Malmquist Very simply, the pilots generally are just overwhelmed with more changes than they can manage. We're putting them in places where there's numerous variables and numerous things changing all at once that is outside anything that they have encountered or been trained for before.

Kai So more automation may very well mean more of these overwhelming situations.

Shem Malmquist Automation has been involved at some level with pretty much all of them.

Sandra So what automation does is to change what is required of pilots.

Stuart Beveridge I think it changed the nature of our job. It requires a deep understanding of an automated system's logic, and what it will achieve with what kinds of inputs.

Sandra So really, automation requires a lot more of pilots

Kai Yeah, because modern planes have lots more automation than just the autopilot. Pilots today don't actually control the plane directly, it's fly-by-wire. What they do now is actually mediated by automation, and that controls the engines, the flight controls, and it stabilises the plane, for example, to make flying more comfortable for the passengers.

Sandra But what that means is that when something goes wrong, say a sensor is faulty, what happens might be completely unpredictable for the pilots, right?

Kai Yes, so the plane might suddenly pull up or down and pilots have no clue why this is happening. So even though the automated system might still react, the way it's meant to, the way it's designed, now it's getting a faulty sensor input, and things become very unpredictable.

Sandra So pilots will now need to understand much more about how the automation actually works.

Shem Malmquist The main thing is to provide the pilots with the information that they need to put the airplane in a safe spot and give them the training so they can better understand how the systems are interacting. The way automation is designed today, pilots really need to understand, have a really good mental model of how it's working, the different modes that it goes through, different factors that will affect how it's going to behave. And that requires some extra training.

Kai Even Captain Chesley Sullenberger,

Sandra That is actual Captain Sully, not Tom Hanks in the movie.

Kai Even Captain Sully said in a recent interview with Shem that it requires much more training and experience, not less to fly highly automated planes.

Sandra Again, automation makes work harder.

Stuart Beveridge Rather than look at reduced training requirements when introducing new automated systems, but actually more training requirements with automated systems.

Kai So automation means pilots need more training, they need to train harder.

Sandra Yeah, because on the one hand, pilots like Stuart must have a mental model of both the aircraft and its primary system.

Stuart Beveridge Pilots do, as you mentioned, need to understand how to fly the plane themselves.

Sandra But on the other hand, pilots like Stuart now also need to know how the flight automation works.

Stuart Beveridge And then an ideal training program would train the pilots in understanding in all the levels of automation right up to the highest level of automation, where the aircraft is essentially flying itself with pre-programmed guidance by the pilot.

Sandra But yet again, common sense prompts us to assume that with more automation, we would need to know or do less.

Kai And that belief is actually quite widespread.

Stuart Beveridge I believe that there is an implicit understanding, not necessarily just in companies, but I think in society, in general, that increased automation requires less involvement from the human operator.

Sandra And that's often reflected in the way that pilots are being trained.

Stuart Beveridge Training programs are constantly being squeezed and often the level of understanding by the pilot of the automation has been lacking. I think that's been demonstrated in a number of accidents in the recent couple of decades.

Kai So again, we need to unlearn what we know about automation. Rather than assuming that automation needs less involvement from the pilots and less training, it actually needs more.

Sandra We've heard about all the different ways in which automation makes work harder, or less enjoyable, or more stressful, even though common sense would tell us that automation would make our life easier because we would be automating away all the tedious or the hard bits. And while it might take away some jobs, at least the remaining Work is of higher quality.

Kai But cognitive automation is different. And so we must unlearn what we used to know about automation.

Sandra And while we could easily end the episode here,

Kai Making Megan's life a bit easier,

Sandra Telling you to set aside common sense, you might rightly ask, so what?

Kai Well, automation makes work harder.

Sandra But it doesn't have to.

Kai There are things we can do about it once we've unlearned old ideas.

Sandra One thing we can do is not pit machines against humans, although that would be fun, but look at work as a whole.

Kai So that robots are not having all the fun. What we need is understanding what humans are doing,

Shem Malmquist Understanding what humans are doing, and what they're not doing, what we're doing right now, is we're designing systems, and then forcing people to adapt to those systems.

Kai That was Shem. Again, because in cognitive automation, robots are only doing part of the job, we need to design work systems that include both robots and humans.

Sandra And that means going beyond just thinking about automating tasks where we replace humans with robots.

Fred Benenson The reason we did it at Kickstarter was that the team was under stress, we built this as a function of them feeling stressed out by having too much work. But balancing what the actual stresses are, right, too much work versus too little, a certain type of quality versus other, but I actually think it belies like a more nuanced understanding of labour.

Kai And like our colleague Alex points out, not designing humans out of the system.

Alex Veen There needs to be some human intervention within these systems and workers need to have a point of contact to try and sort of at least escalate issues.

Sandra But that also means that humans have to understand what's going on, what the algorithm is doing.

Kai And that's our second point,

Stuart Beveridge You also need to design the automated systems to keep the human operator in the loop by providing information on key processes.

Sandra And that's the second thing we can do, give people a better understanding of what's going on.

Kai Yeah, that's right, because then with more transparency, you have less frustration and also more productivity.

Alex Veen Then workers could make more informed decisions about where, when, and how they will work and for what periods of time. So I think that overcomes some of that information asymmetry that currently exists.

Kai And that was our colleague, Alex Veen, once again. And overcoming this information asymmetry is important in all industries.

Stuart Beveridge You need systems that create transparency on what they do and that also allows informed intervention, you still need the human ability to intervene, but you need, the keyword there is informed, the human needs to know what they're doing when they are intervening.

Sandra So humans need to know what the algorithm is doing to be prepared to take over, to know how to interact with it and interpret its decisions.

Kai And that is true, really, for any kind of automated system, whether it's a pilot flying a Boeing 737 Max, an Uber driver making decisions about when to work, a bank manager giving out a loan to a small business, or a doctor making a diagnosis with an automated system.

Sandra There's one more thing we can do when automating cognitive work to make sure we don't end up worse off, redesigning roles.

Kai We need to recognise that working with algorithms requires new skills. And that means more and different training needs.

Stuart Beveridge You have to counter that natural tendency with increased, as I said increased training, increased understanding of the system, and increased inflammation of the system on key processes as it operates.

Kai So in those professions where automation disrupts established ways of learning on the job,

Sandra Or career pathways,

Kai We need to deliberately rethink how people update their skills or create new kinds of learning experiences.

Sandra In the end, I think even though we're all techno-optimists, it does seem we have our work cut out for us with them, algorithms and robots.

Keith Chen Have a tendency to be techno-optimists and you know, maybe a little bit more optimistic about where technology is taking us in society than we should be. I do think that like even a techno-optimist has to pause and say that with the advent of new technology, at the very least kind of users, we're all going to have to grow up a little bit in our use of technology and understanding its implications for kind of how we live.

Kai And the first step is unlearning old ideas about automation.

Sandra So that we're prepared when robots come to make our work harder.

Sandra Next time on The Unlearn Project: why music is no longer just for listening.

Kai But then what is it for? We'll hear from producers, artists, creators and key figures in the industry and figure it out.

Ole Obermann Gets us to what I think is a really profound change in the role of music, because it's all about setting this mood and this vibe and many more people feel that they can be creative using music today than ever in history. I'm Ole Obermann, I'm the Global Head of music at ByteDance and TikTok.

Sandra Oh yeah, and there's…

Ole Obermann There's a great Australian artist Masked Wolf.

Harry Michael What up, it's your boy Masked Wolf and you might know me for my hit Astronaut in the Ocean.

Outro This was The Unlearn Project. Our sound editor was Megan "that was good, now do it better" Wedge. And this episode, and additional nerdy stuff, was written by Sandra Peter and Kai Riemer. We had help with bits and pieces from the entire SBI team. Kishi Pan has done everything China, she is also the only one of us who speaks Mandarin. If you're wondering about the music you're hearing right now, it's one of the Bach Goldberg Variations, a public domain recording made possible by a Kickstarter project and used by us because it's beautiful, and more importantly, free. If you want to know a little more about the topics and research in our podcasts, or for a full nerd out, our shownotes are available sbi.sydney.edu.au/unlearn. The Unlearn Project is a production of Sydney Business Insights, an initiative of the University of Sydney Business School. You can follow us on LinkedIn, Twitter and WeChat. You can subscribe, like, or leave us a positive rating wherever you get your podcasts.

Sandra We hope you have enjoyed this episode from our other series, The Unlearn Project, and we hope you subscribe to The Unlearn Project wherever you get your good podcasts.

Kai Go do it. Do it now.

Sandra And we will see you back here on The Future, This Week.

Kai After our break, and that's all we have time for today. Thanks for listening.

Sandra Thanks for listening.

Outro This was The Future, This Week, an initiative of The University of Sydney Business School. Sandra Peter is the Director of Sydney Business Insights and Kai Riemer is Professor of Information Technology and Organisation. Connect with us on LinkedIn, Twitter and Flipboard and subscribe, like or leave us a rating wherever you get your podcasts. If you have any weird and wonderful topics for us to discuss, send them to sbi@sydney.edu.au.