What happens when a company adopts AI

A group of economists conducted one of the first empirical studies of "generative AI" at a real-world company. They found it had big effects.

View this email online AI Enters The Workplace by Greg Rosalsky Lately, it’s felt like technological change has entered warp speed. Companies like OpenAI and Google have unveiled new Artificial Intelligence systems with incredible capabilities, making what once seemed like science fiction an everyday reality. It’s an era that is posing big, existential questions for us all, about everything from literally the future of human existence to — more to the focus of Planet Money — the future of human work. “Things are changing so fast,” says Erik Brynjolfsson, a leading, technology-focused economist based at Stanford University.  Back in 2017, Brynjolfsson published a paper in one of the top academic journals, Science, which outlined the kind of work that he believed AI was capable of doing. It was called “What Can Machine Learning Do? Workforce Implications.” Now, Brynjolfsson says, “I have to update that paper dramatically given what's happened in the past year or two.” Sure, the current pace of change can feel dizzying and kinda scary. But Brynjolfsson is not catastrophizing. In fact, quite the opposite. He’s earned a reputation as a “techno-optimist.” And, recently at least, he has a real reason to be optimistic about what AI could mean for the economy.  Last week, Brynjolfsson, together with MIT economists Danielle Li and Lindsey R. Raymond, released what is, to the best of our knowledge, the first empirical study of the real-world economic effects of new AI systems. They looked at what happened to a company and its workers after it incorporated a version of ChatGPT, a popular interactive AI chatbot, into workflows. What the economists found offers potentially great news for the economy, at least in one dimension that is crucial to improving our living standards: AI caused a group of workers to become much more productive. Backed by AI, these workers were able to accomplish much more in less time, with greater customer satisfaction to boot. At the same time, however, the study also shines a spotlight on just how powerful AI is, how disruptive it might be, and suggests that this new, astonishing technology could have economic effects that change the shape of income inequality going forward.   Mohamed Hassan/Pixabay The Rise Of Cyborg Customer Service Reps The story of this study starts a few years ago, when an unnamed Fortune 500 company — Brynjolfsson and his colleagues have not gotten permission to disclose its identity — decided to adopt an earlier version of OpenAI’s ChatGPT. This AI system is an example of what computer scientists call “generative AI” and also a “Large Language Model,” systems that have crunched a ton of data — especially text — and learned word patterns that enable them to do things like answer questions and write instructions. This company provides other companies with administrative software. Think like programs that help businesses do accounting and logistics. A big part of this company’s job is helping its customers, mostly small businesses, with technical support. The company’s customer support agents are based primarily in the Philippines, but also the United States and other countries. And they spend their days helping small businesses tackle various kinds of technical problems with their software. Think like, “Why am I getting this error message?” or like, “Help! I can’t log in!”  Instead of talking to their customers on the phone, these customer service agents mostly communicate with them through online chat windows. These troubleshooting sessions can be quite long. The average conversation between the agents and customers lasts about 40 minutes. Agents need to know the ins and outs of their company’s software, how to solve problems, and how to deal with sometimes irate customers. It’s a stressful job, and there’s high turnover. In the broader customer service industry, up to 60 percent of reps quit each year.  Facing such high turnover rates, this software company was spending a lot of time and money training new staffers. And so, in late 2020, it decided to begin using an AI system to help its constantly churning customer support staff get better at their jobs faster. The company’s goal was to improve the performance of their workers, not replace them.  Now, when the agents look at their computer screens, they don’t only see a chat window with their customers. They also see another chat window with an AI chatbot, which is there to help them more effectively assist customers in real time. It advises them on what to potentially write to customers and also provides them with links to internal company information to help them more quickly find solutions to their customers’ technical problems. This interactive chatbot was trained by reading through a ton of previous conversations between reps and customers. It has recognized word patterns in these conversations, identifying key phrases and common problems facing customers and how to solve them. Because the company tracks which conversations leave its customers satisfied, the AI chatbot also knows formulas that often lead to success. Think, like, interactions that customers give a 5 star rating. “I’m so sorry you’re frustrated with error message 504. All you have to do is restart your computer and then press CTRL-ALT-SHIFT. Have a blessed day!” Equipped with this new AI system, the company’s customer support representatives are now basically part human, part intelligent machine. Cyborg customer reps, if you will.  Lucky for Brynjolfsson, his colleagues, and econ nerds like us at Planet Money, this software company gave the economists inside access to rigorously evaluate what happened when customer service agents were given assistance from intelligent machines. The economists examine the performance of over 5,000 agents, comparing the outcomes of old-school customer reps without AI against new, AI-enhanced cyborg customer reps. What Happened When This Company Adopts AI The economists’ big finding: after the software company adopted AI, the average customer support representative became, on average, 14 percent more productive. They were able to resolve more customer issues per hour. That’s huge. The company's workforce is now much faster and more effective. They’re also, apparently, happier. Turnover has gone down, especially among new hires. Not only that, the company’s customers are more satisfied. They give higher ratings to support staff. They also generally seem to be nicer in their conversations and are less likely to ask to speak to an agent’s supervisor. So, yeah, AI seems to really help improve the work of the company’s employees. But what’s even more interesting is that not all employees gained equally from using AI. It turns out that the company's more experienced, highly skilled customer support agents saw little or no benefit from using it. It was mainly the less experienced, lower-skilled customer service reps who saw big gains in their job performance.  “And what this system did was it took people with just two months of experience and had them performing at the level of people with six months of experience,” Brynjolfsson says. “So it got them up the learning curve a lot faster — and that led to very positive benefits for the company.”  Brynjolfsson says these improvements make a lot of sense when you think about how the AI system works. The system has analyzed company records and learned from highly rated conversations between agents and customers. In effect, the AI chatbot is basically mimicking the company’s top performers, who have experience on the job. And it’s pushing newbies and low performers to act more like them. The machine has essentially figured out the recipe for the magic sauce that makes top performers so good at their jobs, and it’s offering that recipe for the workers who are less good at their jobs. That’s great news for the company and its customers, as well as the company’s low performers, who are now better at their jobs. But, Brynjolfsson says, it also raises the question: should the company’s top performers be getting paid even more? After all, they’re now not only helping the customers they directly interact with. They’re now also, indirectly, helping all the company’s customers, by modeling what good interactions look like and providing vital source material for the AI. “It used to be that high-skilled workers would come up with a good answer and that would only help them and their customer,” Brynjolfsson says. “Now that good answer gets amplified and used by people throughout the organization.”  The Big Picture While Brynjolfsson is cautious, noting that this is one company in one study, he also says one of his big takeaways is that AI could make our economy much more productive in the near future. And that’s important. Productivity gains — doing more in less time — are a crucial component for rising living standards. After years of being disappointed by lackluster productivity growth, Brynjolfsson is excited by this possibility. Not only does AI seem to be delivering productivity gains, it seems to deliver them pretty fast.  “And the fact that we're getting some really significant benefits suggests that we could have some big benefits over the next few years or decades as these systems are more widely used,” Brynjolfsson says. When machines take over more work and boost our productivity, Brynjolfsson says, that’s generally a great thing. It means that society is getting richer, that the economic pie is getting larger.  At the same time, Brynjolfsson says, there are no guarantees about how this pie will be distributed. Even when the pie gets bigger, there are people who could see their slice get smaller or even disappear. “It’s very clear that it’s not automatic that the bigger pie is evenly shared by everybody,” Brynjolfsson says. “We have to put in place policies, whether it’s in tax policy or the strategy of companies like this one, which make sure the gains are more widely shared.”  Higher productivity is a really important finding. But what’s probably most fascinating about this study is that it adds to a growing body of evidence that suggests that AI could have a much different effect on the labor market than previous waves of technological change.  For the last few decades, we’ve seen a pattern that economists have called “skill-biased technological change.” The basic idea is that so-called “high-skill” office workers have disproportionately benefited from the use of computers and the internet. Things like Microsoft Word and Excel, Google, and so on have made office workers and other high-paid professionals much better at their jobs.  Meanwhile, however, so-called “low-skill” workers, who often work in the service industry, have not benefited as much from new technology. Even worse, this body of research finds, new technology killed many “middle-skill” jobs that once offered non-college-educated workers a shot at upward mobility and a comfortable living in the middle class. In this previous technological era, the jobs that were automated away were those that focused on doing repetitive, “routine” tasks. Tasks that you could provide a machine with explicit, step-by-step instructions how to do. It turned out that, even before AI, computer software was capable of doing a lot of secretarial work, data entry, bookkeeping, and other clerical tasks. And robots, meanwhile, were able to do many tasks in factories. This killed lots of middle class jobs.  The MIT economist David Autor has long studied this phenomenon. He calls it “job polarization” and a “hollowing out” of the middle class. Basically, the data suggests that the last few decades of technological change was a major contributor to increasing inequality. Technology has mostly boosted the incomes of college-educated and skilled workers while doing little for — and perhaps even hurting — the incomes of non-college-educated and low-skilled workers. Upside Downside But, what’s interesting is, as Brynjolfsson notes, this new wave of technological change looks like it could be pretty different. You can see it in his new study. Instead of experienced and skilled workers benefiting mostly from AI technology, it’s the opposite. It’s the less experienced and less skilled workers who benefit the most. In this customer support center, AI improved the know-how and intelligence of those who were new at the job and those who were lower performers. It suggests that AI could benefit those who were left behind in the previous technological era.  “And that might be helpful in terms of closing some of the inequality that previous technologies actually helped amplify,” Brynjolfsson says. So one benefit of intelligence machines is — maybe — they will improve the know-how and smarts of low performers, thereby reducing inequality.  But — and Brynjolfsson seemed a bit skeptical about this — it’s also possible that AI could lower the premium on being experienced, smart, or knowledgeable. If anybody off the street can now come in and — augmented by a machine — start doing work at a higher level, maybe the specialized skills and intelligence of people who were previously in the upper echelon become less valuable. So, yeah, AI could reduce inequality by bringing the bottom up. But it could also reduce inequality by bringing the top and middle down, essentially de-skilling a whole range of occupations, making them easier for anyone to do and thus lowering their wage premium.  Of course, it’s also possible that AI could end up increasing inequality even more. For one, it could make the Big AI companies, which own these powerful new systems, wildly rich. It could also empower business owners to replace more and more workers with intelligent machines. And it could kill jobs for all but the best of the best in various industries, who keep their jobs because maybe they’re superstars or because maybe they have seniority. Then, with AI, these workers could become much more productive, and so their industries might need fewer of these types of jobs than before. The effects of AI, of course, are still very much being studied — and these systems are evolving fast — so this is all just speculation. But it does look like AI may have different effects than previous technologies, especially because machines are now more capable of doing “non-routine” tasks. Previously, as stated, it was only “routine” tasks that proved to be automatable. But, now, with AI, you don’t have to program machines with specific instructions. They are much more capable of figuring out things on the fly. And this machine intelligence could upend much of the previous thinking on which kinds of jobs will be affected by automation.  Next week, in the Planet Money newsletter, we speak with MIT’s David Autor, who pioneered much of the economic thinking about technological change, automation, inequality, and upward mobility in the past few decades. What’s he thinking now? Stay tuned! Not subscribed? Subscribe to this newsletter. Want to send this to others? Share the web-version of this newsletter on social media. Want more Planet Money? Listen to our podcasts. Newsletter continues after sponsor message Get Bonus Content Subscribe to Planet Money+ for bonus episodes with behind the scenes takes, extended interviews, and extra facts we couldn’t fit into the main show. Plus, it’s ad free. You’ll get The Indicator and Planet Money Summer School too - all while supporting our nerdy, ambitious journalism. Learn more at Plus.npr.org/PlanetMoney.  On Our Podcasts Why it's so hard to mass produce houses in factories — Imagine if we built cars the same way we build houses. First, a typical buyer would meet with the car designer, and tell them what kind of car they want. Then the designer would draw up plans for the car. Listen here The hidden history of race and the tax code — This past January, researchers uncovered that Black taxpayers are three to five times as likely to be audited as everyone else. One likely reason for this is that the IRS disproportionately audits lower-income earners who claim a tax benefit called the earned income tax credit. And this, says law professor Dorthy Brown, is just one example of the many ways that race is woven through our tax system, its history, and its enforcement. Listen here The dark side of the influencer industry — When your lifestyle is your brand, the line between work and life can start to get blurred. From hate comments and sneaky contracts to prejudice and discrimination, it's not all sunshine and brand deals in the influencer industry. The Indicator has the story. Listen here Also on The Indicator: Gen Z's dream job in the influencer industry; The economics of the influencer industry; and The origins of the influencer industry Listen to your local NPR station. Visit NPR.org to hear live radio from WBEZ (edit station). Subscribe to Planet Money+. Your support helps make our show possible and unlocks access to our sponsor-free episodes. What do you think of today's email? We'd love to hear your thoughts, questions and feedback: planetmoney@npr.org Enjoying this newsletter? Forward to a friend! They can sign up here. Looking for more great content? Check out all of our newsletter offerings — including Daily News, Politics, Health and more! You received this message because you're subscribed to Planet Money emails. This email was sent by National Public Radio, Inc., 1111 North Capitol Street NE, Washington, DC 20002 Unsubscribe  |  Privacy Policy