The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale
Leo Feler, Senior Economist, UCLA Anderson Forecast
John List, Professor of Economics at the University of Chicago and Chief Economist at Walmart
Professor of Economics at the University of Chicago and Chief Economist at Walmart
This month, our podcast features a conversation with John List, Professor of Economics at the University of Chicago and Chief Economist at Walmart. We discuss his book, “The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale”.
Leo Feler: John, thanks so much for joining us for this edition of the UCLA Anderson Forecast Direct.
You are this person that when I think of the scale at which you operate – you’re a professor, you publish papers, you served as the Chief Economist of Uber and Lyft, you’re now the Chief Economist of Walmart – you’re involved in so many different projects and great ideas. We’re going to talk mostly about your book, The Voltage Effect, but we also want to bring in your experience running experiments in the various places and organizations where you’ve worked.
When it comes to developing and scaling ideas, you’ve helped startups scale up. Now in your sandbox you have the largest retailer in the US and one of the largest private sector employers in the US.
Let’s start with figuring out if an idea can scale. What do you normally think about when you’re trying to figure out, “Is this a good idea? Can I scale this?”
John List: Five years ago, I thought differently about scale than I do now. Five years ago, I largely thought, “well, the idea worked, and it seems to have really good signal.” And what I mean by signal, is it had a big treatment effect. And, it seemed to be with a general group of people, so I think it will scale. The last five or so years, I’ve been working on the science of scaling. What that means is when you look at an idea, what are the features of that idea that will scale? And what are the features of that idea that will cause it not to scale? Today, I look at an idea and say, there are five fundamental features – which in the book I call Five Vital Signs – and if you check all of these boxes, this is an idea that has a chance to scale.
You still have to execute. So, the first half of the book is about the features of an idea. And the second half of the book is about what I would call four behavioral economics secrets to execution.
But before we get there, let’s talk about the five features. That’s the way I would think about any idea – does it have these five vital signs?
Leo Feler: The five vital signs. First is “false positives”. Second is “know your audience.” Third is “the chef or the ingredients” – and I’ll pause there for a second, because this was one of my favorites. I thought to myself, John List’s projects all seem to scale. Is it because of John List or is it something about his projects? Fourth is “spillovers,” and then fifth is “cost traps.” Those are the five. Let’s go through them.
John List: Think about a Covid test. If it says you’re positive for Covid, it might be wrong. When it’s wrong, that’s called a false positive. As researchers, we try to control the false positive rate to be five percent. That’s the best-case scenario – that five percent of the time, when we say a program works, it doesn’t actually work. Even in the best-case scenario, one in twenty ideas that look great in the petri dish, aren’t so great when you scale them. That’s how I want you to think about false positives. I think in the real world, false positives are much, much greater than five percent, because of things like confirmation bias. If we really think the idea is going to work, and we get data that says it’s not so good, we tend to say the data are wrong. If we get data that say it’s good, we say look at the data! It’s in line with what I want. That’s called confirmation bias. A lot of times as humans, we have these biases that really lead us down a bad path of taking on more false positives. So that’s Chapter One, False Positives.
Chapter Two is really know your market. Know the extent of market and be honest about determining with a current product and service, how big of a market do I actually have? Most of the time we overestimate the extent of market or the market size.
Chapter Three, which you brought up, is one of my favorite chapters, because it’s titled “Is it the Chef or Is it the Ingredients?” It moves from Chapter Two, which is really about the population of people, to talk about the population of situations. It really asks, when you had your idea and you tested it out, what were the fundamental features, or the non-negotiables of that idea, and can you have the same inputs in those non-negotiables that you had in the petri dish – are those available at scale? Think about “is it the chef or is it the ingredients” – a lot of restaurants try to scale and a lot of restaurants fail. Every time they try to scale, but the initial success was because of the chef, it won’t scale. Why? Because unique humans don’t scale. Now at this point, you might be thinking “Wait a second, John – you’ve scaled your operations reasonably well. What’s going on here? You’re telling people unique humans don’t scale.” That’s right.
Think about Uber and Lyft. As you mentioned, I was a Chief Economist at both Uber and Lyft. If we needed drivers that drive like Danica Patrick or Jeff Gordon or Al Unser Jr. or Michael Schumacher, that’s not going to scale, right? Because those types of drivers are unique, and there’s no way we can hire enough of them at scale. But, we can hire drivers like me who will scale.
It gives you a sense about how, in my own operation, I set up teams. I make sure what I tell my teams to do in the different pods – I have my Walmart pod, I have my early education pod (I started a preschool in Chicago Heights), I have my University of Chicago pod – I say, “here is what your job is, here is what I’d like you to do.” Any task that takes my efforts, I don’t give to anyone else. Because what I can offer, my comparative advantage, won’t be scalable. So, it’s an understanding that you have to be able to delegate, but you have to delegate the features that the unique human isn’t responsible for.
Leo Feler: I think one of the reasons you’re able to scale is that you teach. This is a form of scaling, because you write one book, and millions of people can read this book and learn those ideas. You teach a class, your students learn the skillset you’ve taught them, they also go on and teach others. That’s a different way of scaling, too, which is this knowledge-sharing you’re providing to your audience.
John List: One hundred percent. Sometimes unique humans can teach what is unique about their knowledge. In this case, I can teach you, these are the five vital signs, this is what you can do for execution. That will work in general, but in many cases there’s a twist of one of those that you need to apply to make sure you get the highest voltage possible, and that’s nearly impossible to teach, because the unique difference comes at you in so many different ways.
Sure, I can teach you about principles of economics. I can teach you about the science of scaling, but to actually apply it will take a tweak of what you learn from the book. A lot of the time, what you learn from the book will get you 90% of the way there. But to get the last 10%, you might need an extra bit of knowledge or an understanding about how to generalize a principle. That’s what I mean by needing that extra. For example, in many cases in a restaurant, a dish, whether it’s good or bad, is almost a knife-edge case. Those knife-edge cases really need that unique individual to make a difference. Economics is different in the sense that if I can get you to the neighborhood of good results, then you’re generally okay.
Now, vital sign number four is that every idea will have a spillover or a set of spillovers. In the book, I talk about what happened with Uber when we scaled tipping. In 2017, my team rolled out tipping in the Uber app. Before that, there was no tipping in the app. My team did a series of large-scale field experiments around tipping. When, for example, only a small group of drivers get tips in a market – say 5-10%, versus when they all receive tips, the outcomes are very different. If only 5% of drivers are able to receive tips, they earn more money and they work more. But if all of the drivers are able to receive tips, you have a bunch more people who now choose to work more, but that depresses the wages. It doesn’t really matter if you’re receiving tips, you receive the same amount of money as before because it generates a new market equilibrium. So, that’s an idea that doesn’t scale well because of market spillovers.
And then the fifth one is what I call the supply side of scaling, and that’s whether your ideas have economies of scale.
Leo Feler: A lot of the results of scaling seem to depend on particular circumstances, not only of the audience but of the person doing the scaling and of the circumstances in which you are doing the scaling.
Really fascinating for us to think about not only how ideas can scale, but maybe it scales for a particular group or particular audience, and the idea has to change in order for it to be scalable for different groups of people.
John List: That’s a keen insight. Right now, in the social sciences, we are in the heterogeneity revolution. This revolution is saying that one size really doesn’t fit all. There might be a group of people where you need one kind of product and one group of people where you need a slightly different product. People are different. This is also true with public policies, early childhood programs, etc.
Just as important, if not more important, is heterogeneity of situations. There are fundamental features, whether it’s a teacher teaching program, or a home visitor in an early childhood program, or features of the product, or any general feature of the situation – what I’ve found in my own work is those elements are even more important than the differences between people.
So, to back up, what I’m telling you is that while we constantly worry about the differences between populations of people, that is much less of an issue in scaling than when you compare it to the populations of situations. It’s much more difficult to generalize across situations than it is across people. That’s an important point to always understand, as someone who’s trying to scale.
Leo Feler: Which comes to the point that you always have to adapt, right? It’s kind of like a shark. You die if you stay still. You have to make sure your product keeps changing to suit the changing economic situation.
John List: A hundred percent. But, when you say you constantly have to adapt, that needs to be based on science. You constantly have to generate and test data and look at new product configurations to figure out, how do I need to change? So, you’re right to say, “I constantly have to adapt,” but what does that mean? What it means is to constantly test and re-test and figure out all the time, “do I need a different algorithm,” whether it’s a pricing algorithm, or different product in terms of product differentiation, or a different product mix.
If you’re a retailer like Walmart, there are a lot of different questions that if you stay stagnant, you’re dead. The best-case scenario is that you’re constantly testing and re-testing and saying, “how can I do a little bit better?” Because you have to constantly be thinking of opportunity cost of time. Every minute that goes by that you are not providing the right product mix, or you’re not pricing correctly, that’s an extra minute that you’re losing money or you’re losing the chance to make the world a better place. If we constantly think about this as an opportunity cost of time, then I think you have a real urgency to continually generate data, analyze data, and make decisions from data.
Leo Feler: We’re shifting into the second part of your book, which is once you have a great idea that can scale, how do you scale it? This is what you call “high voltage scaling.”
John List: You need to be able to execute to make your business work or to make your organization work. Why am I writing the second half of this book, which is all about execution? Because, here’s what I’ve learned. I’ve been around a long time. I’ve worked in the White House, I’ve worked in a lot of different firms. All organizations tend to make the same four mistakes when it comes to executing.
The first mistake is they don’t appreciate the breadth and depth of how they can use incentives. That’s Chapter Six. It starts out by talking about non-financial incentives. At Uber, when we did the Uber tipping experiments, it’s kind of crazy the data around how many people tip. Only 1% of people tip on every trip. Three out of five people never tip. You heard that right. Three out of five people in our data never, ever tip. If I take those three out of five people who never tip on Uber, and I look at their tipping for a yellow taxi cab, where at the end of the trip, we settle up face to face, 90% of them will tip. You can say, “what’s going on, it’s the same people?” The situation has changed. In one case with Uber, you’re deciding to tip after the trip, when you’re home, you’re in your office, but you’re not face to face with the driver. The social pressure, the social norm, your self-image, social image – these are all different than when you’re doing it face to face, and that causes you to act in a very different way. The general lesson here is that these types of incentives that are non-financial in nature can be very powerful, and they’re very powerful in terms of scaling. I also talk about ways to use financial incentives, like claw-backs. I did this with a teaching experiment, I do this with a journal that I run right now. The general idea is give the bonus up-front, tell people what they have to do to keep the bonus, and if they don’t perform, you take the bonus back. It’s called clawing-back, it’s really effective, and it leverages loss aversion.
Leo Feler: This is that famous paper that talks about gifting someone a mug, and then asking them how much money they would need to receive in order to give the mug back. It’s always a higher number than what they’d be willing to pay to buy that mug to begin with. But because now they have that mug, and they have somehow developed an attachment to it, they don’t want to give it up. That’s loss aversion.
John List: Exactly. I’ve taken some of these insights, I’ve done large-scale field experiments with them, and I’ve found that leveraging people’s loss aversion can be an effective incentive. That’s Chapter Six.
Chapter Seven is Thinking on the Margins. The idea here is we’re programmed to think in averages. Any time you see someone present an academic paper or statistics, they always give you averages. Sometimes they say, “I have a very large group, so look at my average, how precise it is.” That’s the wrong way to think about decision making. I advocate for taking that dataset and slicing it up into thinner slices or thinner pieces, and then try to make decisions on the marginal piece rather than the average piece. The example there is an example from the White House, on how we should spend public dollars. But, I also talk about examples I observed at Lyft and Uber, where people are constantly thinking in terms of averages when they really should be thinking about marginal effects. That’s a typical mistake people make.
The next chapter is about quitting, and that really gets to the opportunity cost of time. I explain in this chapter that people don’t quit enough, or, in the business world, that people don’t pivot enough. People tend not to quit because society tells them it’s repugnant – but that’s society’s problem. The other reason people tend not to quit is that we neglect our opportunity cost of time. I did a survey of recent people who quit their jobs. I asked them, “give me the reasons why you quit.” Reason number one: I lost meaning of work. Reason number two: I didn’t get the promotion I thought I deserved. Reason number three: I didn’t get the pay raise I thought I deserved. All the way to reason number ten: I didn’t like my cubicle. Every reason was: my current job got soiled. Nobody said: my opportunity set got better. I just couldn’t afford to stay the chief economist at Lyft, because my opportunity set got better, and I moved to Walmart. We’re just not programmed to think that way. This is science as well; it’s not art. In this chapter I talk about our science, and I show how and why people should quit more. Whether it’s a relationship, an apartment, a job, let’s say an entrepreneur, we just don’t pivot or quit enough.
Leo Feler: Let’s dig in some more on quitting, because now we’re seeing this in a really interesting way in the economy. People who quit their jobs in this past year have done better than people who remained in their jobs. I wanted to get your thoughts on this.
John List: I put together an experiment where we had people sign up who were on the margin of quitting something – maybe a relationship, maybe a job, maybe they just wanted to move cities. And they just weren’t sure, they were grueling over what to do.
We had them flip a coin. And if it came up one way, we said they should change. And if it didn’t, we said don’t change. For example, you flip a head, you should change; if you flip a tail, you don’t change. We told them, you need to execute that. We couldn’t force them to, but the majority did. And what we find is 6, 9, or 12 months later, they’re happier. They’re happier they made the change.
I don’t know what’s going on in the data you’re talking about, Leo, because in that case, job quitting wasn’t random. There’s selection bias. Those are people selecting to quit. In my example, I’m randomly having people quit and comparing them to the random group that did not quit. So, I can say something hopefully a little stronger about causality. What I can say is there’s evidence that people don’t quit enough – more people who are on the margin of quitting should be quitting. If you’ve thought about it for that long, you should quit. That would be the advice I’d give to people.
It feels like the data you’re talking about are at least consistent with what I’m talking about in terms of those people who did make a change during the Covid era are now better off, because they’re happier. That feels consistent with – it’s not the same evidence because it’s not random selection, and you should expect that to be a stronger treatment effect than what I have – but they both point in the same direction, that once a person who’s grueling over a decision decides to make a change, not all the time, but on average, they’re going to be happier.
Leo Feler: Let’s talk about the last chapter, which is Scaling Culture.
John List: This was a fun chapter. I’ve done work in the gender pay gap for 20-25 years. I’ve done work in the area of discrimination. I’ve done field experiments for 20 years. I’ve done a lot of work on the science of diversity and inclusiveness. These are all scientific topics that when you take them whole-cloth and start thinking about them as a large, important entity, it rolls up into culture. It rolls up into what kind of culture do you want your organization to have and what are the features of that culture that you can control, from the very beginning of when you’re building an organization. And I talk about simple things like in a job advertisement, saying something like “wages are negotiable.” That actually matters a lot. Because when you say wages are negotiable, what we find is that women will negotiate more than when you leave that sentence out of the job ad. And when women negotiate more, they come in with more equal pay compared to men. One of the reasons we have a gender pay gap is because men negotiate a lot more than women, on average. So, if we can set up simple things like designing a job ad or designing the workplace in a way that makes it more inclusive and fair, we should do that. We have science around these issues, the science of building a culture, the science of building an equitable workplace.
Leo Feler: Last question to wrap things up: what do you envision at Walmart? What are your big ambitions about how to run experiments and scale ideas at Walmart?
John List: So, you might be thinking, “John’s crazy. He’s worked with startups, and very nimble startups, and now he’s going to a Fortune One company. So, what in the world is John doing?” Here is what I’m doing. In the end, I want to first of all do science. I want to use science to help change the world for the better. Walmart gives you the biggest sandbox in the world to work on labor issues, to work on customer side issues, to work on sustainability. I have a paper with coauthors, from a few years ago, about how we saved millions of gallons of fuel for Virgin Atlantic Airways by using three behavioral economic insights. Once we showed them, here’s how it works, now we’ve saved millions and millions of gallons of fuel. When I saw that result, a small tweak (let’s call it marginal thinking) that leads to huge impact, I started to think about how Walmart has that example all over the place. Supply chain, customer side, delivery – Walmart is now the second largest e-commerce site in the US, it touches 150 million consumers every week. So, you have the real chance that with behavioral insights, simple economic insights, if you can make even small changes, it reverberates throughout the entire economy. You have a real chance to make significant change. Then I started to think about, there are a lot of new products Walmart is trying out, there are a lot of stores. A big point that I make in the book is that to scale quickly, you want to do multi-site trials. Well, viola! You have that in spades at Walmart because you have 4,500 stores. You can quickly assess how different situations matter. Does something work in Laramie, Wyoming the same way it works in New Jersey and the same way it works in Mississippi? Maybe, maybe not. Maybe the situational features or the people cause it to be different. You learn something fundamental in terms of how the economic science applies in these areas, and you can also help the firm, and help customers of that firm. All of those together led me to say that the opportunity cost of staying at Lyft was too high, so I’m going to quit, I’m going to pivot, and I’m going to go to Walmart. I followed my own advice.
Leo Feler: I’m excited to see what comes next in the world of economic experiments with Walmart’s data, suppliers, and customers. John, thank you so much, we really appreciate the time, and I loved reading your book.
John List: Leo, thanks so much for having me, and you made my day. Anyone who reads the book and likes something in it, it warms my heart. So, thank you so much.
Leo Feler: One of the things you’ve taught me is that I should quit more often. Because I’m definitely one of those people who hangs on to their passions long after their opportunity cost has told them they should give them up.
John List: Every passion you hang on to is one less passion that you can take on. Opportunity cost is important!