Episode 8: Deinstitutionalization

Dr. Thomas Scherer and Dan Spokojny discuss their article, Foreign Policy Should be Evidence-Based.

Transcript

Written robotically. Errors are likely.

Alex Bollfrass
The fp21 founder and CEO Dan Spokojny is back this week. He is talking with Thomas Scherer, who is the Deputy Director of the Center for Peace and Security Studies at the University of California, San Diego. They are discussing a recent article they co-wrote for War on the Rocks. The link will be in the show notes. And the title is "foreign policy should be evidence based." You will first hear Dan read a favorite selection from the article. After that Thomas also reads a piece. Then they go on to discuss what prompted them to read the article. What kind of open questions still remain in their minds on the topic. And finally they begin to address some of the reactions that they've received during the writing. And after the article’s publication. This is part one of that conversation. The second part will follow next week.

Dan Spokojny
in Ascendant China, revanchist Russia, the failure of the wars in Iraq and Afghanistan to achieve US objectives. Climate change the threat of nuclear proliferation, rising authoritarianism. The challenges to US influence on the world stage have become so numerous serious and complex that some experts see the unraveling of American power. Faced with this perilous strategic landscape, some are calling for re-examination of the way in which foreign policy is conducted. These analyses from some of the nation's most capable diplomats are valuable, but the recommendations do not go far enough, revitalizing US foreign policy will require more than a renewed commitment to diplomacy. Instead, we believe that foreign policymakers should embrace the evidence-based policy movement, the nation's ability to persevere through today's International problems may depend on it.

Thomas Scherer
When we advocate for evidence-based policymaking, we're not talking about changes to individual policies, but rather changes to the policy process itself. Features of an evidence-based policymaking process include an emphasis on fact-finding citations of the available evidence for various options, metrics of success, in all policy memos, an emphasis on transparency of decision making, and an emphasis on learning from successes and failures. But we should be clear, not all evidence is created equal. Arguments based on anecdotes, analogies, and simple descriptive statistics can be highly misleading or flat-out wrong. Too often, policymakers rush from the discovery of a few facts to forming a confidently held conclusion, or worse yet, start with the conclusion and then go hunting for supportive evidence. This article started for me seeing the memos coming out from the Biden administration, and learning about the evidence-based policymaking act of 2018, which has some great stuff, but then digging a little deeper, and finding out the timelines of that act. And what was actually required were even that we haven't started to see what should be the fruits of its labor. Maybe everything's been fixed. But it's the kind of requirements laid out in that law from four years ago, are just now coming to their due date. So that was interesting to find out that just when we think we see some movement, some progress, that we don't actually know, we don't have any good information. Some people look at these two memos and say, Look, we're taking care of it. We're on it. And for me, I read these memos as the buyer and administration trying to address some issues that they've seen recently. So the memorandum on revitalizing the foreign policy and national security workforce seem to be a memo telling the foreign policy community. Look, we have your back. We just saw a lot of people in foreign policy being punished or some well-liked of respected diplomats being sidelined for speaking what they saw as truth and in the nation's interest. So That first memo seemed to be addressing that. And the other memo about restoring trust around scientific integrity in government, it seemed to be more of a response to issues in the Trump administration, around hiding data, moving data drawing on maps about where a hurricane Mike go. It's both are well and good. But what we see time and time again in the foreign policy community is things around scientific integrity, they'll say we're not science, maybe we're an art, we don't have this kind of data, this memo on scientific integrity doesn't really pertain to us. So both of these memos are great. And one could read them generously and say there's a space between them that is addressing some of these issues that we're talking about. But my sense is, unless things are very explicit, to say, no foreign policy community, this scientific integrity and evidence-based policymaking applies to you too.

Dan Spokojny
As we say, in the paper, there's been a beachhead established within the foreign policy community for evidence-based policymaking. And some really great things are happening within the administration within the bureaucracy, the Center for Analytics, for instance, is growing rapidly and taking on a lot of resources and trying to introduce more data and evidence into the policy process into the management process at the Department of State, similar things are happening across the bureaucracy. But that starting point isn't quite what we're talking about for a bigger vision of evidence-based policymaking. That's going to require not just saying, okay, here's the evidence now make decisions, but requires new thinking at each different stage, from the collection of information from how we manage that knowledge from how we analyze that information, how we make decisions, and then how we evaluate those decisions to learn and evaluate and monitor implementation. To understand next time, what worked here and what didn't, and to feed that knowledge back into the system.

What did you learn from this process of writing this article?

Thomas Scherer
I appreciated some of the comments we got early on asking us for more evidence on why an evidence-based process would lead to better outcomes, why wouldn't more evidence be useful, especially if you have a little thought experiment, you can take it at the extremes? If you knew everything about the world, you could have a better prediction of what outcomes and action would lead to and be able to choose the outcome appropriately versus if you know nothing about the world, your decision making is pretty much a coin flip that so how could it be that moving further along in that direction of knowing more about the world could be objectionable or something that's not obvious. And yet, doing research in this space is very difficult. There isn't a lot of at least especially in the foreign policy area, not a lot of academic work. At this level. There's some good academic work on decision-making processes and the importance of organizational bureaucracy. And we cite some literature from other fields, the business world is a bit easier, because you can boil everything down to dollars. But it's still striking how scant the evidence is for the arguments we're making. Why is that the case? If you take a policy and an outcome, and try to evaluate what's the success or failure, and what's evidence used, we don't really know what the ultimate goal was of that process, we don't ultimately get to look into the head of the decision-maker, and what they were ranking as their preferences for outcomes. And then we also don't get to look inside their head and see how much the information that was coming into them inform their decision. We also can't do some experiment where give them a piece of evidence or give them a different piece of evidence and see the different outcomes. So there's a lot of steps to the process that make it hard to apply the scientific method, which we're advocating for, to this very case, now the research group, but Fp 21, of refocusing our efforts on understanding these cases of success and failure in cases of where evidence made a difference or lack of evidence seem to have made a difference in a clear way. And to do this all in a non-partisan way. It's tough.

Dan Spokojny
One of the challenges that we received when we were writing this from the reviewers was generally this is hard to dispute that more evidence is better. But can you please give me more historical examples of evidence-based policy processes working, or where an evidence-based policy process might have worked? what you're talking about Thomas is that it's really hard to look in from the outside of a policy process. And even when you are on the inside and say, here's the explicit goal we were trying to achieve with our policy. So if we don't know what the goal is, how do we even evaluate what success or failure is? Next? If we know what the goal is, how do we know what the right set of things were that caused that success? Was it our policy, our beautiful strategy? Was it just luck? Was it something else and gathering evidence as academics want to do on those questions on the evidence For evidence based policymaking is really challenging. It's something I think we've had some really productive conversations on and hope to write more about in the future.

Thomas Scherer
Should we get into some of the reactions and responses we've seen since the pieces come out?

Dan Spokojny
I just loved the engagement we got and I think some of these questions are questions that we ourselves grapple with daily in this work, whether we're working within Fp 21, or in our roles as policymakers or or researchers who care about the impact of our policies. Somebody said, I would feel better about this article, if more of our politics in general, we're fact base, we need to fix that before we can hope to fix US foreign policy. Another person says this overlooks that so many decisions are often shaped by domestic political considerations.

Thomas Scherer
How do you take that point? First off, that these aren't mutually exclusive politics and domestic considerations are always going to play a role in foreign policy decision making, then I think that you would have to make the jump decision-makers are only making decisions based on domestic political considerations to have this idea that improving the processes around evidence wouldn't have any effect. And that that doesn't seem right. Even if a decision-maker is very concerned about political considerations, again, as they should be, then they would want the best evidence possible in order to achieve the outcome that they think is best for them. Given those political considerations. I don't see them mutually exclusive at all. I only see these in working together. The other kind of top-line reactions I would have is it's always interesting to me that this idea that politics stops at the water's edge to steal a famous line, the idea that foreign politics aren't really that important to the average voter, or at least the day to day stuff, I would just question of how much of the 1000s of decisions being made over a year, how heavily do domestic political considerations weigh? at any level? I'm not really sure, or at least in academia, we think that voters don't really vote on foreign policy issues. All of that's moot, because as we say, domestic considered political considerations should be part of a decision-making process. But I just also wonder about the premise that how much do we think domestic political considerations are making into these other levels of US foreign policymaking?

Dan Spokojny
I have two quick responses here. First, evidence can help shape the scope of possible decisions that a policymaker could take that science provides something closer to objective truth of, here's what works, and here's what doesn't work, or here's what's likely to occur if we do X, Y, or Z or do nothing. And so it helps shape the issue set at hand for a policymaker. And I think that's really vital. And then, of course, democracy and the opinions of citizens and the judgment and politics of our legislators and policymakers should play a role here. But their ideas are constrained by some conception of what's going to work and what's not going to work. The closer those two worlds are brought together, the evidence and the politics, the better, the healthier, our policymaking process is going to be. And that leads me to my second point, which is that I think that we've witnessed the consequences of what I call the deinstitutionalization of our professional diplomatic bureaucracy. The American Academy of diplomacy has been screaming about this since 2014. Since 2015, the rising rates of political appointees within the institution of the State Department, political appointees, people who are loyal to the party, not the bureaucracy, not the merit of the issues. We've seen rising rates of politicization of the bureaucracy for a long time. political appointees, ambassadors is one piece of that, but it's a broader problem. And what this leads to is a bureaucracy, which has a harder and harder time distinguishing between what is politics, and what is the best judgment of the bureaucracy. And so I think that I think that's one of the things that that evidence can help fix here, and we shouldn't be afraid of politics, but we should be able to understand this is what politics is. The problem with the deinstitutionalization of our professional bureaucracy of increasing politicization is it removes the guardrails on the political process that a merit driven professional bureaucracy can offer. We saw this play out a little bit during Coronavirus in the epidemiological field that there was a push back between scientists who were saying, Hey, wait a second, we really know how infectious disease works here. And you can't just erase our scientific knowledge now how we deal with that knowledge within a messy body politic of course is always going to be contested as it should be, but that the professional bureaucracy that the see See the NIH were able to bring a lot of political authority to the table, because of their position within the bureaucracy because of their protected position within the political bureaucracy

Thomas Scherer
We're assuming some level of democracy, in essence, where voters or more likely their congressional representatives have some oversight into US foreign policy processes. And if there's questions about a certain decision, they can go back and look into what decision was made and why and ask some questions about it. And we can get into some problems, if someone could make a decision based entirely on some personal goal and then declare to the world that it was based on the best evidence possible and justify their decision in that way. And no one can know. And we'll just have to believe him or her and take him or her at their word that can lead to some problems. And that's, I think that kind of process is exactly what I was trying to work against that, for this all to work, the process has to be more transparent. There has to be some kind of accountability and oversight and transparency.

Dan Spokojny
The process today is the thing that many folks are worried about, which is that you have policymakers show up and say, trust me, this is the right decision, we're going into Iraq, or we're gonna go into Syria, or we're going to back away from this conversation. because trust me, that's highly classified. And I'm an expert, and I'm an expert, because I'm in the seat, and I'm in the seat because I'm an expert. It's this very circular logic, if you can't look at the evidence and engage with the engagement, the theories and the reasoning, the logic that they use, at its worst, that's what foreign policy sometimes resembles. This sets up the next category of criticisms or skepticism, reframed our article is evidence. And it's really curious why so many people say, Yeah, but data can't do this, or data can't do that. So there's a first-order question. Why do you think that people think about, or have this reaction where when you say evidence, they jump to data. And to be fair, we do talk a little bit about data in our report.

Thomas Scherer
My sense is that people feel like they've gotten burned recently, by data, one of the responses we got was specifically pointing to coin or counterinsurgency in Afghanistan. So the question was, so main metrics for coin in Afghanistan, what did that get us? You have this very prominent case in US foreign policy, where there seemed to have been a lot of data. And there seem to have been a failure. It's pretty easy, then to jump that data, obviously, isn't the cure, or perhaps that is even making things worse. And

Dan Spokojny
Most of us have read in books about Vietnam. And the way McNamara tried to quantify success in these very messy ways in Vietnam, according to body counts and number of villages destroyed. And the evidence was ridiculous in those circumstances, and certainly didn't lead to any agreed upon measure of success in Vietnam. And some people feel something similar happened in Afghanistan, or in Iraq, or other places where we started seeing the use of data and counterinsurgency operations just because data exists, and we fell short in these wars doesn't mean data caused failure. Part of it also is about this attitude amongst many scientists, sometimes scientists come in the room and say, Listen, I know all the answers, because I've read the literature on this. And that paints a big target on academics when we're wrong, or when our promises fall short of fixing the challenges we attend to.

Thomas Scherer
I have a lot of concerns about the use of data in my field and political science. We've seen recently some crises around scientific evidence and data, the replication crisis through a couple of different fields, including fields where we think they actually have things that they can measure Well, in economics and medicine, we still see a number of issues around the use of data, and the pitfalls that you can get into you and I both know data is not a panacea. Even regardless of the quality of data you're able to collect and the evidence you have a level of humbleness of caution as important. This gets back to the importance of training, of being able to read and respect and understand data and its limitations.

Alex Bollfrass
If you haven't had a chance to open this week's newsletter, you will find it in your inbox under failing to evolve is a recipe for mediocrity. It starts off by wishing the Department of State a happy belated 232nd birthday, and also includes an article suggesting that the institution could use a fundamental renovation. One of the headline numbers of the study, written by a foreign service officer but published at the Georgetown Institute for the Study of diplomacy suggests that about a third Foreign Service officers are planning on quitting the State Department. Then there are some additional articles in there about recent activity in Congress that affect the executive branch's ability to plan and conduct foreign policy. So it was a good read. If you haven't subscribed yet, I'd encourage you to do so. And I look forward to continuing with part two of the conversation between Dan and Thomas share next week.