How to Fix the State Department
By: Santi Ruiz and Dan Spokojny | March 27, 2024
This interview was conducted by and originally published on Statecraft.
From Statecraft, by Santi Ruiz:
This month, Statecraft has focused on America’s attempts to build nations in Iraq and Afghanistan. We think these stories are a great opportunity to get a closer look at what we mean by “state capacity.” Massive stateside bureaucracies attempted to build new, stable institutions in foreign countries. How did those bureaucracies perform? How about the institutions we built abroad?
In our final installment, we wanted to zoom out, and look at the State Department as a whole. So we got in touch with Dan Spokojny, a former Foreign Service officer and diplomat. Spokojny is also the founder and CEO of fp21, a nonprofit, nonpartisan think tank that advocates for more data and evidence in foreign policy decision-making.
In today’s interview, we had a productive argument about exactly that topic.
Santi: Presidential administrations lean pretty heavily on the National Security Council (NSC) as a vehicle for foreign policymaking, instead of on the State Department. There's often more tension with State than there is with the National Security Council.
Help me understand why that is.
Dan: Starting with a little history might be helpful. The National Security Council didn't exist as a coordinating body until after World War II, when President Truman saw the need for a body that would link the different military services, the State Department, the Department of Treasury, and all instruments of national security policy so they’d pull in the same direction. The NSC, passed as part of the National Security Act of 1947, was born as this coordination mechanism, owned by the president.
Now, fast forward 80 years, and the NSC has grown to being a primary policymaking body in itself. It's still a relatively small organization: between 200-400ish staff over the last couple decades, which isn't so large when you consider the sweep of issues they have to manage from intelligence to defense, to diplomacy, development, economics, etc.
Yet there are some interesting complications here. When you work at the NSC, you serve “at the pleasure of the president.” You're not Senate-confirmed. There's no real congressional oversight. So it becomes much easier for the President and the National Security Advisor to exercise political control and unified direction of the national security issues.
That's great, except you're pushing a lot of decision-making authority to a small group of people, who can't possibly know the issues as well as the executive agencies that they control.
Those executive agencies have congressional oversight. That’s for reasons we could debate, but generally, I think, it’s for good democratic purposes. They’re large, developed bureaucracies with Senate-confirmed heads.
There's a trade off between executive control and competency, and the National Security Council has become this place where a lot of executive control is exercised. Instead of creating interagency procedures where goals can be taken on by different organizations and move in the same direction, currently each organization is its own universe, its own world, its own Cabinet agency with its own control, and they may push in very different directions.
So the National Security Council is like a bandaid on top of this uncoordinated system to try to bring it together at the end. From defense to diplomacy to assistance to economic instruments, these things tend to work in stovepipes and then come together at the very top.
I will want to get back to this relationship between legislative control and executive authority. But first, what is it about how State has worked over the last 60 years that makes presidents want to look elsewhere to implement their foreign policy visions? What about State has made presidents say, “okay, no, we're taking a different route”?
I have to begin by being complimentary of an institution that developed me professionally. The State Department has been integral in shaping U.S. foreign policy. It's the oldest executive agency, it’s been around since almost the founding of this country. It's an extraordinary place that holds extraordinary expertise.
That said, let's pick it apart. Nixon called State “a bunch of eunuchs.” JFK said it's a “bowl of jelly.” Presidents from both sides of the aisle have overtly or implicitly pulled authority away from its diplomats. The State Department is supposed to be the lead author of national security policy, but it hasn't played that role in a very long time.
Now, the reasons for this are complicated. One could say it's about the amount of money and reach and influence that our military has. The Department of Defense is massive, so you're going to trust the institution with more resources and reach at the end of the day.
Same with the intelligence community, which is actually a collection of more than a dozen agencies [how this number is calculated varies]. Across the national security apparatus, you're going to lean on these infrastructures that are more well-resourced than the State Department.
There are some other structural challenges with the Department of State. Creating diplomatic policies to confront controversial, high-profile issues tends to be less politically satisfying than tough rhetoric from the military. A carefully crafted diplomatic intervention is never going to be as sexy as a clear statement of “thou shall not cross our red line.” So the work of the Department of State is less politically cool to talk about for presidents.
And further, there's a history here that dates back to Joseph McCarthy, that the State Department came under a really aggressive political attack, many of its people have been fired. It was accused of losing China because it was too soft on communism. The diplomats, perhaps by actual political preference or temperamentally, lean more towards soft civilian interventions, and they've tended to be seen as a little more left-leaning. That perhaps undermined their ability to play a role in a complicated bipartisan operation.
Lastly, we can go to culture. Diplomats are not known for being loud spoken. They're not known for being clever political actors. The Foreign Service is not terribly known for taking big risks. I think diplomats have failed to fully accept being real foreign policy leaders in a more complicated role.
You're outlining a couple different kinds of issues here, right? One, is State politically unaccountable or riddled with communists? I think there’s a case to be made for both of these.
But in your day-to-day work, you focus on reforming the State Department’s processes. Tell me why we should think process is a top tier priority for foreign policy reform, as opposed to, say, clearing out the communists.
My view is that State should take responsibility for its own shortcomings. The way it's done business hasn't changed for a hundred years. The way that it makes decisions remains, in the words of diplomats, an art and not a science.
Decisions tend to come from the intuitions of our leaders. They tend to come from people who are dedicated, hardworking, and who often have deep cultural expertise, knowledge, language ability, have lived in these countries. But that sort of intuitive, idiosyncratic, ad hoc approach to policymaking is ill-suited for a more complicated century and world. It leads to a great deal of bias in the way that decisions are made and information is taken in.
Just imagine being a foreign policy official at the top of the food chain, trying to manage issues from across the world. The amount of information – news that's breaking, analysis from think tanks, journalists and intelligence and your own people – is absolutely overwhelming.
Without a system that thoughtfully manages that insight, boils it down to clear understandings of how the world is moving, and creates clear doctrine or tradecraft – none of which the State Department has – your official is overwhelmed and has to rely on easily biased heuristics.
That's part of why the State Department has been unable to prove to presidents, Congress, and perhaps even the American people for some decades now that it can take its endowed role of being the leader of foreign policy seriously.
So I've suggested that instead of just demanding more power, the State Department needs to retool, and take on new methods of decision-making and strategy and operationalizing their work.
What you're describing sounds like a cybernetic vision for policymaking. Is that accurate?
That's right on. Yeah, this is like old Herbert Simon academic research on cybernetic approaches to decision-making. We’re not calling for eliminating intuition or creativity, or the human relational aspect of foreign policy, but to create methods within the process of diplomacy to capture what works and to really focus on learning within the world of foreign policy.
Let me give an example of a thing that we can focus on and create more rigor around. We have ceasefire negotiations, mediations, peace processes going on all around the world. There's no central body of knowledge at the Department of State that says, “Hey, here's some case studies of success and failure.” There's no central manual that says, “Here's some best practices,” or a checklist for how you might go about managing this process, or the expertise of the team that you might want to recruit in order to manage this negotiation.
There's no training for how to be a good leader in this space. We let our diplomats – very thoughtful, dedicated expert people – just custom-make their responses to each of these situations.
Let me ask you a question I'm sure you get a lot. Someone might say, “I don't want foreign policy decisions by an elected official to be constrained by an unaccountable, internal, bureaucratic process.
“We elected this president, and he appointed Joe Schmoe. If State bureaucrats are telling him, ‘Don't go to this negotiation, it's not best practice,’ it shouldn't be State's call to make.”
What's your response?
We live in a democracy, and in a democracy, your elected leaders respond to the will of the people. They are representatives, not just conduits of civilian preferences. So you still need some expertise in these bureaucracies to direct the process.
Let me use an analogy. If I'm a bureaucrat at NASA, my job is to engineer engines that work and to build spaceships that fly and won't explode. Now, that doesn't mean that I'm going to override the will of the political leaders of NASA. Those leaders still get to decide, “How much money are we going to spend? What missions do we take on? How do we evaluate risk? Where do we source our materials from?”
But you still have engineers at some point to say, “Listen, I can tell you this engine is going to work and this engine's not. If you want to land on Mars, we're going to have to plot this trajectory.”
I think we can make that same assumption about how foreign policy works. “You tell us what you want to accomplish broadly, Mr. President and Congress. And it's our jobs to say, ‘let me build you a policy that's actually going to work to achieve those goals.’”
“And if you want to change that policy and say, ‘We're no longer going to be at war in this place. We're actually going to shift our resources to another location.’ Fine. We can try to explain to you what we think the implications of the decision are, and we can design a policy that's going to be effective to achieve that pivot.”
That's how bureaucracies work in a healthy way. In the academic literature here, there is a discussion about tradeoffs between competency and control. The more political influence in a bureaucracy you have – politically appointed ambassadors, for example – the more control the president will have.
And control is a good thing. You're elected to shift policy. Let's say we were running the Iraq War for many years, and the people spoke and said, “We want to get out of this war.” And a new political leader comes up and pivots us away from the war. A president wants to be able to control a bureaucracy that often carries a lot of inertia.
On the other hand, the research suggests as you get more and more control, it actually washes out some of the competency that is expressed by that organization. You sacrifice effectiveness. One has to properly balance that.
With this Department of State, there are a high ratio of political leaders, more than just about any other Cabinet agency. Many of our ambassadorships and high-level positions are not led by careerists, but by appointed political leaders, so perhaps the needle is tipped too far in one way.
I'm going to come back to a couple of things you said here, but first, on your NASA analogy: orbital mechanics is a hard science, political science is famously not.
To me, there's a clear weakness in your analogy. Most people think the NASA engineer is more likely to be correct when he says, “Boss, that rocket won't fly,” than when the State bureaucrat says “Boss, that ceasefire won't work.” Tell me why I'm wrong.
Sure. Yes. Clearly orbital mechanics is considered a hard science and political science is considered a soft science. Yet you'll actually find NASA is a great example. NASA had a couple of rockets explode in the last millennia. Those were not, it turned out, engineering problems. The infamous O-ring on the Challenger that exploded was not an engineering failure, really.
The engineers knew there was a weak mechanism there and a high risk of failure due to abnormal weather conditions, and they said, “Hey, please do not launch this rocket.” And the decision makers who weren't taking in that risk properly said, “I've got a launch on schedule. The president's breathing down my neck. It's a perfect day. We've got the film crews ready. Let's go ahead and launch.” So it turned out it wasn't an engineering problem. It was a decision-making problem. It was a problem of not taking the knowledge in to properly calibrate the risk.
Whether it's a hard science or a soft science, one still needs to properly calibrate risk and uncertainty. I think that's where often the State Department fails, is that it's unable to properly calibrate its risk and understand the likely impact of its tools.
If one thinks more scientifically in the craft of foreign policy, it's not about saying this will 100 percent work or 100 percent fail, but, based upon the evidence that we've gathered, we can be more calibrated with our risk and our uncertainty. Rather than just saying, “I know what to do with the evil Russians, we must push back against them.” That sort of gut instinct, normatively-driven policy often makes for great talking points, I think, but it often falls short in the way that we manage our foreign policy.
You've written about how other industries in the private sector, even other parts of the government, are more evidence-based than State. I believe that.
Are there other nations with foreign policy processes that do better evidentiary work? Can we tell that they're better at foreign policy? I believe businesses are generally better run than the Department of State, that seems like a no brainer.
But are there other international success cases for what you're talking about?
I don't have a great answer for you. I haven't done or seen a really thorough analysis. Part of the challenge is, these are really opaque bureaucracies. There's very little rigorous research on the function of foreign affairs bureaucracies in the US or elsewhere. It’s very hard to learn about what really works and to hold the institutions accountable for success and failure.
That said, I think we can point to some interesting processes. The European Union External Action Service said, “We want to prioritize more thoughtfully in the way that we design strategies. And we really want those strategies to be driven from a bottom-up understanding of challenges on the ground, particularly when it comes to conflict.”
So they created this conflict analysis program with a structured set of questions that they sent to each one of their missions in each country around the world, asking, “What do you see as the biggest potential threats? How will those threats unravel?”
These questions were driven by social science understanding of the indicators of instability or potential civil war. They were able to gather rigorous, comparable information from every one of their missions, and aggregate that upwards and say, “Year on year we're able to detect where we think the biggest instability is coming from.”
Do they use that knowledge in a more effective way? I can't say, Santi. But that sort of process really appeals to me. We currently have more of an instinctual response: we’ve got to pay attention to Somalia today and we’ve got to pay attention to Brazil tomorrow.
I'm looking at this paper, “Less Art, More Science,” that you wrote.
That's our manifesto.
You have a case study here, on strategic planning in the Sahel, that seems similar to the EU case you're describing.
That's right. That case study was written by a former colleague who worked in the Bureau of Conflict and Stabilization Operations (CSO). It’s a fairly new structure at State tasked with thinking strategically about conflict prevention, and preventing mass atrocities and genocide.
I learned a great deal from working in this bureau. They bring in a lot of social scientific approaches. They're really, in some ways, the leaders within the department of bringing more strategic thinking to problems.
I think that they were the first State office to build an advanced analytics operation that used more computer-assisted, quantitative, and social scientific methods. Yet CSO is not very well integrated into State.
The way State works is, regional bureaus own the turf: it's the Africa Bureau and the Western Hemisphere Bureau and so on, and those are the power centers of the department. It's where people get their jobs from to go out to embassies. It's where the Foreign Service officers like to live.
The functional bureaus work on cross-cutting issues: Democracy, Rights and Labor, CSO itself and others, they have to ask permission from the regional bureaus to go work into those organizations. Because they don’t own the turf, the functional bureaus tend to have less authority over shaping international policy. It's one of the structural choices that State has made to organize itself: it's still built on a very bilateral model.
So executing foreign policy is country-to-country, rather than transnational. If one wants to make climate change policy, for instance, a priority for the State Department, you need to convince ambassadors in every single relevant country to prioritize this issue, which is a much harder process. Other countries do it differently, and make their cross-cutting issues central.
There's a lot there. Let me go back to the case you mentioned, of a more rigorous analytic process for making strategy in the Sahel.
Some people would say that our entire policy since 2017, after the strategic changes you mentioned, has been a debacle. Mali, Niger, and Burkina Faso are all run by Russian-backed juntas. Violence in northern Nigeria has gotten much worse.
[NB: Since this interview was conducted, Niger revoked its military cooperation agreement with the US, following a “very tense” meeting with a State Department official.]
To me, that suggests it's hard to show these programs are effective.
Yes, I think that's fair. Yes, assessing counterfactuals is a core precept of scientific understanding. It's very complicated in complex environments, which characterizes basically any foreign policy objective. But that doesn't mean that we shouldn't do it.
And it doesn't mean that it's impossible. Right now at State, there's almost zero lessons learned programs. There's almost no “hotwash” program. There's almost no evaluation of strategic effectiveness. We don't promote our officials on whether they were successful in achieving objectives.
It seems like you could divide up the reforms you push for into a soft version and a hard version. The soft version is something like, “Do more retrospective analysis, look back at what we've done, learn from mistakes.”
And then there's a harder version, which is, “Do retrospective analysis according to a modern polisci, scientific method process, using some rigorous academic procedure.”
Wouldn’t it be valuable for State to start by just doing more accountability and internal assessment, even if it wasn't according to this contemporary political science model?
Absolutely. Yes. There's really no perfect way to do these evaluations, and political scientists, social scientists, historians, and sociologists will fight endlessly about different methods. That's healthy in a process, and there will always be disagreement in how you evaluate these policies. Learning requires feedback. It's a core finding from cognitive science.
In the absence of feedback, Santi, one gets more and more confident, but actually not more capable. It's a really interesting human effect, like a human flaw of judgment. So yes, any feedback we can provide to our policymakers is going to make them more capable.
Presumably, even in its current form, the foreign policy establishment wants to get better at things. Currently, how does learning happen at State?
I hate to do this, but I'm going to contest the framing. I don't know if the organization does want to learn from failure. I'm not trying to suggest that people don't want to make the world a better place, they do.
But when you really look at the incentive structures and the behaviors of the organization, there's really not a lot of space for people to accept failure, or to admit that the policy didn't work.
Now, the Department of Defense, for instance, is clearly an imperfect institution, but it has some of these things built in. It has doctrine processes, where it studies the way it’s conducted business in basically every aspect of the organization.
We bring those to our system of war colleges. We try to think hard about how to create patterns of instructions that we can then train and teach the next generation. Doctrine means “that which is trained.” So they capture lessons, they do hotwashes, they review, and they push that back to the next generation. Those processes build in opportunities for learning. State, again, has nothing like this in its bureaucracy.
I can imagine someone saying, “This is great. We do need more accountability. We need to know where decisions were made, how they were made, and how to do them better… and to do that, we actually need to empower political appointees, who can be held accountable.
And we need to disempower civil servants, who can pass stuff off and blame it on the process, and who aren't trotted out in public to explain their decisions.”
What would you say to that?
I think there's really important discussions to have about the proper balance of politicals vs careerists, but on core questions of learning, I actually don't feel that there’s a lot of divergence.
In the culture of American foreign policy – within the bureaucracy but also within the broader world of think tanks and master's programs – we're taught to study history, to become deeply knowledgeable about the places in which we engage, and then use our intuition and judgment to form opinions about the issues of the day.
That process doesn't have a lot of method built into it, whether you're talking about bureaucrat careerists or you're talking about political appointees. It's not like one side or the other is asking for something very different. We need to think at scale about the way we see foreign policy as an art and not a science.
Let me push on that art and science framing a little bit. I don't even want to challenge it necessarily here –
I think challenge is healthy!
OK, I will challenge. I don't like the framing.
But I've got another question. Breaking down foreign policy, are there areas where a quantitative approach would be better than a qualitative one, or vice versa? In what areas do you feel really confident saying a quantitative approach would obviously be better?
Qualitative methods are a science and they're really vital. And there are really bad ways of doing data. So it's not about quant vs qual. But what I think what you're asking is, are there places for intuition vs places for a more rigorous, scientific process.
I think actually there's a really interesting science about intuition here. One of my favorite scholars, Gary Klein, writes about when intuition is effective. He finds two necessary conditions for intuition to thrive: one, you need clear feedback fairly soon. When you hit a golf ball, does it go down the fairway or does it go into the trees? Okay. It went into the trees. That was bad. To map that onto foreign policy, when you try an intervention, there needs to be clear feedback on whether you’re receiving a positive result or not.
The second thing is, you need to have space for “deliberative practice,” to receive feedback on what you did wrong and practice doing it better, to work on your weaknesses and emphasize your strengths.
So just golfing every day for 50 years won’t necessarily make you a better golfer. You need a coach who says on your backswing, you turn your wrists over, or whatever it is. With feedback and practice, intuition can actually grow very powerful, and some places in foreign policy might fit this space.
I also think there's spaces in which science isn't going to help us. For instance, if it comes to ethics, or morals, or right or wrong, or weighing the life of a group of American soldiers. This is what humans are: we have ethics, we have beliefs. So the point isn't to eliminate art and replace it with science, or vice versa. There needs to be a healthy relationship between them.
Just because we want to do something well doesn't mean that the intervention is actually going to be successful. That's a good place for science to study the policy and say, “Hey, listen, we need to do this, but let’s evaluate whether our tool will actually work.”
I wonder about the golf analogy. Individual golf strokes, over the course of my career, are not causally dependent on each other. I might get the yips after a bad stroke, but generally speaking, they're all individual events.
But foreign policy decisions from one person or one bureau are causally linked, right? It's like I'm moving the fairway with each stroke.
So while I would like to take the science of individual decision-making from chess or golf and apply it here, moves in those games aren’t causally linked the way foreign policy decisions are, right?
Yeah, I think what you're saying is, how do we evaluate a stroke in foreign policy when, say, our policy in Thailand is executed in one way, but actually our policy shifts in China next door in a very different way, which causes the whole region to adjust. Okay, how do we evaluate when there are multiple inputs?
Or even if we keep it at the individual level: I'm a mid-career civil servant in Thailand, and I'm trying to get feedback on my own actions. If I successfully built a relationship in one case, maybe that's the reason that the next case is successful. And if I failed in one case, maybe that’s something else outside of my control.
It seems even harder to measure at the individual level, let alone if our guy in China made a decision, and now that's affecting our guy in Malaysia.
Yes. If your argument is, these things are really complicated, then absolutely. If your suggestion is because it's complicated, we shouldn't evaluate anything. I would disagree. And if your suggestion is because it's complicated, just using our intuition is going to bring us to a more accurate place: I would also disagree.
I think it's a question of what's our alternative, and I have real concerns with the status quo approach. But yeah, you're right. The more scientific approach isn't going to be a silver bullet that's going to somehow illuminate the correct path to world peace.
If you try to evaluate the impact of any individual action, so many factors are going to affect it, it's going to be really hard to study. But perhaps a clever social scientist might be able to think of some sort of method design that will give us more leverage to understand the effectiveness.
Let's say we've got to deliver a démarche: a set of instructions from the United States to our host country. We've got to deliver the same exact démarche to every country on the planet to say, “Hey, we need your support in this upcoming vote in the United Nations on X issue.”
Say we deliver half of our instructions from the Secretary of State via a phone call, and deliver half from the ambassador in the field. If that's distributed randomly, this is the power of statistics, you've controlled for all the other complexity. And now we can study the results, which of these interventions work better.
Is that going to tell us we should never use the Secretary of State, etc? No, of course not. But if you go to Silicon Valley, this is how they think about everything. How can we implement this policy in a way that we're going to be able to learn from it? How can we A/B test our rollout of our new website? If we implement that approach into every aspect of our foreign policy, we're going to start learning a lot more about what works under what conditions.
Tell me more about that. In brief, if we were going to really focus on making State better at learning from its own behavior institutionally, how would you do that?
Yeah, I don't think there's any one single intervention that's going to work. I think you need to think about building improvements at every stage of the process.
Here's five stages in which I would think about it. First, you need to collect and manage your information in a thoughtful way.
The second stage is analysis. Third stage is decision-making and strategy. Fourth stage is implementation and learning. Finally you feed that learning into the workforce and you hire, promote and train people.
Okay. So, first stage, we've got to think more like scientists in the way that we collect information and knowledge. Right now we collect information just by reading newspaper articles and sending cables. There's this extraordinary amount of unstructured text. So the first stage is information management.
And the second stage, we've got to conduct analysis of that information. Just reading the newspaper every day is an okay way of managing the world. That's what we've done since the beginning of time. But maybe we should structure our knowledge and actually use some scientific method, some statistics, some process tracing, maybe network analysis to understand the world in more objective ways.
We've got to use the best available analytic tools. This is something the intelligence industry already does. They pay a great deal of time and attention to their analytical tradecraft. There's no similar training or appreciation for analytical tradecraft in the creation of foreign policy analysis.
In the third stage, we've got to build strategies. There's almost no training for creating or operationalizing strategy within State. Folks like Phil Tetlock who study forecasting methodology have rigorous evidence that shows there are more effective ways about thinking about the future and likely outcomes. The State Department should invest in the more effective ways.
In the fourth stage, we learn from these policies as they're being implemented. In the foreign assistance world, there's this thing called monitoring and evaluation. You create a rigorous plan if you're going to spend money on a project for what success looks like; the metrics we're going to pay attention to to understand success or failure.
This is absolutely standard at USAID (United States Agency for International Development), it's standard in a lot of the money expenditure programs at the Department of State. We do absolutely none of this for high level policy or strategy, and we should.
We feed that learning into the final stage, which is hiring based upon the skills we think are necessary. We should be promoting based on a more rigorous understanding of success and failure. We should be training skills that are necessary for competition for success in the 21st century based on learning what actually works in the field.
At each stage of the process, we need to support creating a more structured learning environment.
Let's go back to that first stage, information intake. Where does information go?
A typical policymaker will show up in the morning and boot up their email. They have a classified and an unclassified system, and they're going to receive just an extraordinary amount of information about the world.
Here are diplomatic cables that came in from the field: here's what's going on in the country. There's going to be intelligence cables coming in from intelligence officials receiving information from covert and overt methods, they may just be reading the news and reading diplomatic tables and conducting analysis. They're going to get emails from their friends in the think tank world. They're going to be reading the front page of the Washington Post and the New York Times and the Financial Times and the India Daily, etc.
They're going to be flooded with information and they're going to just read through it, use their own heuristics and mental models for capturing what they think is important, trying to store as much of that as they can in their memory banks. And there you have it. That's the information structure of the Department of State.
Is there a central database for cables? I want to go back and see what this guy was sending last year before I got on base, for example. How am I accessing that information?
Yes, but effectively, no.
Let me differentiate just to be precise. There are two separate systems for foreign policy decision makers. This is true at State, and basically the same thing at the National Security Council.
You get cables and intelligence reports sent to you, which are analytical products. There's also policy memos, and there's this big clearance process, that suggests what we should do in a particular country. So these two systems are disconnected from one another. One is, how are we seeing the world, and the other is, how do we analyze what already happened to the world?
The cable system is a very old, outdated system. There's an archive, but it's extremely difficult to search. Somebody who has invested in learning the Boolean search terms maybe able to find particular cables from the past, but it’s very difficult to do. It's like Alta Vista circa 2000.
On the memo process, it's even harder to understand past policy decisions, because those memos flow to the secretary's office. There's an office called the Executive Secretariat. They own the archive and presumably they have a little bit more access to what's been written, but that's really not accessible to the vast majority of policymakers.
So it creates this fishbowl effect at the Department of State. You care about what's in front of your face, and you don't have encouragement to go look into history, to trace decisions or past analysis over time.
What fixes to that policy memo clearance process would you make to incentivize somebody to develop a better sense of the history?
NASA created one of the world's most respected knowledge management efforts. What they did was they treated it like an engineering problem. They said, “What does a good outcome look like? How do we engineer a solution that is going to be effective?” The core of that is to prioritize the knowledge that's most useful to policymakers.
You need to think about what decisions need to be informed by past information. Then you have to structure that information and separate it from the huge wash of other information and store it.
Right now, policy memos are essentially like a newspaper article. They tell a story, they have an argument in them, but they're not structured in a rigorous way, which makes them not very machine-readable. Imagine surfing old microfiche records of past newspaper articles to understand the history of a particular event. It's just really hard to find information that way.
A more modern way of managing information is to think about structuring comparable information. We've designed this prototype of a new clearance system, the “Bayes Brief,” that asks people to very explicitly label their evidence, assessments, and recommendations.
If this is held in a database, you can see the connection between evidence in different places and recommendations in different places, so there's a database network of knowledge that somebody could more easily surf and visualize.
One obvious challenge of pushing for more citation of sources is that, if those sources are weak or poor science, then the decision-making won’t get any better. Garbage in, garbage out, right? You’re a political scientist, you deal with this every day.
So let me push you a bit. In that first FP21 report, in the section on rewarding merit, you call for State to appoint a Chief Inclusion Officer to “improve diversity and inclusion” and to “hold the bureaus accountable” on “annual areas for improvement.”
The papers that are cited there – Herring (2009) and this McKinsey report that everybody cites – purport to show that diverse teams perform better in the business world. And I'm not sure those are good studies. Some meta analyses of that literature show much more mixed findings.
Just to zoom out, you want us to be more rigorous. At the same time, the political science literature you lean on is riddled with bad research.
Yeah, one of the fundamental features of the scientific process is that, even in the hard sciences, there's no theory that's ever survived permanently. All theories update, we throw out the old stuff when we learn new stuff. This is in contravention to a process in which we cannot learn from our mistakes over time.
In the field of diversity, there's been what I consider very excellent studies on the effect of mandatory diversity training in businesses and government organizations. This is a really popular intervention that got a lot of attention recently.
All the research says mandatory diversity training does not work. In fact, it often harms the people that you're trying to help. [For a fairly comprehensive review of the literature, see here.] Voluntary diversity training actually goes a lot farther.
Okay, one could argue about the relative merits of having diverse organizations. I happen to believe that it's really important both ethically and for the effectiveness of organizations, but let's center the effects of those beliefs on the best available research. We have to allow ourselves to learn over time: hey, the first wave of research suggested it was a good idea. We studied it more, turns out it's a bad idea. And we're going to have the courage to say, we're going to get rid of this program and try something else because this doesn't work. And there are examples from the diversity space of exactly that happening.
I want to ask you about global development. In my lifetime, that field has gotten much better at self-assessment.
And on the other side, the military is quite good at self-assessment. It’s strange that State does lots of development work, and it interfaces with the military regularly, but it hasn’t improved as much as either of those fields.
That's exactly right. Some could make an argument that somehow foreign policy is more complicated, and therefore doesn't deserve this sort of self-assessment. I find that argument completely unconvincing.
As you suggest, over the past 30 years we’ve seen the introduction of much more rigorous standards for evaluating the impact of philanthropic endeavors. And is that a perfect science? Is it like measuring gravity where you can exactly predict where the projectile is going to land?
Of course not. But over time we can build a pretty extraordinary sense that this type of intervention seems to do really well in saving human lives or improving health or eliminating this disease.
I think it's a matter of incentives, the reason why foreign policy doesn't do this. The incentives in the philanthropic world, or when Congress gives money to USAID to spend on a specific program, they want to say, “Show me evidence that you spent this money well. Prove to me that this worked.” So there's a built-in accountability mechanism.
Is it perfect? No. Are there distortions in that process? Of course. But we're improving over time. With diplomacy, with strategy, there's nobody holding a purse string who says, “Demonstrate to me that this thing worked.”
