Foreign Policy Should Be Evidence-Based
By: Dan Spokojny and Thomas Scherer, July 26, 2021
Originally published at War on the Rocks
An ascendant China. A revanchist Russia. The failure of the wars in Iraq and Afghanistan to achieve U.S. objectives. Climate change. The threat of nuclear proliferation. Rising authoritarianism. The challenges to U.S. influence on the world stage have become so numerous, serious, and complex that some experts see the “unraveling” of American power.
Faced with this perilous strategic landscape, some are calling for a reexamination of the way in which U.S. foreign policy is conducted. Career Foreign Service officer, and now CIA director, William Burns suggests “recovering the lost art of American diplomacy,” a sentiment echoed by President Joe Biden. Three highly experienced U.S. diplomats have warned of a “crisis” inside the State Department, characterized by “a reluctance to speak truth to power, a lack of individual accountability … [and] an aversion to professional education and training.” Similarly, Uzra Zeya and Jon Finer — both now senior officials in the Biden administration — argued in late 2020 that a “decades-long failure to implement essential reforms” has produced a “policy environment that has, in some priority areas, evolved beyond the core competencies of most Foreign and Civil Service officers.”
These analyses from some of the nation’s most capable diplomats are valuable. But their recommendations do not go far enough. Revitalizing U.S. foreign policy will require more than a renewed commitment to diplomacy. Instead, policymakers should embrace the evidence-based policy movement. Its methods would enhance American leaders’ ability to achieve national security objectives by reducing costly inefficiencies, reducing misperceptions, and transforming U.S. institutions into organizations that can continually learn. Simply put, yesterday’s tools may not be up to the task of solving today’s problems.
Everyone seems to agree that foreign policy should be based on the best available evidence. Yet, during our work in government and with our think tank fp21, we have seen how the lip-service paid to evidence-based policymaking is not supported by the organizational and cultural changes that would actually enable the effective use of evidence. Such evolution can be achieved if foreign policy professionals begin to think differently about the conduct of policymaking.
American officials should follow a more scientific approach to decision-making, grounded in more transparent standards of evidence. The nation’s ability to persevere through today’s international problems may depend on it.
What’s Wrong with the Current Process?
The State Department houses extraordinary public servants whose hard-earned expertise derives from years of experience, study, and training. Such leaders know a great deal about the world. But, in recent decades, their ability to make effective use of information has not kept up with the vast increase in the volume of that information. The intelligence community has adopted clear standards of analysis, but no analogous standards exist in the policymaking process. Instead, the U.S. system asks officials to rely on their own judgement, and virtually no training or guidance is provided regarding best practices in decision-making.
When each official relies on their own judgement while engaging in the policy process, decisions are grounded more on personality and opinion than most care to admit. A range of predictable biases takes hold. Information and intelligence that aligns with one’s preexisting view of the situation is privileged, while inconvenient facts are cast aside. Carefully selected anecdotes, factoids, and statistics can be assembled to back up just about any policy claim.
All foreign policy professionals will agree that evidence is vital to our work. Yet, the policy process incentivizes us to think about evidence like courtroom lawyers. In order to win a case, each side marshals facts to defend their position to the jury. Facts contrary to that position are disparaged or ignored. Adversarial legalism may be a reasonable approach for criminal justice. It does less well in advancing American strategic interests abroad.
There are real consequences when policymakers ignore readily available evidence. Take the State Department’s failed attempts to contain the civil war in Yemen, for instance. Amid some of the deadliest fighting of the war in October 2016, then-Secretary of State John Kerry announced, “This is the time to implement a ceasefire unconditionally and then move to the negotiating table.” By pushing an agreement with which neither side was willing to comply, and in the absence of any mechanisms to guarantee the ceasefire, this announcement did not comply with best practices related to the timing for a diplomatic intervention. Nor was the peace strategy informed by research on durable peace agreements.
Instead, America’s diplomatic machinery applied the same failed approach repeatedly. Five earlier ceasefires had failed to contain the violence. The sixth ceasefire agreement, signed shortly after Kerry’s statement, was marked by repeated violations by both sides, and expired after only three days. A seventh ceasefire agreement, agreed to a month later, collapsed within 48 hours. Evidence suggests that the approach caused more harm than good — both sides used the pauses in fighting to prepare new attacks, while trust in the political process plummeted. One survey of Yemenis affected by the conflict found, “Every time they announce a ceasefire fighting intensifies.”
Our goal is not to second guess the good-faith attempts of former policymakers. Instead, we hope that the State Department can seize the opportunity for growth. As the scholar Dan Reiter noted, “Like good carpenters, foreign policymakers need to know their tools.”
What Is Evidence-Based Policymaking?
From Silicon Valley to financial services, political campaigns to baseball, today’s most successful enterprises build their cultures around harnessing the best available information and insights. Organizations that adopt modern techniques thrive. Those that don’t, fail.
A 2012 study by Andrew McAfee and Erik Brynjolfsson showed that organizations can radically improve their performance if they seize the opportunities associated with quantitative data (of course, this is only one type of evidence among many). According to their research on 330 publicly traded companies, this requires a serious commitment to building an organization that prioritizes data, rather than simply an organization that subordinates data to traditional decision-making approaches. The authors distinguished between businesses “pretending to be more data-driven than they actually are” and organizations that had built quantitative analytics into the DNA of the company. The results were conclusive: “The more companies characterized themselves as data-driven, the better they performed on objective measures of financial and operational results.”
The use of evidence in the public sector is growing. Evidence-based approaches are standard practice in public health, economic policy, education, and more. In the national security space, evidence-based methods are common in international development, in the intelligence community, and at the Department of Defense. Not all parts of government have embraced evidence-based methods, but the decision-making apparatus of the National Security Council and the Department of State stand out as particularly resistant to change.
When we advocate for evidence-based policymaking, we are not talking about changes to individual policies but rather changes to the policy process itself. Features of an evidence-based policymaking process include: an emphasis on fact-finding; citations of the available evidence for various options; metrics of success in all policy memos; an emphasis on transparency of decision-making; and an emphasis on learning from successes and failures.
But we should be clear: Not all evidence is created equal. Arguments based on anecdotes, analogies, and simple descriptive statistics can be highly misleading or flat-out wrong. Too often, policymakers rush from the discovery of a few facts to forming a confidently held conclusion, or, worse yet, start with a conclusion and then go hunting for supporting evidence.
Evidence-based policymaking demands a high standard. It directs our attention to the quality of the evidence used in the policy process. Evidence-based policymaking uses the scientific method to subject its evidence and claims to constant scrutiny. The scientific method is straightforward. It asks us to think of all claims as hypotheses that require testing rather than as conclusions that need defending. This subtle shift of mindset is the core of evidence-based policymaking.
The first step of the scientific method is something policymakers are already good at — generating hypotheses. This is the stage that policymakers call the “art” of diplomacy. A great way to do this in foreign policy is to read a lot, immerse oneself in a foreign culture, and stay curious about the future. More systematic approaches can challenge policymakers to be even sharper.
The next step is to test the hypotheses by evaluating evidence on all sides of the argument. This is where the existing policymaking process so often falls short. Overconfidence blinds decision-makers to alternative approaches, contrary evidence, and misplaced assumptions. Deep biases in their evidence and analysis lead them away from the truth. When policymakers fail to test their claims, they make mistakes that could have been avoided.
Evidence can vary greatly in type and quality in ways that are not readily apparent. Evidence can be qualitative, data-driven, comparative, historical, or the result of a randomized control trial, but the principles of the scientific method remain unchanged. This is where training as a scientist — and that includes in social science — really pays off. But debates about the specific methods or tools used to conduct evidence-based policymaking are often used as an excuse to ignore evidence entirely. That is dangerous.
One of the biggest misconceptions about evidence-based foreign policy is that science should provide the “right” answers to the problems we face. It will not. It cannot. Anyone who tells you otherwise is trying to sell you something.
Instead, the goal of evidence-based foreign policy is to continually strive for better-informed answers. When policymakers demand transparency about the set of facts used to support recommendations and invite scrutiny about potential sources of bias, they make better policy in the present. When decision-makers subject policy recommendations to evaluation and have the courage to change their minds in the face of new evidence, they will make better policy in the future.
There are many valid concerns about the shortcomings of evidence-based approaches. When we speak with policymakers, most of them lament the lack of demand for evidence in the current policy process. However, some express concern that existing evidence does not apply to their unique policy challenges, or fails to offer actionable recommendations. Others see evidence-based processes as too slow for the real world, or believe that the real world is too complex to apply “lessons from a lab.”
But skeptics miss the point. The question is not whether evidence should be used in the policy process — all policymakers use evidence — but rather how to continue to raise the bar on the quality of the American foreign policy process.
Evidence of Progress, with More Work to be Done
The good news is that a beachhead has already been established in foreign policy for the evidence-based policy movement. Congress passed the Foundations for Evidence-Based Policymaking Act of 2018 — also known as the Evidence Act — that requires executive branch departments to produce evidence-building plans. The Biden administration’s Memorandum on Restoring Trust in Government Through Scientific Integrity and Evidence-Based Policymaking declared that the administration’s decisions across all agencies will be evidence-based and guided by the best available science and data. And the Memorandum on Revitalizing America’s Foreign Policy and National Security Workforce, Institutions, and Partnerships was a rallying cry for core principles including integrity, transparency, and accountability throughout the foreign policy apparatus. Further, the State Department has a promising new Center for Analytics and has recently named its first chief data officer. Although the State Department has yet to release its Enterprise Data Strategy, it is under review.
Documenting progress on the use of evidence today in the bureaucracy is difficult, but the State Department’s FY 2020 Annual Performance Report offered some clues about where things stand. In 2018, the department committed to “increase the use of evidence to inform budget, program planning and design, and management decisions.” By December 2020, well past the deadlines set, only 63 percent of the department’s bureaus had documented how their major programs are expected to lead to desired outcomes and only 55 percent had identified relevant indicators and opportunities for evaluation. “Programs” refer to foreign assistance projects with explicit budgets — statistics do not exist for strategy and policy because little effort is made to track and evaluate high-level policymaking.
Wielding robust evidence to address complex policy challenges is not simple. It will require a cultural shift within U.S. foreign policy institutions. Nevertheless, the groundwork laid by the 2018 Evidence Act and the Biden administration’s commitment to making evidence-based policy offers an opportunity to make serious progress.
We encourage the Biden administration to more explicitly promote evidence-based policymaking for foreign policy processes, including by taking the steps proposed in the fp21 report Less Art, More Science: Transforming U.S. Foreign Policy Through Evidence, Integrity, and Innovation. Biden can supercharge his foreign policy by incentivizing research and analysis, fostering a culture of evidence and empiricism, and making more systematic use of today’s superabundance of information.
The ultimate goal is that future policy debates be won not merely by the force of one’s conviction or one’s position in the hierarchy, but by the strength of one’s evidence. We commend the progress that has been made toward this vision. We hope that Congress, the State Department, and the White House will work together to mainstream evidence-based policymaking.