The Bayes Brief: Designing a Modern Policy Memo Process
By: Alex Bollfrass, Ellice Huang, and Dan Spokojny | January 23, 2022
The policy memo sits at the heart of decision-making and knowledge management at State––yet it has not meaningfully evolved in decades. As the State Department adage suggests, “The paper process is the policy process.”
Two prominent recent studies of the State Department crystalize long-simmering calls for reform of this process. A report from the Council on Foreign Relations suggests “clearance hell” contributes to the State Department’s “lousy reputation for policy effectiveness.” They call for “a completely new approach” to the policy clearance process. And a team of experienced ambassadors at working at Harvard’s Belfer Center calls for a “dramatic streamlining of clearance processes.”
The Bayes Brief is fp21’s attempt to design the future of the policy memo. The Bayes Brief re-imagines the policy memo by capturing and summarizing evidence in a systematic format conducive for policy success in the information age. It is a simple intervention into the policy process that could positively impact the effectiveness of U.S. foreign policy.
Knowledge is the most important strategic asset for foreign policy, and it must be prioritized and leveraged as such. The Bayes Brief achieves this by breaking down the typical policy memo into functional moving parts so that knowledge is easily stored, located, and retrieved. It facilitates a policy process that focuses attention onto evidence of likely policy success rather than flimsy opinions. The Brief also provides a dynamic platform to ensure policy decisions can quickly react to changes on the ground, and decisions can be monitored and evaluated across time and policy space to ensure success.
Let’s first evaluate the challenges with the status quo policy memo, and then we’ll introduce the key features of the Bayes Brief. We also encourage you to check out our recent report on knowledge management at the State Department; research which helped inform the Bayes Brief.
The Problem: An Outdated Process
Our research suggests there is virtually no effective knowledge management system at the State Department.
Legacy knowledge management systems at State have atrophied as decision-makers are flooded with endless information. Informal processes may have made sense when the Department was much smaller and technological capabilities were much more limited, but that is no longer the case. State’s reliance on tacit knowledge — that which is not written down — means that individual insight is prioritized over collective knowledge. Tacit knowledge cannot be disseminated, trained, evaluated, or replicated. Outside observers must simply trust that the decision-maker knows best. Many of these problems are caused by an outdated policy memo process.
We see three pain points leading stemming from — and contributing to — an inefficient decision-making process:
An inability to retrieve past decisions and knowledge, leading to a constant replication of effort. There is no systematic, accessible repository of policy decisions and evidence. Knowledge from past memos that should be leveraged in future decisions is lost, inaccessible, and rendered useless. As a result, analysts are forced to reinvent the wheel and rebuild existing knowledge in each iteration of the policy process.
An inability to track the impact of policy decisions over time. Once a policy is adopted, its impact is seldom monitored or evaluated throughout implementation. The majority of memos and policy decisions are never reexamined, leaving little trace of how the policy evolves over time except in the memories of the most experienced and learned policymakers.
An absence of standards for evidence and analysis, leading to ad hoc, idiosyncratic, risk-averse decision making. In today’s system, policy-makers depend on their own judgment and experience to subjectively evaluate the quality of a recommendation. There is no accountability for the quality of analysis, and best available intelligence is usually absent from policy decision memos. This outdated process excludes reference to the underlying evidence supporting arguments and incentivizes lowest-common-denominator policy decisions that prioritize consensus over impact.
The Solution: A Modern Policy Memo
The Bayes Brief is an updated policy memo template that empowers evidence in the policy creation process. The memo is designed to systematically, transparently, and reproducibly incorporate qualitative and quantitative evidence from multiple sources, including where the evidence is contradictory or absent.
The Bayes Brief is named after Bayes’ Rule, a foundational concept in statistics that updates existing knowledge based on the arrival of new, relevant pieces of evidence.
Here’s a closer look at the features:
1. Breaking down the typical policy memo into functional moving parts.
The typical policy memo is static and one dimensional (Figure 1). It is designed for one-off decisions, and treats knowledge as single-use and disposable. In order to compare evidence and decisions across time and space, the Brief breaks down the memo and draws on its discrete, recurring components (Figure 2).
Figure 1. A typical policy memo at State.
Figure 2. The recurring components of a typical memo.
Each of those components is stored in a knowledge database so that policymakers can quickly locate, retrieve, and build upon the existing foundation of knowledge (Figure 3). This way, evidence is not lost and overlooked in future policy decisions; in fact, it is leveraged in order to create more powerful, impactful evidence-based policies.
Figure 3. The extraction, storage, and application of existing evidence in a memo drafting process.
A simple use-case applies the Bayes Brief is demonstrated below in Figure 4. How might an official decide the policy question: Should we send aid to Country A? Using the Brief, a policy-maker easily locates and retrieves the relevant past memos, extracts the recommendations and evidence from each memo, and bolsters the new policy recommendation with historical knowledge. For a more detailed explanation of this example, click here.
Figure 4. Applying the Brief to answer a foreign policy question.
2. Visualization of relationships between all parts of the policy process, over time and space.
A network graphing function displays the complex linkages between actors, recommendations, predictions, and evidence (Figure 5). One can imagine how this network graph will expand as new policy decisions are made. Users can follow the flow of knowledge across actors and issue areas.
Importantly, if a core piece of knowledge is invalidated or deemed inaccurate, users can easily determine and update which policy decisions and recommendations are affected.
Figure 5. The network graphing feature of the Brief application.
3. Monitoring and evaluating with confidence in the evidence and recommendations.
The Bayes Brief instills methods proven to improve human judgment into the policy making process by requiring decision makers to explicitly outline their assumptions, evidence behind each recommendation, and the level of confidence they hold in their claims (Figure 6).
The Brief thus provides a pathway for predicting and evaluating the impact of policies as they are decided and implemented, creating a system that is constantly learning from its own shortcomings and successes.
Figure 6. In drafting policy memos via the Brief application, policy-makers indicate their confidence level for the positions they take and the evidence they use.
Conclusion and Next Steps
The Bayes Brief is just a prototype at this stage. Our goal with the Bayes Brief is not to sell software — instead, we hope to generate conversation about the anachronistic nature of the current policy memo process at the State Department. Its weaknesses have contributed to the inability of the State Department to play an effective role in supporting U.S. foreign policy, and have undermined the quality of our diplomacy. We know there is a better way, and the Bayes Brief is the first step.
We hope to gather feedback from our target audiences – academics, practitioners, and technologists – and iterate and scale our prototype based on those responses. We also hope to release a beta version of the application for testing in academic communities, educational settings, and ultimately within the State Department.
fp21’s mission is to transform the foreign policymaking process into one that uses best practices and continually learns from itself. Diplomats are taught that “diplomacy is an art, not a science,” but the 21st century is leaving the foreign policy bureaucracy in the dust. The Bayes Brief project is the rare initiative that explores the nexus between science, technology, and foreign policy.
