The Challenge:
Responding to requests for substantive revisions to consultancy work that go against one’s findings, analysis, or professional opinion
The Background and Policy Setting:
King is a political scientist committed to conducting policy-relevant research related to war, peace, development, and education. Over the past 15 years, alongside her university-based work, she has partnered with a number of governments, non-governmental organizations, funders, grant-makers, and research organizations, usually in a consultancy framework. This is sometimes pro bono, oftentimes paid. King describes the policy-setting as one where scholars are increasingly trying to work on projects that are simultaneously socially-scientifically relevant, and practically useful in answering pressing real-world questions. In the past decades, the number of events and multiple organizations that explicitly help match scholars based at universities with policy and programming partners has grown significantly. King is grateful to have had the opportunity to learn from, and contribute to, a diversity of partnerships.
The Engagement:
King has engaged in multiple types of policy-setting consultancies that include impact evaluation work, studying the effectiveness of interventions for program implementers and their funders; authoring backgrounders to assess need prior to programs; desk studies of an evidence base; qualitative case studies exploring program implementation and the experience of participants, and more. Much of her work has been focused in sub-Saharan Africa. While King has sometimes worked alone, she has more often worked in partnership with other scholars, including some based in the countries of focus, as well as doctoral students and postdoctoral fellows from her university. Her reflections here do not arise from any specific engagement, but rather speak to challenges she has encountered many times across this type of engagement.
King describes multiple stakeholder reviews as a routine component of these consultancies. Imagine receiving comments on an impact evaluation design by USAID headquarters, a USAID country office, the relevant ministry in the country in question, a national NGO overseeing grant disbursement and program implementation, and a more local implementing partner who actually implements the program. Such various parties review the products of research and provide comments, to which the research team replies. Sometimes the stakeholders review simultaneously, sometimes one after another. Sometimes there is one round of revisions, oftentimes multiple. There are typically comments that are changes in phrasing or format, that may not be the authors’ first choice, but that are largely stylistic. There are also always useful substantive questions that push one to rethink, clarify or explain.
The challenge, though, comes when stakeholders request substantive revisions that run counter to one’s best academic assessment or interpretation of the findings or facts. Read against some of the challenges that she suspects others in this series will write about, King notes that this is somewhat of a small challenge – it might be someone asking to strike a paragraph, change the emphasis or tone – but it is one that keeps coming up in different guises and that always brings stress and needs careful reflection to address.
King recalls one instance when a government representative requested changes to reflect a more positive description of their country and its human rights record. Variants of this request have come up multiple times. Another example is a program implementer who wanted an impact evaluation report to emphasize certain more favorable results of program effectiveness over less favorable findings. Another instance is a common request to be more assertive in recounting findings, despite a research design that doesn’t support causal claims. All together, these requests are often in the vein of wanting to make things sound more positive, and less critical, than originally written. They do not come from all stakeholders or represent most requested revisions. But, they are challenging.
What were the implications of the engagement?
What to do? Remember this reflection is on proposed changes with which one substantively, and consequently typically ethically, disagrees. While sharing some approaches she has found helpful, King emphasizes that there is no one-size-fits-all answer.
One valuable step is to do our utmost to build positive, communicative, relationships with partners to set the groundwork for conversation. Examples include getting to know the people working on policy and programming with which one is becoming involved, working with partners on understanding the scope of work, “teach-ins” on research design and its implications, and keeping stakeholders informed throughout. In addition to being good practice, these efforts make for stronger collaborations and probably reduce substantive disagreements on products. They do not fully eliminate the challenges discussed in this brief.
Another piece is to try to more fully understand the pressures under which the stakeholders find themselves. Government officials may be held accountable by their superiors for not well representing their interests. Funding usually follows favorable evaluations. Program implementers have more experience on the ground in a given context than King and colleagues and sometimes deeply feel that the program was successful, or that the needs may be different, even if the findings do not support their conclusions.
In revising writing, King explains that she generally tries to accept as many suggested changes as possible, those that don’t evoke the challenges from this brief, to give more opportunity to push on those that really matter – in a friendly and respectful way.
Sometimes there is space for compromise. In one instance of a government asking for a more favorable portrayal of their country, King’s team compromised by citing third party sources for the most negative portrayals; “Freedom House International rates… Human Right Watch reports explain that…” to frame the context of the country’s record. She recalls several back and forths and concedes that this probably went in a long footnote rather than in the main text. This was a reasonable compromise because while King felt that this context was essential to the overall picture, and wasn’t willing to give it up, it wasn’t the core focus of the work product.
Other times there is less room to maneuver. In response to a request to highlight the more positive evaluation findings and to downplay the less positive, King and her colleagues advocated – over multiple rounds of revisions – to stick closely to their original framing. This prolonged the timeline and effort on the project. But they also complemented these exchanges with more conversations and presentations with different stakeholders to build buy-in to the research approach and to help them be able to present the results to their teams and funders as learning, rather than judgment. Instead of highlighting "positive findings" they tried to highlight "the positive in the results", focused on the learning that can take place moving forward.
While she hesitated to include it, since there were special deadlines and pandemic-related circumstances that produced the problem, King shared one instance in which time had run out for her to review, let alone work through edits on a large report. Even though earlier versions were of high quality and coming along well, she considered it unethical not to have fully reviewed important substantive proposed changes. In this circumstance, and given the deadline, she asked not to be listed as author on the report, even though she meaningfully contributed and was both proud of the earlier work and happy to be part of the team. She recounts that a compromise certainly could have been reached, and that all were cooperative, but that review was not possible. Requesting to be removed from authorship - while somewhat drastic and not without consequence on all sides - could be an appropriate response in some other circumstances, too.
In the end, King reminds us that the fact that this challenge arises for researchers – disagreeing with edit requests and confronting the challenge of what to do – doesn’t typically mean that they have done anything wrong. It is often difficult to decide what to do. But the fact that it is a challenge often means researchers are doing something right in thinking carefully about the meaning and implications of what we produce and endorse and the impact we have the privilege to try to make in the policy world.