3 reasons evidence-based policy is hard
Basing federal and state healthcare policy on evidence-based practices seems like a no-brainer.
But data are never simple.
Not helping the matter is that academics and policymakers are coming at the challenge from different angles. Working against them is time, the power of anecdotes and the difficulty of interpreting data.
Academics and political advisers traded ideas on the biggest challenges to using sound evidence as a basis for successful policy at a panel this week at the National Health Policy Conference in Washington.
The question of certainty
Researchers, as a requirement, are clear about the limitations of their data and the conclusions that can and cannot be drawn from it.
“The conclusion of every academic study is more study is needed,” Katherine Baicker, dean of the University of Chicago School of Public Policy said.
Policymakers and politicians, by contrast, are looking for assurances they can give constituents.
Footnotes, caveats and cautions don’t work well for campaign ads and soundbites. That’s just a reality for political advisers, said Douglas Holtz-Eakin, a former director of the Congressional Budget Office, appointed by former President George W. Bush.
"You can’t expect them to put politics aside,” Holtz-Eakin, now president of the conservative American Action Forum, added.
Policymakers and researchers are also working on different timelines. Careful research takes time and needs adequate review.
“But the challenge is, policy can’t wait for perfect evidence,” Baicker said.
Holtz-Eakin's takeaway after years advising policymakers: It’s better to be on time for the debate and 95% sure of your results than it is to be 100% sure and two years late.
Researchers can and should make educated guesses backed by evidence, Baicker said.
“We can’t let uncertainty get in the way of giving a good guess, because if we’re not willing to guess, somebody else will,” she said.
It's important to remember the research and conclusions won’t please everyone, and public policy almost always includes tradeoffs, she added.
“There are very few policies we can point to where everybody’s a winner and it's free,” Baicker said.
Evidence needs to be translated
Evidence can be difficult to interpret even for those relatively well-versed in a subject. Technical language can be a barrier, and charts and graphs can be misleading without the proper context.
Policymakers should also be careful not to take the conclusions they’re trying to reach and push data that can’t fully back them up. Research isn't necessarily designed and performed to answer a specific policy question, Holtz-Eakin said.
“When I talk to politicians, the very first thing I say is, 'The data never speak for themselves, so don’t expect to be given the data and know what to do,'” he said.
Drawing those specific policy conclusions from data is challenging, and some academics, frankly, don’t have any interest in it, Baicker said.
And it can stand in the way from meeting career expectations for academics under pressure to publish or perish. “When you do that translation, when you write those briefs or spend time talking to policymakers, that’s one less paper you’re going to produce,” she said.
Those who do choose to help translate evidence in the interest of making good policy also need to push aside personal preferences.
It doesn't help anyone to make an argument that can’t be supported. “As a voter, I sure ought to have views on all of these policies,” Baicker said. “And I do, and I vote on them. I try very hard to present analyses that are immune from that.”
Narratives are powerful — but misleading
Politicians love to find a great story that illustrates the need for a certain policy. It can help get public support behind an initiative and make complex ideas a bit easier to understand. But anecdotes aren't evidence.
An entirely accurate story may be an outlier, or have extenuating circumstances, and it doesn't represent the evidence. But popular rhetoric is hard to overcome, Baicker said. “The idea that anecdotes are often a more powerful way to communicate things is an extra challenge to keeping academic rigor,” she said.
Narratives are about people taking a certain action, but evidence needs to take into account what people don’t do to get the full picture, said Sherry Glied, dean of New York University's Robert F. Wagner Graduate School of Public Service. That’s a much harder story to tell.
Also, policymakers are often trying to please a coalition of interests, with desires that don’t always align. That makes it trickier for analysts to give pertinent information.
“If you’re trying to do evidence-based policymaking and no one will give you a goal, that’s a really tough job,” she said.
Glied said that although meeting the increasing demand for policy informed by solid research is far more easier said than done, she still thinks there are "tremendous possibilities to use evidence and make more tenable and approachable policy."
Follow Shannon Muchmore on Twitter