James Heckman and Burton Singer have written an important AER P&P piece on "evidence". This piece borders on philosophy as it touches on the deep issue of how do we as economists update our beliefs about how the world works. Put simply; "what is convincing?". Towards the end of the article, Heckman makes a nice point about one of his own famous early 1990s AER papers;
"Another example of the abductive process is
the work of Donohue and Heckman (1991), who
addressed the complex question of whether the
1964 Civil Rights law advanced the economic
position of African Americans. Multiple factors
were at work at the same time: (i) a tight labor
market fueled by the war in Vietnam and policies
in the 1960s; (ii) the advancing level of
education of African Americans; (iii) the Civil
Rights Act and the civil rights movement. Using
numerous historical datasets, as well as examining
multiple outcomes and clues from newspaper
stories and contemporary accounts joined
with basic economic analysis, they establish the
case for a powerful beneficial role for federal
government activism. No single bit of evidence
is decisive. It is the ensemble of evidence and
the substantiation of additional hypotheses that
makes the story compelling."
In this age of the field experiment, how does one take a "clean estimate" of a specific program and combine this with other evidence to tell a compelling story? Or is the estimate the story? How do we "aggregate" different pieces of evidence?