Reviewed by Caroline Fiennes.

Much rubbish is spoken about randomized controlled trials (RCTs). And many of the views ascribed to the ‘randomistas’ who run them are not in fact held by them. For instance, it is often said that RCTs are the only research worth doing, or that they are the gold standard for research, which few randomistas would say. 

So it’s great that somebody talked to a pile of randomistas to document what they actually believe, and say, and do.  

This book is verbatim transcripts of 20 interviews with development economists, some of whom run RCTs and some of whom are sceptical about them. 

All the interviewees are based in the US. This choice is not explained, nor even mentioned. Ogden says just that his choice of interviewees is ‘idiosyncratic’.

It weakens the book considerably in my view, because people in many countries run RCTs, many of whom would probably have agreed to be interviewed.

To me, this apparently arbitrary choice makes the book oddly parochial, as though the author hasn’t realized that real work goes on in the world’s other 195 countries.  

Worse, it plays somewhat into the hands of two common criticisms of randomistas. One is that they experiment on other people, sometimes treating them more like lab-rats than people they’re trying to help.

This view is shared by a Nobel laureate: Angus Deaton expresses his discomfort that so many of these experiments are ‘done by rich people on poor people’. This could have been avoided by including interviews with researchers based in Jakarta or Cape Town or Delhi.  

The other criticism (often unwarranted) is that randomistas assume that the results of their studies apply to other places. Tim gives no discussion to whether his findings about randomistas based in the US might be true of randomistas based elsewhere.  

If I had been asked to review this book before its publication, I’d have advised it to use less economics jargon – that it be written in language that more people will understand. For instance, the first sentence by the first interviewee talks about ‘econometric identification’: until a few years ago when I started working with economists – including several of Tim’s interviewees – and thereby learned their language, I’d have had no idea.

This problem pervades the book: ‘copays… power (which isn’t used to mean what people normally mean by power); confounders, enumerators…’; ‘my smokeless cookers failure of impact work’; ‘Ec101’ and so on.  I fear that this will conceal much of the book’s value from many readers. 

Nonetheless, there are great insights in here. The different reasons that people set up business in developing countries rather than in developed countries: the young-ness of government systems in many less developed countries; whether one can balance rigorous evidence from somewhere else with non-rigorous evidence from here; why Harvard’s Lant Pritchett thinks that (and has written about why) only a ‘desperate’ charity would agree to do an RCT. 

Perhaps inevitably, I would have wanted to explore some things that the interviewees say that Ogden doesn’t.

Why is Angus Deaton ‘not a fan’ of the Cochrane Collaboration, the international entity that creates and shares systematic reviews of evidence in health; why is Esther Duflo, a decorated economist at MIT, ‘not prepared’ to ‘make a judgement about the importance of making people healthy vs having them buy a roof’, given that funders and policy-makers must make such judgements all the time; and is it really or always true that evaluations initiated by funders cannot promote learning?  

These are thoughtful people, and it’s worth hearing from them directly.  

Caroline Fiennes is director of Giving Evidence and writes the How To Give It column in the Financial Times. Email caroline.fiennes@giving-evidence.com @carolinefiennes  

About the book
Published by: The MIT Press
Price: $45
ISBN: 9780262035101
To order: https://mitpress.mit.edu/books/experimental-conversations