The Social Priming Studies in “Thinking Fast and Slow” are not very replicable

Train wreck at Montparnasse Station, at Place de Rennes side (now Place du 18 Juin 1940), Paris, France, 1895. Wikimedia commons

" data-medium-file="https://mindhacksblog.files.wordpress.com/2017/02/train_wreck_at_montparnasse_1895.jpg?w=840?w=200" data-large-file="https://mindhacksblog.files.wordpress.com/2017/02/train_wreck_at_montparnasse_1895.jpg?w=840?w=200" class="alignright size-full wp-image-33872" src="https://mindhacksblog.files.wordpress.com/2017/02/train_wreck_at_montparnasse_1895.jpg?w=840" alt="train_wreck_at_montparnasse_1895" srcset="https://mindhacksblog.files.wordpress.com/2017/02/train_wreck_at_montparnasse_1895.jpg 200w, https://mindhacksblog.files.wordpress.com/2017/02/train_wreck_at_montparnasse_1895.jpg?w=125 125w" sizes="(max-width: 200px) 85vw, 200px" />In Daniel Kahneman’s “Thinking Fast and Slow” he introduces research on social priming – the idea that subtle cues in the environment may have significant, reliable effects on behaviour. In that book, published in 2011, Kahneman writes “disbelief is not an option” about these results. Since then, the evidence against the reliability of social priming research has been mounting.

In a new analysis, ‘Reconstruction of a Train Wreck: How Priming Research Went off the Rails‘, Ulrich Schimmack, Moritz Heene, and Kamini Kesavan review chapter 4 of Thinking Fast and Slow, picking out the references which provide evidence for social priming and calculating how statistically reliable they:

Their conclusion:

The results are eye-opening and jaw-dropping.  The chapter cites 12 articles and 11 of the 12 articles have an R-Index below 50.  The combined analysis of 31 studies reported in the 12 articles shows 100% significant results with average (median) observed power of 57% and an inflation rate of 43%.  …readers of… “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness.

The argument is that the pattern of 100% significant results is near to impossible, even if the effects known were true, given the weak statistical power of the studies to detect true effects.

Remarkably, Kahneman responds in the comments:

What the blog gets absolutely right is that I placed too much faith in underpowered studies. …I have changed my views about the size of behavioral priming effects – they cannot be as large and as robust as my chapter suggested.

The original analysis, and Kahneman’s response are worth reading in full. Together they give a potted history of the replication crisis, and a summary of some of the prime causes (e.g. file draw effects), as well as showing off how mature psychological scientists can make, and respond to critique.

Original analysis: ‘Reconstruction of a Train Wreck: How Priming Research Went off the Rails‘, Ulrich Schimmack, Moritz Heene, and Kamini Kesavan. (Is it a paper? Is it a blogpost? Who knows?!)

Kahneman’s response