His own work held up very well to replication. It's when he is citing the work of other scholars (in particular, that of social psychologists) that doesn't hold up well to replication.
As someone working in this space, I largely agree with your take.
I'll only add that compared to, say, traditional economics we don't have a sense of how unique the problem is to social psych/organizational behavior. In the past decade psychology has done a lot more replication attempts and data auditing than have other fields in the social sciences. So it could be that other fields are equally problematic, but we don't know.
This is a fair point. I would like to see more replications AND better incentives around replications.
FWIW... In plenty of economics papers you can't do a meaningful replication b/c there may be no analogous natural experiment. You can only do analysis of the same data. Such "replications" do not always lead to the same results.
The more experimental parts of the econ literature are now very big on pre-registration (esp. in development, e.g.). Not a panacea by any means, but an improvement for sure.
And there are generally higher standards around making code available now. If you work in this area and read the AER you may be aware of a recent retraction in PF due to coding malpractice. The system has improved but is far from perfect.
Yup, completely agree. I think research practices (and incentives for sound research practices) are starting to tilt in the right direction, though perhaps at a slower pace than I'd like.
Most of Kahneman's own work has held up quite well to replication, and that makes up a large portion of the book. But a lot of the social psych findings he discusses have not fared well to replication attempts.
We tested both ideas (altruism and theft aversion) in the paper. Short answer is we find evidence for both. For somewhat technical reasons, we think the "viewing oneself as a thief" component is probably necessary in order to observe higher return rates for wallets with money than wallets without money. But certainly both elements are at play.
It's a fair point, but we've tried to test this issue in a number of different ways. Based on the available data, cross country differences in email usage doesn't seem to have a meaningful impact on our results. See my comment above to gyf304.
(1) This issue doesn't affect the treatment effect (the difference in return rates for money vs no money), which was the main focus of the paper. If email usage is low in a particular country, that should affect return rates equally in both the money and no money conditions.
(2) We've done a number of robustness checks on the point about email usage and have not been able to find evidence that it has a meaningful impact on the results. For instance, when looking at cross country differences in wallet return rates, the rank order correlation between the "raw" data and one that statistically adjusts for email penetration rates (based on World Bank data) is 0.95.
(1) Noted. The main graphic emphasizes reporting rate over the delta, in both the axes selection and the ranking - and indeed the ranking is the main takeaway from many of the readers. If you rank by percentage delta of report rate, the graphic would be drastically different. This newer graphic wouldn't be "fair" either, as countries with higher control (no money) report rate would have a relatively low percentage delta.
(2) The World Bank Enterprise Survey data is listed under the title "Percent of firms using e-mail to interact with clients/suppliers" (for which China is at 85%) which I think is not the same as email penetration rate (which in the above reference in my parent comment, is at <40%). I understand your reasoning that cross country data is hard to come across.
I also read your reply in Science. I believe that while there are multiple limitations conducting the research - all perfectly reasonable - the limitations nonetheless affected the credibility of Fig. 1 - the main figure of the paper.
It is also mentioned in the reply that creating new social accounts is unfeasible. While I think this is true, wouldn't a single account per platform suffice? Most social platforms allow anonymity for display names / handles. If the social account name does not bear resemblance to an actual name, I don't think the participants would notice.
Actually one of our research assistants was temporarily detained (in Kenya, I believe) for suspicious activity. As you can imagine, having a bunch of "lost" wallets on hand required some explaining.
Definitely a concern. We took steps to examine if there were specific "experimenter effects", that is whether response rates were affected by one research assistant rather than another, and don't find evidence that was a factor. But that doesn't address your broader concern that things might be different if the person dropping off the wallet was native to the country. FWIW the wallets were designed to signal that the owner was probably a local and not a foreigner.
On targeting front-desk employees, we did this both to allow relatively portable cross-country comparisons and for reasons of internal validity (e.g., if we placed wallets on the ground, then can participants "select" into the study which compromises our ability to draw causal inferences).
On your last point about whether it's dishonest to hold onto a wallet. The question is about the treatment effect --- all things equal, is it more dishonest to hold onto a wallet with money vs no money? We polled nationally representative samples in the US, UK, and Poland and in all three countries most people thought so.
Amazing study, was really interesting to read. How do you get to do this studies? How does someone get into social science like this? How did you pay for all the tickets for all the cities?
