How social science ends up as urban myth

[snopes]

Turn up the lights, and the workers work harder. Turn them down again, and they work harder still. The “Hawthorne Effect” is named after Western Electric’s titanic Hawthorne Works in Cicero near Chicago, where a series of productivity trials was carried out between 1924 and 1932. Led by Elton Mayo, a professor at Harvard Business School, they are among the most famous experiments in social science. Not every social scientist is impressed.

Richard Nisbett, a social psychologist at the University of Michigan, complained to The New York Times a decade ago about the study’s fame, calling it a “glorified anecdote”. He had a point. Among managers, the study is generally held to demonstrate that people respond to change: whatever you do, output rises for a while, as long as you do something. Inside academia, “the Hawthorne Effect” refers to the idea that people work hard once you start experimenting on them. Both beliefs are surprising enough to be interesting, while nicely confirming the prejudices of those who hold them.

Interested psychologists have known for a while that all was not well with the study.

http://www.ft.com/cms/s/2/6d1a313e-5...44feabdc0.html

[ganzfeld]

That was fascinating. I have to take issue with the last couple sentences:

Trying to replicate old results is rarely regarded as social science worth publishing in the top journals. That must change.

"Becoming a myth" or becoming accepted fact based on one or a few studies is just one way it happens. It can also be the case that the same or very similar experiments are replicated to get similar results but are all based on a false premise or miss some important alternative hypothesis. So I don't think simply replicating an old experiment should be encouraged at all. Instead, different results that lead to similar (or different) conclusions should be encouraged. I don't know if journals are the problem. As far as I know, they still do publish plenty of that kind of work, especially if it convincingly refutes old conclusions. I think researchers themselves are more to blame for the same misunderstanding represented by this quote from the article -- that journals simply won't publish such results. (Or rereading it, maybe it isn't blaming the journals exactly. That was my first interpretation. That darn passive tense!) They will if the method used is novel or the results show a new way of looking at the older results.

[Jahungo]

I also object to this statement:

Trying to replicate old results is rarely regarded as social science worth publishing in the top journals. That must change.

A lot of Ganzfeld's reasons were good ones. But also there's the plain and simple fact that uncreatively replicating previous studies is not exciting, influential research in any way and simply does not belong in top journals (unless perhaps an unexpected result arises). Looking at the same issue in a different way may be.

[ganzfeld]

Originally Posted by Jahungo

But also there's the plain and simple fact that uncreatively replicating previous studies is not exciting, influential research in any way and simply does not belong in top journals (unless perhaps an unexpected result arises).

Agreed. How about just encouraging graduate students to do such replication with a critical eye? If they find something interesting or develop a new method then take it to the next level and publish if it turns out to be worth it.