Study delivers bleak verdict on validity of psychology experiment results by Ian Sample

http://www.theguardian.com/science/2015/aug/27/study-delivers-bleak-verdict-on-validity-of-psychology-experiment-results?CMP=fb_gu

Of 100 studies published in top-ranking journals in 2008, 75% of social psychology experiments and half of cognitive studies failed the replication test

Psychology experiments are failing the replication test – for good reason

A major investigation into scores of claims made in psychology research journals has delivered a bleak verdict on the state of the science.

An international team of experts repeated 100 experiments published in top psychology journals and found that they could reproduce only 36% of original findings.

The study, which saw 270 scientists repeat experiments on five continents, was launched by psychologists in the US in response to rising concerns over the reliability of psychology research.

“There is no doubt that I would have loved for the effects to be more reproducible,” said Brian Nosek, a professor of psychology who led the study at the University of Virgina. “I am disappointed, in the sense that I think we can do better.”

“The key caution that an average reader should take away is any one study is not going to be the last word,” he added. “Science is a process of uncertainty reduction, and no one study is almost ever a definitive result on its own.”

All of the experiments the scientists repeated appeared in top ranking journals in 2008 and fell into two broad categories, namely cognitive and social psychology. Cognitive psychology is concerned with basic operations of the mind, and studies tend to look at areas such as perception, attention and memory. Social psychology looks at more social issues, such as self esteem, identity, prejudice and how people interact.

In the investigation, a whopping 75% of the social psychology experiments were not replicated, meaning that the originally reported findings vanished when other scientists repeated the experiments. Half of the cognitive psychology studies failed the same test. Details are published in the journal Science.

Even when scientists could replicate original findings, the sizes of the effects they found were on average half as big as reported first time around. That could be due to scientists leaving out data that undermined their hypotheses, and by journals accepting only the strongest claims for publication.

Despite the grim findings, Nosek said the results presented an opportunity to understand and fix the problem. “Scepticism is a core part of science and we need to embrace it. If the evidence is tentative, you should be sceptical of your evidence. We should be our own worst critics,” he told the Guardian. One initiative now underway calls for psychologists to submit their research questions and proposed methods to probe them for review before they start their experiments.

John Ioannidis, professor of health research and policy at Stanford University, said the study was impressive and that its results had been eagerly awaited by the scientific community. “Sadly, the picture it paints - a 64% failure rate even among papers published in the best journals in the field - is not very nice about the current status of psychological science in general, and for fields like social psychology it is just devastating,” he said.

But he urged people to focus on the positives. The results, he hopes, will improve research practices in psychology and across the sciences more generally, where similar problems of reproducibility have been found before. In 2005, Ioannidis published a seminal study that explained why most published research findings are false.

Marcus Munafo, a co-author on the study and professor of psychology at Bristol University, said: “I think it’s a problem across the board, because wherever people have looked, they have found similar issues.” In 2013, he published a report with Ioannidis that found serious statistical weaknesses were common in neuroscience studies.

Nosek’s study is unlikely to boost morale among psychologists, but the findings simply reflect how science works. In trying to understand how the world works, scientists must ask important questions and take risks in finding ways to try and answer them. Missteps are inevitable if scientists are not being complacent. AsAlan Kraut at the Association for Psychological Science puts it: “The only finding that will replicate 100% of the time is likely to be trite, boring and probably already known: yes, dead people can never be taught to read.”

There are many reasons why a study might not replicate. Scientists could use a slightly different method second time around, or perform the experiment under different conditions. They might fail to find the original effect by chance. None of these would negate the original finding. Another possibility is that the original result was a false positive.

Among the experiments that stood up was one that found people are equally adept at recognising pride in faces from different cultures. Another backed up a finding that revealed the brain regions activated when people were given fair offers in a financial game. One study that failed replication claimed that encouraging people to believe there was no such thing as free will made them cheat more.

Munafo said that the problem of poor reproducibility is exacerbated by the way modern science works. “If I want to get promoted or get a grant, I need to be writing lots of papers. But writing lots of papers and doing lots of small experiments isn’t the way to get one really robust right answer,” he said. “What it takes to be a successful academic is not necessarily that well aligned with what it takes to be a good scientist.”

Health Experts Say Coca-Cola is Funding its Own Science to Deliberately Mislead the Public by George Dvorsky

http://io9.com/health-experts-say-coca-cola-is-funding-its-own-science-1723404812

The Global Energy Balance Network— a research institute supported by Coca-Cola—is claiming that exercise, and not diet, is the best way to prevent weight gain. It’s a dubious and self-serving message that’s not going over well amongst diet and obesity experts.

“Most of the focus in the popular media and in the scientific press is that they’re eating too much, eating too much, eating too much, blaming fast food, blaming sugary drinks and so on,” claims Dr. Steve Blair at the Global Energy Balance Network website. “And there’s really virtually no compelling evidence that that in fact is the cause. Those of us interested in science, public health, medicine, we have to learn how to get the right information out there.”

As reported at CBC News, Blair’s extraordinary claim, along with an accompanying video, recently caught the attention of Ottawa-based obesity expert Dr. Yoni Freedhoff. During his ensuing investigation to find the origin of these claims, he discovered that the network is receiving financial and logistical support from Coca-Cola, which isn’t something that was previously disclosed. Alarmed, Freedhoff contacted Anahad O’Connor from The New York Times to get the word out.

In his ensuing article, “Coca-Cola Funds Scientists Who Shift Blame for Obesity Away From Bad Diets,” O’Connor writes:

Health experts say this message is misleading and part of an effort by Coke to deflect criticism about the role sugary drinks have played in the spread of obesity and Type 2 diabetes. They contend that the company is using the new group to convince the public that physical activity can offset a bad diet despite evidence that exercise has only minimal impact on weight compared with what people consume.

This clash over the science of obesity comes in a period of rising efforts to tax sugary drinks, remove them from schools and stop companies from marketing them to children. In the last two decades, consumption of full-calorie sodas by the average American has dropped by 25 percent.

“Coca-Cola’s sales are slipping, and there’s this huge political and public backlash against soda, with every major city trying to do something to curb consumption,” said Michele Simon, a public health lawyer. “This is a direct response to the ways that the company is losing. They’re desperate to stop the bleeding.”

Over at Scientific American, Dina Fine Maron spoke to diet and behavior expert Charlotte Markey to learn if people can lose weight with exercise alone. Here’s what Markey had to say:

I find everything going on here very troubling. In the promotional video from Coke’s group, linked to by the NYT, exercise scientist Steve Blair says we don’t know what is causing obesity and we need more research. That message is oversimplified and terribly misleading. We actually know a great deal about what leads to obesity. It’s not a great mystery. People are eating too much and not exercising enough…that makes it inevitable that people will be obese. The group’s emphasis on physical activity is misleading based on what the data shows. There’s no data to support saying if you exercise for 30 minutes three times a week that this will take care of the problem. We have data refuting that.

In reality, we need people to stop drinking sugary beverages like soda. Soda is the one consumable beverage that is repeatedly cited as having the biggest impact on obesity rates. From a public health standpoint, we want soda out of schools and we want cities to really decrease intake of soda—and Coca-Cola knows this and knows they are being proactive and defensive against taxes on soda and other limitations.

Very disturbing. This issue bears a startling resemblance to the efforts of cigarette manufacturers to deliberately mislead the public about the health risks of smoking.

Jane Goodall On The Importance Of Empathy In Science

Publishers withdraw more than 120 gibberish papers by Richard Van Noorden

Conference proceedings removed from subscription databases after scientist reveals that they were computer-generated.

The publishers Springer and IEEE are removing more than 120 papers from their subscription services after a French researcher discovered that the works were computer-generated nonsense.

Over the past two years, computer scientist Cyril Labbé of Joseph Fourier University in Grenoble, France, has catalogued computer-generated papers that made it into more than 30 published conference proceedings between 2008 and 2013. Sixteen appeared in publications by Springer, which is headquartered in Heidelberg, Germany, and more than 100 were published by the Institute of Electrical and Electronic Engineers (IEEE), based in New York. Both publishers, which were privately informed by Labbé, say that they are now removing the papers.

Among the works were, for example, a paper published as a proceeding from the 2013 International Conference on Quality, Reliability, Risk, Maintenance, and Safety Engineering, held in Chengdu, China. (The conference website says that all manuscripts are “reviewed for merits and contents”.) The authors of the paper, entitled ‘TIC: a methodology for the construction of e-commerce’, write in the abstract that they “concentrate our efforts on disproving that spreadsheets can be made knowledge-based, empathic, and compact”. (Nature News has attempted to contact the conference organizers and named authors of the paper but received no reply*; however at least some of the names belong to real people. The IEEE has now removed the paper).

*Update: One of the named authors replied to Nature News on 25 February. He said that he first learned of the article when conference organizers notified his university in December 2013; and that he does not know why he was a listed co-author on the paper. "The matter is being looked into by the related investigators," he said. 

 

How to create a nonsense paper

Labbé developed a way to automatically detect manuscripts composed by a piece of software called SCIgen, which randomly combines strings of words to produce fake computer-science papers. SCIgen was invented in 2005 by researchers at the Massachusetts Institute of Technology (MIT) in Cambridge to prove that conferences would accept meaningless papers — and, as they put it, “to maximize amusement” (see ‘Computer conference welcomes gobbledegook paper’). A related program generates random physics manuscript titles on the satirical website arXiv vs. snarXiv. SCIgen is free to download and use, and it is unclear how many people have done so, or for what purposes. SCIgen’s output has occasionally popped up at conferences, when researchers have submitted nonsense papers and then revealed the trick.

Labbé does not know why the papers were submitted — or even if the authors were aware of them. Most of the conferences took place in China, and most of the fake papers have authors with Chinese affiliations. Labbé has emailed editors and authors named in many of the papers and related conferences but received scant replies; one editor said that he did not work as a program chair at a particular conference, even though he was named as doing so, and another author claimed his paper was submitted on purpose to test out a conference, but did not respond on follow-up. Nature has not heard anything from a few enquiries.

“I wasn’t aware of the scale of the problem, but I knew it definitely happens. We do get occasional e-mails from good citizens letting us know where SCIgen papers show up,” says Jeremy Stribling, who co-wrote SCIgen when he was at MIT and now works at VMware, a software company in Palo Alto, California.
“The papers are quite easy to spot,” says Labbé, who has built a website where users can test whether papers have been created using SCIgen. His detection technique, described in a study¹ published in Scientometrics in 2012, involves searching for characteristic vocabulary generated by SCIgen. Shortly before that paper was published, Labbé informed the IEEE of 85 fake papers he had found. Monika Stickel, director of corporate communications at IEEE, says that the publisher “took immediate action to remove the papers” and “refined our processes to prevent papers not meeting our standards from being published in the future”. In December 2013, Labbé informed the IEEE of another batch of apparent SCIgen articles he had found. Last week, those were also taken down, but the web pages for the removed articles give no explanation for their absence.
Ruth Francis, UK head of communications at Springer, says that the company has contacted editors, and is trying to contact authors, about the issues surrounding the articles that are coming down. The relevant conference proceedings were peer reviewed, she confirms — making it more mystifying that the papers were accepted.
The IEEE would not say, however, whether it had contacted the authors or editors of the suspected SCIgen papers, or whether submissions for the relevant conferences were supposed to be peer reviewed. “We continue to follow strict governance guidelines for evaluating IEEE conferences and publications,” Stickel said.

 

A long history of fakes

Labbé is no stranger to fake studies. In April 2010, he used SCIgen to generate 102 fake papers by a fictional author called Ike Antkare [see pdf]. Labbé showed how easy it was to add these fake papers to the Google Scholar database, boosting Ike Antkare’s h-index, a measure of published output, to 94 — at the time, making Antkare the world's 21st most highly cited scientist. Last year, researchers at the University of Granada, Spain, added to Labbé’s work, boosting their own citation scores in Google Scholar by uploading six fake papers with long lists to their own previous work².

Labbé says that the latest discovery is merely one symptom of a “spamming war started at the heart of science” in which researchers feel pressured to rush out papers to publish as much as possible.
There is a long history of journalists and researchers getting spoof papers accepted in conferences or by journals to reveal weaknesses in academic quality controls — from a fake paper published by physicist Alan Sokal of New York University in the journal Social Text in 1996, to a sting operation by US reporter John Bohannon published in Science in 2013, in which he got more than 150 open-access journals to accept a deliberately flawed study for publication.

Labbé emphasizes that the nonsense computer science papers all appeared in subscription offerings. In his view, there is little evidence that open-access publishers — which charge fees to publish manuscripts — necessarily have less stringent peer review than subscription publishers.

Labbé adds that the nonsense papers were easy to detect using his tools, much like the plagiarism checkers that many publishers already employ. But because he could not automatically download all papers from the subscription databases, he cannot be sure that he has spotted every SCIgen-generated paper.

http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763

The changing face of psychology

 

 Psychology is championing important changes to culture and practice, including a greater emphasis on transparency, reliability, and adherence to the scientific method. Photograph: Sebastian Kaulitzki/Alamy

http://www.theguardian.com/science/head-quarters/2014/jan/24/the-changing-face-of-psychology

After 50 years of stagnation in research practices, psychology is leading reforms that will benefit all life sciences.

In 1959, an American researcher named Ted Sterling reported something disturbing. Of 294 articles published across four major psychology journals, 286 had reported positive results – that is, a staggering 97% of published papers were underpinned by statistically significant effects. Where, he wondered, were all the negative results – the less exciting or less conclusive findings? Sterling labelled this publication bias a form of malpractice. After all, getting published in science should never depend on getting the “right results”.

You might think that Sterling’s discovery would have led the psychologists of 1959 to sit up and take notice. Groups would be assembled to combat the problem, ensuring that the scientific record reflected a balanced sum of the evidence. Journal policies would be changed, incentives realigned.

Sadly, that never happened. Thirty-six years later, in 1995, Sterling took another look at the literature and found exactly the same problem – negative results were still being censored. Fifteen years after that, Daniele Fanelli from the University of Edinburgh confirmed it yet again. Publication bias had turned out to be the ultimate bad car smell, a prime example of how irrational research practices can linger on and on.

Now, finally, the tide is turning. A growing number of psychologists – particularly the younger generation – are fed up with results that don’t replicate, journals that value story-telling over truth, and an academic culture in which researchers treat data as their personal property. Psychologists are realising that major scientific advances will require us to stamp out malpractice, face our own weaknesses, and overcome the ego-driven ideals that maintain the status quo.
Here are five key developments to watch in 2014.
  
1. Replication

The problem: The best evidence for a genuine discovery is showing that independent scientists can replicate it using the same method. If it replicates repeatedly then we can use it to build better theories. If it doesn't then it belongs in the trash bin of history. This simple logic underpins all science – without replication we’d still believe in phlogiston and faster-than-light neutrinos.

In psychology, attempts to closely reproduce previous methods are rarely attempted. Psychologists tend to see such work as boring, lacking in intellectual prowess, and a waste of limited resources. Some of the most prominent psychology journals even have explicit policies against publishing replications, instead offering readers a diet of fast food: results that are novel, eye catching, and even counter-intuitive. Exciting results are fine provided they replicate. The problem is that nobody bothers to try, which litters the field with results of unknown (likely low) value.

How it’s changing: The new generation of psychologists understands that independent replication is crucial for real advancement and to earn wider credibility in science. A beautiful example of this drive is the Many Labs project led by Brian Nosek from the University of Virginia. Nosek and a team of 50 colleagues located in 36 labs worldwide sought to replicate 13 key findings in psychology, across a sample of 6,344 participants. Ten of the effects replicated successfully.

Journals are also beginning to respect the importance of replication. The prominent outlet Perspectives on Psychological Science recently launched an initiative that specifically publishes direct replications of previous studies. Meanwhile, journals such as BMC Psychology and PLOS ONE officially disown the requirement for researchers to report novel, positive findings.

2. Open Access 

The problem: Strictly speaking, most psychology research isn’t really “published” – it is printed within journals that expressly deny access to the public (unless you are willing to pay for a personal subscription or spend £30+ on a single article). Some might say this is no different to traditional book publishing, so what's the problem? But remember that the public being denied access to science is the very same public that already funds most psychology research, including the subscription fees for universities. So why, you might ask, is taxpayer-funded research invisible to the taxpayers that funded it? The answer is complicated enough to fill a 140-page government report, but the short version is that the government places the business interests of corporate publishers ahead of the public interest in accessing science.
 
How it’s changing: The open access movement is growing in size and influence. Since April 2013, all research funded by UK research councils, including psychology, must now be fully open access – freely viewable to the public. Charities such as the Wellcome Trust have similar policies. These moves help alleviate the symptoms of closed access but don’t address the root cause, which is market dominance by traditional subscription publishers. Rather than requiring journals to make articles publicly available, the research councils and charities are merely subsidising those publishers, in some cases paying them extra for open access on top of their existing subscription fees. What other business in society is paid twice for a product that it didn’t produce in the first place? It remains a mystery who, other than the publishers themselves, would call this bizarre set of circumstances a “solution”.

3. Open Science

The problem: Data sharing is crucial for science but rare in psychology. Even though ethical guidelines require authors to share data when requested, such requests are usually ignored or denied, even when coming from other psychologists. Failing to publicly share data makes it harder to do meta-analysis and easier for unscrupulous researchers to get away with fraud. The most serious fraud cases, such as Diederik Stapel, would have been caught years earlier if journals required the raw data to be published alongside research articles.

How it’s changing: Data sharing isn’t yet mandatory, but it is gradually becoming unacceptable for psychologists not to share. Evidence shows that studies which share data tend to be more accurate and less likely to make statistical errors. Public repositories such as Figshare and the Open Science Framework now make the act of sharing easy, and new journals including the Journal of Open Psychology Data have been launched specifically to provide authors with a way of publicising data sharing.

Some existing journals are also introducing rewards to encourage data sharing. Since 2014, authors who share data at the journal Psychological Science will earn an Open Data badge, printed at the top of the article. Coordinated data sharing carries all kinds of other benefits too – for instance, it allows future researchers to run meta-analysis on huge volumes of existing data, answering questions that simply can’t be tackled with smaller datasets.

4. Bigger Data

The problem: We’ve known for decades that psychology research is statistically underpowered. What this means is that even when genuine phenomena exist, most experiments don’t have sufficiently large samples to detect them. The curse of low power cuts both ways: not only is an underpowered experiment likely to miss finding water in the desert, it’s also more likely to lead us to a mirage.

How it’s changing: Psychologists are beginning to develop innovative ways to acquire larger samples. An exciting approach is Internet testing, which enables easy data collection from thousands of participants. One recent study managed to replicate 10 major effects in psychology using Amazon’s Mechanical Turk. Psychologists are also starting to work alongside organisations that already collect large amounts of useful data (and no, I don’t mean GCHQ). A great example is collaborative research with online gaming companies. Tom Stafford from the University of Sheffield recently published an extraordinary study of learning patterns in over 850,000 people by working with a game developer.

5. Limiting Researcher "Degrees of Freedom"

The problem: In psychology, discoveries tend to be statistical. This means that to test a particular hypothesis, say, about motor actions, we might measure the difference in reaction times or response accuracy between two experimental conditions. Because the measurements contain noise (or “unexplained variability”), we rely on statistical tests to provide us with a level of certainty in the outcome. This is different to other sciences where discoveries are more black and white, like finding a new rock layer or observing a supernova.

Whenever experiments rely on inferences from statistics, researchers can exploit “degrees of freedom” in the analyses to produce desirable outcomes. This might involve trying different ways of removing statistical outliers or the effect of different statistical models, and then only reporting the approach that “worked” best in producing attractive results. Just as buying all the tickets in a raffle guarantees a win, exploiting researcher degrees of freedom can guarantee a false discovery.

The reason we fall into this trap is because of incentives and human nature. As Sterling showed in 1959, psychology journals select which studies to publish not based on the methods but on the results: getting published in the most prominent, career-making journals requires researchers to obtain novel, positive, statistically significant effects. And because statistical significance is an arbitrary threshold (p<.05), researchers have every incentive to tweak their analyses until the results cross the line. These behaviours are common in psychology – a recent survey led by Leslie John from Harvard University estimated that at least 60% of psychologists selectively report analyses that “work”. In many cases such behaviour may even be unconscious.

How it’s changing: The best cure for researcher degrees of freedom is to pre-register the predictions and planned analyses of experiments before looking at the data. This approach is standard practice in medicine because it helps prevent the desires of the researcher from influencing the outcome. Among the basic life sciences, psychology is now leading the way in advancing pre-registration. The journals Cortex, Attention Perception & Psychophysics, AIMS Neuroscience and Experimental Psychology offer pre-registered articles in which peer review happens before experiments are conducted. Not only does pre-registration put the reins on researcher degrees of freedom, it also prevents journals from selecting which papers to publish based on the results.

Journals aren’t the only organisations embracing pre-registration. The Open Science Framework invites psychologists to publish their protocols, and the 2013 Declaration of Helsinki now requires public pre-registration of all human research “before recruitment of the first subject”.

Feynman on Scientific Method

Now I'm going to discuss how we would look for a new law. In general, we look for a new law by the following process: first, we guess it, no, don’t laugh, that’s the truth. Then we compute the consequences of the guess, to see what, if this is right, if this law we guessed is right, to see what it would imply and then we compare the computation results to nature or we say compare to experiment or experience, compare it directly with observations to see if it works.

If it disagrees with experiment, it’s wrong! In that simple statement is the key to science. It doesn’t make any difference how beautiful your guess is, it doesn’t make a difference how smart you are, who made the guess, or what his name is… If it disagrees with experiment, it’s wrong. That’s all there is to it.

It is therefore not unscientific to make a guess, although many people who are not in science think it is. For instance, I had a conversation about flying saucers, some years ago, with a layman — because I am scientific I know all about flying saucers! I said “I don’t think there are flying saucers”. So the other...my antagonist said, “Is it impossible that there are flying saucers? Can you prove that it’s impossible?” “No, I can’t prove it’s impossible. It’s just very unlikely”. At that he said, “You are very unscientific. If you can’t prove it impossible then how can you say that it’s unlikely?” But that is the way that is scientific. It is scientific only to say what is more likely and what less likely, and not to be proving all the time the possible and impossible. To define what I mean, I might have said to him, "Listen, I mean that from my knowledge of the world that I see around me, I think that it is much more likely that the reports of flying saucers are the results of the known irrational characteristics of terrestrial intelligence than of the unknown rational efforts of extra-terrestrial intelligence." It is just more likely. That is all, and it is a very good guess. And we always try to guess the most likely explanation, keeping in the back of our minds the fact that if it does not work, then we must discuss the other possiblities.

There was, for instance, for a while, a phenomenon called super-conductivity, there still is the phenomenon, which is that metals conduct electricity without resistance at low temperatures and it was not at first obvious that this was a consequence of the known laws with these particles. Now that it has been thought through carefully enough, it is seen in fact to be fully explainable in terms of our present knowledge.

There are other phenomena, such as extra-sensory perception, which cannot be explained by our knowledge of physics here. However, that phenomenon has not been well established, and we cannot guarantee that it is there. If it could be demonstrated, of course, that would prove that physics is incomplete, and it is therefore extremely interesting to physicists whether it is right or wrong. Many, many experiments exist which show that it doesn't work. The same goes for astrological influences. If it were true that the stars could affect the day that it was good to go to the dentist - in America we have that kind of astrology - then the physics theory would be wrong, because there is no mechanism understandable in principle from these things that would make it go. That is the reason that there is some scepticism among scientists with regard to those ideas.

Now you see of course that with this method we can disprove any definite theory. We have a definite theory, a real guess, from which you can clearly compute consequences which could be compared to experiment and in principle we can get rid of any theory. You can always prove any definite theory wrong. Notice however that we never prove it right.

Suppose you invent a good guess, calculate the consequences, and discover every time that the consequences you have calculated agree with experiment. The theory is then right? No, it is simply not proved wrong.

Another thing I must point out is that you cannot prove a vague theory wrong. If the guess that you make is poorly expressed and rather vague, and the method that you use for figuring out the consequences is a little vague —you are not sure, and you say, “I think everything’s right because it’s all due to so and so, and such and such do this and that more or less, and I can sort of explain how this works...” then you see that this theory is good, because it cannot be proved wrong! Also if the process of computing the consequences is indefinite, then with a little skill any experimental results can be made to look like the expected consequences. You are probably familiar with that in other fields. ‘A’ hates his mother. The reason is, of course, because she did not caress him or love him enough when he was a child. But if you investigate you find out that as a matter of fact she did love him very much, and everything was all right. Well then, it was because she was overindulgent when he was a child! By having a vague theory it is possible to get either result. The cure for this one is the following: if it were possible to state exactly, ahead of time, how much love is not enough, and how much love is over-indulgent, then there would be a perfectly legitimate theory against which you could make tests. It is usually said when this is pointed out--when you are dealing with psychological matters things can’t be defined so precisely. Yes, but then you cannot claim to know anything about it.

Twenty tips for interpreting scientific claims by William J. Sutherland, David Spiegelhalter & Mark Burgman

http://www.nature.com/news/policy-twenty-tips-for-interpreting-scientific-claims-1.14183

Differences and chance cause variation. The real world varies unpredictably. Science is mostly about discovering what causes the patterns we see. Why is it hotter this decade than last? Why are there more birds in some areas than others? There are many explanations for such trends, so the main challenge of research is teasing apart the importance of the process of interest (for example, the effect of climate change on bird populations) from the innumerable other sources of variation (from widespread changes, such as agricultural intensification and spread of invasive species, to local-scale processes, such as the chance events that determine births and deaths).

No measurement is exact. Practically all measurements have some error. If the measurement process were repeated, one might record a different result. In some cases, the measurement error might be large compared with real differences. Thus, if you are told that the economy grew by 0.13% last month, there is a moderate chance that it may actually have shrunk. Results should be presented with a precision that is appropriate for the associated error, to avoid implying an unjustified degree of accuracy.

Bias is rife. Experimental design or measuring devices may produce atypical results in a given direction. For example, determining voting behaviour by asking people on the street, at home or through the Internet will sample different proportions of the population, and all may give different results. Because studies that report 'statistically significant' results are more likely to be written up and published, the scientific literature tends to give an exaggerated picture of the magnitude of problems or the effectiveness of solutions. An experiment might be biased by expectations: participants provided with a treatment might assume that they will experience a difference and so might behave differently or report an effect. Researchers collecting the results can be influenced by knowing who received treatment. The ideal experiment is double-blind: neither the participants nor those collecting the data know who received what. This might be straightforward in drug trials, but it is impossible for many social studies. Confirmation bias arises when scientists find evidence for a favoured theory and then become insufficiently critical of their own results, or cease searching for contrary evidence.

Bigger is usually better for sample size. The average taken from a large number of observations will usually be more informative than the average taken from a smaller number of observations. That is, as we accumulate evidence, our knowledge improves. This is especially important when studies are clouded by substantial amounts of natural variation and measurement error. Thus, the effectiveness of a drug treatment will vary naturally between subjects. Its average efficacy can be more reliably and accurately estimated from a trial with tens of thousands of participants than from one with hundreds.

Correlation does not imply causation. It is tempting to assume that one pattern causes another. However, the correlation might be coincidental, or it might be a result of both patterns being caused by a third factor — a 'confounding' or 'lurking' variable. For example, ecologists at one time believed that poisonous algae were killing fish in estuaries; it turned out that the algae grew where fish died. The algae did not cause the deaths².

Regression to the mean can mislead. Extreme patterns in data are likely to be, at least in part, anomalies attributable to chance or error. The next count is likely to be less extreme. For example, if speed cameras are placed where there has been a spate of accidents, any reduction in the accident rate cannot be attributed to the camera; a reduction would probably have happened anyway.

Extrapolating beyond the data is risky. Patterns found within a given range do not necessarily apply outside that range. Thus, it is very difficult to predict the response of ecological systems to climate change, when the rate of change is faster than has been experienced in the evolutionary history of existing species, and when the weather extremes may be entirely new.

Beware the base-rate fallacy. The ability of an imperfect test to identify a condition depends upon the likelihood of that condition occurring (the base rate). For example, a person might have a blood test that is '99% accurate' for a rare disease and test positive, yet they might be unlikely to have the disease. If 10,001 people have the test, of whom just one has the disease, that person will almost certainly have a positive test, but so too will a further 100 people (1%) even though they do not have the disease. This type of calculation is valuable when considering any screening procedure, say for terrorists at airports.

Controls are important. A control group is dealt with in exactly the same way as the experimental group, except that the treatment is not applied. Without a control, it is difficult to determine whether a given treatment really had an effect. The control helps researchers to be reasonably sure that there are no confounding variables affecting the results. Sometimes people in trials report positive outcomes because of the context or the person providing the treatment, or even the colour of a tablet³. This underlies the importance of comparing outcomes with a control, such as a tablet without the active ingredient (a placebo).

Randomization avoids bias. Experiments should, wherever possible, allocate individuals or groups to interventions randomly. Comparing the educational achievement of children whose parents adopt a health programme with that of children of parents who do not is likely to suffer from bias (for example, better-educated families might be more likely to join the programme). A well-designed experiment would randomly select some parents to receive the programme while others do not.

Seek replication, not pseudoreplication. Results consistent across many studies, replicated on independent populations, are more likely to be solid. The results of several such experiments may be combined in a systematic review or a meta-analysis to provide an overarching view of the topic with potentially much greater statistical power than any of the individual studies. Applying an intervention to several individuals in a group, say to a class of children, might be misleading because the children will have many features in common other than the intervention. The researchers might make the mistake of 'pseudoreplication' if they generalize from these children to a wider population that does not share the same commonalities. Pseudoreplication leads to unwarranted faith in the results. Pseudoreplication of studies on the abundance of cod in the Grand Banks in Newfoundland, Canada, for example, contributed to the collapse of what was once the largest cod fishery in the world⁴.

Scientists are human. Scientists have a vested interest in promoting their work, often for status and further research funding, although sometimes for direct financial gain. This can lead to selective reporting of results and occasionally, exaggeration. Peer review is not infallible: journal editors might favour positive findings and newsworthiness. Multiple, independent sources of evidence and replication are much more convincing.

Significance is significant. Expressed as P, statistical significance is a measure of how likely a result is to occur by chance. Thus P = 0.01 means there is a 1-in-100 probability that what looks like an effect of the treatment could have occurred randomly, and in truth there was no effect at all. Typically, scientists report results as significant when the P-value of the test is less than 0.05 (1 in 20).

Separate no effect from non-significance. The lack of a statistically significant result (say a P-value > 0.05) does not mean that there was no underlying effect: it means that no effect was detected. A small study may not have the power to detect a real difference. For example, tests of cotton and potato crops that were genetically modified to produce a toxin to protect them from damaging insects suggested that there were no adverse effects on beneficial insects such as pollinators. Yet none of the experiments had large enough sample sizes to detect impacts on beneficial species had there been any⁵.

Effect size matters. Small responses are less likely to be detected. A study with many replicates might result in a statistically significant result but have a small effect size (and so, perhaps, be unimportant). The importance of an effect size is a biological, physical or social question, and not a statistical one. In the 1990s, the editor of the US journal Epidemiology asked authors to stop using statistical significance in submitted manuscripts because authors were routinely misinterpreting the meaning of significance tests, resulting in ineffective or misguided recommendations for public-health policy⁶.

Study relevance limits generalizations. The relevance of a study depends on how much the conditions under which it is done resemble the conditions of the issue under consideration. For example, there are limits to the generalizations that one can make from animal or laboratory experiments to humans.

Feelings influence risk perception. Broadly, risk can be thought of as the likelihood of an event occurring in some time frame, multiplied by the consequences should the event occur. People's risk perception is influenced disproportionately by many things, including the rarity of the event, how much control they believe they have, the adverseness of the outcomes, and whether the risk is voluntarily or not. For example, people in the United States underestimate the risks associated with having a handgun at home by 100-fold, and overestimate the risks of living close to a nuclear reactor by 10-fold⁷.

Dependencies change the risks. It is possible to calculate the consequences of individual events, such as an extreme tide, heavy rainfall and key workers being absent. However, if the events are interrelated, (for example a storm causes a high tide, or heavy rain prevents workers from accessing the site) then the probability of their co-occurrence is much higher than might be expected⁸. The assurance by credit-rating agencies that groups of subprime mortgages had an exceedingly low risk of defaulting together was a major element in the 2008 collapse of the credit markets.

Data can be dredged or cherry picked. Evidence can be arranged to support one point of view. To interpret an apparent association between consumption of yoghurt during pregnancy and subsequent asthma in offspring⁹, one would need to know whether the authors set out to test this sole hypothesis, or happened across this finding in a huge data set. By contrast, the evidence for the Higgs boson specifically accounted for how hard researchers had to look for it — the 'look-elsewhere effect'. The question to ask is: 'What am I not being told?'

Extreme measurements may mislead. Any collation of measures (the effectiveness of a given school, say) will show variability owing to differences in innate ability (teacher competence), plus sampling (children might by chance be an atypical sample with complications), plus bias (the school might be in an area where people are unusually unhealthy), plus measurement error (outcomes might be measured in different ways for different schools). However, the resulting variation is typically interpreted only as differences in innate ability, ignoring the other sources. This becomes problematic with statements describing an extreme outcome ('the pass rate doubled') or comparing the magnitude of the extreme with the mean ('the pass rate in school x is three times the national average') or the range ('there is an x-fold difference between the highest- and lowest-performing schools'). League tables, in particular, are rarely reliable summaries of performance.

Nature 503, 335–337 (21 November 2013)

References

1. Doubleday, R. & Wilsdon, J. Nature 485, 301–302 (2012).
2. Borsuk, M. E., Stow, C. A. & Reckhow, K. H. J. Water Res. Plan. Manage. 129, 271–282 (2003).
3. Huskisson, E. C. Br. Med. J. 4, 196–200 (1974)
4. Millar, R. B. & Anderson, M. J. Fish. Res. 70, 397–407 (2004).
5. Marvier, M. Ecol. Appl. 12, 1119–1124 (2002).
6. Fidler, F., Cumming, G., Burgman, M., Thomason, N. J. Socio-Economics 33, 615–630 (2004).
7. Fischhoff, B., Slovic, P. & Lichtenstein, S. Am. Stat. 36, 240–255 (1982).
8. Billinton, R. & Allan, R. N. Reliability Evaluation of Power Systems (Plenum, 1984).
   
9. Maslova, E., Halldorsson, T. I., Strøm, M., Olsen, S. F. J. Nutr. Sci. 1, e5 (2012).

Why is this pioneering researcher fighting to have his work retracted?

http://io9.com/why-is-this-pioneering-researcher-fighting-to-have-his-493704664

When Rutgers evolutionary biologist Robert Trivers began to suspect that his co-author, William Brown, had faked the data on a widely circulated study, he was placed in the unenviable position of bringing the fraud to light. It has not been easy.

Trivers first began to suspect that something was amiss in 2007, two years after the study – which found Jamaican teens with a high degree of body symmetry were more likely to be rated "good dancers" by their peers – had been featured on the cover of Nature. As Nature News' Eugenie Samuel Reich reports, Trivers has been fighting since 2008 to have the results withdrawn from the scientific literature, at the occasional expense of his reputation: 

In seeking a retraction, Trivers self-published The Anatomy of a Fraud, a small book detailing what he saw as evidence of data fabrication. Later, Trivers had a verbal altercation over the matter with a close colleague and was temporarily banned from campus.

An investigation of the case, completed by Rutgers and released publicly last month, now seems to validate Trivers’ allegations. Brown disputes the university’s finding, but it could help to clear the controversy that has clouded Trivers’ reputation as the author of several pioneering papers in the 1970s. For example, Trivers advanced an influential theory of ‘reciprocal altruism’, in which people behave unselfishly and hope that they will later be rewarded for their good deeds. He also analysed human sexuality in terms of the investments that mothers and fathers each make in child-rearing.

Steven Pinker, a psychologist at Harvard University in Cambridge, Massachusetts, calls the dancing paper “a lark” and “journalist bait” that lacks a firm basis in theory. “It was cute rather than deep,” he says. But he describes Trivers’ earlier work as “monumental”, and says that it would be a travesty if Trivers became known for one controversial study rather than his wider contributions to evolutionary biology. “Trivers is one of the most important thinkers in the history of the biological and social sciences,” Pinker says.

Scientificamerican.com: Research in the Digital Age: It’s More Than Finding Information… By Jody Passanisi and Shara Peters

http://blogs.scientificamerican.com/guest-blog/2013/05/02/research-in-the-digital-age-its-more-than-finding-information/

The Role of Research in the Digital Age 

We all know that the Internet has led to an explosion of available information. When students search for information about a topic, they are met with a plethora of articles, from both credible and non-credible resources. The skill of research has always been considered to be a pillar of the social studies discipline, though the nature of research itself has been rapidly changing as the Internet develops and our society becomes less dependant on paper-bound books. As social studies teachers, it is our job to be cognizant of how these changes are having an impact on our discipline.

The Encyclopedia

Gone are the days of consulting the ever-trustworthy Encyclopedia Britannica; there used to be an inherent trust we could have that the information we found was the most relevant to our query, was presented in a (relatively) unbiased way, and was accurate. Now, finding the information is only a small fraction of the challenge of research. Students must now discern if the source they found contains accurate, factual, and documented information. Once they have done that, they must determine what the purpose of their source is, and whether or not it is presenting the information in a significantly skewed manner. This skill set is commonly found as part of university-level history curriculum, but now students as young as 4th grade need to begin developing this proficiency.

The Value of a Website

After receiving too many research papers that relied solely on Wikipedia, we realized that these skills needed to be explicitly taught, and that they needed to be developed in our social studies class. When looking for previously published curricula about Internet skills, we found Common Sense Media’s Test Before You Trust materials, which were exactly what we were looking for. They guide students through asking tough questions about each source: Is the bias readily apparent? Who paid for the website? How many sources are cited for their information? Thanks to this material, students can at least ask the right questions about the online source.


In order to have the skills to evaluate a source found on the Internet, we need to not only teach tools to do this –like those found in Common Sense Media’s Test Before You Trust materials– but we need to teach how to evaluate the perspective of the sources students read, and to students even younger than before. In other words: we need to teach about bias.

Perspective

Obviously, before the advent of the Internet, historians wrote from particular perspectives. The perspective of the author of a primary source was written from the perspective of  personal experience. The letters of Abigail Adams reflect her perspective on politics, women’s  rights, and slavery in a different way from the writings of Thomas Jefferson.  Throughout history, historians have looked at events through the lens of their own biases– their writings are colored by their politics, culture, and experience. Also, the availability of certain information to those historians limited what they could and couldn’t write about.  It wasn’t as often though, when we were in middle school, that students encountered a secondary source or tertiary source beyond the encyclopedia–so teaching about bias wasn’t as necessary.

Instead now, secondary and tertiary sources on the Internet can be found by anyone and written by anyone–evaluating the bias of the source plays an important part in evaluating whether the site is useful. Since the Internet is not peer reviewed like academic journals, students are going to have to do the evaluation themselves. We teach our history students to evaluate bias by reading two different sources writing from different perspectives on the same historical event. Students find the details in the text that help shed light on what a source’s perspective is. Students find telling adjectives, figure out what information is included, what is omitted. Everything is data.

Analysis and Evaluation in Social Studies Research

The tools used for detecting the bias of a source, and the critical thinking skills they require, must become part of social studies curriculum, and earlier now than ever before. However,  critical thinking skills of evaluation and analysis that are required to detect bias aren’t necessarily developed until students reach the formative operations stage described by Piaget. While the seeds of perspective analysis need to be planted early, some students may not yet be developmentally ready for learning how to discern on their own. To assist them, there are tools to help sort through the vast amount of resources available. For example, search engines like SweetSearch only display results appropriate for students (though that doesn’t mean the sites they find are without bias).

Today, people are not necessarily considered knowledgeable based on how much information they know, but by how much facility they have with that information. As teachers in the discipline of history we have to own the idea that teaching students how to analyze and evaluate the information they find is more important than gathering that information together in one place. We ask our students to research, but it is not simply about finding information anymore. Students will need to sift through multiple perspectives on the Internet, and ultimately decide which perspectives are valuable and useful for their purpose. As social studies teachers, we have to show them HOW to research.

From idea to science: Knowing when you’ve got a good idea

http://arstechnica.com/science/2013/03/from-idea-to-science-the-human-factor/

by Aurich Lawson

One of the great untold stories in science is the process of science itself. I don't mean stories about what scientists have discovered and what that discovery tells us; we (and many others) cover those every day. I also don't mean stories about the pure joy of discovery and the excitement of finding out that everything you thought you understood was total bollocks. We cover that here at Ars occasionally, and there are plenty of books on it if you're hungry for more.
What's missing is the background for these stories of discovery. How do you take an idea from its very beginning as a casual musing through to an actual research program? What's involved in that process? How do you sort out good ideas from bad and choose what to pursue and what to abandon? That is the story that I want to tell.
Since this is the story of science-as-a-process rather than science-as-a-result, I will be using myself as an example. I am, as some of you may know, a tenure track faculty member at a research institute in the Netherlands. Being a researcher in the Netherlands is not that different from being a researcher anywhere else, so a lot of what I discuss will be familiar to scientists everywhere. Since I recently hopped on the tenure track, I have the next few years to prove that I am able to not only carry out research, but to start and manage entire research programs. And, as yet, I have no research program to manage.
What will follow is a series of posts that document my success or failure in this particular endeavor. It will not be a blog as such; instead, I am aiming to give you a flavor of what goes through our minds when we come up with an idea, and what happens afterwards. It's not enough just to have an idea—it has to meet all sorts of criteria, only some of which have anything to do with science. As such, we have to refine and structure an idea into something that could become a coherent body of research. Then we have to convince other people that it's a good idea.
This article will mainly be background: what does it mean to be a physics researcher in the Netherlands? What conditions do I have to meet? What sort of time-scale are we talking about in terms of viable ideas? As for the rest of the series:

• The next post will discuss some of the specifics of the research program I want to build, the sort of physics that gets me out of bed in the morning.
• The third post will be about getting the resources I need to carry out that research. How do I sell my idea, and to whom?
• The fourth post will eventually describe the outcome of my salesmanship, and the perpetual question: what next?

There will be quite a delay between the latter posts in the series (it takes a while for grants to be evaluated). And, if I have not failed in the most abject manner possible, the process may continue from there. Success never ensues immediately after the money arrives.

The human factor

To understand how I choose between good ideas and bad ideas, we need to step back from actual physics and science and take a look at the structure of research community that I work in. Research takes resources. I don't mean money—all right, I do mean money—but it also requires time and people and lab space and support. There is a human and physical infrastructure that I have to make use of. I may be part of a research organization, but I have no automatic right of access to any of this infrastructure.
In the Netherlands, doctoral candidates are not students (nor are they, as many think of students, free labor). Instead, they are full-time staff with four-year contracts. What does this mean? First, it is very difficult to organically grow an idea from small scale research projects into something larger that has a doctoral student attached to it. The timing just doesn't work out with that four-year limit. It is difficult to begin a project with a masters or undergraduate student who will "just take a look" and then hand that over seamlessly to a PhD student should it appear promising.
This also has implications for scale. A PhD student has the right to expect a project that generates a decent body of work within those four years. A project that is going to take eight years of construction work before it produces any scientific results cannot and should not be built by a PhD student. On the other hand, a project that dries up in two years is equally bad. In other words, no matter what idea I come up with, I need to be able to say that all the candidates I hire should find enough material to write a thesis and graduate—no matter what the experimental outcome.
This means that any big idea I come up with also needs to be partitioned into chunks of the right size. If it can't, then it doesn't work in an academic institution.
Since all experimental results need to be thesis-worthy, the questions I want to answer should be open enough to accommodate failure. For instance, my ideas are often based on a single experiment: if we conduct experiment "a," we could measure property "b," and that would be so cool! But, what if "a" doesn't work? Does the student go home?
So, the core idea also needs to be structured so, should certain experiments not work, they still build something that can lead to experiments which do work. Or, if the cool new instrument we want to build can't measure exactly what I intended, there are other things it can measure. One of those other things must be fairly certain of success.
To put it bluntly: all paths must lead to results of some form.

A ticking clock

So I owe it to my students to come up with research ideas that will generate some success in the right time frame  But there's another human side—mine. As a tenure tracker, time is a big boundary condition. If I choose to forego PhD students, I could come up with an idea that involves eight years of construction before the first results might be expected (and first students hired). That would be acceptable in terms of meeting their requirements.
Unfortunately, my time would be up by then. Part way through the eight years, the director of the institute would look at my performance and promptly tell me to seek work elsewhere. The tenure track is there to give researchers a limited time to prove that they can do everything a tenured researcher should be able to. I must succeed with medium scale projects or many small scale projects rather than a big, long-term project.
That doesn't mean, however, these projects can't be pieces of some longer-term big project. I simply have to ensure that the project delivers results at all time-scales. In my case, the projects should be in the one-to-six PhD student range and should not require more than a year of instrumentation building (unless building the instrument can be counted as doing science). It should, from today, deliver lots of results within a four-year period. Or I'll be looking for another job.

Fitting in

Finally, there are institutional goals and resources. At the moment, I am in an institute that is going through a major restructuring and relocation. The institute will split up, with pieces moving to three different universities. Each piece will have a very different focus: energy research at Eindhoven, soft X-Ray optics at the University of Twente, and a free electron laser facility at University of Nijmegen. At the moment, I am part of the soft X-Ray optics group, so, my research should fit within that theme. On the other hand, as a tenure track researcher, I need to demonstrate some independence. My research still needs to be distinct from what the group already does.
These considerations, which are largely political in nature, are surprisingly important. They make the difference between enthusiastic institutional support (above and beyond what you are entitled to) and grudging assistance, limited to exactly what you are entitled to and delivered on someone else's schedule. In other words, the enthusiasm with which an institute supports a research program may very well be the difference between success and failure.
In the next installment, I will talk about the idea that I am planning: how it originated, and how I expanded it into a program that meets the requirements outlined in this post. As you will see, the raw idea, as expressed in its original form, doesn't fit well to the goals of the research group that I am a part of. My job became making it fit.

Getting a paper accepted

Do objects of different masses really fall at the same rate? The Nordtvedt effect posits they don’t

http://io9.com/5980308/do-objects-of-different-masses-really-fall-at-the-same-rate-the-nordtvedt-effect-posits-they-dont

by Esther Inglis-Arkell

It's been demonstrated since the 1500s that, when falling toward a certain body, objects fall at the same rate. Everyone from Galileo in Pisa to David Scott on the moon demonstrated that. But what if they're wrong? The Nordtvedt effect posits exactly that.

One of the most famous science legends has Galileo dropping two different-sized cannonballs off the leaning tower of Pisa demonstrating that objects of different mass fall at the same rate. Actually, he rolled two balls down ramps to demonstrate the effect and the tower had nothing to do with it. At the end of the Apollo 15 mission, commander David Scott dropped a hammer and a feather on the moon - where there is no air resistance - to prove the same thing. The mass of the object falling doesn't matter. What matters is the mass of the thing making the object fall. Whether a planet is grabbing a cannonball or a feather, the object falls at the same rate. This is called the equivalence principle, and has been held as scientific truth for about four hundred years.

Just to show that no physics principle is sacred, Professor Kenneth Nordtvedt of Montana State University proposed the idea that objects fall at different rates due to their mass. Or actually, he outlined exactly what we'd see happening between the moon, the Earth, and the sun if an object's mass was taken into account in the gravitational pull between it and the body it was orbiting. The Earth, being the more massive orbiting body, would fall towards the sun at a faster rate than the moon.

The effect justifies this by playing with three different concepts. The first concept is gravitational self-energy. This is roughly the idea that all the little pieces of an object have an effect on each other. A solid ball, for example, could be carved up into little shells inside each other like Russian nesting dolls, all of which are pulling at each other. This energy would be larger for big objects than for small objects.

The second two concepts are two different views of mass. There's inertial mass. Imagine an object is on perfectly greased wheels on a perfectly smooth, level floor. If you were to reach out and push it, you would have to exert enough force to move its inertial mass. Then there's gravitational mass. Imagine you now have to pick up that object against the pull of gravity. Outwardly, this seems like a harder task. I could easily push a large friend along on, for example, a wheeled office chair. I'd have a hard time picking them (and the chair) up off the ground. I'd have to use more force. But the difference in perceived mass is just because, when you lift, you're working against a force. Earth's gravity is pulling down. We know the force of Earth's gravity, and we know the mass of the Earth - take those away and I'd be using the same force pulling the person up into my arms as I would pushing them across a level floor. And if I'm using the same force - I must be moving the same mass. In other words, the inertial mass (mass of an object floating in space) and the gravitational mass (mass of an object sitting on the Earth), are the same. It's just the gravity of the Earth that's making the difference.

Every experiment has found these two masses to be identical, but Nordtvedt's idea of gravitational self-energy might change that. He posited that all those little pieces pulling at each other with their gravity might contribute to an object's gravitational mass, and not its inertial mass. Since a larger object has more mass, it would have more gravitational mass. So now, even accounting for the pull of gravity, more force is being exerted to lift a mass than to push it. And the bigger the mass is, the bigger the gap between the force required to lift it and the force required to shove it is. So the Earth is exerting more force on bigger objects than smaller ones, and bigger objects fall faster.

At least that's the idea. The Nordtvedt effect has been tested, and so far no evidence has been found that the more massive Earth is falling towards the sun faster than the moon. If there is an effect, it's very slight. But if it's there, everything we know about motion, and even relativity, changes. Wouldn't that be cool?

Introducing “The Nine Circles of Scientific Hell” — a real figure in a real scientific journal

This figure is the only figure in an article entitled "The Nine Circles of Scientific Hell" — an article that seeks to "take a humorous look at the fate that awaits scientists who sin against best practice," and was just published in Perspectives on Psychological Science. To be fair, the article is supposed to be funny, so whether it can official contend for the Most Amusing title with journal figures from more "serious" academic papers is up for debate

http://io9.com/5959152/introducing-the-nine-circles-of-scientific-hell--a-real-figure-in-a-real-scientific-journal