Scientific journals should insist on more robust experimental processes, say biologists after reviewing nearly 900,000 experiments
The team found that non-blind experiments - that is, where scientists knew which samples they were recording - averaged a 27 per cent stronger result than blind trials.
However their review suggests that less than one in four experiments used blind data recording.
"We found that non-blind papers tended to exaggerate differences between the experimental group and the control group," said lead researcher Dr Luke Holman, from the Research School of Biology at The Australian National University (ANU).
"For example, a non-blind trial of a new drug might conclude that it is way more effective than a placebo, when in fact the drug's true effect is rather modest, simply because the researchers' expectations biased the results."
The paper is published in PLOS Biology at a time when experimental processes are under the microscope following increased levels of retractions and some journals are reviewing their peer review procedures.
In the largest study of its kind, the team analysed nearly 900,000 papers from the PubMed life sciences database, using automated 'data mining'. They also - in a blind trial, of course - compared 83 pairs of evolutionary biology papers on similar topics, in which the data was collected blind in one, and not in the other.
The team also found that non-blind studies rejected the null hypothesis more strongly, said Dr Holman.
"Non-blind studies more confidently concluded that differences between treatment and control groups were real, and not just due to chance variation."
Co-researcher Dr Megan Head, also from ANU Research School of Biology said self-reflection is important.
"Science is still the best method we have for understanding the world, and we have to keep working to make it better."
Dr Holman and his colleagues believe that journals should insist on blind trials more strongly, perhaps by making prominent statements to authors and peer reviewers about the necessity of using blind trials.
Scientists are aware of their biases, and use techniques such as blind trials to minimise them, but the pressure to get things done faster leads to some people skimping on experimental design, said Dr Head.
"It is not necessarily slower to take data blind, you just need to be a little creative," she said.
Dr Holman suggested better training is part of the solution.
"Many researchers are unaware that their expectations can introduce such strong bias, and so they don't feel the need to work blind," he said.