If you’re a fan of “Why Most Published Research Findings Are False” (2005; Ioannidis), you’ll be interested to know that Professor Ioannidis’s work has in fact been reproduced. See “Many Psychology Findings Not as Strong as Claimed, Study Says” (nytimes) and “Estimating the reproducibility of psychological science” (the paper; full text available).
One of my MIT graduate school classmates commented on this article: “When there were only a couple of climate scientists, they were probably reasonably good. Now that it is a popular discipline, they are approaching in quality the average of the population… And that is without taking fads and funding pressure into account.”
See also: clinical trials. 57% of those published before 2000 reported positive effects from the treatments, but that figure plunged to just 8% in studies that were conducted after 2000. Since 2000 researchers must state their trial methods and outcome measures before collecting data, preventing cherry-picking once the study is over.
http://www.nature.com/news/registered-clinical-trials-make-positive-findings-vanish-1.18181
Regarding the “quality” of climate scientists, listen to what MIT professor Richard Lindzen had to say about it in front of the British House of Commons Energy and Climage Change committee (the following URL should start the video at 2:49:16; if it doesn’t, drag the playhead to that point; the following ~3 minutes is the relevant portion of his comments):
https://youtu.be/6GzNATrGH7I?t=2h49m16s
There was an article about aout pre-clinical cancer research, where only 11% of the published studies (in big journals like Nature, Science, Cell, etc) were reproducible. It was roughly the same for some other therapeutic areas as well. Many ‘landmark’ molecular biology papers are basically rubbish.
The scary thing is that these results are being mined for data analysis (eg. gene sets describing molecular pathway interactions) just by the mere mention of two genes interacting. With such low reproducibility, we will be running in circles for years.
The problem with science publication is that the people who do the best science (ie. the ones who just present the facts) get pushed out by those who are better at writing neat little stories (ie the story tellers). Stories are great for human comprehension, but then it’s easy for science to turn into science fiction.
@GermanL: “The problem with science publication … who are better at writing neat little stories”. That and the fact they use and rely on theories in their studies and publication.
Real scientific advancements are few and far between, but labs need funding every day and you aren’t going to get funding unless your lab publishes results and those results had better not be “We were unable to produce any statistically significant findings.” So the whole industry depends on publishing stuff which is mostly meaningless.
For any given advance there are generally many failures which contributed to the advance in the sense that someone read about the failure and thought “well that didn’t work, lets try this…”. We don’t properly account for those contributions made by folks whose experiments didn’t succeed providing an incentive to cherry pick and exaggerate results.