The Reproducibility Problem

For some years now psychologists have been talking about the Reproducibility Problem. This is where a team of researchers publish a paper that shows some new and interesting result but when another team uses the same method to try to reproduce the result they cannot detect the effect. This is significant because one paper is never proof of anything, one paper can always be an anomaly. The ability to faithfully reproduce an effect is one of the cornerstones of the scientific method and knowing that what we see is a real effect.

An open access article in Science this week reports on a large collaborative effort to try to put some flesh on the bones of this problem. Nosek et al attempted to replicate 100 wide ranging psychological studies performed in 2008 published in three different, high quality psychological journals. They copied the methods of the original studies exactly and even used the same materials where available.

They found that 97% of the original studies had a statistically significant result but that only 36% of replications did; also, the mean effect size more than halved in the replications. This all ties in with what I was talking about in a recent post on P-Hacking, the phenomenon consciously or otherwise of massaging your results into significance. Although this study focussed on psychology I don't think it is unreasonable to assume that many fields of science suffer from a similar problem to some degree. It is a symptom of being rewarded for new and innovative research but not for reliable reproduction of existing research. To reiterate my point from my previous post, I don't think that this means science is broken, but it does mean that it is inefficient.

A large part of the blame lies with journal editors who aren't interested in publishing replications, but also with those in charge of allocating grant money and in Principal Investigators who want to up their research impact. I'm afraid that, at the end of the day, a lot of science is simply not new and sexy, it is about meticulous, hard graft that won't set the world afire but will, slowly, push back the boundaries of our knowledge. Nosek sums up the situation well in his paper:

Reproducibility is not well understood because the incentives for individual scientists prioritize novelty over replication. Innovation is the engine of discovery and is vital for a productive, effective scientific enterprise. However, innovative ideas become old news fast. Journal reviewers and editors may dismiss a new test of a published idea as unoriginal. The claim that “we already know this” belies the uncertainty of scientific evidence. Innovation points out paths that are possible; replication points out paths that are likely; progress relies on both. Replication can increase certainty when findings are reproduced and promote innovation when they are not. This project provides accumulating evidence for many findings in psychological research and suggests that there is still more work to do to verify whether we know what we think we know.

Image used with permission from here

Universal Flu Vaccine: A Step Forward

Great news! We have made another step in the direction of creating a universal flu vaccine. This has always been a goal but an elusive one. For many years now we have had to make an educated guess each year as to what the most prevalent strain of flu will be in the upcoming season, production of a vaccine for that one strain is then ramped up through the summer to try to ensure a good supply for the winter. Remember, each single dose of flu vaccine is created inside a chicken egg. Think how many millions of eggs must be needed each year just to help us stop getting flu!

Anyway, sometimes our best guess isn't as good as it could have been, plus there is no protection from the many other strains of flu that may or may not be doing the rounds at any given time; so it's an imperfect system. Then why haven't we made a universal vaccine before? Well, let's picture an influenza virus. Imagine a ball with lots of lollypops sticking out of it. The sweet, delicious head of the lollipop is the part that can change and vary and evolve and is what makes them so darned hard for our immune systems to recognise and adapt to. If you've heard terms like H5N1 and H3N2 then these refer to this part. The stick part of the lollipop (the haemagglutinin stem) is unchanging and seems to be the same in every strain of flu so far studied. It would seem sensible to create a vaccine that targets this unchanging stalk component rather than the ever evolving blobby head, but this has never been possible because the stalk is physically difficult to access. Imagine flying over a dense forest canopy and trying to find a way to hack at a tree trunk.

In the past some clever people have found a way to decapitate the stalk and so free it up for attack, but it turned out that the stalk then sort of just fell apart leaving nothing for our immune systems to attack. In a new paper in Science a team from Leiden, Germany, detailed how they were able to decapitate the stalk but then modify it so that it was stable thereby leaving a target. Then they created an antibody that is able to recognise the stem and provoke an immune response and protection from the virus.

Mice infected with the lethal H5N1 strain of influenza were fully protected. Macaques still had a bout of the flu but the symptoms were much less severe than normal. The concept, then, has been proved. The next stage will be human trials, but these typically take several years to perform so we are still someway from a universal vaccine for ourselves. It's a prize worth waiting for, though; half a million people generally die from flu each year, and that's in a good year. Last year a mutation in the H3N2 strain, which the vaccine was targeted against, resulted in a vaccine that provided little protection and a commensurate spike in deaths. Perhaps in a decade the flu vaccine will be a part of the childhood vaccination program and that will be all we need for the rest of our lives, that would be a truly wonderful achievement.

Three dimensional influenza structure; image used with permission

3D Hologram Could Map Black Hole Contents says Stephen Hawking

Stephen Hawking has been at it again, being all controversial and awesome. At a meeting of the Royal Institute of Technology in Stockholm he has given a talk that may go someway to explaining the Information Paradox of black holes.

In the 1960s Hawking became famous for proving the existence of black holes and that nothing, not even light, is capable of escaping their vast gravitational pull. In the 1970s he became even more famous for disproving his earlier proof. He then contended that something could indeed escape a black hole, a type of radiation that went on to bear his own name. The idea that black holes can evaporate away via Hawking radiation is one of the concepts behind the Heat Death of the universe, the theory that the universe will 'end' by merely ceasing to contain a single atom of anything several trillion years from now becoming a vast expanse of uniform, cold nothingness.

There is a problem, though; if black holes irrevocably swallow up matter and energy then the information, or properties, of that matter and energy are lost and this violates key tenets of quantum theory which says that information must be preserved. Physicists and mathematicians studying black holes have spent the last several decades trying to find a way around this conundrum.

Hawking, in collaboration with University of Cambridge physicist Malcolm Perry and Harvard theorist Andrew Strominger, have proposed a potential mechanism. Like many things on the cutting edge of physics it sounds a bit abstract and difficult to follow but, who knows, maybe this will be high school level knowledge a decade from now. As far as I understand it, which is by no means fully, they are suggesting that as an object, let's say Iain Duncan Smith, passes the event horizon of a back hole, a record of his black heart and other wretched components will be forever preserved in the form of a three dimensional hologram. The hologram consists of two space dimensions and one time dimension and would need to be capable of storing enough information to theoretically map out the innards of the black hole and everything it has consumed.

They go on to claim that as Hawking radiation leaves the black hole it would somehow pick up a bit of the hologram data and that way information would be preserved and the laws of physics remain intact. However, the data would come out in a random, unreadable format thereby also preserving the idea that nothing useful can escape a black hole. An analogy was made to burning an encyclopedia; if you were to take an encyclopedia, you know, a paper one, and set fire to it and then keep all the ashes then technically you could say you still have all of the information it once contained, but practically speaking there's no way you could actually retrieve it.

I know, it sounds like an imaginative fudge to try and escape a tight spot, only time will tell. Indeed, it's difficult to say anything a this point as we only have a speech to go on, it will likely be some months before an actual paper is published for proper analysis. Once that is released it will no doubt get the proper scrutiny that such an idea warrants. Give it another decade or two and we might have it figured out, but we'll get there in the end.

Image used with permission

Another Nail In The Coffin of Homeopathy

Great news! A recently published judicial review of the decision by the Lothian Health Board to defund homeopathy has given a sound slap in the face to anyone still left out there that thinks homeopathy has a shred of credibility left.

To give a little background; in January 2012 the Lothian Health Board conducted a review of the homeopathy service it was providing and concluded that the service was not worth funding any further. Any sensible person would immediately realise that this was because it would be unethical to provide public money to fund a 'treatment' based solely on pre-Victorian era magic. However, an idiot by the name of Honor Watt decided to challenge the decision by instigating a judicial review. She was aided by an undisclosed charity, the word on the street is that it was the British Homeopathic Association (BHA).

Even hardcore homeopaths know that they can't win an argument on the science, witchcraft doesn't stand up well to logical arguments, so the tactic used was to say that the decision made by the health board was discriminatory. They didn't say which particular characteristic had been discriminated against so I'm going to go right ahead and assume it's stupidity. Now, whenever an NHS body makes any kind of decision like this it is obliged to carry out an impact assessment with regard to potential discrimination. They had tried to gather as much demographic information as possible before the decision was taken but this was somewhat hampered by the fact that the homeopathy clinics didn't bother collecting even the most basic information about their patients, a highly unusual practice when one is supposedly performing a medical intervention. Anyhoo, on the information available they concluded that no one group of people would be unduly affected.

The killer blow came at the end of the judgement (paragraph 32). The judge, Lord Uist, said:

In any event, even if I had concluded that the Board had failed to comply with its PSED, I would have refused to reduce the decision under review.  It is plain that the Board, as it was entitled to do, accepted the view that there was no scientific evidence for the efficacy of homeopathy and that funding for it was a waste of the limited funds at its disposal.  In these circumstances the countervailing factor in this case was so powerful, indeed overwhelming, that no decision other than the one taken by the Board was conceivable.  A different decision, namely, to continue spending money on a service whose efficacy was not established, would have been unreasonable.

This basically means that, even if the decision had been discriminatory, he would not have overturned it as homeopathy has no basis in medical science and it would be unreasonable to fund it under any circumstance. These are strong and unequivocal statements. There are very few boards in the UK that still provide any funding for homeopathy, I think we're in to single digits now, but even one is too many. Hopefully this judgement will provide the impetus required to finally eradicate this wasteful and potentially harmful practice from public healthcare settings.

Samuel Hahnemann, the originator of the nonsense that is homeopathy

ISS to Destroy Planet

Is it me, or is the International Space Station about to throw its javelin of destruction at the earth?

Watch out!