Cognitive dissonance about research reform

An interesting editorial on research practices came out in PLOS Medicine yesterday.  It’s good to hear about reproducibility and reforms we need to see in science from a fellow statistician, John Ioannidis over at Stanford.  Each discipline has its own quirks and accepted practices, but statistics is a common factor in every study.  I believe we statisticians have a unique perspective on the problem: we get to play the role of data advisor on other peoples’ studies and the PIs on our own.

Ioannidis cites examples of things that work in several fields, including reproducibility practices, data registration, and stricter statistical analyses.  Then he proposes a new “structure of scientific careers” that doesn’t just favor old men with fancy titles and big grants.  In this framework,

 Resources and power are seen as opportunities, and researchers need to match their output to the opportunities that they have been offered—the more opportunities, the more the expected (replicated and, hopefully, even translated) output. Academic ranks have no value in this model and may even be eliminated: researchers simply have to maintain a non-negative balance of output versus opportunities. In this deliberately provocative scenario, investigators would be loath to obtain grants or become powerful (in the current sense), because this would be seen as a burden.

I got to this part of the article and thought, “Wait, this sounds crazy?”  It almost seems like there would be no incentive to work hard, like any award would come with some negative consequences and you’d be punished if your work didn’t produce results.  Isn’t that exactly what research reforms are trying to get around?  Maybe a greater emphasis on sharing negative results would get around this problem, but I digress.

After reading this the first time and feeling my knee-jerk disagreement, I took a step back and realized that my negative response is precisely due to my being immersed in the current culture of “publish or perish” and academic hierarchies.  I’m so entrenched in this way of thought that it’s hard to see other models for scientific careers.  However, I’m on Ioannidis’s side and I believe we need to seriously rethink the way research is done in order to have more high quality results.

Frankly my commentary on the subject is pretty useless because it’s a hard question and I’m no expert.  You should just go read the article here.

We’re all Bayesians

Human decisions can be viewed as a probabilistic problem: in a world full of uncertainty, how do we understand what we observe and respond rationally?  Decision-making, inductive reasoning, and learning have all been modeled using Bayes’ theorem, which says that the probability of event A given that event B occurs (the posterior) depends on certain known or estimatable probabilities: the probability, without any other information, of event A happening (the prior), the probability of B occurring given that A has occurred (the likelihood), and the probability of B occurring (the model evidence).  In an article published today, Acerbi et al. looked at factors that make our probabilistic inference suboptimal.

For example, suppose we want to decide whether to wear a dress or pants.  The decision probably depends on whether or not we believe it will be a hot day (event A).  We’d probably just check weather.com to make a rational decision or just choose the outfit we like the best regardless of the weather, but suppose all we have at our disposal is memory of the temperatures last summer (event B).  What we want to know is the chance that today will be hot given our knowledge of what summer weather is typically like.  We probably have a prior idea about what today’s weather will be like, knowledge of typical summer weather, and some level of confidence in our understanding of last summer’s weather.  We have all the quantities needed to apply Bayes’ rule and pick out our outfit for the day.

Of course, nobody sits down and computes the posterior probability of a sunny day every morning by thinking about the prior.  And this is where we have trouble making decisions:

if prior experience has complex patterns, we might require more trial-and-error before finding the optimal solution. This partly explains why, for example, a person deciding the appropriate clothes to wear for the weather on a June day in Italy has a higher chance of success than her counterpart in Scotland.

In this example, the prior distribution of temperatures on a June day in Italy might be centered around a high temperature, whereas the distribution for Scotland might have two peaks or be somewhat flat.  We’d assume that it would be easier to guess the average of a Gaussian distribution than a more complicated or skewed one.  To see how this plays out in rational decision-making, the authors tested peoples’ ability to predict the location of a target using a variety of different prior distributions.  They concluded,

This finding suggests that human efficacy at probabilistic inference is only mildly affected by complexity of the prior per se, at least for the distributions we have used. Conversely, the process of learning priors is considerably affected by the class of the distribution: for instance, learning a bimodal prior (when it is learnt at all) can require thousands of trials, whereas mean and variance of a single Gaussian can be acquired reliably within a few hundred trials.

We might have a harder time figuring out patterns of weather in Scotland than in Italy, but the paper suggests that we’d have an equally difficult time deciding what to wear in either country.  Perhaps we just disregard Bayes’ law and make irrational decisions based on our emotions or personal biases.