p-values and their discontents

What do we teach about statistical inference?

Principles from the 2016 ASA statement on p-values:

  1. p-values can indicate how incompatible the data are with a specified statistical model.
  2. p-values do not measure the probability that the studied hypothesis is true, or the probability that data were produced by random chance alone.
  3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
  4. Proper inference requires full reporting and transparency.
  5. A p-value or statistical significance does not measure the size of an effect or the importance of a result.
  6. By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.

Divide into groups of six and write down what you teach that would inform each of these principles.

Additional discussion

It’s helpful if some people take notes here

  1. How relevant is the ASA concern about p-values to our courses?
  2. Do your students understand hypothesis testing? Your colleagues? -A dialog from Science News about the precise meaning of a p-value.
  3. Student misconceptions

The alternative hypothesis

In this shared document, write your answers to the following questions.

What role does the alternative hypothesis play in hypothesis testing?

What do you teach about the alternative hypothesis?

It’s worse than this

The Nature papers highlight the problem of researchers dismissing a result because the p-value isn’t low enough.

  • Origin?
    • Misconception that p-value is the probability that the Null hypothesis is true.
    • Implicit loss function for hypothesis testing is that only false rejections matter: \(\alpha = 0.05\) and power = ????
    • Perhaps failure to emphasize conditional probabilities.
  • But we also face:
    • Large \(n\). Any non-zero effect size can be significant.
    • Large number of potential explanatoery variables
    • Increasing popularity of models that learn, e.g. step-wise regression, random forests, … Vastly increases possibilities for false discovery without the labor that p-hacking used to require.

Homework

  1. Read Section 3: There are many do’s from the lead editorial to review their recommendations for what to do.

  2. Read one of the two papers (one and two)focussing on education in the TAS special issue, or pick any other paper from the issue. (Section 7 of the lead editorial gives author’s summaries of their recommendations, which can help you pick a paper that will be relevant to you.)