Back in the days before fake news and alternative facts were popularized, the preferred method of public-policy falsehood was “lying with statistics” - presenting accurate data in a misleading way, or selectively drawing upon individually-plausible studies to back a conclusion which was much less likely than any of the steps in the assertion. My goal with this series is to review several of these techniques so that you can recognize them or resolve competing claims. There are several technical variations on how misleading statistics come about:
Scientists are generally well-respected, especially when combined with other mathematically-literate professions like engineers, doctors and teachers. Other respected professions are associated with presumably non-partisan and unbiased data, such as judges, lawyers, police officers, and accountants. One of the motivations for deliberate / semi-deliberate “lying with statistics” is to leverage the respect accorded to these professions and their methods in making an argument: “If respected professionals who have reviewed the evidence believe X, then so should you”, often with a hefty does of “You don’t need to look too closely at the evidence of X or try to understand it yourself.”
Part of what distinguishes “lying with statistics” from other falsehoods is the underlying statistics can be accurate. The various problems are well-known to statisticians and scientists. There are various ways of addressing them - most of which involve a lot of work, additional research, and ongoing debates with those skilled in the appropriate arts. These things take a lot of time and dedication to work and they’re not foolproof. The difficulty of obtaining absolute certainty is why statisticians and scientists are careful to hedge their bets with measures of uncertainty, meta-analyses, and other tricks. Important research gets cross-checked and validated in many ways by many different groups, sometimes in surprising ways.
However: journalists, pundits, and politicians have different biases from scientists, and are under increased pressure to generate popular and/or surprising results. This often leads them to de-emphasize technical details like uncertainties and context. To be fair, it’s not that reporters or politicians are necessarily lying - they may legitimately believe what they say, but haven’t taken the time to understand why they’re taking the original study out of context.
The defense against misleading statistics, as with other forms of propaganda and misinformation, is to approach data with a critical eye: Is what’s being measured or reported actually what’s important to the argument? (The difference between mean, median, mode, and case studies is important here, as are problems involving sampling.) Dig into the details, and stay as close as possible to what the original researchers are saying. When possible, think about different ways the same important effect can be measured and check those as well. Draw news from reputable sources which focus on reporting facts and analysis.