Metrology is the science of taking accurate measurements. Measuring the effect of public policies is a key part of ensuring those policies get long-term support, or of making sure that problem policies are swiftly and effectively fixed.
This may sound straightforward, but in practice there are many situations in which accurate measurements are both difficult and impossible. These may include cases where we desire data about a future or alternate state, such as
Cases where the question concerns a future or hypothetical state. These are usually addressed by creating a model, plugging in as much relevant data as possible, and simulating the rest. Different models are likely to give different results, and controversies may arise from the models’ details.
What will happen if we implement this policy?
What would have happened if we had tried something different from our historical policies?
What will next year’s weather be like?
Cases where the question involves value judgments. These are usually addressed by trying to define “good” and “bad” in terms of measurable quantities such as individuals’ health, happiness, and finances. There are multiple ways of measuring health, happiness, and finance - and an overall “good” versus “bad” assessment may require estimated trade-offs between those quantities. Both the choices of individual measurements and of the exchange rates between them may also be debated.
Should we reform the tax code according to Senator A’s proposal?
Who will benefit and how much if we legalize recreational marijuana? Who will be hurt and how much?
How much money should be spent reducing the sorts of pollution which lead to lung disease? Which industries (or government agencies) should bear those financial costs? What is the target balance between human health and human finances?
Cases where the individual components are measurable, but they are too numerous and varied for a coherent, unambiguous measurement. These often resort to using a mean, median, or other form of data reduction. No matter how the data gets simplified, there are sure to be individual cases with an atypical response: Someone will be unhappy with the resulting policy even if it performs exactly as advertised for the vast majority of people.
How much money do people make?
How healthy are food choices?
How bad is pollution?
Cases where the ability to measure a quantity is dependent on that quantity. (Compare the Weak Anthropic Principle of cosmology.)
Policy wonks and other researchers must find a way to cut through these difficulties and conflicts, and politicians must find a way to sell what they judge as a “best option”, even knowing there may be disagreement in how different people view the options. Conversely, misinformation relies on taking advantage of complex & confusing situations, partisan assumptions and simplified logic to paint a distorted view of the world.
Some guidelines from metrology:
Measure the quantities of interest as directly as possible
Clearly communicate the way the measurements were done
Research alternate models and alternate measurement methods.
Find theoretical justifications from multiple authors which show the measurements are plausible
Enumerate and catalog various sources of bias, including your own assumptions - and try to quantify these biases and correct for them.