Causal Analytics for Applied Risk Analysis Louis Anthony Cox, Jr

Summary and Conclusions: Potential Roles of Judicial Review in Transforming Regulatory Causal Inference and Prediction
This chapter has argued that more active and stringent judicial review of the causal reasoning and claims advanced in support of regulations can increase the net social benefits from regulations bycorrecting problems that currently promote unrealistically large estimates of the benefits caused by regulations. Among these are the following:

1. 
   Ignoring risk-aversion and risk premiums for correlated losses.When it is not certain that reducing exposure to a regulated substance or activity will actually cause the expected health, economic, or other benefits attributed to such reductions, and when the regulation affects a large number of economic agents, then the risk-adjusted value of the uncertain benefits can be much less than their expected value. This difference, called the risk premium in decision analysis, is due to risk-aversion, which penalizes large numbers of correlated losses. This reduction in benefits due to uncertainty about causation is not accounted for in benefits assessments and BCA calculations that focus on expected net benefits while ignoring risk aversion.
   
2. 
   Tyranny of extreme perceptions.Regulatory agencies may attract and retain employees who believe that the uncertain net benefits caused by regulation are higher than most other people do. If so, these relatively extreme perceptions are likely to shape agency beliefs and benefits assessments for regulations.
   
3. 
   Use of unvalidated and simplistic models of benefits caused by regulations.Confronted with uncertainty and complexity in the causal networks that link regulations to their consequences (both intended and unintended), regulators, like other people, often adopt simplistic, inaccurate, or unproved modeling assumptions, such as that adverse health effects will decrease in proportion to reductions in regulated exposures, or that positive exposure-response associations represent manipulative causation. These assumptions can lead to large but inaccurate predictions of the benefits from regulation. Such estimates are then amplified by media reports and public concerns in which the assumption-based numbers are treated as facts, without discounting them for uncertainty.
   
4. 
   Failure to focus on manipulative causality. Theepidemiological evidence of harm caused by regulated exposures, and estimates of the presumed benefits of reducing exposures,are based almost entirely on associational-attributive causal findings in important real-world examples such as the Irish coal-burning bans, the US EPA Clean Air Act Amendments, and the FDA ban of animal antibiotics. As previously discussed, such findings have no necessary implications for predictive or manipulative causation. They do not provide a logically sound basis for risk assessments or benefits estimates for proposed future changes in regulations to reduce exposures. Moreover, associational causation can almost always be found by making modeling choices and assumptions that create a statistically significant exposure-response association (“p-hacking”), even in the absence of predictive or manipulative causation. Thus, conflating evidence of associative and attributive causation with evidence of manipulative causation can lead to routinely exaggerated estimates of the benefits caused by regulations.
   
5. 
   Failure to learn effectively from experience.Health, safety, and environmental regulations are usually evaluated during the rule-making process based on prospective modeling and prediction of the desirable effects that they will cause. This prospective view does not encourage learning from data via retrospective evaluation, or designing regulations to be frequently modified and improved in light of experience. But such adaptive learning and policy-refinement have been found to be essential for effective decision-making and forecasting under uncertainty in other areas such as high-reliability organizations, superforecasting, and control of systems under uncertainty. As illustrated by the example of the Irish coal burning bans, the relatively rare retrospective evaluations of the effectiveness of regulatory interventions that are currently conducted are prone to unsound design and confirmation bias. Rigorously designed data collection, evaluation, and modification based on performance feedback are not routinely incorporated into the implementation of most regulations. Thus, estimates of the benefits caused by a costly but ineffective regulation may remain exaggerated for years or decades, leading to widespread perceptions that it was effective and to adoption of similar measures elsewhere, as in the case of the Dublin coal-burning bans that are now being advocated for nation-wide adoption.
   

The preceding problems have a single root cause: reliance on fallible and overconfident human judgments about causation. Such judgments tend to over-estimate the benefits of regulations and neglect or underestimate uncertainties about them, thus promotingmore regulation than needed to maximize net social benefits. We have argued that, fortunately, more objective and trustworthy data-driven estimates of the effects actually caused by regulations and of uncertainties about those effects are now technically possible, and that they are also organizationally possible and practical if judicial review of causal reasoning and claims is strengthened. Advances in data science have yielded demonstrably useful principles and algorithms for assessing and quantifying predictive causation from data. Stronger judicial review that incorporates lessons from these methods into the review and application of causal reasoning used to support contested regulations can help to correct the preceding problems and to obtain many of the benefits of more accurate and trustworthy estimates of the impacts caused by regulations.

The following recommendations suggest how courts can promote better regulatory benefits assessment, impact evaluation, and adaptive learning to increase the net social benefits of regulations.

1. 
   Insist on evidence of manipulative causation. Rules of evidence used in determining whether it is reasonable to conclude that a proposed regulation will probably cause the benefits claimed for it should admit only evidence relevant for manipulative causation. This includes evidence of predictive causation, insofar as manipulative causation usually implies predictive causation. It also includes evidence on causal pathways and mechanisms whereby changes in exposures in turn change harm, based on well-validated and demonstrably applicable causal laws, mechanisms, processes, or paths in a causal network. Reject regulatory actions proposed without evidence of manipulative causation. Insofar as they provide no sound reason to believe that the proposed actions will actually bring about the consequences claimed for them, they should be viewed as arbitrary and capricious.
   
2. 
   Exclude evidence based on associational and attributive causation. Such evidence is not a logically, statistically, or practically sound guide for predicting effects of regulatory interventions.
   
3. 
   Encourage data-driven challenges to current benefits estimates. Producing relevant (manipulative causation or predictive causation) information about the impacts caused by regulations can improve risk assessments and benefits estimates, but is expensive for those who undertake to develop such information. To the extent that it can increase the net social benefits of regulation by more accurately revealing the impacts of changes in regulations, such information has a social value – a positive externality – and its production and use to improve regulations should therefore be encouraged. One way to do so might be to grant legal standing to parties who seek to challenge current estimates of regulatory impacts based on new information or analyses of manipulative causation (at least if they also bear either costs or predicted benefits of proposed regulations). A second way might be to emphasize that the burden of proof for changing a regulation can be met by any stakeholder with standing who can show that doing so will increase net social benefits.
   
4. 
   Discourage reliance on expert judgments of causation. Do not defer to regulatory science and expertise based on professional or expert judgements. Instead, insist on data-driven evidence of manipulative causation (including tests for predictive causation and elucidation of causal pathways or mechanisms) as the sole admissible basis for causal claims and estimates of the impacts caused by regulations.
   

A joint regulatory and legal system that encourages data-driven challenges to the assumptions and benefits estimates supporting current regulatory policies can create incentives for stakeholders – whether advocates or opponents of a change in regulation – to develop and produce the information needed to improve the effectiveness and net social benefits of regulation. It can also create incentives for regulators to adopt more of the habits of high-reliability organizations, regarding current policies as temporary and subject to frequent change and improvements based on data. Setting expectations that judicial review will provide independent, external, rigorous review of causal claims in light of data whenever stakeholders with standing insist on it may also encourage development of lighter-weight regulations that are less entrenched and difficult to change and that are more open to learning from experience and revising as needed to maximize net social benefits.

Of course, there is a large overhead cost to changing regulations that makes too-frequent change undesirable (Stokey, 2008). However, thethreat of rigorous judicial review and court-enforced revisions when data show that estimates of benefits caused by regulations are either unsound or obsolete would encourage regulators to develop sounder initial causal analyses and more modest and defensible estimates of the benefits of actions when manipulative causality – and hence the true benefits from regulation – are uncertain. This provides a useful antidote to the above factors that currently promote over-estimation of uncertain regulatory benefits, with little penalty for being mistaken and little opportunity for stakeholders to correct or improve estimates based on the judgments of selected experts.

In summary, more active judicial review of causal claims about regulatory impacts, with data-driven evidence about manipulative causation being the price of entry for affecting decisions, creates incentives to expandthe socially beneficial role of stakeholders as information collectors. Simultaneously, active and rigorous judicial review of causal claims provides a mechanism to help regulators learn to perform better. It does so both by serving as an external critic and reviewer of causal reasoning and predictions on which contested actions are predicated, and also by providing an opportunity for new information and different data-informed views to be brought to bear in assessing the actual effects being caused by current contested policies. An adversarial system allows different stakeholders to produce relevant information, both confirming and disconfirming, for evaluating the hypothesis that current regulatory policies are performing as predicted in causing desired effects. Active judicial review of causal claims supporting contested regulations by a court that is known to apply BCA or law-and-economics principles provides incentives for the stakeholders to produce such information with the intent of reinforcing, revising, or overturning current regulations as needed to increase net social benefits. Doing so always coincides with increasing the net benefits to at least some of the stakeholders, since increasing the sum of net benefits received by all affected individuals implies increasing the net benefits received at least some of them. Thus, judicial review can promote production and use of causally relevant information and help regulatorsto learn from experience how to make regulations more beneficial. This is not a role that can easily be played by other institutions.

If courts develop, maintain, and routinely apply expertise in dispassionate, data-driven causal inference, both the threat and the reality of judicial review will help to overcome the significant drawbacks of current judgment-based approaches to causal inference for regulatory benefits assessment. Such review will also provide both regulators and stakeholders affected by regulations with incentives and ability to improve the net social benefits from regulations over time.
REFERENCES
Aliferis CE, Statnikov A, Tsamardinos I, Mani S, Koutsoukos XS (2010). Local Causal and Markov Blanket Induction for Causal Discovery and Feature Selection for Classification Part I: Algorithms and Empirical Evaluation. Journal of Machine Learning Research 11 (2010) 171-234


Apte JS, Marshall JD, Cohen AJ, Brauer M. Addressing Global Mortality from Ambient PM2.5.Environ Sci Technol. 2015 Jul 7;49(13):8057-66.
Bartholomew MJ, Vose DJ, Tollefson LR, Travis CC. A linear model for managing the risk of antimicrobial resistance originating in food animals.Risk Anal. 2005 Feb;25(1):99-108.
Bontempi G, Flauder M. (2015). From dependency to causality: A machine learning approach. Journal of Machine Learning Research. 16: 2437-2457. https://arxiv.org/abs/1412.6285
Cover TM, Thomas JA. (2006) Elements of Information Theory 2nd Ed. John Wiley & Sons. Hoboken, NJ. ISBN-13 978-0-471-24195-9ISBN-10 0-471-24195-4. https://archive.org/details/ElementsOfInformationTheory2ndEd (Last accessed 1-9-2018)
Cox LA Jr, Popken DA. Has reducing fine particulate matter and ozone caused reduced mortality rates in the United States?Ann Epidemiol. 2015 Mar;25(3):162-73. 
Cromar KR, Gladson LA, Perlmutt LD, Ghazipura M, Ewart GW. American Thoracic Society and Marron Institute Report. Estimated Excess Morbidity and Mortality Caused by Air Pollution above American Thoracic Society-Recommended Standards, 2011-2013.Ann Am Thorac Soc. 2016
Dockery DW, Rich DQ, Goodman PG, Clancy L, Ohman-Strickland P, George P, Kotlov T; HEI Health Review Committee. Effect of air pollution control on mortality and hospital admissions in Ireland.Res Rep Health Eff Inst. 2013 Jul;(176):3-109.
Dockery D, Pope C, Xu X, et al. An association between air pollution and mortality in six US cities. N Engl J Med. 1993;329:1753–1759
Dominici F, Greenstone M, Sunstein CR. Science and regulation. Particulate matter matters.Science. 2014 Apr 18; 344(6181):257-9.
EPA, 2011a. The Benefits and Costs of the Clean Air Act from 1990 to 2020: Summary Report. U.S. EPA, Office of Air and Radiation. Washington, D.C. www.epa.gov/air/sect812/aug10/summaryreport.pdf
EPA, 2011b. The Benefits and Costs of the Clean Air Act from 1990 to 2020. Full Report. U.S. EPA, Office of Air and Radiation. Washington, D.C. http://www.epa.gov/oar/sect812/feb11/fullreport.pdf
Fedak KM, Bernal A, Capshaw ZA, Gross S. Applying the Bradford Hill criteria in the 21st century: how data integration has changed causal inference in molecular epidemiology.Emerg Themes Epidemiol. 2015 Sep 30;12:14. 
Frey L, Fisher D, Tsamardinos I, Aliferis CF, Statnikov A, (2003). Identifying Markov Blankets with Decision Tree Induction. Proceedings of the Third IEEE International Conference on Data Mining, Melbourne, FL Nov 19-22 2003. pp 59-66
Friston K, Moran R, Seth AK. Analysing connectivity with Granger causality and dynamic causal modelling.CurrOpinNeurobiol. 2013 Apr;23(2):172-8.
Furqan MS, Siyal MY. Random forest Granger causality for detection of effective brain connectivity using high-dimensional data.J IntegrNeurosci. 2016 Mar;15(1):55-66. 
Gardner, D. (2009). The Science of Fear: How the Culture of Fear Manipulates Your Brain. Penguin Group. New York, New York.
Gelman A, Zelizer A. (2015) Evidence on the deleterious impact of sustained use of polynomial regression on causal inference. Research and Politics. January-March 2015: 1–7. http://www.stat.columbia.edu/~gelman/research/published/rd_china_5.pdf
Giannadaki D, Lelieveld J, Pozzer A. Implementing the US air quality standard for PM2.5 worldwide can prevent millions of premature deaths per year.Environ Health. 2016 Aug 23;15(1):88.
Greenland S. Multiple-bias modelling for analysis of observational data J. R. Statist. Soc. A (2005) 168, Part 2, pp. 267–306
Halliday DM, Senik MH, Stevenson CW, Mason R.Non-parametric directionality analysis - Extension for removal of a single common predictor and application to time series.JNeurosci Methods. 2016 Aug 1;268:87-97.
Hammond PJ. Harsanyi’s utilitarian theorem: A simpler proof and some ethical connotations. In R. Selten (Ed.) Rational Interaction: Essays in Honor of John Harsanyi. 1992.
Harsanyi JC. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. The Journal of Political Economy (1955): 309-321.
Hill J. 2016 Atlantic Causal Inference Conference Competition: Is Your SATT Where It's At? http://jenniferhill7.wixsite.com/acic-2016/competition
Höfler M. The Bradford Hill considerations on causality: a counterfactual perspective.Emerg Themes Epidemiol. 2005 Nov 3;2:11.
Höfler M, Lieb R, Wittchen HU. Estimating causal effects from observational data with a model for multiple bias.Int J Methods Psychiatr Res. 2007;16(2):77-87.
Hoover, K. D. Causal structure and hierarchies of models. Studies in History and Philosophy of Biological and Biomedical Sciences (2012), http://dx.doi.org/10.1016/j.shpsc.2012.05.007http://public.econ.duke.edu/~kdh9/Source%20Materials/Research/Causal%20Hierarchies%20published%20version.pdf
Hurd HS, Malladi (2008). A stochastic assessment of the public health risks of the use of macrolide antibiotics in food animals. Risk Analysis. 28 (3): 695–710.
Kelly O. (2016) How the coal ban dealt with Dublin’s burning issue. The prohibition of ‘smoky’ coal in 1990 resulted in 350 fewer annual deaths in city. The Irish Times. Saturday, September 26, 2015. /www.irishtimes.com/news/environment/how-the-coal-ban-dealt-with-dublin-s-burning-issue-1.2367021
Kleinberg S, Hripcsak G. A review of causal inference for biomedical informatics. J Biomed Inform. 2011 Dec;44(6):1102-12. 
Laden F, Schwartz J, Speizer FE, Dockery DW. Reduction in fine particulate air pollution and mortality: extended follow-up of the Harvard Six Cities study. Am J RespirCrit Care Med. 2006;173:667–672
Lo WC, Shie RH, Chan CC, Lin HH. Burden of disease attributable to ambient fine particulate matter exposure in Taiwan.JFormos Med Assoc. 2016