Causal Analytics for Applied Risk Analysis Louis Anthony Cox, Jr

Replicating Selection of Predictors Based on Significant Assay-Endpoint Combinations

Kleinstreuer et al. (2013) determined which chemical/assay combinations would be considered “significant” for predicting carcinogenic potential usingodds ratio values, as well as their 95% confidence intervals, from two sources: univariate odds ratios, and average “permuted” odds ratios, as described below and inparagraph 3 of Materials and Methods in the article. In several places where the article’s discussion was insufficient for us to reproduce results, the R software code provided by the authors gave the needed clarification.

Kleinstreuer at al. first computed an odds ratio for each assay/endpoint combination (664 x 11) using the vector of AC50 values for the assays (with non-zeros converted to 1) and the 0/1 vector of endpoint values. In accordance with the study procedures, any chemical rows corresponding to an “NA” endpoint result and any assay column with fewerthan 10 hits over the 292 chemicals used in the training set wereeliminated. We used software that we developed independently in the Python programming language to check whether we could reproduce the study results.

Replicating the Univariate Odds Ratios Calculations

For each assay-endpoint combination, we fed the two binary vectors into a Python function that computed contingency table values n₀₀, n₀₁, n₁₀, and n₁₁ corresponding to the counts of vector elements falling into each celln_xy (with the convention that x = assay, y=endpoint). For instances where all 4 cell counts were non-zero, the function returned the odds ratio:

OR = n₀₀*n₁₁/n₀₁*n₁₀,

and the standard error:

SE = sqrt(1/n₀₀ + 1/n₁₁ + 1/n₀₁ + 1/n₁₀)

In the case of any zero cell elements, OR = 1 and SE = 0 was returned (the computation was indeterminate and flagged for non-use). We computed the upper and lower 95% confidence intervals using the following standard asymptoticformulas:

LCI₉₅ =exp(log(OR) – 1.96*SE)

UCI₉₅ = =exp(log(OR) + 1.96*SE)

These computations, verified as the same used by Kleinstruer et al. via the software code, correspond to the “Wald” method for calculating odds ratios based on a normal approximation to the log odds ratios. Other methods are available, including the exact Fisher (conditional maximum likelihood), midpoint (median unbiased), and small sample adjusted methods. Such alternative methods are appropriate when one or more cells have small values, as happens often in this study. These alternative methods are readily available in the R language epitools module (http://cran.r-project.org/web/packages/epitools/index.html). Experimentation showed that they often provide wider confidence intervals than the Wald method, potentially producing fewer false positives (in which an artificially narrow confidence interval wrongly excludes the null hypothesis of no significant relation, OR = 1).We verified that our software was computing the same odds ratios and confidence intervals by checking against the intermediate output files provided by Dr. Judson.

Replicating the Permuted Odds Ratios Calculations

Kleinstreuer et al. (2013) also describe odds ratios, one per endpoint, determined by “permuting the endpoints and calculating the OR values for all assays”. The 95% confidence intervals for each endpoint are said to be calculated from the “OR distribution across all assays”. The precise meaning of these statements, described below, was made apparent by studying the provided R software code.

A single “permutation” involves taking a random permutation of the 0/1 endpoint vector, and computing the odds ratios for the permuted vector versus each of the 664 chemical assays. To obtain the result for a single endpoint, 10000 permutations are performed, and the odds ratio vectors of results from each are concatenatedinto a single long vector (any indeterminate elements are removed). The vector is then sorted and the elements at 0.025 and 0.975 of the vector length are extracted. These values determine the 95% confidence intervals associated with that endpoint.

We developed the same procedure in Python using the same inputs. Table 8.3, produced using our Python code, provides results very similar (but not identical) to those of the original authors.
Table 8.3. Comparison of Computed Versus NCCT-Provided Endpoint CI Values

End Point	Computed LCI	NCCT LCI	Computed UCI	NCCT UCI
MLiver2Pre	0.326	0.326	2.65	2.65
MLiver3Neo	0.279	0.274	2.80	2.80
MLung2Pre	0.310	0.296	3.88	3.88
MSpleen2Pre	0.316	0.304	3.80	3.80
RKidney2Pre	0.291	0.291	3.23	3.23
RLiver2Pre	0.302	0.302	2.65	2.65
Rliver3Neo	0.314	0.314	3.60	3.60
RTestes2Pre	0.303	0.303	3.25	3.25
RTestes3Neo	0.304	0.310	3.53	3.53
RTroid2Pre	0.303	0.303	2.94	2.92
RTroid3Neo	0.299	0.299	3.12	3.03

The small differences observed above are plausibly due to differences in random number generation, which resulted in different permutation sets; they did not affect the selection of significant assay-endpoint combinations.