InterModel Vigorish (IMV)

Compare two sets of predicted probabilities for the same binary outcomes. The IMV expresses how much better the second model predicts than the first, on the scale of a weighted coin: each model's geometric mean likelihood is converted to the weight of a coin with the same entropy, and the IMV is the proportional gain in that weight.

Usage

irw_imv(data, p1, p2, resp = "resp", eps = 1e-06)

Arguments

data

A data frame holding the outcomes and both sets of predicted probabilities, or a numeric vector of binary outcomes. When data is a vector, p1 and p2 must be numeric vectors of the same length.

p1

Predictions from the baseline model: a column name in data, or a numeric vector when data is a vector of outcomes.

p2

Predictions from the comparison model, in the same form as p1.

resp

The binary outcome. Ignored when data is a vector; otherwise a column name in data, defaulting to "resp".

eps

Clamping tolerance for predicted probabilities. Default 1e-6.

Value

A single numeric value: the proportional gain in coin weight from p1 to p2.

Details

An IMV of 0 means the two models predict equally well; 0.05 means model 2 is equivalent to a coin 5\ is not symmetric: irw_imv(data, "p1", "p2") is the gain from moving from p1 to p2, and swapping the arguments does not simply flip the sign.

Predicted probabilities are clamped to [eps, 1 - eps] before the likelihoods are computed, so predictions of exactly 0 or 1 do not produce an infinite log-likelihood.

Out-of-sample predictions are the intended use: comparing in-sample predictions will favour the more flexible model by construction. The "Comparing Models Out of Sample with the IMV" article works through a cross-validated comparison of a Rasch model against a 2PL.

References

Domingue, B. W., Rahal, C., Faul, J., Freese, J., Kanopka, K., Rigos, A., Stenhaug, B., & Tripathi, A. (2021). InterModel Vigorish (IMV): A novel approach for quantifying predictive accuracy with binary outcomes. doi:10.31235/osf.io/8sgz5

See also

irw_predict for producing p1 and p2 from fitted mirt models.

Examples

set.seed(1)
truth <- rbinom(500, 1, 0.7)
# A model that knows the base rate beats one that guesses at random.
irw_imv(truth, p1 = rep(0.5, 500), p2 = rep(0.7, 500))
#> [1] 0.4194643