Reliability and Scalability with IRW Data
Source:vignettes/articles/scaling-reliability.Rmd
scaling-reliability.RmdBefore fitting a measurement model it is worth asking whether the
items hang together at all. This article covers internal consistency
with psych and nonparametric scalability with
mokken, using a real IRW table that ships with the
package.
The data
swmd_mokken is bundled with irw, so this
article runs on genuine warehouse data with no network access. Fetching
it yourself would look like this:
df <- irw_fetch("swmd_mokken")
df <- swmd_mokken
str(df)
#> Classes 'tbl_df', 'tbl' and 'data.frame': 4557 obs. of 4 variables:
#> $ resp : int 1 1 1 1 1 1 1 1 1 1 ...
#> $ item : chr "Item2" "Item1" "Item3" "Item1" ...
#> $ rater: int 12 13 17 22 25 25 28 44 45 45 ...
#> $ id : int 1000303 1000303 1001302 1001302 1001302 1001302 1001302 1002504 1002504 1002504 ...Note the rater column: each person-item pair was scored
by several raters, so there are many more rows than the 30 x 7 = 210
person-item combinations.
Collapsing raters
irw_long2resp() aggregates duplicate
id-item pairs, and how it does so matters
here. The default for a numeric response is "mean", which
produces fractional scores that neither package below will accept as
ordinal categories. "mode" keeps the responses on their
original 1-5 scale by taking each rater panel’s most common score:
wide <- irw_long2resp(df, agg_method = "mode", id_density_threshold = NULL)
#> NOTE: This dataset contains 'rater' information with 651 unique raters.
#>
#> Dropped 4347 duplicate response(s) from 210 id-item pair(s) (out of 210 total pairs). Aggregating using agg_method='mode'.
dim(wide)
#> [1] 30 8
wide[1:3, ]
#> id item_Item1 item_Item2 item_Item3 item_Item4 item_Item5 item_Item6
#> 1 1000303 3 2 3 3 3 3
#> 2 1001302 2 2 3 4 3 3
#> 3 1002504 3 3 3 3 3 3
#> item_Item7
#> 1 3
#> 2 3
#> 3 3Internal consistency
alpha(resp)$total
#> raw_alpha std.alpha G6(smc) average_r S/N ase mean sd
#> 0.7920768 0.7935834 0.7916973 0.3545158 3.844571 0.05446908 3.214286 0.3554145
#> median_r
#> 0.3388695
omega(resp, nfactors = 1, plot = FALSE)$omega.tot
#> Loading required namespace: GPArotation
#> Omega_h for 1 factor is not meaningful, just omega_t
#> Warning in schmid(m, nfactors, fm, digits, rotate = rotate, n.obs = n.obs, :
#> Omega_h and Omega_asymptotic are not meaningful with one factor
#> [1] 0.799235Mokken scalability
Scalability coefficients ask whether the items form a cumulative scale. Guttman’s rule of thumb reads H below 0.3 as unscalable, 0.3-0.4 as weak, 0.4-0.5 as moderate, and above 0.5 as strong.
H <- coefH(as.data.frame(resp))
#> Warning in check.data(X): Varying numbers of item scores were observed across the items.
#> Either the items have the same number of response categories but some item categories were not endorsed;
#> or the items have
#> different numbers of categories by design.
#> In the latter case, the sum score cannot be used for (ordinal) measurement.
#> $Hij
#> item_Item1 se item_Item2 se item_Item3 se item_Item4
#> item_Item1 0.504 (0.152) 0.676 (0.195) 0.545
#> item_Item2 0.504 (0.152) 0.577 (0.114) 0.677
#> item_Item3 0.676 (0.195) 0.577 (0.114) 0.583
#> item_Item4 0.545 (0.310) 0.677 (0.205) 0.583 (0.272)
#> item_Item5 0.643 (0.295) 1.000 (0.000) 0.464 (0.187) 0.583
#> item_Item6 1.000 (0.000) 0.628 (0.141) 0.620 (0.229) 0.545
#> item_Item7 0.605 (0.312) 0.659 (0.287) 0.464 (0.257) 0.375
#> se item_Item5 se item_Item6 se item_Item7 se
#> item_Item1 (0.310) 0.643 (0.295) 1.000 (0.000) 0.605 (0.312)
#> item_Item2 (0.205) 1.000 (0.000) 0.628 (0.141) 0.659 (0.287)
#> item_Item3 (0.272) 0.464 (0.187) 0.620 (0.229) 0.464 (0.257)
#> item_Item4 0.583 (0.360) 0.545 (0.258) 0.375 (0.513)
#> item_Item5 (0.360) 0.737 (0.230) 0.375 (0.291)
#> item_Item6 (0.258) 0.737 (0.230) 0.605 (0.328)
#> item_Item7 (0.513) 0.375 (0.291) 0.605 (0.328)
#>
#> $Hi
#> Item H se
#> item_Item1 0.640 (0.084)
#> item_Item2 0.642 (0.061)
#> item_Item3 0.580 (0.088)
#> item_Item4 0.580 (0.235)
#> item_Item5 0.640 (0.100)
#> item_Item6 0.673 (0.120)
#> item_Item7 0.516 (0.186)
#>
#> $H
#> Scale H se
#> 0.616 (0.059)
H$Hi
#> Item H se
#> item_Item1 0.640 (0.084)
#> item_Item2 0.642 (0.061)
#> item_Item3 0.580 (0.088)
#> item_Item4 0.580 (0.235)
#> item_Item5 0.640 (0.100)
#> item_Item6 0.673 (0.120)
#> item_Item7 0.516 (0.186)
H$H
#> Scale H se
#> 0.616 (0.059)Every item clears 0.5 and the scale coefficient is comfortably strong.
Automated item selection confirms that the items belong to a single scale – every item is assigned to scale 1 at the default lower bound of 0.3:
aisp(as.data.frame(resp))
#> Warning in check.data(X): Varying numbers of item scores were observed across the items.
#> Either the items have the same number of response categories but some item categories were not endorsed;
#> or the items have
#> different numbers of categories by design.
#> In the latter case, the sum score cannot be used for (ordinal) measurement.
#> 0.3
#> item_Item1 1
#> item_Item2 1
#> item_Item3 1
#> item_Item4 1
#> item_Item5 1
#> item_Item6 1
#> item_Item7 1aisp() warns that the items show varying numbers of
observed response categories. With only 30 respondents some of the five
categories simply never came up for some items. That is a property of
this small table rather than a mistake, but on a larger table it is
worth checking with irw_check_resp(), which reports sparse
categories item by item:
checks <- irw_check_resp(df)
names(checks$sparse_category_items)
#> NULLMonotone homogeneity
The monotonicity check needs enough people per rest-score group to
estimate item-step response functions. With only 30 respondents,
check.monotonicity() stops at its default group size, so
the group size has to be set by hand:
mono <- check.monotonicity(as.data.frame(resp), minsize = 10)
#> Warning in check.data(X): Varying numbers of item scores were observed across the items.
#> Either the items have the same number of response categories but some item categories were not endorsed;
#> or the items have
#> different numbers of categories by design.
#> In the latter case, the sum score cannot be used for (ordinal) measurement.
summary(mono)
#> ItemH #ac #vi #vi/#ac maxvi sum sum/#ac zmax #zsig crit
#> item_Item1 0.64 0 0 NaN 0 0 NaN 0 0 0
#> item_Item2 0.64 0 0 NaN 0 0 NaN 0 0 0
#> item_Item3 0.58 1 0 0 0 0 0 0 0 0
#> item_Item4 0.58 1 0 0 0 0 0 0 0 0
#> item_Item5 0.64 0 0 NaN 0 0 NaN 0 0 0
#> item_Item6 0.67 1 0 0 0 0 0 0 0 0
#> item_Item7 0.52 0 0 NaN 0 0 NaN 0 0 0No violations, but treat that as illustrative rather than conclusive.
The #ac column counts the comparisons the check was
actually able to make, and at this sample size there are very few of
them: minsize = 5 yields 18 active comparisons and 3
violations, minsize = 10 yields 3 and 0, and
minsize = 15 yields 1 and 0. A clean result from three
comparisons is weak evidence. A serious monotonicity assessment wants a
larger table; irw_filter() will find one:
irw_filter(n_participants = c(500, 5000), density = c(0.8, 1))