Interpret the following statistics from reliability studies.   Check all results that reflect good reliability.

Measurement reliability. For attributes that are measured on a continuous scale, two common reliability statistics are correlation (r) and paired t-test. The intra-class correlation coefficient (ICC) is interpreted the same way as the Pearson correlation coefficient (e.g. larger ‘r’ means stronger correlation between two different measurement methods, or higher reliability). A paired t-test can be used to compare the difference in value between two measurement methods on the same set of subjects. If the paired t-test p-value is <0.05, the two measurement methods are significantly different in how they measure the same attribute (i.e., “poor” reliability).

For categorical variables (such as diagnosing or predicting a condition as yes/no), a common reliability statistic is Cohen’s Kappa. Kappa measures category agreement between two raters. Kappa usually ranges from 0 to 1 (it is possible to get a kappa <0, but it is not common). Although there are many ways to interpret Kappa values, a common method is as listed below (Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977:33;159-174).

• <0 = agreement less than “chance” (50/50 chance with a dichotomous variable, like a coin toss) – this is possible but does not occur often (Viera AJ, Garrett JM Understanding interobserver agreement: the kappa statistic. Fam Med 2005;37;360-63.)
• 0 = agreement equivalent to “chance”
• <0.4 = slight to fair reliability
• 4-0.6 = moderate reliability
• 61-0.8 = substantial reliability
• 81-1 = near perfect agreement

Use your knowledge from previous modules and module 7 lectures, and the above information to answer the questions below.

• Interpret the following statistics from reliability studies.   Check all results that reflect good reliability. (1 each, total 7)

__ r = -0.85

__ ICC = +0.9

__ Paired t-test p=.006

__ Paired t-test p=.85

__ kappa=0.2

__ kappa=0.9