# Inter-Rater Agreement In Statistics

Therefore, the common probability of an agreement will remain high, even in the absence of an “intrinsic” agreement between the councillors. A useful interrater reliability coefficient (a) is expected to be close to 0 if there is no “intrinsic” agreement and (b) increased if the “intrinsic” agreement rate improves. Most probability-adjusted match coefficients achieve the first objective. However, the second objective is not achieved by many well-known measures that correct the odds. [4] For variables with more than two measurements, we also assessed the impact of using an ordinal scale instead of a nominal scale on predicted reliability. As Fleiss`K offers no possibility of escalation of ordination, we did this analysis only for the Krippendorff Alpha. Alpha`s estimates have increased by 15-50% if an ordinal scale is used against a nominal scale. However, the use of an ordinal scale gives for these correct variable estimates of Alpha, since the data were collected ordinally. Here, we were able to obtain point estimates ranging from 0.70 (HER-2) to 0.88 (estrogen group), indicating a significant convergence between advisors.

The CCI evaluation (McGraw- Wong, 1996) was conducted using an inter-mediated CCI to assess the extent to which coders provided consistency in their sensitivity beyond the subjects. The resulting CCI was in the excellent ICC range of 0.96 (Cicchetti, 1994), indicating that the coders had a high degree of convergence and indicate that empathy was assessed similarly in donors. The high CCI suggests that independent coders have introduced a minimum amount of measurement errors and that, therefore, statistical performance for subsequent analyses is not significantly reduced. Empathy assessments were therefore deemed appropriate to be used in the hypothesis testing of the hypothesis in this study. The consistency observed in the case study showed significant differences between the parameters studied (Table 2) ranging from 10% (MIB-1 proliferation rate) to 96% (estrogen receptor group). The parameters based on a semi-objective count (i.e. hormone receptor groups and MIB-1 proliferation) were no more consistent than those based on a pure estimate. LeBreton, J.M., and Senter, J. L. (2008).