Cohen's Kappa
Two annotators labelled the same items. cohensKappa(a, b) measures their agreement corrected for chance over the two equal-length label lists:
po = fraction of items where a[i] == b[i] pe = sum over labels of (share of a with that label) × (share of b with that label) kappa = (po − pe) / (1 − pe), and 1 when pe is 1
Labels can be any values. Results are compared after rounding to six decimals.
cohensKappa(["a", "a", "b", "b", "a", "b"], ["a", "b", "b", "b", "a", "a"]) -> 0.333333 cohensKappa([1]*19 + [0], [1]*20) -> 0 // 95% raw agreement, zero kappa
Hints
Hint 1
Count each label's occurrences in a and in b. Chance agreement pe is the sum over every label of (count in a / n) × (count in b / n).
Hint 2
When both annotators always use the same single label, pe is 1 and the formula divides by zero — the definition makes that case 1.