Fig. 2
Simulation scheme for reliability measures under one of the proposed scenarios (scenario 2). In the figure, four conditions/tissues are considered (A,B,C,D) in which two raters (1,2) measure 500 subjects. The cross-tabulation tables between raters are shown, all with marginal frequencies across raters given by ∼ 1/j (j=1,2,3,4 and A=1,B=2,C=3,D=4), corresponding to the simulation scenario number 2 described in “Materials and methods” section. Initially, all 500 individuals are correctly classified between raters. 3000 permutations are performed, such that the marginals frequencies are conserved. Every 30 permutations reliability measures (P0, κ, λ and r) are computed. The process is repeated 50 times. A second simulation is performed from an initial cross-tabulation of perfect disagreement, where diagonal terms are all 0. A total of 10,000 values of each measure are then obtained for simulations that cover the whole reliability interval (0,1) under each scenario