For three specific scenarios, we removed (quite by chance) a predefined proportion of data (10, 25 and 50%) to assess the impact of the absence of values in the TRAC (Missing Completely at Random) hypothesis. The selection criteria for the scenarios were an empirical probability of coverage of almost 95% for Fleiss`K and Krippendorffs Alpha, a sample size of 100, as well as concordance variations, categories and evaluators on the scenarios. Intraclass correlation (ICC) is one of the most frequently used statistics to assess IRR for ordinal, intermittent and relational variables. CICs are suitable for studies involving two or more coders and can be used if all subjects are evaluated in a study by several programmers or if only a portion of subjects are evaluated by several programmers and the rest by one encoder. CICs are suitable for completely cross-design or if a new set of encoders is randomly selected for each participant. Unlike Cohens (1960) kappa, who quantifies the IRR on the basis of an all-or-nothing agreement, CICs take into account the size of the disagreement in calculating irr estimates, with larger disagreements leading to smaller ONES. Many research designs require IRR assessments in order to show the extent of the correspondence between programmers. Appropriate unachievable statistics should be carefully selected by researchers to ensure that their statistics coincide with the design and purpose of their study and that the statistics used are appropriate based on the distribution of observed evaluations. Researchers should use validated IRR statistics when assessing the IRR, instead of using compliance percentages or other indicators that do not take into account random concordance and do not provide information on statistical relevance.

The in-depth analysis and reporting of irr test results will more clearly state the results of the research community. Missing values cannot be taken into account by Fleiss`K, unless all observations with missing values are excluded. On the other hand, for Krippendorffs Alpha, all observations are taken into account in the calculation with at least two evaluations. We investigated the robustness of the two coefficients in the absence of values under TRAC conditions with respect to mean bias and type 1 empirical error for three scenarios (I. . . .