Weigle (2002), who discusses "factors in writing assessment," outlines a scheme in which the candidate (the writer or the person being assessed) works through a particular instrument to produce a performance, and a rater
examines the performance with the use of a rating scale to arrive at a score (see Figure 1).
For example, if one rater
'strongly disagrees' and another 'strongly agrees' this must be considered a greater level of disagreement than when one rater
'agrees' and another 'strongly agrees'.
The rating scale for rater
confidence was reverse scored from the direction of the skills rating scale in order to reduce the possibility of response bias.
If either or both of these criteria were not met, the portfolio was referred to a third rater
, usually one of the four table leaders.
Thus different decisions may have decreased the inter rater
Also, the probability of rater
disagreement when scoring the QUEST was considered extremely low.
We suggest more detail on training methodology for adequate reliability in raters
with the modified TJA (Dudley et al.
Regular training of raters
with timely feedback is required to improve interrater reliability.
The results in Table 2 (Model 3) show that the interaction between a rater
with prevention focus and the leader's moral attentiveness was significant.
Google's guidebook for quality raters
gives several guidelines and examples for them to use.
Spearman's correlation among different rater
pairs ranged from 0.
The second author is the expert for the training but was also a rater
for the study.