The expert panel that developed the weighting scheme analyzed information from sample student responses, and set cut scores to distinguish four levels of student understanding of the conservation of matter benchmark:
When the weighting scheme is applied, the 10 items in COMA reliably target the conservation of matter benchmark, with Cronbach's alpha = 0.85 for the entire sample (N = 2,282).
The pattern of data for various subgroups for both the 1st and 2nd years of implementation indicated that subgroups of students entered a unit of study on conservation of matter with significantly different pretest scores (except for gender), Then, presented with the CTA curriculum materials, subgroups of students learned more about the target idea than did their matched peer comparison subgroup.
Frankly, in thinking hard about the concepts that must be understood in grappling with the conservation of matter benchmark, it is difficult to conceive of how anyone could make sense of this "big idea" without direct experiences with physical phenomena that are structured and scaffolded, making the case that matter is conserved in chemical reactions.
On average, most students in this implementation study did not master the conservation of matter benchmark, but rather scored in the low to mid ranges of understanding on COMA.
CTA is centered on one important big idea in science, conservation of matter. The entire focus of this unit is on this big idea.
The CTA unit carefully structures and sequences the phenomena that students experience in each lesson, leading to an emerging understanding of conservation of matter. The lessons provide nontrivial questions to induce discussions and to guide student reasoning.
Conservation of matter assessment manual Unpublished manuscript, The George Washington University.