Connecting Classroom and Large Scale Assessment
The CLR Moderation Process

Findings from the 1995 CLR moderations conducted by the CLR Core Development Group and the Center for Language in Learning for the California Department of Education and CLR-registered schools

Fieldwork: April-June, 1995

WHAT PERCENTAGE OF CLR STUDENTS STATEWIDE ARE READERS AT THE END OF GRADE 3?

A total of 108 student records were moderated at this grade span, where the emphasis of assessment is on tracing children's progress toward becoming independent readers. The sampling is small and, with 10 of the 11 participating schools receiving Title 1 funds, not representative of the total state population. Data compiled from the moderations, however, are used in the chart below to illustrate the kind of report which can help schools and parents learn what students can do in terms of shared criteria.

If this data had been drawn from results at one school, they would learn, for instance, that 29 (91%) of the 32 third graders whose records were scored in the CLR moderations this year demonstrated that they are readers. Five (16%) showed evidence of being exceptionally fluent or advanced readers, 9 (28%) are fluent or proficient readers, and 15 (47%) are moderately fluent or partially proficient readers. Only 3 third graders were judged as not yet fluent as opposed to 13 at Grade 1. Each of the five developmental levels on the K-3 scale is defined below.

A chart demonstrating reading achievement, K-3

1 Beginning reader: Uses just a few successful strategies for tackling print independently. Relies on having another person to read the text aloud. May still be unaware that text carries meaning.

2 Not yet fluent reader: Tackling known and predictable text with growing confidence but still needing support with new and unfamiliar ones. Growing ability to predict meanings and developing strategies to check predictions against other cues such as the illustrations and the print itself.

3 Moderately fluent reader (partially proficient): Well launched on reading but still needs to return to a familiar range of reader text. At the same time, beginning to explore new kinds of texts independently. Beginning to read silently.

4 Fluent reader (proficient): A capable reader who now approaches familiar texts with confidence but still needs support with unfamiliar materials. Beginning to draw inferences from books and stories. Reads independently. Chooses to read silently.

5 Exceptionally fluent reader (advanced): An avid and independent reader who is making choices from a wider range of material. Able to appreciate nuances and subtlety in text.

WHAT PERCENTAGE OF CLR STUDENTS ARE EXPERIENCED READERS BY THE END OF THE 4-8 GRADE SPAN ?

This year's sample is too small to tell us. A total of 66 student records were moderated at these grade levels, where the emphasis of assessment is on tracing the growth of student reading experience. While the number of records is too small for a representative sampling of student performance, the chart below uses the compiled data to illustrate the kind of report possible for monitoring and improving school- or program-wide progress toward the goal that all students become experienced readers, as defined by reaching Levels 3, 4 or 5 in eighth grade curriculum and materials .

A chart demonstrating reading achievement, 4-8

1 Inexperienced reader: Experience as a reader has been limited. Generally chooses to read a very easy and familiar text where illustrations play an important part. Has difficulty with any unfamiliar materials and yet may be able to read own dictated texts confidently. Needs a great deal of support with the reading demands of the classroom. Over dependent on one strategy when reading aloud, often reads word by word. Rarely chooses to read for pleasure.

2 Less experienced reader: Developing fluency as a reader and reading certain kinds of material with confidence. Usually chooses short books with simple narrative shapes and with illustrations. May read these silently; often re-reads favorite books. Reading for pleasure often includes comics and magazines. Needs help with the reading demands of the classroom and especially with using reference and information books.

3 Moderately experienced reader (partially proficient): A confident reader who feels at home with books. Generally reads silently and is developing stamina as a reader. Is able to read for longer periods and cope with more demanding texts, including novels. Willing to reflect on reading and often uses reading in own learning. Selects books independently and can use information books and materials for straightforward reference purposes, but still needs help with unfamiliar material, particularly non-narrative prose.

4 Experienced reader (proficient) : A self-motivated, confident and experienced reader who may be pursuing particular interests through reading. Capable of tackling some demanding texts and can cope well with the reading of the curriculum. Reads thoughtfully and appreciates shades of meaning. Capable of locating and drawing on a variety of sources in order to research a topic independently.

5 Exceptionally experienced reader (advanced) : An enthusiastic and reflective reader who has strong established tastes in fiction and non-fiction. Enjoys pursuing own reading interests independently. Can handle a wide range and variety of texts, including some adult material. Recognizes that different kinds of text require different styles of reading. Able to evaluate evidence drawn from a variety of information sources. Is developing critical awareness as a reader.

WHAT ARE THE LEVELS OF STUDENT ACHIEVEMENT IN HIGH SCHOOLS USING THE CLR?

The CLR reading performance scale for Grades 9-12 was tested in classrooms this year. The scale focuses assessment on tracing student progress toward becoming accom-plished readers. At two of the moderations held across California this year, high school teachers shared their students' records and discussed the value of the scales for assessing the work of their students. On the whole, the teachers felt the scale reflected the range of desired high school levels of performance. In spring, 1996, teachers at CLR-registered high schools will use the scale to moderate records at Grades 9-12. Scale levels and their definitions are as follows:

1 Literal reader: May be able to derive meaning from a variety of texts, yet usually takes them at face value. Is unaware that he/she may rightly challenge the writer's claims, evidence, or ideas, or may legitimately critique a text for style, logic, organization, etc. Expects or settles on one single view of the text. Sees most text as unrelated to life outside of school. May express frustration with density of course texts. Frequently abandons the reading of books, even those he or she has ostensibly chosen. May rely on non-print media to collect information. Relies on others for interpretation of text meanings. Shows a lack of familiarity with common text organizers, e.g., headings, index. Defines him or herself as one who dislikes reading.

2 Less accomplished reader: May see self as a poor reader but can read course text with preparation and support of visual and/or auditory supplement, e.g., graphics, oral readings. Will read assigned texts but does not read for pleasure or for purposes outside of school. May still rely on getting course information from media other than text, including collaborative groups and film or tapes. Is beginning to collaborate with others to construct meaning in text. Tentative use of advance organizers and genre schemas. Can apply prior experience to some aspects of stories and biographies but may be unable to relate his or her own past experience to abstract ideas presented without context or "hands-on" application.

3 Moderately accomplished reader (partially proficient): Often tentative about sharing own interpretation of texts. With preparation and support, can read aloud expressively. Has developed a sense of genre. Shows a willingness to persist with some difficult texts. Is beginning to make associations between texts and personal experience. Can explain the way particular texts are organized to help the reader derive meaning. Is aware, in interpreting texts, of the influence of the context (i.e. period of time, gender/status of author) in which they were written. Developing skill in using text ideas and challenging text assertions. Reads assigned texts.

4 Accomplished reader (proficient): An effective reader of particular genres, can provide convincing evidence of comprehension. Has strategies for unlocking difficult text, including the sharing of initial interpretations with other readers and the author's use of print conventions (punctuation, headings, index). Able to evaluate information from multiple sources, including text and personal experience. Able to explain the bases of contradictory interpretations and previously held misconceptions. Brings knowledge gained from text read outside of class to bear on course work. Selects books for a wide range of purposes. Can manage the reading of long texts outside of class.

5 Exceptionally accomplished reader (advanced): An avid reader who reads easily across the range of purposes for reading: from seeking information to exploring experience. Elaborates on connections he or she is making with text, able to explain how they aid understanding. Can weigh and compare relative strength and weakness, style, structure, credibility, or aesthetics among texts. Able to demonstrate texts' potential for multiple interpretations. Can explain the significance of the social, cultural or political history of a text. Reads aloud fluently, with appropriate expression.

HOW ACCURATE ARE CLR TEACHER JUDGMENTS OF STUDENT PROGRESS?

Results from this year's moderations indicate high rates of agreement among teachers in their judgments of student performance. With scale placement as well as student and teacher identification masked on each record prior to the moderations at both school site and regional sessions, 174 out of 198 records (88%) received two matched scores within two readings. Twenty-four records were not scored because regional readers determined that there was not enough evidence to assign scale placement (20), the teacher was ill so only site and regional placement could be made (3), and one Spanish language record was mistakenly left as a split without a third reading. Of the number scored, 135 judgments (79%) made by the originating classroom teachers were corroborated by two readings. The remaining 35 records were ranked one point higher or lower and required third readings.

Some trends appeared which will be studied for their impact on procedures for next year's moderations. Preliminarily, it can be said that in the cases where judgments between the original teacher and those at the regional readings differed, 30 of the 35 records (86%) were scored higher by the original teacher than by the regional readers. While this finding may seem to confirm a belief that some judgments are less rigorous than others, the discrepancies may be due to inexperience with this kind of assessment. For most of the 125 participants, this year brought not only their first moderation but also the first year of keeping individual student records meant to be understood by others in and beyond the school. The most common response teachers made to the moderation experience (66 of 84 respondents) was that the evidence to defend their judgments of student performance was known and available, but it had been overlooked and omitted. As one teacher put it, "I learned that it is important to justify observations with data. Children's perceptions of themselves as learners (quotes, comments), samples of their written work, and text samples is vital information to be included."

HOW DO THE MODERATIONS INFLUENCE TEACHING AND LEARNING?

One of the criticisms of conventional assessments is that their use narrows the curriculum to what can be easily measured. To study the effects of the CLR and the moderations on classroom practice, teachers were asked to describe the changes they will make in their use of the CLR next year. A complete analysis of their responses is underway, but a preliminary look reveals that most plan to reorganize their classrooms to permit more student involvement in learning and assessment. One teacher, who described the moderation experience as "very intense and productive," spoke for many who learned just "what I need to collect as evidence." Another said she needed "to look at class structure differently and modify teacher talk" because she "learned with these 3 students* what works with some and not others." Specifically, this teacher noted, "I need to make my room/lessons more student-centered. . . . This process has helped to redefine the teacher's role."

Another criticism of conventional assessments is that they require secrecy so they cannot provide useful, timely information. A school board member, having observed the moderation process, attested to the transparency of the records as to what's being taught and learned. "I'm told, 'They're not teaching reading anymore.' I can see by these records evidence of learning word attack skills, comprehension, etc. And I can use the scales when I visit classrooms." Appreciation of this open view of classroom learning was echoed by teachers, many of whom said things like this one: "It's great to be able to . . . look at other teachers' records, especially from teachers whom I don't usually work with, and learn about what they're doing and why they do it. It validates our roles not only as 'teachers' but as researchers, observers, and learners."

Among the positive comments, surprisingly few teachers expressed dismay at the profound nature of the changes in their practice the CLR requires. We hope to document, in studies underway, evidence of the kinds of professional support teachers need to meet the challenges of more complex assessment.

*The first year of using the CLR, teachers keep records on just 3-5 students in order to learn the system .

Overview of the 1995 CLR Moderation Process

In May and June of 1995, 125 teachers met for 4-5 hours at five sites, from Petaluma to San Diego, to read and score student records during the second annual CLR regional moderations. The moderation process began at their schools, where, with classroom teachers' scores masked, they had shared their own students' records and moderatedĄthat is, they discussed and sometimes modifiedĄtheir interpretations of the performance scales for measuring reading achievement in the light of these concrete examples. At the regional sessions, teachers representing each site used the same scales to read and score again the records from each other's schools. School and regional placements had to agree or the records went to a third reader, a certified CLR teacher-leader. Records judged as having "not enough evidence" were not scored. The purposes of the annual moderations are

bullet to monitor increases and declines in student achievement at CLR-registered schools,

bullet to validate the judgments of student progress made by classroom teachers using the CLR,

bullet to determine the support needed by teachers as they use the CLR's comprehensive and authentic assessment system.

What's in a California Learning Record?

Although the CLR system does not require teachers and students to follow a prescribed curriculum and/or products menu, there is a standardized format which includes: bullet parent descriptions of home literacy bullet student work samples and reflections on oneself as a learner bullet teacher observations/analyses bullet teacher summaries of student achievement with recommendations about what is next to be learned. All of this data is collected in the CLR, except copies of selected work from student portfolios, which are attached for moderation purposes.

ACKNOWLEDGMENTS

California Learning Record Core Development Group members and others who led the moderations this year are as follows: Leader Lisa Damico with panel members Gretchen Covington and Bea Curran, K-6, Bay Area and northern California; Co-leaders Dana Arntson and Susan Miserlis with panel member Nancy Hazen, K-6, San Diego county; Leader Sally Thomas with panel members Lisa Graham and Alice Kakuda, K-6, Orange, Los Angeles and Riverside counties; Co-leaders Darby Williams, Janet Ghio and Lynda Peddy, K-12, Sacramento county; and Leader Kiyo Masuda with panel member Phyllis Hallam, 7-8, Bay Area. Other Core Development members conducted site moderations only: Kathy Thompson and Carol Hendsch, 9-12, Fresno; Judy Lynch and Bev Ruby, K-6, Sacramento; Marla Heeren, K-6, Fresno.

Much appreciation is due the guests who contributed their observations to a study of the moderation process: Judy DeChesere, high school teacher in Petaluma; Dr. John Fredricksen, University of California at Berkeley, Department of Education and Project Director at Educational Testing Service; Mae Gundlach, Consultant in Language Arts Unit at the California Department of Education; Charlotte Higuchi, Los Angeles Unified teacher on special assignment to the University of California at Los Angeles CRESST Center; Marie McKittrick, student teacher in the Cross-cultural Language and Academic Development Program at University of California at Berkeley; Marjorie Pogue, Fullerton Unified School District Board member; Dr. Amauri Rodrigues, principal of Lincoln Elementary in Pomona, a CLR-registered school; Mary Weber, Title I Resource Teacher, San Diego City Schools; and Dr. Sandy Williams, Director of Evaluation Services, San Diego County Office of Education.

Our thanks to staffs at the following schools/district, which served as sites for the 1995 CLR moderations: Cherry Valley School in Petaluma, Darnall E-Campus in San Diego, Laguna Road Elementary in Fullerton, Lincoln Unified District Office in Stockton, and Martin Luther King Jr. Middle School in Berkeley. Special thanks are also due Phyllis Hallam, UC Berkeley doctoral student, for her analyses of moderation data and the California Department of Education and schools throughout California for partial funding of the 1995 CLR Moderations.

Reading Scales 1 & 2 have been adapted with permission by the copyright holder, Centre for Language in Primary Education, Webber Row, London SE 8 QW. The original scales appear in the Primary Language Record Handbook for Teachers, distributed in the U.S. by Heinemann Educational Books.

Reading Scale 3 is produced and copyrighted by the Center for Language in Learning.

Responsibility for the data interpretations and opinions expressed in this publication rests solely with the Center for Language in Learning. For further information about the CLR assessment system, please call (619) 443-6320 or write the Center, 10610 Quail Canyon Road, El Cajon, CA 92021.

This was a publication of the Center for Language in Learning