University of Illinois at Urbana-Champaign

Facebook

Send comments
to the ECRP Editor.

          
Share HomeJournal ContentsIssue Contents
Volume 5 Number 1
©The Author(s) 2003

Applying an Analytic Writing Rubric to Children's Hypermedia "Narratives"

Michael Seth Mott & Cynthia Etsler
Purdue University Calumet

Deondra Drumgold
Bank Street College of Education School for Children

Abstract

In an effort designed to guide and improve the assessment of a newly developed writing environment, the reliability and developmental and concurrent validity of a previously validated rubric developed for pen-and-paper-created narratives, Writing What You Read (WWYR), was determined when applied to hypermedia-authored narratives of children in second and third grade. Children (n = 60) from four intact classrooms produced hypermedia narratives (text, audio, graphic, and video elements) over a 4-month period in a school-based computer laboratory. Raters (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion. Two analyses were used to determine reliability: percentages of agreement and Pearson correlations. Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for ±0 and .99 for ±1). Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales (Theme, Character, Setting, Plot, and Communication), the r values were .59, .55, .49, .50, and .50, respectively. Developmental validity of the WWYR scores was examined with one-way MANOVA to evaluate the WWYR scores of children grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill. Evidence for the developmental validity of the WWYR scores was supported across the three ability groups, F(2, 36) = 2.59, p <. 01. Concurrent validity was examined through correlational analysis between students' mean WWYR score and ITBS score. Scores from the two measures were positively correlated, r = .83, p < .01, providing evidence of the sensitivity of the WWYR assessment to measurethe developmental literacy competency of the third-grade students. Results support teachers' use of a validated rubric developed for pen-and-paper-created narratives applied to hypermedia narratives, despite additional visual and audio narrative elements inherent to hypermedia. Implications for literacy teaching and learning with hypermedia address core questions about the similarities and differences between written textual expression and visual and verbally recorded expression. A revised and expanded WWYR rubric is proposed to begin to address these core questions generated by teachers' use of the hypermedia writing environment.

Introduction

The very notion of "writing" is increasingly being transformed by new digital computer technologies in society, homes, businesses, and schools. Today, one's ability to represent thought electronically is more important than ever. The extreme pace of change in society dictates that early childhood educators consider how to facilitate children in learning to express themselves digitally in computer environments in developmentally appropriate ways. This article addresses the curriculum, instruction, and assessment of children's written expression in hypermedia, a computer environment that supports text, audio, video, and graphics. The hypermedia curriculum and instruction are described, and a technical analysis of a proposed assessment is applied and discussed.

Emerging Trends

Prior research has addressed the reliability and validity of a narrative rubric useful for both teaching and learning (value) and large-scale (utility) literacy assessment. As literacy curriculum and instruction continue to transform to include new computer-based learning environments, researchers must correspondingly develop and technically evaluate assessments for the new environments. Hypermedia represents a powerful electronic environment through which literacy expression can be facilitated by the manipulation of text, graphics, audio, and video elements, and thistechnology and integration are reflected in the new literacy K-12 standards (IRA and NCTE, 2001). The qualities of hypermedia that support higher-level cognitive processes such as synthesis, organization, evaluation, and reflexivity have been well documented (Yang, 1996; Mott, Sumrall, & Hodges, 1997). However, there is a glaring absence of ways to reliably and validly assess students' hypermedia products. To address this absence, a narrative process writing curriculum and instruction environment was merged with hypermedia. These terms are defined as follows:

This article addresses the reliability and developmental and concurrent validity of a previously validated narrative writing rubric, Writing What You Read (WWYR) (Wolf & Gearhart, 1994;Novak, Herman, & Gearhart, 1996), applied to hypermedia narratives created by students in grades 2 and 3 (see Table 1). The technical qualities of the process-oriented classroom rubric, valuable for teaching and learning on a day-to-day basis, are linked to the utility of the narrative rubric for measuring elementary students' literacy competencies as identified via a validated large-scale instrument, the Iowa Test of Basic Skills—Literacy Competency. Thus, five main issues are addressed relating to rater judgments of elementary students' hypermedia narratives (stories with text, graphics, audio, and video elements authored using HyperStudio hypermedia software):

Table 1
Writing What You Read Narrative Rubric (Wolf & Gearhart, 1994)
Theme Character Setting Plot Communication
Explicit-Implicit Flat-Round Backdrop-Essential Simple-Complex Context-bound
Didactic-Revealing Static-Dynamic Simple-Multifunctional Static-Conflict Literal-Symbolic
1: Not present or not developed through other narrative elements 1: One or two flat, static characters, with little relationship between characters 1: Backdrop setting with little or no indication of time or place ("There was a little girl. She liked candy.") 1: One or two events with little or no conflict ("Once there was a cat. The cat liked milk.") 1: Writing bound to context (You have to be there) and often dependent on drawing and talk to clarify the meaning
2: Meaning centered in a series of list-like statements ("I like my mom. And I like my dad. And I like my...") 2: Some rounding, usually in physical description; relationship between characters is action driven 2: Skeletal indication of time and place often held in past time ("Once there was..."); little relationship to other narrative elements 2: Beginning sequence of events, but out-of-sync occurrences; events without problem; problem without resolution 2: Beginning awareness of reader considerations; straightforward style and tone focused on getting the information out
3: Beginning statement of theme, often explicit and didactic ("The mean witch chased the children and she shouldn't have done that.") 3: Continued rounding in physical description, particularly stereotypical features ("wart on the end of her nose") 3: Beginning relationship between setting and other narrative elements (futuristic setting to accommodate aliens and spaceships) 3: Single, linear episode with clear beginning, middle, and end; the episode contains a problem, emotional response, action, and outcome 3: Writer begins to make sense of explanations and transitions ("because" and "so"); literal style centers on description
4: Beginning revelation of theme on both explicit and implicit levels through the more subtle things characters say and do 4: Beginning insights into motivation and intention that drive the feeling and action of main characters often through limited omniscient point of view 4: Setting becomes more essential to the development of the story in explicit ways: characters may remark on the setting, or the time and place may be integral to the plot 4: Plot increases in complexity with more than one episode; each episode contains problem, emotional response, action, and outcome 4: Increased information and explanation for the reader (linking ideas as well as episodes); words more carefully selected to suit the narrative's purpose
5: Beginning use of secondary themes, often tied to overarching theme, but sometimes tangential 5: Further rounding (in feeling and motivation); dynamic features appear in central characters and between characters 5: Setting may serve more than one function, and the relationship between functions is more implicit and symbolic 5: Stronger relationships between episodes (with resolution in one leading to a problem in the next) 5: Some experimentation with symbolism (particularly figurative language), which shows reader considerations
6: Overarching theme multilayered and complex; secondary themes integrally related to the primary themes 6: Round, dynamic major characters through rich description of affect, intention, and motivation 6: Setting fully integrated with the characters, action, and theme 6: Overarching problem and resolution supported by multiple episodes 6: Careful crafting of choices of story structure as well as vocabulary demonstrate considerate orchestration of all resources

Research Components

Methods

Children (n = 60) from four intact classrooms (two second-grade and two third-grade classrooms) produced hypermedia narratives over a 4-month period in a school-based computer laboratory equipped with 10 Windows-based microcomputers. Rater/(Teachers) (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. An interactive hypermedia software tutorial program was developed and used to train the teachers (n = 4) in the implementation of process writing techniques in conjunction with the use of hypermedia features as part of their elementary curriculum. Raters participated in a 3-hour training and rating session in a university computer laboratory equipped with five Power Macintosh microcomputers. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion.

Materials

The WWYR rubric shown in Table 1 contained five evaluative scales designed to assess students' developing competencies in narrative writing: Theme, Character, Setting, Plot, and Communication. The vertical analytical evaluative scales (1-6 for each competency) were designed to enable teachers to make instructional decisions on specific narrative components needing reinforcement and were not intended as a method for assigning a numerical value to a narrative. Teachers merely had to shade a box in the rubric to denote where a child's narrative was along each competency. The ITBS (Linn & Wilson, 1990) Form J was used as a basic battery for grades K-9 and included language skills directly related to writing: word analysis, vocabulary, spelling, and reading comprehension. Reliability coefficients for Form J ranged from .70-.90 for the language skills components. Additionally, the ITBS met high standards of overall technical quality and has been a widely accepted standardized measure of cognitive skill.

Hypermedia Narratives

HyperStudio Presentation Software (Wagner, 1997-2001) was used to support the children in expressing themselves with text, audio, video, and graphic elements in their narrative productions (see Figures 1-4). Children composed on paper and computer with teachers structuring the process with Writing Workshop (Graves, 1983), a method for organizing writing into discrete and recursive stages. These stages are (1) brainstorming, (2) revising, (3) drafting, (4) peer conferencing, (5) editing, and (6) publishing (see Mott & Klomes, 2001, for a detailed description of a program similar to the program addressed in the current study).

Elements shown are (1) graphics text, (2) paint bucket tool (blue), (3) line tool: freehand, (4) hyperconnection or link to "Page 1" via button.

HyperStudio narrative "page" with hypermedia elements

Figure 1. HyperStudio narrative "page" with hypermedia elements: Grade 2.

Page with the following features: textbox/word processed text with scrolling and a graphic (cat) inserted into Shape

Figure 2. Page with the following features: textbox/word processed text with scrolling and a graphic (cat) inserted into Shape.

HyperStudio page. Note control of font, color, background color, and font size

Figure 3. HyperStudio page. Note control of font, color, background color, and font size.

This page includes a scanned photograph

Figure 4. This page includes a scanned photograph.

HyperStudio was selected for this project for a number of reasons: (1) it supports high-end features such as video and animation; (2) it is relatively easy to use, even by young children (grades 1-3); (3) it contains a flexible interface; and (4) it is the most widely used multimedia/hypermedia platform in elementary education with over 100,000 users (Wagner, 1997-2001). HyperStudio contains several programming features that support children's hypermedia programming (see Figures 5-10). Programming instruction (and writing workshop) occurred through mini-lessons over a 4-month period. Figures 5 through 10 contain examples from a mini-lesson.

This page contains the programming line tool

Figure 5. This page contains the programming line tool.

The line tool was introduced to children during their first programming mini-lessons.

This page contains the hyperconnection programming tool

Figure 6. This page contains the hyperconnection programming tool.

The "hyper" programming tool, button-creation, enabled children to connect cards (narrative pages) to other cards. Button-creation supported linear links, (card 1 to 2 to 3), suitable for early childhood.

This page contains the graphics tool

Figure 7. This page contains the graphics tool. Children inserted graphics and scanned images from art representing many media: watercolor, acrylic paint, crayons, and colored pencils.

This page contains the textbox tool (word processing)

Figure 8. This page contains the textbox tool (word processing).

Writing was accomplished via the textbox tool that functions as a word processor with editing tools, cutting, pasting, and other word processor capabilities.

This page contains the digital audio deck

Figure 9. This page contains the digital audio deck.

Children inserted audio clips, programmed via button-creation, to enrich their text. Audio clips were recorded by the author or downloaded from HyperStudio for special effects such as an alarm clock sound.

This page contains video control

Figure 10. This page contains video control.

Video clips were inserted, programmed via button-creation, to supplement the narrative. One example was a video clip of Mars used in a science fiction narrative by a third-grade student.

Reliability

Percentages of Agreement

Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for ±0 and .99 for ±1) (see Table 2). The ±0 and ±1 percentages of agreement across 10 pairs of raters were higher than the ±0 and ±1 agreement levels found in both the Gearhart, Herman, Novack, and Wolf (1995) and Novak, Herman, and Gearhart (1996) WWYR reliability studies (compared in Table 3). The high percentages of agreement found in this study may be attributed to the raters' use of only the first three WWYR rubric evaluative subscale levels. The WWYR rubric contains six subscale levels that are developmentally sequenced according to the varied writing competencies of students in grades K-6. Because students in this study were in second and third grade, raters typically applied only levels 1, 2, and 3 out of five total levels. This narrow range of values independently applied by raters limited the number of choices. Hence, high percentages of agreement between raters would be expected based on the limited number of scale levels used.

Table 2
Percentages of Agreement for all Five Subscales of the WWYR Rubric
WWYR Subscale ±0 ±1 n
Theme .70 .96 60
Character .78 .99 60
Plot .73 .99 60
Setting .67 .99 60
Communication .68 .99 60

Table 3
Percentages of Agreement for the WWYR Rubric Averaged across All Subscales
WWYR Rating Material and Grade ±0 ±1 n
Hypermedia Narratives: Grades 2-3
Mott & Sumrall (1998)
.71 .98 60
Pen-and-Paper Narratives: Grades 1-6
Gearhart, Herman, Novak, & Wolf (1995)
.46 .96 120
Collections of Pen-and-Paper Narratives: Grades 2-5
Novak, Herman, & Gearhart (1996)
.25 .94 52

The percentages of agreement that were revealed in the current study, although higher than those found in the Gearhart, Herman, Novak, and Wolf (1995) study, should be considered descriptive information. Gearhart et al. remarked that percentages of agreement found for the WWYR should not be interpreted as "strong evidence of reliability" (p. 224). Rather, percentages of agreement can be used to help identify the existence of widely varying patterns of rater judgments, both across WWYR subscales and across all rater pairs. No such widely varying patterns were found in the current study. The limitations of analyses involving percentages of agreement analysis were discussed by Abedi (1994), who argued that, although percentages of agreement can reveal the existence of widely varying patterns of agreement among raters, they can also yield different results from other analyses such as Pearson correlations.

Pearson Correlations

Pearson correlations were used to further examine reliability of rater judgments. Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales, Theme, Character, Setting, Plot, and Communication, the r values were .59, .55, .49, .50 and .50. Table 4 contains the results of the Pearson correlations for WWYR rubric scoring across all rater pairs for the current study and for the Gearhart et al. (1995) study. An examination of correlation scores for hypermedia narrative productions revealed that interrater reliability for four of the five WWYR subscales (Theme, Character, Setting, and Plot) was comparable to the interrater reliability levels found in the Gearhart et al. (1995) WWYR reliability study for pen-and-paper-created narratives. For the fifth subscale (Communication), however, the correlational coefficient value was .16 higher in the Gearhart et al. study than in the current study. Despite the lower value found in the current study for Communication, Gearhart et al. related that an average subscale correlation higher than .50 could be considered adequate for a rubric such as the WWYR. Table 5 summarizes the comparison of WWYR correlations across all subscales for the current study and those found in the literature (Gearhart et al., 1995; Novak et al., 1996).

Table 4
Average Pearson Correlations for WWYR Rubric Scoring across 10 Pairs of Raters
WWYR Rating Material and Grade   Theme Character Setting Plot Communication
Hypermedia Narratives: Grades 2-3 r .59 .55 .49 .50 .50
Mott & Sumrall, 1998(n = 60) SD .25 .31 .25 .29 .24
Pen-and-Paper Narratives Grades 1-6 r .64 .59 .48 .57 .66
Gearhart et al. (1995) (n = 120) SD .10 .10 .12 .14 .10

Table 5
Comparison of WWYR Subscale Correlations: Pen-and-Paper Narratives versus Hypermedia
Subscale Theme Character Setting Plot Communication

Samples (n = 60)Mott & Sumrall (1998)
Hypermedia

Theme -- .86* .79* .79* .73*
Character -- -- .74* .74* .76*
Setting -- -- -- .75* .68*
Plot -- -- -- -- .78*
Communication -- -- -- -- --

Samples (n = 120)Gearhart et al. (1995)
Pen-and-Paper Narratives

Theme -- .83* .81* .83* .86*
Character -- -- .82* .87* .86*
Setting -- -- -- .73* .86*
Plot -- -- -- -- .85*
Communication -- -- -- -- --
Note: *p < .001.

The WWYR correlations observed in this study, as well as in the Gearhart et al. (1995) study, demonstrated that ratings were highly correlated across all subscales. The r values were low in this study as well as in the Gearhart et al. (1995) and Novak et al. (1996) studies. However, set guidelines for what is an acceptable level of interrater reliability do not exist. Nonetheless, both Gearhart et al. and Novak et al., whose studies analyzed holistic scores derived from the combined r values of Theme, Character, Plot, Setting, and Communication, argued that r values that fell within the .50 to .70 range were acceptable for analytic writing rubrics. In the current study, the interrater reliability for Theme, Character, Plot, and Communication subscales fell within the .50 and .59 range, but the level of interrater reliability (r = .49) for the Setting subscale did not. It is important to note that, in the Gearhart et al. (1995) study, a low coefficient value for the subscale of Setting was also found (r = .48).1 The acceptable interrater reliabilities for Theme, Character, Plot, and Communication in this study were comparable to the acceptable levels found in the Gearhart et al. (1995) study, and the r values for the Setting subscale in both this study and the Gearhart et al. (1995) study were not acceptable (albeit by large-scale standards). It is important to note that interrater reliability levels for Theme, Character, and Plot in this study may have been lower than the r values in the Gearhart et al. (1995) study because the researcher applied more stringent rating procedures in this study. Raters in the Gearhart et al. (1995) study were permitted to resolve differences greater than one scale point through discussion, whereas raters in this study were not permitted to resolve differences. In the current study, all ratings were included in the final data set.

The highly correlated rater judgments, along all five WWYR subscales for the current study and for the Gearhart et al. (1995) study, provided further evidence of the reliability of WWYR raters' judgments. The true function of a writing rubric is that it "enables raters to apply standard criteria in making judgments about the quality of students' work" (Abedi, 1994, p. 8). Gearhart et al. (1995), Novak et al. (1996), and Abedi (1994) argued that highly correlated scores across rubric subscales can be viewed as a positive indication that raters' judgments are being consistently applied.

Validity

Developmental Validity

Developmental validity of WWYR scores was examined via one-way MANOVA conducted on the low-, medium-, and high-ability vectors of WWYR subscale scores. The assumption for this analysis was based upon the technical qualities of the ITBS—Literacy measure to delineate the developmental literacy levels of the children. Results indicated a statistically significant difference between the three ability groups (F(2, 36) = 2.59, p = .01). Table 6 provides descriptive statistics, and Table 7 provides an additional summary of these results across each of the five WWYR subscales.

Table 6
Descriptive Statistics: WWYR Subscales across ITBS Ability Level
Statistics Dependent Variables
n Theme Character Setting Plot Communication
Mean Vectors            

ITBS Ability Level

Low 13 2.31 1.80 1.96 2.10 2.14
Medium 13 2.80 2.34 2.32 2.66 2.52
High 14 2.86 2.60 2.66 2.74 2.77

Variance-Covariance Matrix

Theme   .21 .15 .19 .13 .11
Character   -- .23 .13 .15 .13
Setting   -- -- .19 .14 .11
Plot   -- -- -- .19 .14
Communication   -- -- -- -- .16
Table 7
Mean WWYR Subscale Scores for Low-, Medium-, and High-Ability Grade-3 Students
WWYR Subscale ITBS NPR/Literacy Category Mean Score SD n F Sig
Theme Low 2.31 .62 16 6.19 .01
Medium 2.80 .28 10
High 2.86 .31 13
Character Low 1.80 .50 16 10.77 .01
Medium 2.34 .38 10
High 2.60 .51 13
Setting Low 1.96 .42 16 9.34 .01
Medium 2.32 .56 19
High 2.66 .34 13
Plot   Low 2.10 .54 16 9.28 .01
Medium 2.66 .34 10
High 2.74 .34 13
Communication Low 2.14 .47 16 9.20 .01
Medium 2.52 .34 10
High 2.77 .35 13

Tukey HSD tests were conducted on the mean vector scores of the three ability groups for all five WWYR subscales to follow up these results. For the WWYR subscale of Theme, low-ability students (M = 2.31, SD = .62) received lower scores than both medium-ability students (M = 2.80, SD = .28) and high-ability students (M = 2.86, SD = .31). For the WWYR subscales of Character, Setting, Plot, and Communication, all differences were significant. Therefore, low-ability students' scores were significantly lower than medium-ability students' scores, which were significantly lower than the high-ability students' scores. The significant differences revealed between low, medium, and high ITBS groups and the WWYR subscale scores provided evidence for the sensitivity of the WWYR to measure the development of students' hypermedia/writing competence. The significant results of the one-way MANOVA suggest that raters' judgments were evaluating students' skills as message producers (communication through text and other meaning-based symbol systems). The one-way MANOVA did not yield results that would enable the researcher to describe the degree of relatedness of raters' WWYR judgments and students' ITBS scores. In order to describe the relationship between WWYR scores and literacy skill (as measured by the ITBS), additional correlational analyses were conducted.

Concurrent Validity

The observed Pearson r correlation revealed a positive relationship between students' average WWYR score (averaged across the subscales of Theme, Character, Setting, Plot, and Communication) and their ITBS National Percentile Rank (literacy skills score), r = .83, p > .001. The positive correlation (r = .83) between students' WWYR scores and ITBS scores revealed in this analysis provided evidence for the concurrent validity (the degree to which test scores are related to the scores on an already established test) of WWYR raters' judgments of hypermedia productions. According to Messick (1992), establishment of the concurrent validity of a measure can be a stepping-stone toward establishment of the content-related validity (the degree to which scores evaluate the specific domain they were designed to evaluate) of a measure. Hence, the developmental and concurrent validities established for WWYR raters' judgments of hypermedia productions represented an important initial attempt toward eventually establishing the content-related validity of the WWYR when applied to hypermedia productions.

The strong, positive, linear relationship between ITBS literacy skill scores and WWYR rater judgments of hypermedia productions indicated that the hypermedia writing curriculum used in the current study involved literacy-based activities. The fact that students in this study expressed themselves through hypermedia features, and not solely through text, indicated that students' literacy skill can be enhanced through student expression via hypermedia and multimedia features. Table 8 provides additional information on the students' utilization of the hypermedia features used in their writing. This finding supported the claims of Daiute and Morse (1994), who observed that students who engaged in hypermedia writing developed literacy skill through the manipulation of text and other symbols. A weakness of the developmental and concurrent validity analyses was that evidence for obtaining the degree to which rater judgments of students' hypermedia productions evaluated textual features as well as textual and other hypermedia features (audio, hypermedia links, graphics, etc.) could not be determined.

Table 8
Frequency of HyperStudio Multimedia Features Used in Students' Hypermedia Narrative Productions*
Hypermedia/Multimedia Feature
Grade Level Button with Hypermedia Link Button with Audio Button with Video Text Box Graphics Text Scanned Art Graphics Objects (Clip Art)
2 (n = 20) 100% 81% 0% 100% 45% 96% 82%
3 (n = 40) 100% 100% 5% 100% 64% 100% 100%
*Note. In three out of the four classrooms where hypermedia/writing occurred, students' use of hypermedia/multimedia features was controlled by the teachers.

Assessment Value and Utility

The results of this study suggest several important implications for the assessment of students' hypermedia products. Having a reliable and valid assessment for evaluating students' hypermedia-based writing serves two general purposes: (1) to enhance classroom instruction (value), and (2) to inform, to a lesser extent, educational policy (utility). The positive results yielded in this study concerning the reliability and validity of the WWYR provide an avenue for teachers to accurately and consistently evaluate their students' hypermedia narrative productions by applying the WWYR assessment. The value of an assessment is the degree to which it enhances teacher instruction by linking teachers' comments to their instructional objectives (Wolf & Gearhart, 1994). Therefore, in order for teachers to properly evaluate both student outcomes and the instructional effectiveness of their hypermedia/writing curricula, it is useful for all educators to apply a reliable and valid instrument. Furthermore, the positive correlation between the students' ITBS literacy skill score and WWYR average score for hypermedia productions indicated that students who were engaged in a hypermedia/writing curriculum improved their literacy skills.

Table 9
Hypermedia-WWYR
Hypermedia Theme Character Setting Plot Communication
Elements Explicit-Implicit Flat-Round Backdrop-Essential Simple-Complex Context-bound
Didactic-Revealing Static-Dynamic Simple-Multifunctional Static-Conflict Literal-Symbolic
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
1: Not present or not developed through other narrative elements 1: One or two flat, static characters, with little relationship between characters 1: Backdrop setting with little or no indication of time or place ("There was a little girl. She liked candy.") 1: One or two events with little or no conflict ("Once there was a cat. The cat liked milk.") 1: Writing bound to context (You have to be there) and often dependent on drawing and talk to clarify the meaning
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
2: Meaning centered in a series of list-like statements ("I like my mom. And I like my dad. And I like my...") 2: Some rounding, usually in physical description; relationship between characters is action driven 2: Skeletal indication of time and place often held in past time ("Once there was..."); little relationship to other narrative elements 2: Beginning sequence of events, but out-of-sync occurrences; events without problem; problem without resolution 2: Beginning awareness of reader considerations; straightforward style and tone focused on getting the information out
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
3: Beginning statement of theme, often explicit and didactic ("The mean witch chased the children and she shouldn't have done that.") 3: Continued rounding in physical description, particularly stereotypical features ("wart on the end of her nose") 3: Beginning relationship between setting and other narrative elements (futuristic setting to accommodate aliens and spaceships) 3: Single, linear episode with clear beginning, middle, and end; the episode contains a problem, emotional response, action, and outcome 3: Writer begins to make sense of explanations and transitions ("because" and "so"); literal style centers on description
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
4: Beginning revelation of theme on both explicit and implicit levels through the more subtle things characters say and do 4: Beginning insights into motivation and intention that drives the feeling and action of main characters often through limited omniscient point of view 4: Setting becomes more essential to the development of the story in explicit ways: characters may remark on the setting, or the time and place may be integral to the plot 4: Plot increases in complexity with more than one episode; each episode contains problem, emotional response, action, and outcome 4: Increased information and explanation for the reader (linking ideas as well as episodes); words more carefully selected to suit the narrative's purpose
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
5: Beginning use of secondary themes, often tied to overarching theme, but sometimes tangential 5: Further rounding (in feeling and motivation); dynamic features appear in central characters and between characters 5: Setting may serve more than one function, and the relationship between functions is more implicit and symbolic 5: Stronger relationships between episodes (with resolution in one leading to a problem in the next) 5: Some experimentation with symbolism (particularly figurative language), which shows reader considerations
-Text-
-Hypertext-
-Graphic-
-Audio-
-Video-
6: Overarching theme multilayered and complex; secondary themes integrally related to the primary themes 6: Round, dynamic major characters through rich description of affect, intention, and motivation 6: Setting fully integrated with the characters, action, and theme 6: Overarching problem and resolution supported by multiple episodes 6: Careful crafting of choices of story structure as well as vocabulary demonstrate considerate orchestration of all resources

Note

1. A note on correlations averaged across raters: (1) A relatively small number of raters (n = 5) were used in this study and the Gearhart, Herman, Novak, and Wolf (1995) study, which may have contributed to the lower r values across all subscales. The attenuation of correlational coefficients may be another explanation for the low levels of interrater reliability (Gay, 1996). Accordingly, coefficients tend to be lower when a restricted range of values is utilized (e.g., the narrow range of only 3 out of a possible 6 WWYR subscale levels utilized by raters in this study). Thus, the more narrow the range of scores utilized by raters, the lower the coefficients. On the other hand, Gearhart et al. argued that if the number of raters was statistically increased five-fold, r values in the .50 to .60 range for Theme, Character, Setting, Plot, and Communication would be changed to .87, .89, .82, .86, and .89, respectively. Gearhart et al. used decision-study (multiplication of sample scores and aggregation of the results) coefficients to determine the number of raters needed to attain high reliability coefficients. (2) The r value for the Communication subscale in this study was considerably lower than the r value in the Gearhart et al. study (r = .50 versus .66). This sizable disparity in the level of interrater reliability may have been the result of the contrasting features of hypermedia-created narrative productions versus pen-and-paper-created narratives. The Communication subscale text primarily consisted of evaluative prompts designed to guide teachers in the assessment of writing style (see Table 1). Perhaps, in the current study, raters solely viewed textual features at the expense of the hypermedia features of graphics, sounds, buttons, and scanned art.

References

Abedi, Jamal. (1994). Final report of achievement, section A (report to the National Center for Education Statistics, contract no. RS90159001). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

Daiute, Colette, & Morse, Frances. (1994). Access to knowledge and expression: Multimedia writing tools for students with diverse needs and strengths. Journal of Special Education Technology, 12(3), 221-256. EJ 492 959.

Gay, Lorrie R. (1996). Educational research: Competencies for analysis and application (5th ed.). Englewood Cliffs, NJ: Prentice-Hall.

Gearhart, Maryl; Herman, Joan L.; Novak, John R.; & Wolf, Shelby A. (1995). Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubric. Assessing Writing, 2(2), 207-242. EJ 562 433.

Graves, Donald. (1983). Writing: Teachers and children at work. Portsmouth, NH: Heinemann. ED 234 430.

International Reading Association (IRA) and National Council of Teachers of English (NCTE). (2001). Standards for the English language arts. Urbana, IL: NCTE; Newark, DE: IRA.

Linn, R., & Wilson, V. (1990). Review of the Iowa Test of Basic Skills form J. In Jane Close Conoley and James C. Impara (Eds.), Mental measurement yearbook (9th ed.). Lincoln: University of Nebraska Press.

Messick, Stephen. (1992). Validity of test interpretation and use. In M. Alkin (Ed.), Encyclopedia of education research (6th ed., pp. 1487-1495). New York: Macmillan.

Mott, Michael S., & Klomes, Jeanine. (2001). The synthesis of writing workshop and hypermedia-authoring: Grades 1-4. Early Childhood Research & Practice [Online], 3(2). Available: http://ecrp.illinois.edu/v3n2/mott.html [2003, February 6]. ED 458 043.

Mott, Michael S., & Sumrall, William J. (1998). Scientists are presenters: Tech trek: Interactive media. Science Scope, 21(7), 42-45.

Mott, Michael S.; Sumrall, William J.; & Hodges, M. Lee. (1997, November). Process and computer-based elementary writing curriculum: A review of methods and assessments. Paper presented at the annual meeting of the Mid-South Educational Research Association, Memphis, TN. ED 415 255.

Novak, John R.; Herman, Joan L.; & Gearhart, Maryl. (1996). Establishing validity for performance-based assessments: An illustration for collections of student writing. Journal of Educational Research, 89(4), 220-233. EJ 528 634.

Wagner, Roger. (1997-2001). HyperStudio multimedia presentation software. Palo Alto, CA: Roger Wagner Publishing.

Wolf, Shelby A., & Gearhart, Maryl. (1994). Writing What You Read: A framework for narrative assessment. Language Arts, 71(6), 425-445. EJ 490 740.

Yang, S. C. (1996). A dynamic reading-linking-to-writing model for problem solving within a constructive hypermedia learning environment. Journal of Educational Multimedia and Hypermedia, 5(3/4), 263-283.

Author Information

Dr. Michael Seth Mott is an assistant professor of literacy/early childhood education at Purdue University Calumet. Dr. Mott conducts research in literacy and technology integration and designs materials for teaching both literacy and foundations of education courses.

Dr. Michael Seth Mott
Assistant Professor of Literacy
Purdue University Calumet
2200 169th Street
Hammond, IN 46323
Email: mott@calumet.purdue.edu

Cynthia Etsler is a visiting instructor in the School of Education at Purdue University Calumet. In addition to mentoring new faculty in the School of Education, Professor Etsler is designing digital courseware for courses in literacy and science methods for preservice and inservice teachers.

Deondre Drumgold is an administrator in Bank Street College's Computing and Information Services (CIS) Department where he provides computing, audiovisual, telecommunications, and networking support for the College's academic and administrative communities.

This article has been accessed 15,459 times through June 1, 2007.