Volume 5 Number 1
©The Author(s) 2003
Applying an Analytic Writing Rubric to Children's Hypermedia "Narratives"
Abstract
In an effort designed to guide and improve the assessment of a newly developed writing environment, the reliability and developmental and concurrent validity of a previously validated rubric developed for pen-and-paper-created narratives, Writing What You Read (WWYR), was determined when applied to hypermedia-authored narratives of children in second and third grade. Children (n = 60) from four intact classrooms produced hypermedia narratives (text, audio, graphic, and video elements) over a 4-month period in a school-based computer laboratory. Raters (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion. Two analyses were used to determine reliability: percentages of agreement and Pearson correlations. Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for ±0 and .99 for ±1). Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales (Theme, Character, Setting, Plot, and Communication), the r values were .59, .55, .49, .50, and .50, respectively. Developmental validity of the WWYR scores was examined with one-way MANOVA to evaluate the WWYR scores of children grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill. Evidence for the developmental validity of the WWYR scores was supported across the three ability groups, F(2, 36) = 2.59, p <. 01. Concurrent validity was examined through correlational analysis between students' mean WWYR score and ITBS score. Scores from the two measures were positively correlated, r = .83, p < .01, providing evidence of the sensitivity of the WWYR assessment to measurethe developmental literacy competency of the third-grade students. Results support teachers' use of a validated rubric developed for pen-and-paper-created narratives applied to hypermedia narratives, despite additional visual and audio narrative elements inherent to hypermedia. Implications for literacy teaching and learning with hypermedia address core questions about the similarities and differences between written textual expression and visual and verbally recorded expression. A revised and expanded WWYR rubric is proposed to begin to address these core questions generated by teachers' use of the hypermedia writing environment.
Introduction
The very notion of "writing" is increasingly being transformed by new digital computer technologies in society, homes, businesses, and schools. Today, one's ability to represent thought electronically is more important than ever. The extreme pace of change in society dictates that early childhood educators consider how to facilitate children in learning to express themselves digitally in computer environments in developmentally appropriate ways. This article addresses the curriculum, instruction, and assessment of children's written expression in hypermedia, a computer environment that supports text, audio, video, and graphics. The hypermedia curriculum and instruction are described, and a technical analysis of a proposed assessment is applied and discussed.
Emerging Trends
Prior research has addressed the reliability and validity of a narrative rubric useful for both teaching and learning (value) and large-scale (utility) literacy assessment. As literacy curriculum and instruction continue to transform to include new computer-based learning environments, researchers must correspondingly develop and technically evaluate assessments for the new environments. Hypermedia represents a powerful electronic environment through which literacy expression can be facilitated by the manipulation of text, graphics, audio, and video elements, and thistechnology and integration are reflected in the new literacy K-12 standards (IRA and NCTE, 2001). The qualities of hypermedia that support higher-level cognitive processes such as synthesis, organization, evaluation, and reflexivity have been well documented (Yang, 1996; Mott, Sumrall, & Hodges, 1997). However, there is a glaring absence of ways to reliably and validly assess students' hypermedia products. To address this absence, a narrative process writing curriculum and instruction environment was merged with hypermedia. These terms are defined as follows:
- Process writing curriculum and instruction: writing using discrete stages (brainstorming, drafting, conferencing, revising, editing, and publishing) administered via "mini-lessons" applied to address the needs of the individual writers in whole-group instruction. See Graves (1983) for a detailed description.
- Hypermedia: Hypermedia comprises two main components: (1) hyper: the ability to program electronic links, or hyperconnections, to connect information to any other Internet-based source or simply to link locally to a hard drive or diskette, and (2) media: the ability to manipulate multiple meaning-based symbol systems representing a variety of sourcestext, graphics, audio, and video clips.
This article addresses the reliability and developmental and concurrent validity of a previously validated narrative writing rubric, Writing What You Read (WWYR) (Wolf & Gearhart, 1994;Novak, Herman, & Gearhart, 1996), applied to hypermedia narratives created by students in grades 2 and 3 (see Table 1). The technical qualities of the process-oriented classroom rubric, valuable for teaching and learning on a day-to-day basis, are linked to the utility of the narrative rubric for measuring elementary students' literacy competencies as identified via a validated large-scale instrument, the Iowa Test of Basic SkillsLiteracy Competency. Thus, five main issues are addressed relating to rater judgments of elementary students' hypermedia narratives (stories with text, graphics, audio, and video elements authored using HyperStudio hypermedia software):
- Reliability of interrater judgments is examined using the WWYR of hypermedia narrative quality.
- Developmental validity of the WWYR scores is examined with one-way MANOVA used to evaluate the WWYR scores of students grouped as low, medium, or high ability based on their Iowa Test of Basic Skills (ITBS) National Percentile Rank for Literacy Skill.
- Concurrent validity is examined through correlational analysis of students' mean WWYR scores and ITBS scores.
- Value of the WWYR for use as a teaching tool is summarized through reviewing the genesis of the rubric as demonstrated in its path from creation for pen-and-paper narratives to hypermedia narratives as evaluated in multiple studies.
- Utility of the WWYR applied to students' hypermedia narratives is addressed to reveal possible alignment between the innovative curriculum and instruction addressed in the current study with ITBS- and WWYR-identified literacy levels.
Theme | Character | Setting | Plot | Communication |
---|---|---|---|---|
Explicit-Implicit | Flat-Round | Backdrop-Essential | Simple-Complex | Context-bound |
Didactic-Revealing | Static-Dynamic | Simple-Multifunctional | Static-Conflict | Literal-Symbolic |
1: Not present or not developed through other narrative elements | 1: One or two flat, static characters, with little relationship between characters | 1: Backdrop setting with little or no indication of time or place ("There was a little girl. She liked candy.") | 1: One or two events with little or no conflict ("Once there was a cat. The cat liked milk.") | 1: Writing bound to context (You have to be there) and often dependent on drawing and talk to clarify the meaning |
2: Meaning centered in a series of list-like statements ("I like my mom. And I like my dad. And I like my...") | 2: Some rounding, usually in physical description; relationship between characters is action driven | 2: Skeletal indication of time and place often held in past time ("Once there was..."); little relationship to other narrative elements | 2: Beginning sequence of events, but out-of-sync occurrences; events without problem; problem without resolution | 2: Beginning awareness of reader considerations; straightforward style and tone focused on getting the information out |
3: Beginning statement of theme, often explicit and didactic ("The mean witch chased the children and she shouldn't have done that.") | 3: Continued rounding in physical description, particularly stereotypical features ("wart on the end of her nose") | 3: Beginning relationship between setting and other narrative elements (futuristic setting to accommodate aliens and spaceships) | 3: Single, linear episode with clear beginning, middle, and end; the episode contains a problem, emotional response, action, and outcome | 3: Writer begins to make sense of explanations and transitions ("because" and "so"); literal style centers on description |
4: Beginning revelation of theme on both explicit and implicit levels through the more subtle things characters say and do | 4: Beginning insights into motivation and intention that drive the feeling and action of main characters often through limited omniscient point of view | 4: Setting becomes more essential to the development of the story in explicit ways: characters may remark on the setting, or the time and place may be integral to the plot | 4: Plot increases in complexity with more than one episode; each episode contains problem, emotional response, action, and outcome | 4: Increased information and explanation for the reader (linking ideas as well as episodes); words more carefully selected to suit the narrative's purpose |
5: Beginning use of secondary themes, often tied to overarching theme, but sometimes tangential | 5: Further rounding (in feeling and motivation); dynamic features appear in central characters and between characters | 5: Setting may serve more than one function, and the relationship between functions is more implicit and symbolic | 5: Stronger relationships between episodes (with resolution in one leading to a problem in the next) | 5: Some experimentation with symbolism (particularly figurative language), which shows reader considerations |
6: Overarching theme multilayered and complex; secondary themes integrally related to the primary themes | 6: Round, dynamic major characters through rich description of affect, intention, and motivation | 6: Setting fully integrated with the characters, action, and theme | 6: Overarching problem and resolution supported by multiple episodes | 6: Careful crafting of choices of story structure as well as vocabulary demonstrate considerate orchestration of all resources |
Research Components
Methods
Children (n = 60) from four intact classrooms (two second-grade and two third-grade classrooms) produced hypermedia narratives over a 4-month period in a school-based computer laboratory equipped with 10 Windows-based microcomputers. Rater/(Teachers) (n = 5) with knowledge of the teaching of process writing and use of hypermedia software judged the hypermedia narrative productions. An interactive hypermedia software tutorial program was developed and used to train the teachers (n = 4) in the implementation of process writing techniques in conjunction with the use of hypermedia features as part of their elementary curriculum. Raters participated in a 3-hour training and rating session in a university computer laboratory equipped with five Power Macintosh microcomputers. Raters judged all students' (n = 60) hypermedia narrative productions individually without resolving differences through discussion.
Materials
The WWYR rubric shown in Table 1 contained five evaluative scales designed to assess students' developing competencies in narrative writing: Theme, Character, Setting, Plot, and Communication. The vertical analytical evaluative scales (1-6 for each competency) were designed to enable teachers to make instructional decisions on specific narrative components needing reinforcement and were not intended as a method for assigning a numerical value to a narrative. Teachers merely had to shade a box in the rubric to denote where a child's narrative was along each competency. The ITBS (Linn & Wilson, 1990) Form J was used as a basic battery for grades K-9 and included language skills directly related to writing: word analysis, vocabulary, spelling, and reading comprehension. Reliability coefficients for Form J ranged from .70-.90 for the language skills components. Additionally, the ITBS met high standards of overall technical quality and has been a widely accepted standardized measure of cognitive skill.
Hypermedia Narratives
HyperStudio Presentation Software (Wagner, 1997-2001) was used to support the children in expressing themselves with text, audio, video, and graphic elements in their narrative productions (see Figures 1-4). Children composed on paper and computer with teachers structuring the process with Writing Workshop (Graves, 1983), a method for organizing writing into discrete and recursive stages. These stages are (1) brainstorming, (2) revising, (3) drafting, (4) peer conferencing, (5) editing, and (6) publishing (see Mott & Klomes, 2001, for a detailed description of a program similar to the program addressed in the current study).
Elements shown are (1) graphics text, (2) paint bucket tool (blue), (3) line tool: freehand, (4) hyperconnection or link to "Page 1" via button.
HyperStudio was selected for this project for a number of reasons: (1) it supports high-end features such as video and animation; (2) it is relatively easy to use, even by young children (grades 1-3); (3) it contains a flexible interface; and (4) it is the most widely used multimedia/hypermedia platform in elementary education with over 100,000 users (Wagner, 1997-2001). HyperStudio contains several programming features that support children's hypermedia programming (see Figures 5-10). Programming instruction (and writing workshop) occurred through mini-lessons over a 4-month period. Figures 5 through 10 contain examples from a mini-lesson.
The line tool was introduced to children during their first programming mini-lessons.
The "hyper" programming tool, button-creation, enabled children to connect cards (narrative pages) to other cards. Button-creation supported linear links, (card 1 to 2 to 3), suitable for early childhood.
Writing was accomplished via the textbox tool that functions as a word processor with editing tools, cutting, pasting, and other word processor capabilities.
Children inserted audio clips, programmed via button-creation, to enrich their text. Audio clips were recorded by the author or downloaded from HyperStudio for special effects such as an alarm clock sound.
Video clips were inserted, programmed via button-creation, to supplement the narrative. One example was a video clip of Mars used in a science fiction narrative by a third-grade student.
Reliability
Percentages of Agreement
Percentages of agreement for the WWYR rubric averaged across 10 pairs of raters found high percentages of agreement among raters (.70 for ±0 and .99 for ±1) (see Table 2). The ±0 and ±1 percentages of agreement across 10 pairs of raters were higher than the ±0 and ±1 agreement levels found in both the Gearhart, Herman, Novack, and Wolf (1995) and Novak, Herman, and Gearhart (1996) WWYR reliability studies (compared in Table 3). The high percentages of agreement found in this study may be attributed to the raters' use of only the first three WWYR rubric evaluative subscale levels. The WWYR rubric contains six subscale levels that are developmentally sequenced according to the varied writing competencies of students in grades K-6. Because students in this study were in second and third grade, raters typically applied only levels 1, 2, and 3 out of five total levels. This narrow range of values independently applied by raters limited the number of choices. Hence, high percentages of agreement between raters would be expected based on the limited number of scale levels used.
WWYR Subscale | ±0 | ±1 | n |
---|---|---|---|
Theme | .70 | .96 | 60 |
Character | .78 | .99 | 60 |
Plot | .73 | .99 | 60 |
Setting | .67 | .99 | 60 |
Communication | .68 | .99 | 60 |
WWYR Rating Material and Grade | ±0 | ±1 | n |
---|---|---|---|
Hypermedia
Narratives: Grades 2-3 Mott & Sumrall (1998) |
.71 | .98 | 60 |
Pen-and-Paper
Narratives: Grades 1-6 Gearhart, Herman, Novak, & Wolf (1995) |
.46 | .96 | 120 |
Collections
of Pen-and-Paper Narratives: Grades 2-5 Novak, Herman, & Gearhart (1996) |
.25 | .94 | 52 |
The percentages of agreement that were revealed in the current study, although higher than those found in the Gearhart, Herman, Novak, and Wolf (1995) study, should be considered descriptive information. Gearhart et al. remarked that percentages of agreement found for the WWYR should not be interpreted as "strong evidence of reliability" (p. 224). Rather, percentages of agreement can be used to help identify the existence of widely varying patterns of rater judgments, both across WWYR subscales and across all rater pairs. No such widely varying patterns were found in the current study. The limitations of analyses involving percentages of agreement analysis were discussed by Abedi (1994), who argued that, although percentages of agreement can reveal the existence of widely varying patterns of agreement among raters, they can also yield different results from other analyses such as Pearson correlations.
Pearson Correlations
Pearson correlations were used to further examine reliability of rater judgments. Pearson correlations averaged across 10 pairs of raters found acceptable interrater reliability for four of the five subscales. For the five subscales, Theme, Character, Setting, Plot, and Communication, the r values were .59, .55, .49, .50 and .50. Table 4 contains the results of the Pearson correlations for WWYR rubric scoring across all rater pairs for the current study and for the Gearhart et al. (1995) study. An examination of correlation scores for hypermedia narrative productions revealed that interrater reliability for four of the five WWYR subscales (Theme, Character, Setting, and Plot) was comparable to the interrater reliability levels found in the Gearhart et al. (1995) WWYR reliability study for pen-and-paper-created narratives. For the fifth subscale (Communication), however, the correlational coefficient value was .16 higher in the Gearhart et al. study than in the current study. Despite the lower value found in the current study for Communication, Gearhart et al. related that an average subscale correlation higher than .50 could be considered adequate for a rubric such as the WWYR. Table 5 summarizes the comparison of WWYR correlations across all subscales for the current study and those found in the literature (Gearhart et al., 1995; Novak et al., 1996).
WWYR Rating Material and Grade | Theme | Character | Setting | Plot | Communication | |
---|---|---|---|---|---|---|
Hypermedia Narratives: Grades 2-3 | r | .59 | .55 | .49 | .50 | .50 |
Mott & Sumrall, 1998(n = 60) | SD | .25 | .31 | .25 | .29 | .24 |
Pen-and-Paper Narratives Grades 1-6 | r | .64 | .59 | .48 | .57 | .66 |
Gearhart et al. (1995) (n = 120) | SD | .10 | .10 | .12 | .14 | .10 |
Subscale | Theme | Character | Setting | Plot | Communication |
---|---|---|---|---|---|
Samples
(n = 60)Mott
& Sumrall (1998)
|
|||||
Theme | -- | .86* | .79* | .79* | .73* |
Character | -- | -- | .74* | .74* | .76* |
Setting | -- | -- | -- | .75* | .68* |
Plot | -- | -- | -- | -- | .78* |
Communication | -- | -- | -- | -- | -- |
Samples
(n = 120)Gearhart
et al. (1995)
|
|||||
Theme | -- | .83* | .81* | .83* | .86* |
Character | -- | -- | .82* | .87* | .86* |
Setting | -- | -- | -- | .73* | .86* |
Plot | -- | -- | -- | -- | .85* |
Communication | -- | -- | -- | -- | -- |
Note: *p < .001. |
The WWYR correlations observed in this study, as well as in the Gearhart et al. (1995) study, demonstrated that ratings were highly correlated across all subscales. The r values were low in this study as well as in the Gearhart et al. (1995) and Novak et al. (1996) studies. However, set guidelines for what is an acceptable level of interrater reliability do not exist. Nonetheless, both Gearhart et al. and Novak et al., whose studies analyzed holistic scores derived from the combined r values of Theme, Character, Plot, Setting, and Communication, argued that r values that fell within the .50 to .70 range were acceptable for analytic writing rubrics. In the current study, the interrater reliability for Theme, Character, Plot, and Communication subscales fell within the .50 and .59 range, but the level of interrater reliability (r = .49) for the Setting subscale did not. It is important to note that, in the Gearhart et al. (1995) study, a low coefficient value for the subscale of Setting was also found (r = .48).1 The acceptable interrater reliabilities for Theme, Character, Plot, and Communication in this study were comparable to the acceptable levels found in the Gearhart et al. (1995) study, and the r values for the Setting subscale in both this study and the Gearhart et al. (1995) study were not acceptable (albeit by large-scale standards). It is important to note that interrater reliability levels for Theme, Character, and Plot in this study may have been lower than the r values in the Gearhart et al. (1995) study because the researcher applied more stringent rating procedures in this study. Raters in the Gearhart et al. (1995) study were permitted to resolve differences greater than one scale point through discussion, whereas raters in this study were not permitted to resolve differences. In the current study, all ratings were included in the final data set.
The highly correlated rater judgments, along all five WWYR subscales for the current study and for the Gearhart et al. (1995) study, provided further evidence of the reliability of WWYR raters' judgments. The true function of a writing rubric is that it "enables raters to apply standard criteria in making judgments about the quality of students' work" (Abedi, 1994, p. 8). Gearhart et al. (1995), Novak et al. (1996), and Abedi (1994) argued that highly correlated scores across rubric subscales can be viewed as a positive indication that raters' judgments are being consistently applied.
Validity
Developmental Validity
Developmental validity of WWYR scores was examined via one-way MANOVA conducted on the low-, medium-, and high-ability vectors of WWYR subscale scores. The assumption for this analysis was based upon the technical qualities of the ITBSLiteracy measure to delineate the developmental literacy levels of the children. Results indicated a statistically significant difference between the three ability groups (F(2, 36) = 2.59, p = .01). Table 6 provides descriptive statistics, and Table 7 provides an additional summary of these results across each of the five WWYR subscales.
Statistics | Dependent Variables | |||||
---|---|---|---|---|---|---|
n | Theme | Character | Setting | Plot | Communication | |
Mean Vectors | ||||||
ITBS Ability Level |
||||||
Low | 13 | 2.31 | 1.80 | 1.96 | 2.10 | 2.14 |
Medium | 13 | 2.80 | 2.34 | 2.32 | 2.66 | 2.52 |
High | 14 | 2.86 | 2.60 | 2.66 | 2.74 | 2.77 |
Variance-Covariance Matrix |
||||||
Theme | .21 | .15 | .19 | .13 | .11 | |
Character | -- | .23 | .13 | .15 | .13 | |
Setting | -- | -- | .19 | .14 | .11 | |
Plot | -- | -- | -- | .19 | .14 | |
Communication | -- | -- | -- | -- | .16 |
WWYR Subscale | ITBS NPR/Literacy Category | Mean Score | SD | n | F | Sig |
---|---|---|---|---|---|---|
Theme | Low | 2.31 | .62 | 16 | 6.19 | .01 |
Medium | 2.80 | .28 | 10 | |||
High | 2.86 | .31 | 13 | |||
Character | Low | 1.80 | .50 | 16 | 10.77 | .01 |
Medium | 2.34 | .38 | 10 | |||
High | 2.60 | .51 | 13 | |||
Setting | Low | 1.96 | .42 | 16 | 9.34 | .01 |
Medium | 2.32 | .56 | 19 | |||
High | 2.66 | .34 | 13 | |||
Plot | Low | 2.10 | .54 | 16 | 9.28 | .01 |
Medium | 2.66 | .34 | 10 | |||
High | 2.74 | .34 | 13 | |||
Communication | Low | 2.14 | .47 | 16 | 9.20 | .01 |
Medium | 2.52 | .34 | 10 | |||
High | 2.77 | .35 | 13 |
Tukey HSD tests were conducted on the mean vector scores of the three ability groups for all five WWYR subscales to follow up these results. For the WWYR subscale of Theme, low-ability students (M = 2.31, SD = .62) received lower scores than both medium-ability students (M = 2.80, SD = .28) and high-ability students (M = 2.86, SD = .31). For the WWYR subscales of Character, Setting, Plot, and Communication, all differences were significant. Therefore, low-ability students' scores were significantly lower than medium-ability students' scores, which were significantly lower than the high-ability students' scores. The significant differences revealed between low, medium, and high ITBS groups and the WWYR subscale scores provided evidence for the sensitivity of the WWYR to measure the development of students' hypermedia/writing competence. The significant results of the one-way MANOVA suggest that raters' judgments were evaluating students' skills as message producers (communication through text and other meaning-based symbol systems). The one-way MANOVA did not yield results that would enable the researcher to describe the degree of relatedness of raters' WWYR judgments and students' ITBS scores. In order to describe the relationship between WWYR scores and literacy skill (as measured by the ITBS), additional correlational analyses were conducted.
Concurrent Validity
The observed Pearson r correlation revealed a positive relationship between students' average WWYR score (averaged across the subscales of Theme, Character, Setting, Plot, and Communication) and their ITBS National Percentile Rank (literacy skills score), r = .83, p > .001. The positive correlation (r = .83) between students' WWYR scores and ITBS scores revealed in this analysis provided evidence for the concurrent validity (the degree to which test scores are related to the scores on an already established test) of WWYR raters' judgments of hypermedia productions. According to Messick (1992), establishment of the concurrent validity of a measure can be a stepping-stone toward establishment of the content-related validity (the degree to which scores evaluate the specific domain they were designed to evaluate) of a measure. Hence, the developmental and concurrent validities established for WWYR raters' judgments of hypermedia productions represented an important initial attempt toward eventually establishing the content-related validity of the WWYR when applied to hypermedia productions.
The strong, positive, linear relationship between ITBS literacy skill scores and WWYR rater judgments of hypermedia productions indicated that the hypermedia writing curriculum used in the current study involved literacy-based activities. The fact that students in this study expressed themselves through hypermedia features, and not solely through text, indicated that students' literacy skill can be enhanced through student expression via hypermedia and multimedia features. Table 8 provides additional information on the students' utilization of the hypermedia features used in their writing. This finding supported the claims of Daiute and Morse (1994), who observed that students who engaged in hypermedia writing developed literacy skill through the manipulation of text and other symbols. A weakness of the developmental and concurrent validity analyses was that evidence for obtaining the degree to which rater judgments of students' hypermedia productions evaluated textual features as well as textual and other hypermedia features (audio, hypermedia links, graphics, etc.) could not be determined.
Hypermedia/Multimedia Feature | |||||||
---|---|---|---|---|---|---|---|
Grade Level | Button with Hypermedia Link | Button with Audio | Button with Video | Text Box | Graphics Text | Scanned Art | Graphics Objects (Clip Art) |
2 (n = 20) | 100% | 81% | 0% | 100% | 45% | 96% | 82% |
3 (n = 40) | 100% | 100% | 5% | 100% | 64% | 100% | 100% |
*Note. In three out of the four classrooms where hypermedia/writing occurred, students' use of hypermedia/multimedia features was controlled by the teachers. |
Assessment Value and Utility
The results of this study suggest several important implications for the assessment of students' hypermedia products. Having a reliable and valid assessment for evaluating students' hypermedia-based writing serves two general purposes: (1) to enhance classroom instruction (value), and (2) to inform, to a lesser extent, educational policy (utility). The positive results yielded in this study concerning the reliability and validity of the WWYR provide an avenue for teachers to accurately and consistently evaluate their students' hypermedia narrative productions by applying the WWYR assessment. The value of an assessment is the degree to which it enhances teacher instruction by linking teachers' comments to their instructional objectives (Wolf & Gearhart, 1994). Therefore, in order for teachers to properly evaluate both student outcomes and the instructional effectiveness of their hypermedia/writing curricula, it is useful for all educators to apply a reliable and valid instrument. Furthermore, the positive correlation between the students' ITBS literacy skill score and WWYR average score for hypermedia productions indicated that students who were engaged in a hypermedia/writing curriculum improved their literacy skills.
Hypermedia | Theme | Character | Setting | Plot | Communication |
---|---|---|---|---|---|
Elements | Explicit-Implicit | Flat-Round | Backdrop-Essential | Simple-Complex | Context-bound |
Didactic-Revealing | Static-Dynamic | Simple-Multifunctional | Static-Conflict | Literal-Symbolic | |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
1: Not present or not developed through other narrative elements | 1: One or two flat, static characters, with little relationship between characters | 1: Backdrop setting with little or no indication of time or place ("There was a little girl. She liked candy.") | 1: One or two events with little or no conflict ("Once there was a cat. The cat liked milk.") | 1: Writing bound to context (You have to be there) and often dependent on drawing and talk to clarify the meaning |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
2: Meaning centered in a series of list-like statements ("I like my mom. And I like my dad. And I like my...") | 2: Some rounding, usually in physical description; relationship between characters is action driven | 2: Skeletal indication of time and place often held in past time ("Once there was..."); little relationship to other narrative elements | 2: Beginning sequence of events, but out-of-sync occurrences; events without problem; problem without resolution | 2: Beginning awareness of reader considerations; straightforward style and tone focused on getting the information out |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
3: Beginning statement of theme, often explicit and didactic ("The mean witch chased the children and she shouldn't have done that.") | 3: Continued rounding in physical description, particularly stereotypical features ("wart on the end of her nose") | 3: Beginning relationship between setting and other narrative elements (futuristic setting to accommodate aliens and spaceships) | 3: Single, linear episode with clear beginning, middle, and end; the episode contains a problem, emotional response, action, and outcome | 3: Writer begins to make sense of explanations and transitions ("because" and "so"); literal style centers on description |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
4: Beginning revelation of theme on both explicit and implicit levels through the more subtle things characters say and do | 4: Beginning insights into motivation and intention that drives the feeling and action of main characters often through limited omniscient point of view | 4: Setting becomes more essential to the development of the story in explicit ways: characters may remark on the setting, or the time and place may be integral to the plot | 4: Plot increases in complexity with more than one episode; each episode contains problem, emotional response, action, and outcome | 4: Increased information and explanation for the reader (linking ideas as well as episodes); words more carefully selected to suit the narrative's purpose |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
5: Beginning use of secondary themes, often tied to overarching theme, but sometimes tangential | 5: Further rounding (in feeling and motivation); dynamic features appear in central characters and between characters | 5: Setting may serve more than one function, and the relationship between functions is more implicit and symbolic | 5: Stronger relationships between episodes (with resolution in one leading to a problem in the next) | 5: Some experimentation with symbolism (particularly figurative language), which shows reader considerations |
-Text- -Hypertext- -Graphic- -Audio- -Video- |
6: Overarching theme multilayered and complex; secondary themes integrally related to the primary themes | 6: Round, dynamic major characters through rich description of affect, intention, and motivation | 6: Setting fully integrated with the characters, action, and theme | 6: Overarching problem and resolution supported by multiple episodes | 6: Careful crafting of choices of story structure as well as vocabulary demonstrate considerate orchestration of all resources |
Note
1. A note on correlations averaged across raters: (1) A relatively small number of raters (n = 5) were used in this study and the Gearhart, Herman, Novak, and Wolf (1995) study, which may have contributed to the lower r values across all subscales. The attenuation of correlational coefficients may be another explanation for the low levels of interrater reliability (Gay, 1996). Accordingly, coefficients tend to be lower when a restricted range of values is utilized (e.g., the narrow range of only 3 out of a possible 6 WWYR subscale levels utilized by raters in this study). Thus, the more narrow the range of scores utilized by raters, the lower the coefficients. On the other hand, Gearhart et al. argued that if the number of raters was statistically increased five-fold, r values in the .50 to .60 range for Theme, Character, Setting, Plot, and Communication would be changed to .87, .89, .82, .86, and .89, respectively. Gearhart et al. used decision-study (multiplication of sample scores and aggregation of the results) coefficients to determine the number of raters needed to attain high reliability coefficients. (2) The r value for the Communication subscale in this study was considerably lower than the r value in the Gearhart et al. study (r = .50 versus .66). This sizable disparity in the level of interrater reliability may have been the result of the contrasting features of hypermedia-created narrative productions versus pen-and-paper-created narratives. The Communication subscale text primarily consisted of evaluative prompts designed to guide teachers in the assessment of writing style (see Table 1). Perhaps, in the current study, raters solely viewed textual features at the expense of the hypermedia features of graphics, sounds, buttons, and scanned art.
References
Abedi, Jamal. (1994). Final report of achievement, section A (report to the National Center for Education Statistics, contract no. RS90159001). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
Daiute, Colette, & Morse, Frances. (1994). Access to knowledge and expression: Multimedia writing tools for students with diverse needs and strengths. Journal of Special Education Technology, 12(3), 221-256. EJ 492 959.
Gay, Lorrie R. (1996). Educational research: Competencies for analysis and application (5th ed.). Englewood Cliffs, NJ: Prentice-Hall.
Gearhart, Maryl; Herman, Joan L.; Novak, John R.; & Wolf, Shelby A. (1995). Toward the instructional utility of large-scale writing assessment: Validation of a new narrative rubric. Assessing Writing, 2(2), 207-242. EJ 562 433.
Graves, Donald. (1983). Writing: Teachers and children at work. Portsmouth, NH: Heinemann. ED 234 430.
International Reading Association (IRA) and National Council of Teachers of English (NCTE). (2001). Standards for the English language arts. Urbana, IL: NCTE; Newark, DE: IRA.
Linn, R., & Wilson, V. (1990). Review of the Iowa Test of Basic Skills form J. In Jane Close Conoley and James C. Impara (Eds.), Mental measurement yearbook (9th ed.). Lincoln: University of Nebraska Press.
Messick, Stephen. (1992). Validity of test interpretation and use. In M. Alkin (Ed.), Encyclopedia of education research (6th ed., pp. 1487-1495). New York: Macmillan.
Mott, Michael S., & Klomes, Jeanine. (2001). The synthesis of writing workshop and hypermedia-authoring: Grades 1-4. Early Childhood Research & Practice [Online], 3(2). Available: http://ecrp.illinois.edu/v3n2/mott.html [2003, February 6]. ED 458 043.
Mott, Michael S., & Sumrall, William J. (1998). Scientists are presenters: Tech trek: Interactive media. Science Scope, 21(7), 42-45.
Mott, Michael S.; Sumrall, William J.; & Hodges, M. Lee. (1997, November). Process and computer-based elementary writing curriculum: A review of methods and assessments. Paper presented at the annual meeting of the Mid-South Educational Research Association, Memphis, TN. ED 415 255.
Novak, John R.; Herman, Joan L.; & Gearhart, Maryl. (1996). Establishing validity for performance-based assessments: An illustration for collections of student writing. Journal of Educational Research, 89(4), 220-233. EJ 528 634.
Wagner, Roger. (1997-2001). HyperStudio multimedia presentation software. Palo Alto, CA: Roger Wagner Publishing.
Wolf, Shelby A., & Gearhart, Maryl. (1994). Writing What You Read: A framework for narrative assessment. Language Arts, 71(6), 425-445. EJ 490 740.
Yang, S. C. (1996). A dynamic reading-linking-to-writing model for problem solving within a constructive hypermedia learning environment. Journal of Educational Multimedia and Hypermedia, 5(3/4), 263-283.
Author Information
Dr. Michael Seth Mott is an assistant professor of literacy/early childhood education at Purdue University Calumet. Dr. Mott conducts research in literacy and technology integration and designs materials for teaching both literacy and foundations of education courses.
Dr.
Michael Seth Mott
Assistant Professor of Literacy
Purdue University Calumet
2200 169th Street
Hammond, IN 46323
Email: mott@calumet.purdue.edu
Cynthia Etsler is a visiting instructor in the School of Education at Purdue University Calumet. In addition to mentoring new faculty in the School of Education, Professor Etsler is designing digital courseware for courses in literacy and science methods for preservice and inservice teachers.
Deondre Drumgold is an administrator in Bank Street College's Computing and Information Services (CIS) Department where he provides computing, audiovisual, telecommunications, and networking support for the College's academic and administrative communities.