Moreover, high human-computer agreement does not guarantee that the system is meaningfully assessing important aspects of writing, such as evidence use. In fact, human raters can be inconsistent in their assessments of writing and disagree with one another. Human raters have long been considered the "gold standard" for assessing student writing. For the most part, assessments of how well AES and AWE systems perform have focused solely on how closely their scores correspond to human ratings of the same essays. How well AES and AWE systems assess the targeted aspect of writing is rarely considered. AES systems that attend to evidence use and AWE systems that provide feedback supporting high-quality use of evidence would have the potential to guide instructional next steps or students' revision process. Even so, few systems attend to the substantive dimension of evidence use, which is concerned with how the writing is supported by source material. In recent years, systems have started to attend more to substantive dimensions of writing. Holistic scores often rely on surface aspects of writing (e.g., spelling, grammar, syntax, word count) because these aspects can be more reliably evaluated by widely used natural language processing (NLP) techniques. Historically, AES systems have tended to provide a holistic score rather than scores on specific dimensions of writing (e.g., content, organization, style, mechanics). Thorough assessment of analytic text-based writing requires evaluating the content of students' writing, particularly how they marshal evidence from a source text to support their analysis. AES and AWE systems rarely provide guidance for students to improve substantive aspects of their writing.Moreover, students benefit from the more immediate-and consistent-feedback that AWE systems can offer, which supports cycles of revision.ĪES and AWE systems, however, are not without challenges. AES and AWE systems can reduce teachers' time burden related to grading and providing feedback, respectively, enabling teachers to assign more such writing tasks and thus provide more opportunities for students to learn and practice the skills. AES refers to the use of computer programs to assign a score to a piece of writing AWE systems provide feedback on one or more aspects of writing and may or may not also provide scores. The Promise and Perils of Automated Writing Scoring and Feedback SystemsĪutomated essay scoring (AES) and automated writing evaluation (AWE) systems have potential benefits. When students do write, they rarely receive substantive feedback and rarely engage in cycles of revision that require them to apply feedback to strengthen their work. Elementary grades have not historically focused on analytic text-based writing, teachers report feeling underprepared to teach writing, and the time needed to assess student writing is burdensome. Students' opportunities to practice and learn analytic text-based writing are limited, however. As such, it is emphasized in national and state English language arts and writing standards. Specifically, analytic text-based writing-which focuses on analysis and interpretation of fiction or informational texts, using evidence from the source text to support claims and interpretations-is critical to academic success and college readiness (Wang et al., 2018). This brief summarizes some of the contributions of the work, which have been documented in greater detail in peer-reviewed articles. Since 2017, experts at the RAND Corporation and the University of Pittsburgh have been conducting research to advance automated writing scoring and feedback systems for elementary school student writers.
0 Comments
Leave a Reply. |