Neural Automated Writing Evaluation with Corrective Feedback
Abstract
The utilization of technology in second language learning and teaching has become ubiquitous. For the assessment of writing specifically, automated writing evaluation (AWE) and grammatical error correction (GEC) have become immensely popular and effective methods for enhancing writing proficiency and delivering instant and individualized feedback to learners. By leveraging the power of natural language processing (NLP) and machine learning algorithms, AWE and GEC systems have been developed separately to provide language learners with automated corrective feedback and more accurate and unbiased scoring that would otherwise be subject to examiners. In this paper, we propose an integrated system for automated writing evaluation with corrective feedback as a means of bridging the gap between AWE and GEC results for second language learners. This system enables language learners to simulate the essay writing tests: a student writes and submits an essay, and the system returns the assessment of the writing along with suggested grammatical error corrections. Given that automated scoring and grammatical correction are more efficient and cost-effective than human grading, this integrated system would also alleviate the burden of manually correcting innumerable essays. A link to the demonstration video: https://youtu.be/CoqmBqqcXio
Neural Automated Writing Evaluation with Corrective Feedback
Izia Xiaoxiao Wang1∗ Xihan Wu2∗ Edith Coates3††thanks: Equally contributed authors. Min Zeng2 Jiexin Kuang2 Siliang Liu2 Mengyang Qiu4 Jungyeul Park2 1Linguistik Zentrum Zürich, Universität Zürich, Schweiz 2Department of Linguistics, The University of British Columbia, Canada 3Department of Mathematics, The University of British Columbia, Canada 4Department of Psychology, Trent University, Canada
1 Introduction
Second language (L2) instructors and learners both believe that instructors are not fulfilling their duties if they fail to offer sufficient feedback on the essay submissions (Cho, 2018). However, providing and revising feedback can be incredibly time-consuming and laborious tasks for both language instructors and learners. Automated writing evaluation and grammatical error correction tools would offer potential solutions for alleviating the workload for language instructors and reducing the overhead of trying to understand the instructor’s feedback for language learners. These systems help to meet the need for more efficient practices in the digital age, providing instantaneous scoring with corrective feedback.
The basic term AWE is defined as “the process of evaluating and scoring written prose via computer programs” (Shermis and Burstein, 2003). In essence, this system predicts continuous values for a holistic score or a set of trait scores, thereby providing a comprehensive assessment of the writing quality. AWE systems require multiple source prompt (e.g. cross-prompt) scoring instead of prompt-specific scoring. In practical usage, a set of scores across different rubrics is much more in need, rather than a holistic and overall writing quality assessment. Therefore, we argue that an effective AWE system should incorporate cross-prompt rubric scoring for constructive results.
GEC is the automation of detecting and correcting grammatical errors in the text. As the number of second language learners of English rises, and the demand for timely feedback to better facilitate their learning intensifies, GEC has gained increasing popularity and attention in both academia and industry in recent years. Currently, a common approach is to treat GEC as a translation task where incorrect sentences are translated into correct sentences.
In this paper, we aim to demonstrate a system that would contribute to the integration of AWE and GEC and is designed to facilitate the language learning process and promote better learning outcomes. While there exist various approaches that attempt to leverage the efficiency of GEC or AWE systems to give away seemingly instantaneous but virtually momentary solutions for language learners, our objective is to introduce a robust and sensible approach to transform traditional language education. Our approach empowers language learners to "write it right" by enabling simulated examination situations.
2 Previous Work
AWE
Jacobs et al. (1981) first proposed the prototype for a set of components that in conjunction capture the communicative effectiveness of compositions holistically. Later, it was developed and refined by Connor-Linton and Polio (2014). Moreover, a variation of the same holistic evaluation on essays is implicitly used in develo** the different attributes in the ASAP++ data set (Mathias and Bhattacharyya, 2018) for English. Currently, neural models have been dominating AWE systems. Ke and Ng (2019), Ramesh and Sanampudi (2021), and Uto (2021) have summarized recent neural models well. For automatic essay scoring, there are two main model types. Firstly, in RNN-based models, the RNN output is sent to mean-over-time to aggregate the input to the fixed length vector and a linear layer for the scalar value (Taghipour and Ng, 2016) , or alternatively, a simple BiLSTM to the linear layer is used for predicting essay scores (Alikaniotis et al., 2016). Secondly, transformer-based models, e.g., BERT with BiLSTM with attention (Nadeem et al., 2019) or BERT concatenated with handcrafted features (Uto et al., 2020), can be used to predict the score. Fine-tuning BERT using multiple losses, including regression loss and reranking loss, for constraining automated essay scores has been shown to produce state-of-the-art results (Yang et al., 2020).
GEC
Similar to AWE, neural models have also dominated GEC systems, where Wang et al. (2021) presents a comprehensive review. In general, recent state-of-the-art systems can be divided into two categories: sequence-to-sequence NMT-based approaches and sequence tagging edit-based approaches. For NMT-based approaches, a transformer-based encoder-decoder architecture is typically used, where source (or incorrect) sentences are encoded into a hidden state with a multi-head self-attention layer (Vaswani et al., 2017). Unlike a typical translation task, GEC only requires changing a few words in a sentence. Incorporating a copying mechanism into seq2seq (that is, copying unchanged words from original sentences) has been shown to significantly boost GEC performance (Zhao et al., 2019). This unique characteristic of GEC (i.e., the high overlap between the source and target sentences) has also led to the development of edit-based approaches (Omelianchuk et al., 2020). Edit-based models also use a transformer-based architecture. However, instead of predicting complete sentences, these models are trained to predict a series of editing operations (e.g., delete, append, and replace), which has dramatically improved the speed of inference while maintaining high performance. Given that there is a lack of sufficient parallel training data in GEC, data augmentation via artificial error generation has been shown to boost performance in both NMT-based and edit-based approaches (Stahlberg and Kumar, 2021). In addition, incorporating large pre-trained masked language models, such as BERT and its variants (Devlin et al., 2019), in pre-training and/or fine-tuning stages has yielded better results (Kaneko et al., 2020).
Only a few previous work have shown for their efforts to integrate pre-neural AWE and GEC systems together where they have focused on perceived feedback of language learners (Ranalli, 2018; O’Neill and Russell, 2019; Zhang, 2020; Ariyanto et al., 2021; Reynolds et al., 2021). Whereas empirical studies have explored students’ perception, beliefs, and preferences towards feedback, their influential role in their writing performance has remained unknown (Truscott, 1999; Ferris, 1999; Chandler, 2003; Ferris, 2004, 2014). Hence, this paper concentrates on the integration of the AWE and GEC systems and its impact on language education. The combination of neural AWE and GEC systems is introduced for the first time to achieve results that are close to the state-of-the-art (SOTA).
3 System Description
The flowchart in Figure 1 demonstrates the learner’s interactions with the system, from the initial step of submitting the writing to receiving the AWE and GEC results. Adapted to the language learners’ perspectives, the procedures are disassembled as the following:
(i) After completing the writings, language learners submit their writings to the system. (ii) Taken the submitted writing as input, the system compiles both AWE and GEC components with it. (iii) For the GEC, the output presents in-line grammatical error corrections with the original writing. (iv) For the AWE, the output offers one overall and eight rubric scores, which delivers the complete set of the evaluation results from the AWE system. (v) At the output interface, the language learner receives the combined feedback from the two integrated outputs generated from AWE and GEC.
-
(1)
*I gess almost people cannot speaking English.
-
(2)
*I guess most people cannot speak English.
Figure 2 shows an interface screenshot of the integrated AWE and GEC results for the sentence from a language learner’s writing in (1) and grammatical errors corrected in (2). AWE results show the absolute values between 0 and 100 regardless of their proficiency level. These AWE scores can also be normalized based on the learner’s proficiency level using min-max normalization if it is required.
AWE
For AWE models,111Since we are working with L2 English learners, we prefer to use AWE instead of AES (automatic essay scoring), which is designated for L1 English learners. we use Automated Student Assessment Prize (ASAP)222https://www.kaggle.com/competitions/asap-aes and its extension ASAP++ (Mathias and Bhattacharyya, 2018) for English. ASAP was introduced as part of a Kaggle competition in 2012. It has since become widely used in prompt-specific (Alikaniotis et al., 2016; Dong et al., 2017) and cross-prompt (Cummins et al., 2016; Ridley et al., 2021; Jiang et al., 2023; Chen and Li, 2023; Do et al., 2023) automated scoring systems. Since only two of eight sets in ASAP have scores for individual essay attributes, ASAP++ annotates other rubric scoring for six sets of ASAP. The current ASAP and ASAP++ datasets contain eight different prompt essay sets with from three to five rubric human-rated scores: e.g. content, organization, word choice, sentence fluency, conventions, prompt adherence, language, narrativity as well as overall. ASAP consists of only overall scores for Prompts 1-6, and different rubric scores (content, organization, conventions for Prompt 7, and content, organization, word choice, sentence fluency, conventions for Prompt 8). ASAP++ provides various rubric scores for Prompts 1–6 (content, organization, word choice, sentence fluency, convention for Prompts 1-2, and content, prompt adherence, language, narrativity for Prompt 3-6) to supplement the original overall scores from ASAP (Prompt 1-6).
For the current system, we use the denoised ASAP dataset, which we employ simple text denoising techniques using prompting. The original ASAP dataset employed a named entity recognition approach using Stanford CoreNLP (Manning et al., 2014) and a range of pattern matching rules to eliminate personally identifying information from the essays. Consequently, entities within the text are identified and replaced with strings initiated with the ‘@’ symbol, such as @PERSON1. Furthermore, more than 5% of sentences exhibit encoding issues where UTF-8 symbols are not correctly displayed, in addition to the presence of non-word entities. We classify sentences containing encoding issues and non-word entities as noise, and subsequently, undergo a denoising process to address them. Our hypothesis posits that enhancing text quality through denoising will yield improved linear regression results in AES. For this process, we employ two prompts with gpt-3.5-turbo-instruct: one to address encoding errors and another to replace non-word entities with arbitrary entity names sequentially. It is noteworthy that gpt-3.5-turbo-instruct has a tendency to correct grammatical errors in the original text during sentence generation. To restore the original words, we utilize the .m2 annotation generated by ERRANT (Bryant et al., 2017). This annotation delineates the modifications made in the original text, allowing us to retain only the symbols with corrected encoding and the replaced non-word entities. After text denoising, we utilize roberta-base (Liu et al., 2019) for the linear regression task. Detailed prompt-by-prompt results are presented in Table 1, including the evaluation conducted on the original text to ensure a fair comparison (Cleaned). During training, we utilize the default values of the Trainer class,333https://huggingface.co/docs/transformers/main_classes/trainer employing the RoBERTa base model. We normalize the score to the range of 0 and 1, and we multiply the results by 100 to calculate QWK using the standard evaluation script provided by ets.org.
| Prompt 1 | Prompt 2 | Prompt 3 | Prompt 4 | Prompt 5 | Prompt 6 | Prompt 7 | Prompt 8 | Average | |
|---|---|---|---|---|---|---|---|---|---|
| Original | 0.6187 | 0.6308 | 0.6962 | 0.6626 | 0.7158 | 0.5924 | 0.4622 | 0.4586 | 0.6047 |
| Cleaned | 0.7344 | 0.5485 | 0.6978 | 0.6540 | 0.7416 | 0.6156 | 0.4637 | 0.4589 | 0.6143 |
GEC
We build seq2seq GEC models using fairseq (Ott et al., 2019). We explore ensemble models that incorporate different pre-trained large language models (LLMs), including BERT and BART. The key difference between the two LLMs is in their pre-training objectives. BERT is trained using a masked language modelling objective, where a certain percentage of the input tokens are masked, and the model is trained to predict the original tokens. In contrast, BART is trained using a denoising auto-encoding objective, where the model is trained to reconstruct the original input text from a corrupted version of the input. Our current training setup is largely based on Kaneko et al. (2020). We use four GEC datasets – FCE, NUCLE, W&I +LOCNESS, and Lang-8 – provided by the BEA 2019 Shared Task for training (Bryant et al., 2019).444https://www.cl.cam.ac.uk/research/nl/bea2019st All the incorrect sentences are spell-checked and added back in order to augment the size of the original training set (1,157,324 sentence pairs), resulting in 1,718,693 sentence pairs in total. 6,575 sentence pairs from the FCE and W&I+LOCNESS development sets are used as validation. Our best model (using BERT) achieves a 65.29 F0.5 score on the BEA 2019 test set. We have also been exploring various ways of fine-tuning the power of LLMs as well as edit-based approaches such as GECToR (Omelianchuk et al., 2020) and T5 (Rothe et al., 2021), which currently have provided SOTA results for the GEC task. These pre-trained models are prepared for seamless integration into our system. While prompting GPT for the GEC task has also been been investigated, it could not obtain the state of the art results for the GEC task, and this is still prevalent in recent previous work (Loem et al., 2023).
4 Discussion and Future Perspectives
Features and metrics
In the integration, we have examined several automatic metrics as features to describe the attributes of the training dataset. Specifically, these attributes are represented in terms of complexity, fluency, and accuracy features. First, complexity features use quantitative measures such as the number of words and sentences in the text with their numbers and mean lengths. The length of the written text has been considered an essential feature in the learner corpus mostly for identifying proficiency levels. Complexity features can also measure syntactic complexity in L2 writing (Polio, 1997; Ortega, 2003; Lu, 2010). Syntactic complexity measures are also considered, which include Yngve’s depth algorithm (Yngve, 1960), Frazier’s local non-terminal numbers (Frazier, 1985), and the D-level scale (Rosenberg and Abbeduto, 1987; Covington et al., 2006) in L1 writing. Second, fluency, as the potential of language learners to apply their knowledge of grammar to produce intelligible speech and writing, also plays an important role in language production. For fluency, we considered the two metrics defined in Asano et al. (2017) and Ge et al. (2018) as fluency features of AWE. Third, accuracy represents the ability to produce correct sentences using correct grammar and vocabulary. However, it requires linguistic annotation such as grammatical error categories and error correction. Currently, the AWE and GEC models have been trained independently. We plan to introduce GEC results as accuracy features for the AWE system in future works.
Multilingual aspects for GEC
Current system is demonstrated with GEC results for English, but the GEC model is being extended to the multilingual environment, including Chinese and Korean, using multilingual Transformers such as mBART (Liu et al., 2020). For Chinese, our current GEC training set is from NLPTEA 2014-2016 CGED Shared Tasks (Lee et al., 2016).555https://sites.google.com/view/nlptea2018/shared-task It contains 587,050 sentence pairs from writing samples of Chinese L2 language learners (i.e., the writing section of the computer-based Test of Chinese as a Foreign Language (TOCFL) and of the Hanyu Shui** Kaoshi (HSK) (‘Chinese Proficiency Test’)). Chinese GEC sentences are segmented into words using pkuseg in SpaCy666https://github.com/explosion/spacy-pkuseg. For Korean, the grammatical error correction dataset from National Institute of Korean Language777https://www.korean.go.kr contains 57,320 sentence pairs for training from written writing samples of Korean L2 language learners. Korean GEC sentences are preprocessed into the sequence of morphemes as a de facto format for Korean machine translation. With ERRANT (Bryant et al., 2017) being adapted to Chinese (Hinson et al., 2020), Korean (Yoon et al., 2023), and many other languages, it is now possible to streamline grammatical error annotation and evaluation across languages.
Multilingual aspects for AWE and universal rubrics
Similar to the multilingual GEC, we also adopt a multilingual perspective for our AWE system. Although the current AWE is demonstrated to assess English inputs, it is designed to be compatible with multilingual input data. This is made possible by two configurations in the assessment process: (1) having the same underlying scoring scheme for the AWE training data regardless of what language is being used, and (2) the set of evaluation standards designed in the scoring scheme assesses the communicative effectiveness of the writers’ compositions, rather than any language-specific traits. Both the multilingual scoring scheme and the evaluation standards are ascertained from comparing the proposed language-specific AWE rubrics in the literature, and then extracting the shared analytic traits to serve as the basis for multilingual AWE’s training dataset evaluation rubrics. Following the same mechanism, our AWE has the potential to easily expand to incorporate a new language if that language’s proposed AWE evaluation rubrics can be accommodated into our existing ones. In fact, for the Korean AWE system, Lim et al. (2023) already implemented a holistic scoring system based on the learner corpus from L2 learners of Korean (Park and Lee, 2016). We are currently develo** a multilingual rubric scoring system based on the evaluation rubrics from ASAP++ (Mathias and Bhattacharyya, 2018) for English, ACEA dataset’s multi-traits (He et al., 2022) for Chinese, and the rubrics used in the recently released Korean essay evaluation dataset for Korean.888Released in October 2022, https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=545
| Perceived fairness and accuracy of GEC and AWE: | |
| - On a scale of 1-5, how fair do you think the grammar feedback on this writing to be? (1= not fair at all, 5= very fair) | |
| - On a scale of 1-5, how accurate and trustworthy do you think the grammar correction on this writing to be? (1= not accurate/trustworthy at all, 5= very accurate/trustworthy) | |
| - On a scale of 1-5, how fair do you think the grading and evaluation of this writing to be? (1= not fair at all, 5= very fair) | |
| - On a scale of 1-5, how accurate and trustworthy do you think the grading and evaluation of this writing to be? (1= not accurate/trustworthy at all, 5= very accurate/trustworthy) | |
| \hdashline Perceived clarity and helpfulness: | |
| - On a scale of 1-5, how clear do you think the grading and feedback on this writing to be? (1= not clear at all, 5= very clear) | |
| - On a scale of 1-5, how helpful do you perceive the feedback on this writing to be in improving this student’s writing skills? (1= not helpful at all, 5= very helpful) | |
| - On a scale of 1-5, how helpful do you think rubrics with criteria and scores to be in understanding the grading and feedback on this writing assignment? (1= not helpful at all, 5= very helpful) | |
| \hdashline Mental effort: | |
| - On a scale of 1-5, how difficult do you find it to perceive the grading and feedback on this writing to be? | |
| - On a scale of 1-5, how much mental effort does this task (differ for stages 1 and 2) require from you to perceive the feedback? | |
| \hdashline Attitudinal responses: | |
| - On a scale of 1-5, how comfortable do you feel about perceiving/using this system to receive the grading and feedback on this writing? | |
| - On a scale of 1-5, how effective do you feel about perceiving/using this system to receive the grading and feedback on this writing? | |
Teacher’s view
The workflow in Figure 1 focuses on the perspectives of language learners, in which they can prepare for language examinations in a more tangible and efficient way. By simply submitting their complete writing to the system, not only does this workflow enable simulated examination situations, where the language learners receive instant objective scoring results, but also corrective feedback is provided for each submission. In addition to serving language learners, Figure 3 indicates another scope, where the system also facilitates language instructors. In this workflow, instructors can intervene with the AWE and GEC process in correspondence with desired rubrics and conventions before the results are delivered to students. This seemingly minor inclusion of the instructor’s view capacitates the system’s potentiality to extend from simulating examinations for students’ practice to facilitating actual examination situations. Therefore, this integrated system would also alleviate the burden of manually correcting innumerable writings from language instructors. This would enable the instructors to focus their expertise on teaching, thereby providing students with more valuable language learning experiences.
Perception problem
There are several studies which investigated the effectiveness of AWE on improving learners’ writing performances (Chodorow et al., 2010; Lee, 2020) and students’ perceptions towards AWE tools (Chen and Cheng, 2008; Link et al., 2014; Ranalli, 2018; Zhang and Hyland, 2018; O’Neill and Russell, 2019; Zhang, 2020; Ariyanto et al., 2021). The majority of the researches looked into the use of AWE feedback for L2 writing on single-group designs (Dikli and Bleyle, 2014; Lavolette et al., 2015). Thus, it remains unclear to what extent automated feedback provided by the AWE system supports second language acquisition at various proficiency levels. The present study further investigates the impact of AWE feedback on L2 writing performance and students’ perceptions at varying proficiency levels. Therefore, we are planning to conduct the following experiments: (i) student perception experiments: how language learners perceive results from the automated system without human instructor intervention; (ii) instructor perception experiments: how language instructors perceive and evaluate results from the automated system being conveyed to language learners. , in collaboration with Centre for Teaching, Learning and Technology at The University of British Columbia (See Table 4).999https://ctlt.ubc.ca
5 Conclusion
With the seamless integration of neural AWE and GEC, the proposed system will be made available to the general public for the potential to serve as an invaluable language resource for language education. With features that extend from simulating examination settings, this integrated system empowers language learners to improve their writing skills and language proficiency through hands-on mistakes and encourages them to "write it right" to facilitate actual examination situations. This multifaceted tool offers comprehensive approaches that not only promote better learning outcomes but also transform the traditional interaction between language learners and instructors in language education. These innovative approaches make the proposed system a promising language resource for anyone who seeks to improve the L2 learning experience. The proposed system is accessible at https://open-writing-evaluation.github.io/awe/.
Ethics Statement
To confirm Behavioural Research Ethics at the University of British Columbia,101010https://ethics.research.ubc.ca/behavioural-research-ethics authors have obtained a certificate of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (TCPS 2): Course on Research Ethics (CORE-2022).111111https://tcps2core.ca
Acknowledgement
This research is based upon work partially supported by the SoTL Seed Program for Jungyeul Park at The University of British Columbia.
References
- Alikaniotis et al. (2016) Dimitrios Alikaniotis, Helen Yannakoudakis, and Marek Rei. 2016. Automatic Text Scoring Using Neural Networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 715–725, Berlin, Germany. Association for Computational Linguistics.
- Ariyanto et al. (2021) Meilisa Sindy Astika Ariyanto, Nur Mukminatien, and Sintha Tresnadewi. 2021. College Students’ Perceptions of an Automated Writing Evaluation as a Supplementary Feedback Tool in a Writing Class. Jurnal Ilmu Pendidikan, 27(1):41–51.
- Asano et al. (2017) Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017. Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343–348, Taipei, Taiwan. Asian Federation of Natural Language Processing.
- Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 Shared Task on Grammatical Error Correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.
- Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
- Chandler (2003) Jean Chandler. 2003. The efficacy of various kinds of error feedback for improvement in the accuracy and fluency of L2 student writing. Journal of Second Language Writing, 12(3):267–296.
- Chen and Cheng (2008) Chi-Fen Emily Chen and Wei-Yuan Eugene Cheng. 2008. Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in EFL Writing Classes. Language Learning & Technology, 12(2):94–112.
- Chen and Li (2023) Yuan Chen and Xia Li. 2023. PMAES: Prompt-map** Contrastive Learning for Cross-prompt Automated Essay Scoring. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1489–1503, Toronto, Canada. Association for Computational Linguistics.
- Cho (2018) Myounghee Cho. 2018. The Use and Efficacy of Correction Symbols in KFL Writing. The Korean Language in America, 22(1):83–92.
- Chodorow et al. (2010) Martin Chodorow, Michael Gamon, and Joel Tetreault. 2010. The utility of article and preposition error correction systems for English language learners: Feedback and assessment. Language Testing, 27(3):419–436.
- Connor-Linton and Polio (2014) Jeff Connor-Linton and Charlene Polio. 2014. Comparing perspectives on L2 writing: Multiple analyses of a common corpus. Journal of Second Language Writing, 26(December):1–9.
- Covington et al. (2006) Michael A. Covington, Congzhou He, Cati Brown, Lorina Naci, and John Brown. 2006. How complex is that sentence? A proposed revision of the Rosenberg and Abbeduto D-Level Scale. Technical report, University of Georgia, Athens, Georgia.
- Cummins et al. (2016) Ronan Cummins, Meng Zhang, and Ted Briscoe. 2016. Constrained Multi-Task Learning for Automated Essay Scoring. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–799, Berlin, Germany. Association for Computational Linguistics.
- Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- Dikli and Bleyle (2014) Semire Dikli and Susan Bleyle. 2014. Automated Essay Scoring feedback for second language writers: How does it compare to instructor feedback? Assessing Writing, 22(0):1–17.
- Do et al. (2023) Hee** Do, Yunsu Kim, and Gary Geunbae Lee. 2023. Prompt- and Trait Relation-aware Cross-prompt Essay Trait Scoring. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1538–1551, Toronto, Canada. Association for Computational Linguistics.
- Dong et al. (2017) Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based Recurrent Convolutional Neural Network for Automatic Essay Scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 153–162, Vancouver, Canada. Association for Computational Linguistics.
- Ferris (1999) Dana R. Ferris. 1999. The case for grammar correction in L2 writing classes: A response to truscott (1996). Journal of Second Language Writing, 8(1):1–11.
- Ferris (2004) Dana R. Ferris. 2004. The “Grammar Correction” Debate in L2 Writing: Where are we, and where do we go from here? (and what do we do in the meantime …?). Journal of Second Language Writing, 13(1):49–62.
- Ferris (2014) Dana R. Ferris. 2014. Responding to student writing: Teachers’ philosophies and practices. Assessing Writing, 19(1):6–23.
- Frazier (1985) Lyn Frazier. 1985. Syntactic complexity. In David R Dowty, Lauri Karttunen, and Arnold M Zwicky, editors, Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives, pages 129–189. Cambridge University Press, Cambridge.
- Ge et al. (2018) Tao Ge, Furu Wei, and Ming Zhou. 2018. Fluency Boost Learning and Inference for Neural Grammatical Error Correction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1055–1065, Melbourne, Australia. Association for Computational Linguistics.
- He et al. (2022) Yaqiong He, Feng Jiang, Xiaomin Chu, and Peifeng Li. 2022. Automated Chinese Essay Scoring from Multiple Traits. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3007–3016, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Hinson et al. (2020) Charles Hinson, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Heterogeneous Recycle Generation for Chinese Grammatical Error Correction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2191–2201, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Jacobs et al. (1981) Holly L. Jacobs, Stephen A. Zinkgraf, Deanna R. Wormuth, and V. Faye Hearfiel. 1981. Testing ESL composition: a practical approach. Longman Higher Education Division (a Pearson Education company), Rowley, MA.
- Jiang et al. (2023) Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, and Qing Gu. 2023. Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation Learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12456–12470, Toronto, Canada. Association for Computational Linguistics.
- Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4248–4254, Online. Association for Computational Linguistics.
- Ke and Ng (2019) Zixuan Ke and Vincent Ng. 2019. Automated Essay Scoring: A Survey of the State of the Art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6300–6308, Macao. International Joint Conferences on Artificial Intelligence Organization.
- Lavolette et al. (2015) Elizabeth Lavolette, Charlene Polio, and Jimin Kahng. 2015. The accuracy of computer-assisted feedback and students’ responses to it. Language Learning & Technology, 19(2):50–68.
- Lee et al. (2016) Lung-Hao Lee, Gaoqi Rao, Liang-Chih Yu, Endong Xun, Baolin Zhang, and Li-** Chang. 2016. Overview of NLP-TEA 2016 Shared Task for Chinese Grammatical Error Diagnosis. In Proceedings of the 3rd Workshop on Natural Language Processing Techniques for Educational Applications ({NLPTEA}2016), pages 40–48, Osaka, Japan. The COLING 2016 Organizing Committee.
- Lee (2020) Young-Ju Lee. 2020. Effect of Automated Writing Evaluation Feedback on Korean University Students’ Revision Behavior. Foreign Languages Education, 27(4):1–22.
- Lim et al. (2023) KyungTae Lim, Jayoung Song, and Jungyeul Park. 2023. Neural automated writing evaluation for Korean L2 writing. Natural Language Engineering, 29(5):1341–1363.
- Link et al. (2014) Stephanie Link, Ahmet Dursun, Kadir Karakaya, and Volker Hegelheimer. 2014. Towards Better ESL Practices for Implementing Automated Writing Evaluation. CALICO Journal, 31(3):323–344.
- Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8:726–742.
- Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, **gfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR, abs/1907.1.
- Loem et al. (2023) Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023. Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 205–219, Toronto, Canada. Association for Computational Linguistics.
- Lu (2010) Xiaofei Lu. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4):474–496.
- Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. Association for Computational Linguistics.
- Mathias and Bhattacharyya (2018) Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP++: Enriching the ASAP Automated Essay Grading Dataset with Essay Attribute Scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 1169–1173, Miyazaki, Japan. European Language Resources Association (ELRA).
- Nadeem et al. (2019) Farah Nadeem, Huy Nguyen, Yang Liu, and Mari Ostendorf. 2019. Automated Essay Scoring with Discourse-Aware Neural Models. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 484–493, Florence, Italy. Association for Computational Linguistics.
- Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR – Grammatical Error Correction: Tag, Not Rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163–170, Seattle, WA, USA → Online. Association for Computational Linguistics.
- O’Neill and Russell (2019) Ruth O’Neill and Alex M. T. Russell. 2019. Stop! Grammar time: University students’ perceptions of the automated feedback program Grammarly. Australasian Journal of Educational Technology, 35(1):42–56.
- Ortega (2003) Lourdes Ortega. 2003. Syntactic Complexity Measures and their Relationship to L2 Proficiency: A Research Synthesis of College-level L2 Writing. Applied Linguistics, 24(4):492–518.
- Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
- Park and Lee (2016) Jungyeul Park and Junghee Lee. 2016. A Korean Learner Corpus and its Features. Journal of the Linguistic Society of Korea, 75:69–85.
- Polio (1997) Charlene G. Polio. 1997. Measures of Linguistic Accuracy in Second Language Writing Research. Language learning, 47(1):101–143.
- Ramesh and Sanampudi (2021) Dadi Ramesh and Suresh Kumar Sanampudi. 2021. An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, pages 1–33.
- Ranalli (2018) Jim Ranalli. 2018. Automated written corrective feedback: how well can students make use of it? Computer Assisted Language Learning, 31(7):653–674.
- Reynolds et al. (2021) Barry Lee Reynolds, Chian-Wen Kao, and Yun-yin Huang. 2021. Investigating the Effects of Perceived Feedback Source on Second Language Writing Performance: A Quasi-Experimental Study. The Asia-Pacific Education Researcher, 30(6):585–595.
- Ridley et al. (2021) Robert Ridley, Liang He, Xin-yu Dai, Shujian Huang, and Jiajun Chen. 2021. Automated Cross-prompt Scoring of Essay Traits. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13745–13753.
- Rosenberg and Abbeduto (1987) Sheldon Rosenberg and Leonard Abbeduto. 1987. Indicators of linguistic competence in the peer group conversational behavior of mildly retarded adults. Applied Psycholinguistics, 8(1):19–32.
- Rothe et al. (2021) Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A Simple Recipe for Multilingual Grammatical Error Correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 702–707, Online. Association for Computational Linguistics.
- Shermis and Burstein (2003) Mark D. Shermis and Jill C. Burstein. 2003. Introduction. In Mark D. Shermis and Jill C. Burstein, editors, Automated Essay Scoring: A Cross-disciplinary Perspective, pages xiii–xvii. Routledge, New York.
- Stahlberg and Kumar (2021) Felix Stahlberg and Shankar Kumar. 2021. Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 37–47, Online. Association for Computational Linguistics.
- Taghipour and Ng (2016) Kaveh Taghipour and Hwee Tou Ng. 2016. A Neural Approach to Automated Essay Scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
- Truscott (1999) John Truscott. 1999. The case for “The Case Against Grammar Correction in L2 Writing Classes”: A response to Ferris. Journal of Second Language Writing, 8(2):111–122.
- Uto (2021) Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika, 48(2):459–484.
- Uto et al. (2020) Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural Automated Essay Scoring Incorporating Handcrafted Features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077–6088, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6000–6010. Curran Associates, Inc.
- Wang et al. (2021) Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A Comprehensive Survey of Grammatical Error Correction. ACM Transactions on Intelligent Systems and Technology, 12(5):1–51.
- Yang et al. (2020) Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020. Enhancing Automated Essay Scoring Performance via Fine-tuning Pre-trained Language Models with Combination of Regression and Ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1560–1569, Online. Association for Computational Linguistics.
- Yngve (1960) Victor H. Yngve. 1960. A Model and an Hypothesis for Language Structure. Proceedings of the American Philosophical Society, 104(5):444–466.
- Yoon et al. (2023) Soyoung Yoon, Sungjoon Park, Gyuwan Kim, Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2023. Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6713–6742, Toronto, Canada. Association for Computational Linguistics.
- Zhang (2020) Zhe (Victor) Zhang. 2020. Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing, 43(January):100439.
- Zhang and Hyland (2018) Zhe (Victor) Zhang and Ken Hyland. 2018. Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36(April):90–102.
- Zhao et al. (2019) Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and **gming Liu. 2019. Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 156–165, Minneapolis, Minnesota. Association for Computational Linguistics.