Applied and Computational Engineering

- The Open Access Proceedings Series for Conferences


Proceedings of the 4th International Conference on Signal Processing and Machine Learning

Series Vol. 52 , 27 March 2024


Open Access | Article

Effectiveness of finetuning pretrained BERT and deBERTa for automatic essay scoring

Wentao Zhong * 1
1 Chengdu University of Technology

* Author to whom correspondence should be addressed.

Applied and Computational Engineering, Vol. 52, 87-95
Published 27 March 2024. © 27 March 2024 The Author(s). Published by EWA Publishing
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Citation Wentao Zhong. Effectiveness of finetuning pretrained BERT and deBERTa for automatic essay scoring. ACE (2024) Vol. 52: 87-95. DOI: 10.54254/2755-2721/52/20241321.

Abstract

With the growing importance of summary writing skills in the educational system and the inherent complexity of manual assessment, there is an urgent need for automated summary scoring solutions. Pre-trained models are popular nowadays, such as Bidirectional Encoder Representations from Transformers (BERT) and Decoding enhanced BERT with disentangled attention (deBERTa). The performance of direct use with trained models on specific tasks still needs to be improved. This paper focuses on the impact on the performance of summary scoring systems after adding linear and dropout layers to these pre-trained models for feature extraction and dimensionality reduction operations. The paper details the optimization for the particular task of summary scoring automation after using the pre-trained models. This paper focuses on adding linear and dropout layers to perform feature extraction and dimensionality reduction operations. The aim is to make the model more adaptable to this specific educational task. Ultimately, it is hoped that these studies will enhance the pedagogical toolkit for educators and enrich the academic experience for students.

Keywords

automatic essay scoring, BERT, transformer

References

1. Ahn, S. (2022). Developing Summary Writing Abilities of Korean EFL University Students through Teaching Summarizing Skills. English Teaching, 77(2), 25-43.

2. Susanti, M. N. I., Ramadhan, A., & Warnars, H. L. H. S. (2023). Automatic essay exam scoring system: A systematic literature review. Procedia Computer Science, 216, 531-538.

3. Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3), 2495-2527.

4. Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science, 5, e208.

5. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

6. He, P., Liu, X., Gao, J., & Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.

7. Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019). How to fine-tune bert for text classification?. In Chinese Computational Linguistics: 18th China National Conference, 194-206.

8. Mayfield, E., & Black, A. W. (2020). Should you fine-tune BERT for automated essay scoring?. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, 151-162.

9. Yang, R., Cao, J., Wen, Z., Wu, Y., & He, X. (2020,). Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In Findings of the Association for Computational Linguistics, 1560-1569.

10. CommonLit - Evaluate Student Summaries, URL: https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries. Last accessed: 2023/10/13

Data Availability

The datasets used and/or analyzed during the current study will be available from the authors upon reasonable request.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Authors who publish this series agree to the following terms:

1. Authors retain copyright and grant the series right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this series.

2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the series's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this series.

3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See Open Access Instruction).

Volume Title
Proceedings of the 4th International Conference on Signal Processing and Machine Learning
ISBN (Print)
978-1-83558-349-4
ISBN (Online)
978-1-83558-350-0
Published Date
27 March 2024
Series
Applied and Computational Engineering
ISSN (Print)
2755-2721
ISSN (Online)
2755-273X
DOI
10.54254/2755-2721/52/20241321
Copyright
27 March 2024
Open Access
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited

Copyright © 2023 EWA Publishing. Unless Otherwise Stated