Development of a Natural Language Processing Framework for Automated Assessment of Multilingual Academic Texts

Abstract

The increasing number of academic works generated in several different languages in the systems of higher education has resulted in a greater need for assessment methods that would be able to function outside the single-language environment. Manually grading the essays, reports, and short answer questions is laborious, unreliable across several graders and hard to scale if the works are submitted in several languages, which happens often in multilingual education systems that use not only a national language and a regional language but also English for instruction and assessment. In this paper, it presents the design and architecture of a natural language processing system for automatic assessment of multilingual academic papers. The system utilizes multilingual contextual representations, syntactic and discourse features and multi-trait scoring system that includes assessing content relevance, grammar, cohesion and originality of the text. In contrast to automated essay scoring algorithms that are traditionally built and validated for one language only and deliver one overall score, the proposed framework is based on language-independent representations and employs a calibrated and multi-component scoring layer, augmented with human-in-the-loop verification. Comparative analysis of the flow implemented in traditional automated scoring systems versus the proposed flow is provided to demonstrate the difference in structure between these two paradigms. Pilot evaluation protocol with standard measures of agreement and errors is suggested to provide a demonstration of how the framework can be validated in the context of multilingual academic corpora. The discussion highlights the potential practical applications of the framework in the context of multilingual academic institutions as well as the present limitations of the framework and future research directions.