Publication Details
Keywords: Automated descriptive answer evaluation, unsupervised grading, short-answer scoring, semantic similarity, BGE embeddings, BERTopic, K-Means, Gaussian mixture model, agglomerative clustering, quadratic weighted kappa
Abstract
This study examines whether descriptive university answers can be grouped into quality-related score categories without exposing the clustering process to human-assigned marks, thereby adhering to an unsupervised methodology. The analysis utilizes 1,000 digitized answer scripts from a third-semester Operating Systems examination at Bangalore University, resulting in 6,228 answer-level records. The experiment focuses exclusively on five-mark descriptive questions, restricting the admissible score range to 0–5 and establishing an upper limit of six ordered clusters. Each response is represented by contextual BGE embeddings, semantic similarity to the model answer, topic assignments from BERTopic, and two structural features: answer length and token count. Following MinMax normalization, feature vectors are clustered independently for each question using K-Means, Gaussian Mixture Model (GMM), and agglomerative clustering. Human marks are not involved in cluster formation or in determining cluster-to-score mappings. Instead, clusters are ranked by their mean cosine similarity to the model answer and mapped in ascending order to marks 0-5. K-Means demonstrates the highest agreement with human grading, achieving a quadratic weighted kappa (QWK) of 0.5453, a Pearson correlation of 0.6673, and a Spearman correlation of 0.6611. Agglomerative clustering follows with a QWK of 0.5193, while GMM yields a QWK of 0.3703. These findings indicate that a fully label-free clustering pipeline can recover a meaningful ordinal structure from descriptive answers, although the level of agreement varies across questions.
References
- [1] Mohler, Michael, and Rada Mihalcea. "Text-to-text semantic similarity for automatic short answer grading." Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009). 2009. [2] Zesch, Torsten, Michael Heilman, and Aoife Cahill. "Reducing annotation efforts in supervised short answer scoring." Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications. 2015. [3] Reimers, Nils, and Iryna Gurevych. "Sentence-bert: Sentence embeddings using siamese bert-networks." Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019. [4] Bexte, Marie, Andrea Horbach, and Torsten Zesch. "Similarity-based content scoring-how to make S-BERT keep up with BERT." Proceedings of the 17th workshop on innovative use of NLP for building educational applications (BEA 2022). 2022. [5] Haller, Stefan, et al. "Survey on automated short answer grading with deep learning: from word embeddings to transformers." arXiv preprint arXiv:2204.03503 (2022). [6] Divya, Arunima, Vivek Haridas, and Jayasree Narayanan. "Automation of short answer grading techniques: Comparative study using deep learning techniques." 2023 fifth international conference on electrical, computer and communication technologies (ICECCT). IEEE, 2023. [7] RK, Nadesh. "An enhanced framework for smart automated evaluations of answer scripts using NLP and deep learning methods." Multimedia Tools and Applications 84.11 (2025): 8491-8513. [8] Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022). [9] MacQueen, J. "Multivariate observations." Proceedings ofthe 5th Berkeley symposium on mathematical statisticsand probability. Vol. 1. No. 1967. Oakland, CA, USA: University of California press, 1967. [10] Fraley, Chris, and Adrian E. Raftery. "Model-based clustering, discriminant analysis, and density estimation." Journal of the American statistical Association 97.458 (2002): 611-631. [11] Ward Jr, Joe H. "Hierarchical grouping to optimize an objective function." Journal of the American statistical association 58.301 (1963): 236-244. [12] Rousseeuw, Peter J. "Silhouettes: a graphical aid to the interpretation and validation of cluster analysis." Journal of computational and applied mathematics 20 (1987): 53-65. [13] Ruskov, Martin. "Who and How: Using sentence-level NLP to evaluate idea completeness." International Conference on Artificial Intelligence in Education. Cham: Springer Nature Switzerland, 2023. [14] Pathirage, Sandaruwani. "A Statistical Analysis on ChatGPT's Evaluation Bias on Handwritten Diagram Based Exam Scripts." 2026 IEEE International Research Conference on Smart Computing and Systems Engineering (SCSE). Vol. 9. IEEE, 2026. [15] Chamieh, Imran, Torsten Zesch, and Klaus Giebermann. "LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches." Proceedings of the 19th workshop on innovative use of nlp for building educational applications (bea 2024). 2024. [16] Grévisse, Christian. "LLM-based automatic short answer grading in undergraduate medical education." BMC Medical Education 24.1 (2024): 1060. [17] Maya, A. K. R., Javed Nazura, and B. L. Muralidhara. "Recent trends in answer script evaluation–a literature survey." 3rd International Conference on Integrated Intelligent Computing Communication & Security (ICIIC 2021). Atlantis Press, 2021. [18] Xiao, Shitao, et al. "C-pack: Packed resources for general chinese embeddings." Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval. 2024.