When Students Asked ChatGPT Instead of Me: Investigating Generative AI in Programming Education Through NLP and Pedagogical Analytics
Downloads
This study investigates the growing dependence of students on popular generative artificial intelligence tools such as ChatGPT, GitHub Copilot, Jupyter AI, Google Bard, and more for coding assignments, project development, exam preparation, and conceptual learning in computer science higher education. Based on empirical data from various undergraduate and graduate-level programming courses at the Academy of Applied Technical and Preschool Studies in Serbia, the research applies natural language processing (NLP) and builds a machine learning model to examine student engagement and predict potential educational outcomes. A Python-based comprehensive framework integrates CodeBERT method for semantic similarity and plagiarism detection, TF-IDF with cosine similarity for benchmark comparisons, XGBoost for rubric-based classification, and DBSCAN clustering methods for code anomaly detection. Sentiment analysis further captures student attitudes toward frequent AI use. Rather than limiting the use of such approaches, this paper introduces a scalable solution for AI-aware assessment and curriculum design, encouraging responsible and ethical usage of modern generative technologies. The results support an innovative and future-ready model for education in the era of artificial intelligence.
H. Güner and E. Er, “AI in the classroom: Exploring students’ interaction with ChatGPT in programming learning,” Educ. Inf. Tech., vol. 30, pp. 12681–12707, 2025, doi: 10.1007/s10639-025-13337-7.
K. Fuchs, “Exploring the opportunities and challenges of NLP models in higher education: Is ChatGPT a blessing or a curse?,” Front. Educ., vol. 8, p. 1166682, May 2023, Frontiers Media SA,
doi: 10.3389/feduc.2023.1166682.
J. Beltrán and E. Veiga-Zarza, “Evaluating the use of large language models in programming courses: a comparative study,” EDULEARN Proc., vol. 1, pp. 1761–1768, 2025, doi: 10.21125/edulearn.2025.0530.
A. Konak and C. J. S. F. Clarke, “Augmenting critical thinking skills in programming education through leveraging ChatGPT: Analysis of its opportunities and consequences,” in Proc. 2023 Fall Mid Atlantic Conf.: Meeting Our Students Where They Are and Getting Them Where They Need to Be, Ewing, NJ, USA, Oct. 2023, doi: 10.18260/1-2--45117.
S. Folvarochna, Using learning analytics to identify student challenges in programming education. B.Sc. thesis, Dept. Comput. Sci. Inf. Technol., Fac. Appl. Sci., Ukrainian Catholic Univ., Lviv, Ukraine, 2025.
G. Fenu, R. Galici, M. Marras, and D. Reforgiato, “Exploring student interactions with AI in programming training,” in Adjunct Proc. 32nd ACM Conf. User Modeling, Adaptation and Personalization, New York, NY, USA: Assoc. Comput. Mach., 2024, pp. 555–560, doi: 10.1145/3631700.3665227.
B. Ma, L. Chen, and S. Konomi, “Exploring student perception and interaction using ChatGPT in programming education,” in Proc. 21st Int. Conf. Cogn. Explor. Learn. Digital Age (CELDA), 2024, doi: 10.33965/celda2024_202408l005.
J. Xie, “Improving introductory Java programming education through ChatGPT,” J. Comput. Sci. Coll., vol. 40, no. 3, pp. 140–150, Oct. 2024.
R. Yilmaz and F. G. K. Yilmaz, “The effect of generative artificial intelligence (AI)-based tool use on students' computational thinking skills, programming self-efficacy and motivation,” Comput. Educ.: Artif. Intell., vol. 4, p. 100147, 2023,
doi: 10.1016/j.caeai.2023.100147.
F. M. Mboya, G. M. Wambugu, A. M. Oirere, E. O. Omuya, F. M. Musyoka, and J. W. Gikandi, “Enhancing personalized learning in programming education through generative artificial intelligence frameworks: A systematic literature review,” Int. J. Adv. Trends Comput. Sci. Eng., vol. 14, no. 2, pp. 514–522, 2025,
doi: 10.30534/ijatcse/2025/051422025.
S. Boguslawski, R. Deer, and M. G. Dawson, “Programming education and learner motivation in the age of generative AI: Student and educator perspectives,” Inf. Learn. Sci., vol. 126, no. 1/2, pp. 91–109, 2025, doi: 10.1108/ILS-10-2023-0163.
T. Phung, V. A. Pădurean, J. Cambronero, S. Gulwani, T. Kohn, R. Majumdar, and G. Soares, “Generative AI for programming education: Benchmarking ChatGPT, GPT-4, and human tutors,” in Proc. 2023 ACM Conf. Int. Comput. Educ. Res.–Vol. 2, Aug. 2023, pp. 41–42, doi: 10.1145/3568812.3603476
S. Sarsa, P. Denny, A. Hellas, and J. Leinonen, “Automatic generation of programming exercises and code explanations using large language models,” in Proc. 2022 ACM Conf. Int. Comput. Educ. Res. – Vol. 1, Aug. 2022, pp. 27–43,
doi: 10.1145/3501385.3543957.
S. Yazdani, M. Najimi, and M. Ahmadzadeh, “The paradox of generative AI in programming education,” in EDULEARN Proc., 2025, pp. 7775–7784,
doi: 10.21125/edulearn.2025.1927.
D. Franklin, P. Denny, D. A. Gonzalez-Maldonado, and M. Tran, Generative AI in computer science education: Challenges and opportunities. Cambridge, U.K.: Cambridge Univ. Press, 2025.
J. Prather, P. Denny, J. Leinonen, B. A. Becker, I. Albluwi, M. Craig, and J. Savelka, “The robots are here: Navigating the generative AI revolution in computing education,” in Proc. 2023 Working Group Reports Innov. Technol. Comput. Sci. Educ., 2023, pp. 108–159, doi: 10.1145/3623762.3633499.
D. Cambaz and X. Zhang, “Use of AI-driven code generation models in teaching and learning programming: A systematic literature review,” in Proceed. 55th ACM Techn. Sympos. Comput. Sci. Educ. vol. 1, Mar. 2024, pp. 172–178,
doi: 10.1145/3626252.3630958.
Y. Zhan, Q. Liu, W. Gao, Z. Zhang, T. Wang, S. Shen, et al., “CoderAgent: Simulating student behavior for personalized programming learning with large language models,” arXiv preprint, 2025.
doi: 10.48550/arXiv.2505.20642.
The Academy of Applied Technical and Preschool Studies, Lecture notes on Fund. of Programming, Serbia, 2023-2024.
The Academy of Applied Technical and Preschool Studies, Lecture notes on Software Engineering, Serbia, 2023-2024.
The Academy of Applied Technical and Preschool Studies, Lecture notes on Big Data Analytics, Niš, Serbia, 2023-2024.
K. M. G. S. Karunarathna and R. A. H. M. Rupasingha, “Learning to use normalization techniques for preprocessing and classification of text documents,” Int. J. Multidiscip. Stud., vol. 9, no. 2, pp. 69–81, 2022.
P. T. Nguyen, J. Di Rocco, C. Di Sipio, R. Rubei, D. Di Ruscio, and M. Di Penta, “Is this snippet written by ChatGPT? An empirical study with a CodeBERT-based classifier,” arXiv preprint, 2023,
doi: 10.48550/arXiv.2307.09381.
A. Asselman, M. Khaldi, and S. Aammou, “Enhancing the prediction of student performance based on the machine learning XGBoost algorithm,” Interact. Learn. Environ., vol. 29, no. 3, pp. 3360–3379, 2021, doi: 10.1080/10494820.2021. 1928235.
H. Du, S. Chen, H. Niu and Y. Li, "Application of DBSCAN clustering algorithm in evaluating students' learning status," in Proc. 17th Int. Conf. Comput. Intell. Security, Chengdu, China, 2021, pp. 372–376, doi: 10.1109/CIS54983.2021.00084.
