AI Literacy Does Not Eliminate Challenges: Evolving Complexity among English Education Students in AI Integrated ELT Classrooms
DOI:
https://doi.org/10.24256/ideas.v14i1.10416Keywords:
AI literacy, evolving challenges, ELT, mixed-method, critical AI useAbstract
The increasing integration of artificial intelligence (AI) in education has generated a widespread assumption that higher AI literacy reduces the challenges students face. This study challenges that assumption by examining how challenges persist and evolve alongside students’ developing AI literacy. Focusing on 47 English Education students enrolled in an AI-integrated ELT classroom at an Islamic State University in Java, Indonesia, this research investigates: (1) the level and distribution of AI literacy, and (2) how challenges shift across different levels of competence. A sequential explanatory mixed-method design was employed, combining questionnaire data from 47 respondents, adapted from the Meta AI Literacy Scale (MAILS), with semi-structured interviews involving three purposively selected participants representing contrasting literacy levels. Grounded in socio-technical systems theory and critical AI literacy frameworks, the findings reveal that students demonstrate a generally high level of AI literacy (M = 2.97), with the strongest performance in operational use (M = 3.07) and the weakest in generative competence (M = 2.53). More importantly, challenges do not diminish as literacy increases; instead, they become more complex. Lower-level students struggle with basic operational issues such as prompt formulation and output interpretation, while intermediate-level students encounter cognitive difficulties in refining and aligning AI responses. Higher-level students face critical challenges related to accuracy evaluation, bias detection, and uncertainty management. These findings suggest that AI literacy development is not a linear progression toward reduced difficulty, but a layered socio-technical process in which higher competence exposes deeper limitations. This study contributes to AI literacy research by reframing challenges as evolving rather than diminishing and highlights the need for instructional approaches that emphasize critical, reflective, and ethical engagement with AI in educational contexts.
References
Alkharusi, H. (2022). A descriptive analysis and interpretation of data from Likert scales in educational and psychological research. International Journal of Educational Methodology, 12, 13–16.
Ateeq, A., Alzoraiki, M., Milhem, M., & Ateeq, R. A. (2024). Artificial intelligence in education: Implications for academic integrity and the shift toward holistic assessment. Frontiers in Education, 9, 1–11. https://doi.org/10.3389/feduc.2024.1470979
Baxter, G., & Sommerville, I. (2011). Socio-technical systems: From design methods to systems engineering. Interacting with Computers, 23(1), 4–17. https://doi.org/10.1016/j.intcom.2010.07.003
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Carolus, A., Koch, M. J., Straka, S., Latoschik, M. E., & Wienrich, C. (2023). MAILS - Meta AI literacy scale: Development and testing of an AI literacy questionnaire. Computers in Human Behavior: Artificial Humans, 1(2), 100014. https://doi.org/10.1016/j.chbah.2023.100014
Chen, X., Xie, H., Zou, D., & Hwang, G. J. (2020). Application and theory gaps during the rise of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1, 100002. https://doi.org/10.1016/j.caeai.2020.100002
Chyung, S. Y., Roberts, K., Swanson, I., & Hankinson, A. (2017). Evidence-based survey design: The use of a midpoint on the Likert scale. Performance Improvement, 56(10), 15–23. https://doi.org/10.1002/pfi.21727
Crawford, K. (2021). Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. Yale University Press.
Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning & Technology, 26(2), 5–24.
Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promise and implications for teaching and learning. Center for Curriculum Redesign.
Husnaini, H. (2022). Development of Self Esteem-Oriented Micro Teaching Materials for IAIN Palopo English Education Students. IDEAS: Journal on English Language Teaching and Learning, Linguistics and Literature, 10(1), 538-560.
Ivankova, N. V., Creswell, J. W., & Stick, S. L. (2006). Using mixed-methods sequential explanatory design. Field Methods, 18(1), 3–20. https://doi.org/10.1177/1525822X05282260
Jordan, M. I., & Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245), 255–260. https://doi.org/10.1126/science.aaa8415
Kasneci, E., et al. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
Lérias, E., Guerra, C., & Ferreira, P. (2024). Literacy in artificial intelligence as a challenge for teaching in higher education. Information, 15(4). https://doi.org/10.3390/info15040205
Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727
Luckin, R., Holmes, W., Griffiths, M., & Forcier, L. B. (2016). Intelligence unleashed: An argument for AI in education. Pearson.
Masruddin, M., & Nasriandi, N. (2022). Lexical and Syntactical Errors Performed by Junior High School Student in Writing Descriptive Text. IDEAS: Journal on English Language Teaching and Learning, Linguistics and Literature, 10(1), 1094-1100.
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6). https://doi.org/10.1145/3457607
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021). Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence, 2, 100041. https://doi.org/10.1016/j.caeai.2021.100041
Nilsson, N. J. (2010). The quest for artificial intelligence. Cambridge University Press.
Paas, F., & van Merriënboer, J. J. G. (2020). Cognitive load theory: Methods to manage working memory load. Educational Psychologist, 55(2), 63–77. https://doi.org/10.1080/00461520.2020.1735413
Pangrazio, L., & Selwyn, N. (2023). Towards a school-based critical AI literacy. Learning, Media and Technology. https://doi.org/10.1080/17439884.2023.2196603
Pinski, M., & Benlian, A. (2023). AI literacy: Towards measuring human competency in artificial intelligence. Proceedings of the Annual Hawaii International Conference on System Sciences, 165–174. https://doi.org/10.24251/hicss.2023.021
Roll, I., & Wylie, R. (2016). Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education, 26, 582–599. https://doi.org/10.1007/s40593-016-0110-3
Russell, S., & Norvig, P. (2020). Artificial intelligence: A modern approach (4th ed.). Pearson.
Selwyn, N. (2019). Should robots replace teachers? AI and the future of education. Polity Press.
Shneiderman, B. (2022). Human-centered AI. Oxford University Press.
Sweller, J. (2011). Cognitive load theory. Psychology of Learning and Motivation, 55, 37–76. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Syauqi Harsyah, et al. (2024). AI literacy among EFL learners: A survey study. Journal of English Language Teaching and Linguistics, 9(1), 45–59.
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of AI in higher education. International Journal of Educational Technology in Higher Education, 16(1). https://doi.org/10.1186/s41239-019-0171-0
Zhou, Z., Guan, H., Bhat, M. M., & Hsu, J. (2019). Fake news detection via NLP is vulnerable to adversarial attacks. Proceedings of ICAART, 794–800. https://doi.org/10./10.5220/0007566307940800
Downloads
Published
Issue
Section
Citation Check
License
Copyright (c) 2026 Shofia Mardhiyah , Tedi Rohadi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under an Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See the Effect of Open Access)
