Developing an Artificial Intelligence (AI)-Assisted Instrument for Assessing the Short Story Writing Skills of Grade XI Students at Senior High School
DOI:
https://doi.org/10.24256/ideas.v14i2.12887Keywords:
4D model; analytic rubric; artificial intelligence; assessment instrument; short story writingAbstract
Assessing short story writing is prone to subjectivity when it is not supported by a valid and reliable instrument, while the use of artificial intelligence (AI) in drafting assessment instruments has rarely been documented. This study aims to describe the development process and the quality of an AI-assisted instrument for assessing the short story writing skills of grade XI students at SMA Al-Azhar Mandiri Palu. It followed the 4D development model (define, design, develop, disseminate), with AI (Claude Opus 5.5) serving as a drafting aid rather than as a rater of students’ work. Data were collected through interviews, observation, document analysis, validation by five experts, the scoring of 30 short stories by two teachers, and a practicality questionnaire, and were analyzed using feasibility percentages, the intraclass correlation coefficient (ICC), Cronbach’s alpha, percentage agreement, and practicality percentages. The AI generated 32 rubric descriptors from a single prompt. After the researcher’s review, 12 descriptors were accepted unchanged, 17 were revised to make them observable and consistent with the quantitative requirement, and 3 were rejected because they relied on impressionistic criteria. The 12 descriptors accepted unchanged were later revised on the language validator’s advice, so none of the AI-generated descriptors was used unchanged in the final product. The instrument obtained a feasibility percentage of 92.63% (very valid), a single-measures ICC of 0.758 (good; 95% CI 0.541–0.878), and a practicality percentage of 98.33% (very practical). Cronbach’s alpha was low (−0.224 and 0.251) because scores clustered at the top of the scale, while exact inter-rater agreement reached 94.17%. The instrument is feasible for use, and AI is effective as a drafting aid provided that its output is reviewed by the researcher and validated by experts.
References
Ahmad, K., Br Ginting, S. U., & Sidiqin, M. A. (2020). Hubungan penguasaan unsur intrinsik cerpen dengan kemampuan menulis cerpen siswa kelas XI SMK Swasta Maju Binjai tahun pelajaran 2019/2020 [The relationship between mastery of the intrinsic elements of short stories and the short story writing ability of grade XI students of SMK Swasta Maju Binjai in the 2019/2020 academic year]. Jurnal Serunai Bahasa Indonesia, 17(1), 7–19. doi:10.37755/jsbi.v17i1.257
Alfianika, N., & Sitohang, K. (2022). Validitas pengembangan rubrik penilaian menulis paragraf narasi dan deskripsi dalam pembelajaran bahasa Indonesia [The validity of developing assessment rubrics for writing narrative and descriptive paragraphs in Indonesian language learning]. Fon: Jurnal Pendidikan Bahasa dan Sastra Indonesia, 18(2). doi:10.25134/fon.v18i2.5592
Anthropic. (2026). Claude (Version Opus 5.5) [Large language model]. Retrieved from https://claude.ai
Arikunto, S. (2018). Dasar-dasar evaluasi pendidikan [Fundamentals of educational evaluation] (2nd ed.). Jakarta: Bumi Aksara.
Brookhart, S. M. (2018). Appropriate criteria: Key to effective rubrics. Frontiers in Education, 3, 22. doi:10.3389/feduc.2018.00022
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. doi:10.1016/0895-4356(90)90158-L
Hanis, M., & Wahyudin, D. (2024). Pemanfaatan artificial intelligence (AI) dalam penyusunan asesmen pembelajaran bagi guru sekolah dasar [The use of artificial intelligence (AI) in developing learning assessments for elementary school teachers]. Jurnal Ilmiah Profesi Pendidikan, 9(2), 1199–1207. doi:10.29303/jipp.v9i2.2252
Hikmah, S. N. A. (2021). Pengembangan instrumen asesmen keterampilan menulis teks eksposisi [Developing an assessment instrument for expository text writing skills]. Jurnal Tarbiyatuna: Jurnal Kajian Pendidikan, Pemikiran dan Pengembangan Pendidikan Islam, 2(1), 59–69. doi:10.30739/tarbiyatuna.v2i01.975
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., . . . Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. doi:10.1016/j.lindif.2023.102274
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. doi:10.1016/j.jcm.2016.02.012
Maulana, A. (2022). Analisis validitas, reliabilitas, dan kelayakan instrumen penilaian rasa percaya diri siswa [An analysis of the validity, reliability, and feasibility of a student self-confidence assessment instrument]. Jurnal Kualita Pendidikan, 3(3), 133–139. Retrieved from https://journal.kualitama.com/index.php/jkp/article/view/331
Maydiantoro, A. (2021). Model-model penelitian pengembangan (research and development) [Research and development models]. Jurnal Pengembangan Profesi Pendidik Indonesia, 1(2), 29–35.
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. doi:10.1016/j.rmal.2023.100050
Muhajir, M., Rismawati, R., Marlis, M. I., Faisal, S. F. R., Vanezal, V., Raldiastari, S., & Hakil, M. (2026). Pemanfaatan kecerdasan buatan (AI) dalam asesmen pembelajaran mendalam di SDIT Darul Fikri Makassar [The use of artificial intelligence (AI) in deep learning assessment at SDIT Darul Fikri Makassar]. Jurnal Petisi (Pendidikan Teknologi Informasi), 7(2), 302–310. doi:10.36232/jurnalpetisi.v7i2.4417
Nurgiyantoro, B. (2016). Penilaian pembelajaran bahasa berbasis kompetensi [Competency-based assessment in language learning]. Yogyakarta: BPFE.
Nurgiyantoro, B. (2018). Teori pengkajian fiksi [The theory of fiction analysis]. Yogyakarta: Gadjah Mada University Press.
Oktavianus, A. J. E., Naibaho, L., & Rantung, D. A. (2023). Pemanfaatan artificial intelligence pada pembelajaran dan asesmen di era digitalisasi [The use of artificial intelligence in learning and assessment in the digitalization era]. Jurnal Kridatama Sains dan Teknologi, 5(2), 473–476. doi:10.53863/kst.v5i02.975
Rasyid, A., Hartanto, R., & Priyanto, C. (2026). Reliabilitas alat ukur, jenis-jenis, cara pengukuran, dan faktor-faktor yang memengaruhinya [The reliability of measuring instruments: Types, measurement methods, and influencing factors]. Tsaqofah: Jurnal Penelitian Guru Indonesia, 6(2), 1285–1294. doi:10.58578/tsaqofah.v6i2.8511
Santi, A., Silvia, D., & Damaianti, V. S. (2023). Penilaian autentik pembelajaran bahasa Indonesia menulis karya ilmiah: Penggunaan dan pencapaian keterampilan peserta didik [Authentic assessment in Indonesian language learning of scientific writing: Its use and students’ skill attainment]. Fon: Jurnal Pendidikan Bahasa dan Sastra Indonesia, 19(2), 226–238. doi:10.25134/fon.v19i2.7710
Santoso, R. S. B., Suyono, Harsiati, T., & Sari, N. P. (2023). Implementasi instrumen asesmen keterampilan menulis cerpen dengan media lagu inspiratif untuk siswa kelas XI SMA di era merdeka belajar [Implementing a short story writing assessment instrument using inspirational songs for grade XI senior high school students in the Merdeka Belajar era]. Ghancaran: Jurnal Pendidikan Bahasa dan Sastra Indonesia. doi:10.19105/ghancaran.vi.11772
Sauhenda, A. F., Marnina, & Narahawarin, M. F. (2025). Pelatihan penyusunan instrumen penilaian keterampilan menulis cerita pendek di SMP Negeri Urumb Merauke [Training on developing short story writing assessment instruments at SMP Negeri Urumb Merauke]. Jurnal Pelayanan Masyarakat, 2(4), 283–290. doi:10.62951/jpm.v2i4.2743
Sihombing, B., Zamsiswaya, & Sawaluddin. (2024). Model pengembangan 4D (define, design, develop, dan disseminate) dalam pembelajaran pendidikan Islam [The 4D development model (define, design, develop, and disseminate) in Islamic education learning]. Journal of Islamic Education El Madani, 4(1), 11–19. doi:10.55438/jiee.v4i1.135
Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., . . . Olson, C. B. (2024). Comparing the quality of human and ChatGPT feedback of students’ writing. Learning and Instruction, 91, 101894. doi:10.1016/j.learninstruc.2024.101894
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. doi:10.5116/ijme.4dfb.8dfd
Waruwu, M. (2024). Metode penelitian dan pengembangan (R&D): Konsep, jenis, tahapan, dan kelebihan [Research and development (R&D) methods: Concepts, types, stages, and advantages]. Jurnal Ilmiah Profesi Pendidikan, 9(2), 1220–1230. doi:10.29303/jipp.v9i2.2141
Downloads
Published
Issue
Section
Citation Check
License
Copyright (c) 2026 Andy Dahnial Pratama Mz, Ida Nuraeni, Sukma
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under an Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See the Effect of Open Access)
