School of Languages at XJTLU Conference 2024

This study investigates how students' digital literacy influences their preferences, perceived value of AI academic search engines, and academic writing performance. To achieve this, two surveys with embedded tests of Digital Literacy were developed, building upon existing frameworks.

During the summer of 2023, 17 PSE students from an EMI university participated in the study. They completed a pre-survey, received training on AI academic search engines, and took a post-survey afterward. Additionally, 15 students participated in two focus group interviews, and their source use scores from their assessed argumentative essays were recorded.

The survey data were analyzed using SPSS 27.0, while the interviews were transcribed with Whisper and analyzed using Nvivo 12. Reliability analysis demonstrated high internal consistency (0.93) for both the pre-survey and post-survey. Although the paired samples t-tests did not yield statistically significant p-values due to limitations in sample size and intervention constraints, significant improvements were observed in students' knowledge, usage, and perceptions of AI academic search engines. Notably, AI digital literacy scores increased from pre-intervention (M=10.76, SD=2.62) to post-intervention (M=12.12, SD=1.27), with the scale related to AI academic source search engines showing the most substantial increase (MD=0.42, SD=1.14). The interviews revealed that Connected Papers, Scinapse, and Semantic Scholars were the most commonly utilized AI academic search engines among students. Students' identification of limitations in AI academic search engines, including restricted search capabilities and limited article downloads, underscores the necessity for additional training to improve accuracy and efficiency in utilizing these tools.

This study highlights the effectiveness of targeted interventions in improving learners' digital literacy skills and integrating AI technologies into academic practices. The findings serve as a basis for future development in this field, with the potential for expanding the survey to large sample sizes for further explorations.