The Mode Effects and Digital Divide Among Taiwanese Students Based on the TIMSS 2019 Computer-Based Testing
Author: Kuan-Ming Chen (Research Center for Testing and Assessment, National Academy for Educational Research), Tian-Ming Sheu (Department of Education, National Taiwan Normal University), Tsung-Hau Jen (Science Education Center, National Taiwan Normal University)
Vol.&No.：Vol. 70, No. 2
Date：June 2025
Pages：1-41
DOI：https://doi.org/10.6209/JORIES.202506_70(2).0001
Abstract：
Introduction
To meet the demands and advantages of the digital age, most international large-scale assessments have transitioned students’ response modes from paper-and-pencil to computer-based testing (CBT). With the aid of digitalization, CBT can better align with the fundamental principles of authentic assessment (Herrington & Herrington, 1998) by incorporating multimedia presentations and capturing test-takers’ maneuvering behaviors in greater detail. However, when the item characteristics or constructs that an item is designed to assess differ between paper-and-pencil and CBT formats, a mode effect may arise as a result of variations in students’ response modes. If this mode effect correlates with factors related to digital divide, such as disparities in students’ socioeconomic status (SES), the allocation of urban-rural resources, and gender (Cooper, 2006; Hsieh & Ming, 2022; Hu & Chang, 2023; Kuo, 2022), CBT raises concerns regarding educational equity. Therefore, addressing the digital divide in CBT implementations is crucial to ensuring educational equity and accessibility before full adoption.
As the first assessment cycle to implement CBT, in the Trends in International Mathematics and Science Study (TIMSS) 2019, half of the participating countries, including Taiwan, implemented CBT as the primary testing format (eTIMSS) for the main survey while concurrently administering a bridge study using the paper-and-pencil format (paperTIMSS). The purpose of this bridge study was to identify and equate potential mode effects in TIMSS 2019, ensuring item equivalence across modes and supporting robust trend analyses across assessment cycles. Drawing on data of Taiwanese eighth-grade students from TIMSS 2019 and its accompanying bridge study, this research examines concerns regarding educational equity by examining the relationships between mode effects and three students background factors: SES, school location, and gender. Specifically, this study seeks to determine whether Taiwanese eighth-grade students’ performance in mathematics and science differed between the two testing formats and whether a digital divide was evident based on SES, school location, or gender. Additionally, TIMSS 2019 incorporated computerized innovative items to assess problem-solving and inquiry (PSI) tasks (Mullis et al., 2021). Therefore, this study also explores how Taiwanese eighth-grade students from different backgrounds performed on these computerized PSI tasks (eTIMSS-PSI).
Method
Analyzing the performance of Taiwanese eighth-grade students on trend items in mathematics and science from eTIMSS 2019 and its bridge study, paperTIMSS, mode effects were identified by comparing the item average percent correct between the paper-and-pencil and the CBT formats. Subsequently, the digital divide related to demographic factors was examined by comparing scale score performance on trend items between eTIMSS and paperTIMSS among students with varying SES, school locations, and genders. Finally, students’ performance on eTIMSS-PSI items was analyzed using scale scores to assess how Taiwanese eighth-grade students from different backgrounds performed.
Results
Significant reductions in the item average percent correct on trend items were observed in eTIMSS compared to paperTIMSS among Taiwanese eighth-grade students in both mathematics and science, indicating substantial mode effects across both subjects. Based on these identified mode effects, scaling procedures were implemented using the international average mode effect and performance data from previous assessment cycles (which included only paperTIMSS) for each subject to generate scale scores. Consequently, the average scale scores for mathematics and science were adjusted to ensure comparability between eTIMSS and paperTIMSS, as well as between TIMSS 2019 and prior assessment cycles.
By comparing the average scale score performance between eTIMSS and paperTIMSS among Taiwanese eighth-grade students with varying SES, school locations, and genders, no significant difference was found in the relationship between mode effects and students’ background factors in either mathematics or science. Therefore, the digital divide due to mode effects was not evident in the scale scores across both subjects based on SES, school location, or gender.
Furthermore, the analysis of eTIMSS-PSI scale scores for Taiwanese eighth-grade students with varying backgrounds revealed a declining trend in performance among students from higher SES compared to those from lower SES backgrounds, as well as among students attending schools in urban areas compared to those in rural areas.
Conclusions
The performance differences between paper-and-pencil and CBT formats in TIMSS 2019 mathematics and science among Taiwanese eighth-grade students from different backgrounds reveal the following key findings:
1. Overall mode effects: Taiwanese eighth-grade students, similar to their counterparts in most other countries, demonstrated lower average percent correct on items presented in the CBT format compared to the paper-and-pencil format in both mathematics and science. Substantial mode effects were observed, particularly in mathematics.
2. No widening digital divide: After applying scaling procedures to ensure comparability of scale score between the two testing formats, no evidence of a digital divide was found based on SES, school location, or gender, in either mathematics or science. Students from different backgrounds performed similarly across both formats.
3. Performance disadvantage on PSI tasks: As computerized PSI tasks captured test-takers’ maneuvering behaviors in greater detail, students from higher SES backgrounds or urban areas demonstrated a performance disadvantage on PSI tasks. It remains unclear whether this disadvantage results from weaker PSI skills among Taiwanese students in general or potential impacts of interactive nature of CBT items on specific student groups. Future research is needed to explore these factors.
In the digital age, the transition to CBT is inevitable as technology continues to transform education and assessment practices. The integration of digital tools in learning environments makes the adoption of CBT essential for aligning with modern educational needs while offering more interactive and efficient assessment methods. The urgency of digitalization in education become even more apparent during the COVID-19 pandemic, highlighting the need for accessible and equitable digital learning solutions. While CBT offers benefits, concerns related to educational equity, including mode effects and the digital divide, remain. By analyzing the impact of CBT in TIMSS 2019, educators and policymakers can accordingly develop evidence-based strategies to ensure fair, reliable, and valid assessments for all students. With precise scaling procedures to eliminate originally evident mode effects, it is assuring that no widening digital divide was identified. In conclusion, the transition to digital assessments in TIMSS 2019 did not compromise educational equity for Taiwanese eighth-grade students.
Keywords：
international large-scale survey, computer-based testing, digital divide, mode efect