Evaluation of the Psychometric Properties of the General Literacy Test for University Students
Author: Po-Hsi Chen(Department of Educational Psychology and Counseling, National Taiwan Normal University), Hsin-Ying Huang(Science Education Center, National Taiwan Normal University), Yu-Hsin Chen(Department of Educational Psychology and Counseling, National Taiwan Normal University), Tai-Ting Yeh (Department of Educational Psychology and Counseling, National Taiwan Normal University), Shao-Tsu Su(Department of Educational Psychology and Counseling, National Taiwan Normal University)
Vol.&No.：Vol. 60, No. 3
Date：September 2015
Pages：95-126
DOI：10.6209/JORIES.2015.60(3).04
Abstract：
This study evaluated the psychometric properties of the General Literacy Test for University Students. To develop the assessment framework, the educational objectives of general literacy courses of universities in Taiwan as well as the core competencies of Assessment and Teaching of 21st Century Skills were all reviewed and considered. The general literacy test is composed of nine literacy domains: communication and collaboration, esthetics, information, lifelong learning, career, leadership, problem solving, social concerns and citizenship, and scientific thinking. The items of the general literacy test were developed into a multimedia assessment. In addition, booklets including two to three testlets on each of the nine literacy domains were organized. Overall, 10,958 students from 20 universities participated. The results demonstrated that testlet effect could be ignored and the partial credit model could be used to analyze data. In addition, the items fit the partial credit model closely and the test demonstrated favorable construct validity. In addition, small differences existed between gender and grades. The results were consistent with those of previous studies and showed that the test’s criterion-related validity was acceptable. Furthermore, the standard error of ability ranged from .20 to .60 logit and the reliabilities of the booklets were more than .69 in the nine literacy domains. The evidence supports the high degree of reliability and validity of basic literacy assessment.
Keywords：reliability, validity, literacy assessment, scenario based test, online test
《Full Text》
References：
- 王保進（2011）。以學生學習成效品質保證為核心之第二週期系所評鑑作業規範。評鑑雙月刊，33，7-11。【Wang, B.-J. (2011). Student learning outcomes associating with quality assurance mechanism for department and graduate institute evaluation. Evaluation Bimonthly, 33, 7-11.】
- 林煥祥、劉聖忠、林素微、李暉（2008）。台灣參加PISA 2006成果報告。科技部專題研究計畫成果報告（NSC95-2522-S-026-002）。花蓮市：國立花蓮教育大學；高雄市：國立高雄師範大學。【Lin, H.-X., Liu, S.-Z., Lin, S.-W., & Li, H. (2008). PISA 2006 results in Taiwan. Ministry of Science and Technology project report (NSC95-2522-S-026-002). Hualien, Taiwan: National Hualien University of Education; Kaohsiung, Taiwan: National Kaohsiung Normal University.】
- 郭生玉（2012）。心理與教育研究法。新北市：精華。【Guo, S.-Y. (2012). Educational psychology and methodology. New Taipei City, Taiwan: Jin-Hua.】
- 陳儀蓉、黃芳銘（2006）。組織公民行為量表在男女員工群體上之測驗恆等性檢定。測驗學刊，53（2），297-326。【Chen, Y.-J., & Hwang, F.-M. (2006). Assessing measurement invariance of organizational citizenship behavior citizenship behavior scale between male and female staff. Psychological Testing, 53(2), 297-326.】
- 陳柏熹（2011）。心理與教育測驗：測驗編製理論與實務。新北市：精策教育。【Chen, P.-H. (2011). Psychology and educational testing: Theory and practice. New Taipei City, Taiwan: Kinetics.】
» More
- 王保進（2011）。以學生學習成效品質保證為核心之第二週期系所評鑑作業規範。評鑑雙月刊，33，7-11。【Wang, B.-J. (2011). Student learning outcomes associating with quality assurance mechanism for department and graduate institute evaluation. Evaluation Bimonthly, 33, 7-11.】
- 林煥祥、劉聖忠、林素微、李暉（2008）。台灣參加PISA 2006成果報告。科技部專題研究計畫成果報告（NSC95-2522-S-026-002）。花蓮市：國立花蓮教育大學；高雄市：國立高雄師範大學。【Lin, H.-X., Liu, S.-Z., Lin, S.-W., & Li, H. (2008). PISA 2006 results in Taiwan. Ministry of Science and Technology project report (NSC95-2522-S-026-002). Hualien, Taiwan: National Hualien University of Education; Kaohsiung, Taiwan: National Kaohsiung Normal University.】
- 郭生玉（2012）。心理與教育研究法。新北市：精華。【Guo, S.-Y. (2012). Educational psychology and methodology. New Taipei City, Taiwan: Jin-Hua.】
- 陳儀蓉、黃芳銘（2006）。組織公民行為量表在男女員工群體上之測驗恆等性檢定。測驗學刊，53（2），297-326。【Chen, Y.-J., & Hwang, F.-M. (2006). Assessing measurement invariance of organizational citizenship behavior citizenship behavior scale between male and female staff. Psychological Testing, 53(2), 297-326.】
- 陳柏熹（2011）。心理與教育測驗：測驗編製理論與實務。新北市：精策教育。【Chen, P.-H. (2011). Psychology and educational testing: Theory and practice. New Taipei City, Taiwan: Kinetics.】
- 廖添富、黃景裕（1995）。我國大學生公民態度之研究－樣本取自北部地區。公民訓育學報，4，109-127。doi:10.6231/CME.1995(4)05【Liao, T.-F., & Huang, C.-Y. (1995). A study of the civic attitude of college students on Taiwan, R.O.C.－Samples selected from northern Taiwan. Bulletin of Civic and Moral Education, 4, 109-127. doi:10.6231/CME.1995 (4)05】
- 國立臺灣師範大學雲端測驗中心（2014）。大學生基本素養指導手冊。取自http://140.122.69. 221/ntnucit/index.php/services-mainnav/testing-of-students-literacy-mainnav【National Taiwan Normal University Research Center for Internet Testing. (2014). Guilds for basic literacy test for university students. Retrieved from http://140.122.69.221/ntnucit/index.php/services-mainnav/testing-of-students- literacy-mainnav】
- 國家教育研究院（2014）。教育部提升國民素養專案辦公室。取自http://literacytw.naer.edu.tw/ index.php?REFDOCID=0m8mptc3ioz0a9z8【National Acedemy of Educational Research. (2014). Office of enhancing the basic literacy of people. Retrieved from http://literacytw.naer.edu.tw/index.php?REFDOCID=0m8mptc3ioz0a9z8】
- Assessment & Teaching of 21st Century Skills. (2010). 21st century skills white paper. Retrieved from http://atc21s.org/index.php/resources/white-papers/
- Aiken, L. R., & Groth-Marnat, G. (2005). Psychological testing and assessment (12th ed.). Boston, MA: Pearson Education.
- Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397-472). Reading, MA: Addison-Wesley.
- Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431-444. doi:10.1177/014662168200 600405
- Chen, P.-H., Kuo, J.-W., & Sung, Y.-T. (2011, July). Influence of pre-test design on the precision of the parameters estimation in the multidimensional items bank. Paper presented at the IMPS 2011, the 76rd annual meeting of the Psychometric Society, Hong Kong, China.
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New Jersey, NY: Lawrence Erlbaum Associates.
- Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155-159.
- European Association for Quality Assurance. (2009a). ENQA position paper on quality assurance in the EHEA－In view of the leuven and louvainla-neuve meeting of ministers responsible for higher education of 28-29 April 2009. Retrieved from http://www.enqa.eu/files/ENQA_ position_ paper%20(3).pdf
- European Association for Quality Assurance. (2009b). The Bologna process 2020－The European higher education area in the new decade. Retrieved from http://www.enqa.eu/files/Leuven_ Louvain-la-Neuve_ Communique_ April_2009.pdf
- Gardner, H. (1983). Frames of mind. New York, NY: Basic Books.
- Holland, J. L. (1973). Making vocational choices: A theory of careers. Upper Saddle River, NJ: Prentice Hall.
- Liu, T.-T., & Sun, H.-B. (2012). Gender differences on information literacy of science and engineering undergraduates. International Journal of Modern Education and Computer Science, 4(2), 23-30. doi:10.5815/ijmecs.2012.02.04
- Ma, F.-C., Ding, R., & Li, Z.-Z. (2009). Case study: An analysis of information literacy competence of university students in Wuhan area. Document, Information & Knowledge, 1, 24-29. (in Chinese)
- Masters, M. S., & Sanders, B. (1993). Is the gender difference in mental rotation disappearing? Behavior Genetics, 23(4), 337-341. doi:10.1007/BF01067434
- McManus, I. C., & Furnham, A. (2006). Aesthetic activities and aesthetic attitudes: Influences of education, background and personality on interest and involvement in the arts. British Journal of Psychology, 97(4), 555-587. doi:10.1348/000712606X101088
- Organization for Economic Co-operation and Development. (2007a). PISA 2006: Science competencies for tomorrow’s world, volume I analysis. Paris, France: Author.
- Organization for Economic Co-operation and Development. (2007b). PISA 2006: Science competencies for tomorrow’s world, volume II analysis. Paris, France: Author.
- Pope-Davis, D. B., & Twing, J. S. (1991). The effects of age, gender, and experience on measures of attitude regarding computers. Computers in Human Behavior, 7(4), 333-339. doi:10.1016/ 0747-5632(91)90020-2
- Raju, N. S., Price, L. R., Oshima, T. C., & Nering, M. L. (2007). Standardized conditional SEM: A case for conditional reliability. Applied Psychological Measurement, 31(3), 169-180. doi:10. 1177/0146621606291569
- Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.
- Rottinghaus, P. J., Day, S. X., & Borgen, F. H. (2005). The career futures inventory: A measure of career-related adaptability and optimism. Journal of Career Assessment, 13(1), 3-24. doi:10. 1177/1069072704270271
- Rowland, K. D. (2004). Career decision-making skills of high school students in the Bahamas. Journal of Career Development, 31(1), 1-13. doi:10.1023/B:JOCD.0000036702.02293.1a
- Schmidt, W. H., Raizen, S. A., Britton, E. D., Bianchi, L. J., & Wolfe, R. G. (1997). Many visions, many aims: A cross-national investigation of curricular intentions in school science. Dordrecht, the Netherlands: Kluwer Academic.
- Scott, B. P. (1998). Just what in a competency? Training Minneapolis, 35(6), 58-64.
- Sypher, H. E., Applegate, J. L., & Church, S. L. (1981, May). Cognitive differentiation and verbal intelligence: Clarifying relationships. Paper presented at the annual conference of the International Communication Association, Minneapolis, MN.
- Verderber, R. F., & Verderber, K. S. (1995). Inter-act: Using interpersonal communication skills/ voices: A selection of multicultural readings. Belmont, CA: Wadsworth.
- Wang, W.-C., & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29(2), 126-149. doi:10.1177/0146621604271053
- Wright, B. D. (1994). Reasonable mean-square fit values. In B. D. Wright & J. M. Linacre (Eds.), Rasch measurement transactions (Part 2, p. 370). Chicago, IL: MESA.
- Yip, D.-Y., Chiu, M.-M., & Ho, E. S.-C. (2004). Hong Kong student achievement in OECD-PISA study: Gender differences in science content, literacy skills, and test item formats. International Journal of Science and Mathematics Education, 2(1), 91-106. doi:10.1023/B:IJMA.0000026 537.85199.36
| APA Format | Chen, P.-H., Huang, H.-Y., Chen, Y.-H., Yeh, T.-T., & Su, S.-T. (2015). Evaluation of the Psychometric Properties of the General Literacy Test for University Students. Journal of Research in Education Sciences, 60(3), 95-126. doi:10.6209/JORIES.2015.60(3).04 |
|---|