A Methodology for Generating Subquestions for the Force Concept Inventory (and Other Research-Based Assessments)

Michael M. Hull; Jun-ichiro Yasuda; Naohiro Mae

doi:10.51724/ijpce.v16i2.351

Authors

Michael M. Hull University of Alaska Fairbanks https://orcid.org/0000-0003-1551-9553
Jun-ichiro Yasuda Yamagata University, Institute of Arts and Sciences https://orcid.org/0000-0002-2798-2708
Naohiro Mae Osaka University, Research Center for Nuclear Physics https://orcid.org/0000-0001-5118-299X

DOI:

https://doi.org/10.51724/ijpce.v16i2.351

Keywords:

Force Concept Inventory, subquestions, false positives, Newtonian mechanics

Abstract

In this paper, we discuss the methodology we have developed and have been using for creating subquestions for the Force Concept Inventory (FCI). The FCI is a research-based assessment that is used internationally to assess student understanding of Newtonian mechanics. The assessment has been investigated from a number of perspectives and many suggestions have been made for its improvement. One challenge that is becoming more and more pervasive as more learning has transitioned to an online environment and more students are completing the FCI without a proctor is that of test security. If the answers to FCI items become easily accessible, then students will be able to provide correct answers despite lacking in understanding of Newtonian mechanics. One approach to mitigate the effects of items leaking into the public sphere and maintaining test security is the creation and administration of subquestions in place of the original FCI questions. Subquestions have an additional benefit of reducing false positives (answering a survey item correctly without correct understanding) and false negatives (answering incorrectly despite correct understanding). In this paper, we will discuss how we created subquestions for four items on the FCI, informed by survey-based interviews with students and the original intended targets of the items.

Downloads

Download data is not yet available.

References

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66(1), 64-74. https://doi.org/10.1119/1.18809

Hestenes, D., & Halloun, I. (1995). Interpreting the force concept inventory: A response to March 1995 critique by Huffman and Heller. The Physics Teacher, 33(8), 502. https://doi.org/10.1119/1.2344278

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force Concept Inventory. The Physics Teacher, 30(3), 141-158. https://doi.org/10.1119/1.2343497

Lawson, A. E. (1978). The development and validation of a classroom test of formal reasoning. Journal of Research in Science Teaching, 15(1), 11-24. https://doi.org/10.1002/tea.3660150103

Low, D. J., & Wilson, K. F. (2017). The role of competing knowledge structures in undermining learning: Newton’s second and third laws. American Journal of Physics, 85(1), 54-65. https://doi.org/10.1119/1.4972041

Planinic, M., Ivanjek, L., & Susac, A. (2010). Rasch model based analysis of the Force Concept Inventory. Physical Review Special Topics - Physics Education Research, 6(1), Article 010103. https://doi.org/10.1103/PhysRevSTPER.6.010103

Scott, T. F., & Schumayer, D. (2017). Conceptual coherence of non-Newtonian worldviews in Force Concept Inventory data. Physical Review Physics Education Research, 13(1), Article 010126. https://doi.org/10.1103/PhysRevPhysEducRes.13.010126

Scott, T. F., Schumayer, D., & Gray, A. R. (2012). Exploratory factor analysis of a Force Concept Inventory data set. Physical Review Special Topics - Physics Education Research. 8(2), Article 020105. https://doi.org/10.1103/PhysRevSTPER.8.020105

Taniguchi, M.-a., & Yasuda, J.-i. (2014). Quantitative validation of Japanese translation of Force Concept Inventory using subquestions [Japanese]. Journal of the Physics Education Society of Japan, 62(4), 226-231. https://doi.org/10.20653/pesj.62.4_226

Thornton, R. K., Kuhl, D., Cummings, K., & Marx, J. (2009). Comparing the force and motion conceptual evaluation and the force concept inventory. Physical Review Special Topics - Physics Education Research, 5(1), Article 010105. https://doi.org/10.1103/PhysRevSTPER.5.010105

Traxler, A., Henderson, R., Stewart, J., Stewart, G., Papak, A., & Lindell, R. (2018). Gender fairness within the Force Concept Inventory. Physical Review Physics Education Research, 14(1), Article 010103. https://doi.org/10.1103/PhysRevPhysEducRes.14.010103

Wainer, H., Bradlow, E. T., & Wang, X. (2007). What’s a testlet and why do we need them? In Testlet response theory and its applications (pp. 44–59). Cambridge University Press. https://doi.org/10.1017/CBO9780511618765.005

Wang, J., & Bao, L. (2010). Analyzing force concept inventory with item response theory. American Journal of Physics, 78(10), 1064-1070. https://doi.org/10.1119/1.3443565

Wilson, K. F., & Low, D. J. (2015). “On second thoughts…”: Changes of mind as an indication of competing knowledge structures. American Journal of Physics, 83(9), 802-808. https://doi.org/10.1119/1.4928131

Yasuda J.-i, Hull, M. M., & Mae, N. (2023). Visualizing depth of student conceptual understanding using subquestions and alluvial diagrams. Physical Review Physics Education Research, 19(2), Article 020121. https://doi.org/10.1103/PhysRevPhysEducRes.19.020121

Yasuda, J.-i., & Taniguchi, M.-a. (2013). Validating two questions in the Force Concept Inventory with subquestions. Physical Review Special Topics - Physics Education Research, 9(1), Article 010113. https://doi.org/10.1103/PhysRevSTPER.9.010113

Yasuda, J.-i., Hull, M. M., & Mae, N. (2022). Improving test security and efficiency of computerized adaptive testing for the Force Concept Inventory. Physical Review Physics Education Research, 18(1), Article 010112. https://doi.org/10.1103/PhysRevPhysEducRes.18.010112

Yasuda, J.-i., Mae, N., Hull, M. M., & Taniguchi, M.-a. (2018). Analyzing false positives of four questions in the Force Concept Inventory. Physical Review Physics Education Research, 14(1), Article 010112. https://doi.org/10.1103/PhysRevPhysEducRes.14.010112

Yasuda, J.-i., Mae, N., Hull, M. M., & Taniguchi, M.-a. (2021). Optimizing the length of computerized adaptive testing for the Force Concept Inventory. Physical Review Physics Education Research, 17(1), Article 010115. https://doi.org/10.1103/PhysRevPhysEducRes.17.010115

Yasuda, J.-i., Uematsu, H., & Nitta, H. (2011). Validating a Japanese version of Force Concept Inventory [Japanese]. Journal of the Physics Education Society of Japan, 59(2), 90-95. https://doi.org/10.20653/pesj.59.2_90