Automatic Item Generation (AIG), or Automated Item Generation, is a process linking psychometrics with computer programming. It uses a computer algorithm to automatically create test items that are the basic building blocks of a psychological test. The method was first described by John R. BormuthBormuth, J. (1969). On a theory of achievement test items. Chicago, IL: University of Chicago Press. in the 1960s , but was not developed until recently. AIG uses a two-step process: first, a test specialist creates a template called an item model; then, a computer algorithm is developed to generate test items.Gierl, M.J., & Haladyna, T.M. (2012). Automatic item generation, theory and practice. New York, NY: Routledge Chapman & Hall. So, instead of a test specialist writing each individual item, computer algorithms generate families of items from a smaller set of parent item models.Glas, C.A.W., van der Linden, W.J., & Geerlings, H. (2010). Estimation of the parameters in an item-cloning model for adaptive testing. In W.J. van der Linden, & C.A.W. Glas (Eds.). Elements of adaptive testing (pp. 289-314). DOI: 10.1007/978-0-387-85461-8_15.Gierl, M.J., & Lai, H. (2012). The role of item models in automatic item generation. International Journal of testing, 12(3), 273-298. DOI: 10.1080/15305058.2011.635830. In psychological testing, the responses of the test taker to test items provide objective measurement data for a variety of human characteristics.Van der Linden, W.J., & Hambleton, R.K. (1997). Item Response Theory: a brief history, common models, and extensions. In R.K. Hambleton, & W.J. van der Linden (Eds.). Handbook of modern Item Response Theory (pp. 1-31). New York: Springer. Some characteristics measured by psychological and educational tests include academic abilities, school performance, intelligence, motivation, etc. , and these tests are frequently used to make decisions that have significant consequences on individuals or groups of individuals. Achieving measurement quality standards, such as test validity, is one of the most important objectives for psychologists and educators.Embretson, S.E. (1999). Issues in the measurement of cognitive abilities. In S.E. Embretson, & S.L. Hershberger (Eds.). The new rules of measurement (pp. 1-15). Mahwah: Lawrence Erlbaum Associates. AIG is an approach to test development which can be used to maintain and improve test quality economically in the contemporary environment where computerized testing has increased the need for large numbers of test items. Benefits AIG reduces the cost of producing standardized tests,Rudner, L. (2010). Implementing the graduate management admission test computerised adaptive test. In W.J. van der Linden, and C.A.W. Glas (Eds.). Elements of adaptive testing (pp. 151-165). DOI: 10.1007/978-0-387-85461-8_15. as algorithms can generate many more items in a given amount of time than a human test specialist. It can quickly and easily create parallel test forms, which allow for different test takers to be exposed to different groups of test items with the same level of complexity or difficulty, thus enhancing test security. When combined with computerized adaptive testing, AIG can generate new items or select which already-generated items should be administered next based on the test taker's ability during the administration of the test. AIG is also expected to produce items with a wide range of difficulty, fewer errors in construction, and is expected to permit higher comparability of items due to a more systematic definition of the prototypical item model.Irvine, S. (2002). The foundations of item generation for mass testing. In S.H. Irvine, & P.C. Kyllonen (Eds.). Item generation for test development (pp. 3-34). Mahwah: Lawrence Erlbaum Associates.Lai, H., Alves, C., & Gierl, M.J. (2009). Using automatic item generation to address item demands for CAT. In D.J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Web: www.psych.umn.edu/psylabs/CATCentral. Test development (including AIG) can be enriched if it is based on any cognitive theory. Cognitive processes taken from a given theory are often matched with item features during their construction. The purpose of this is to predetermine a given psychometric parameter, such as item difficulty (from now on: ). Let radicals be those structural elements that significantly affect item parameters and provide the item with certain cognitive requirements. One or more radicals of the item model can be manipulated in order to produce parent item models with different parameters (e.g., ) levels. Each parent can then grow its own family by manipulating other elements that Irvine called incidentals. Incidentals are surface features that suffer random variations from item to item within the same family. Items that have the same structure of radicals and only differ in incidentals are usually labeled as isomorphsBejar, I. I. (2002). Generative testing: from conception to implementation in Item Generation for Test Development, eds. S. H. Irvine and P. C. Kyllonen (Mahwah, NJ: Lawrence Erlbaum Associates), 199–217. or clones.Embretson, S.E. (1999). Generating items during testing: psychometric issues and models. Psychometrika, 64(4), 407-433.Arendasy, M. E., and Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of  high-stakes educational and occupational assessment. Learning and individual differences, 22, 112–117. doi: 10.1016/j.lindif.2011.11.005. There can be two kinds of Item Cloning: On the one hand, the item model may consist of an item with one or more open places, and cloning is done by filling each place with an element selected from a list of possibilities. On the other hand, the item model could be an intact item which is cloned by introducing transformations, for example changing the angle of an object of spatial ability tests.Glas, C. A. W., and van der Linden, W. J. (2003). Computerized adaptive testing with item cloning. Applied psychological measurement, 27, 247–261. doi: 10.1177/0146621603027004001. The variation of these items’ surface characteristics should not significantly influence the testee's responses. This is the reason why it is believed that incidentals produce only slight differences among the item parameters of the isomorphs. 50px The material was copied from this source, which is available under a Creative Commons Attribution 4.0 International License. A test of melodic discrimination developed with the aid of the computational model Racchman-Jun 2015fCollins , T., Laney, R., Willis, A., & Garthwaite, P.H. (2016). Developing and evaluating computational models of music style. Artificial intelligence for engineering design, analysis  and manufacturing, 30, 16-43. DOI: 10.1017/S0890060414000687. was administered to participants in a 2017 trial. According to the data collected by P.M. Harrison et al., results demonstrate strong validity and reliability.Harrison, P.M., Collins, T., & Müllensiefen, D. (2017). Applying modern psychometric techniques to melodic discrimination testing: item response theory, computerised adaptive testing, and automatic item generation. Scientific reports, 7(3618), 1-18. Gierl and his colleaguesGierl, M.J., Lai, H., Pugh, D., Touchie, C., Boulais, A.P., & De Champlain, A. (2016). Evaluating the psychometric characteristics of generated multiple-choice test items. Applied measurement in education, 29(3), 196-210. DOI: 10.1080/08957347.2016.1171768.Lai, H., Gierl, M.J., Byrne, B.E., Spielman, A.I., & Waldschmidt, D.M. (2016). Three modelling applications to promote automatic item generation for examinations in dentistry. Journal of dental education, 80(3), 339-347.Gierl, M.J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical education, 47, 726-733. DOI: 10.1111/medu.12202.Gierl, M.J., Lai, H., & Turner, S.R. (2012). Using automatic item generation to create multiple-choice test items. Medical education, 46(8), 757-765. DOI: 10.1111/j.1365-2923.2012.04289.x. used an AIG program called the Item Generator (IGORGierl, M.J., Zhou, J., & Alves, C. (2008). Developing a taxonomy of item mode types to promote assessment engineering. J technol learn assess, 7(2), 1-51.) to create multiple-choice items that test medical knowledge. IGOR-generated items, even when compared to manually-designed items, showed good psychometric properties. Holling, Bertling, and ZeuchHolling, H., Bertling, J.P., & Zeuch, N. (2009). Automatic item generation of probability word problems. Studies in educational evaluation, 35(2-3), 71-76. used probability theory to automatically generate mathematical word problems with expected difficulties. They achieved a RaschRasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press. model fit , and item difficulties could be explained by the Linear Logistic Test Model (LLTMFischer, G.H. (1973). The linear logistic test model as an instrument of educational research. Acta Psychological, 37, 359-374. DOI: 10.1016/0001-6918(73)90003-6.), as well as by the Random-Effects LLTM. Holling, Blank, Kuchenbäcker, and KuhnHolling, H., Blank, H., Kuchenbäcker, K., & Kuhn, J.T. (2008). Rule-based item design of statistical word problems: a review and first implementation. Psychology science quarterly, 50(3), 363-378. made a similar study with statistical word problems but without using AIG. Arendasy and his colleaguesArendasy, M.E., Sommer, M., Gittler, G., & Hergovich, A. (2006). Automatic generation of quantitative reasoning items. A pilot study. Journal of individual differences, 27(1), 2-14. DOI: 10.1027/1614-0001.27.1.2.Arendasy, M.E., & Sommer, M. (2007). Using psychometric technology in educational assessment: the case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items. Learning and individual differences, 17(4), 366-383. DOI: 10.1016/j.lindif.2007.03.005. presented studies on automatically generated algebra word problems and examined how a quality control framework of AIG can affect the measurement quality of items.
Automatic Item Generation (AIG), or Automated Item Generation, is a process linking psychometrics with computer programming. It uses a computer algorithm to automatically create test items that are the basic building blocks of a psychological test. The method was first described by John R. BormuthBormuth, J. (1969). On a theory of achievement test items. Chicago, IL: University of Chicago Press. in the 1960s but was not developed until recently. AIG uses a two-step process: first, a test specialist creates a template called an item model; then, a computer algorithm is developed to generate test items.Gierl, M.J., & Haladyna, T.M. (2012). Automatic item generation, theory and practice. New York, NY: Routledge Chapman & Hall. So, instead of a test specialist writing each individual item, computer algorithms generate families of items from a smaller set of parent item models.Glas, C.A.W., van der Linden, W.J., & Geerlings, H. (2010). Estimation of the parameters in an item-cloning model for adaptive testing. In W.J. van der Linden, & C.A.W. Glas (Eds.). Elements of adaptive testing (pp. 289-314). DOI: 10.1007/978-0-387-85461-8_15.Gierl, M.J., & Lai, H. (2012). The role of item models in automatic item generation. International Journal of testing, 12(3), 273-298. DOI: 10.1080/15305058.2011.635830. In psychological testing, the responses of the test taker to test items provide objective measurement data for a variety of human characteristics.Van der Linden, W.J., & Hambleton, R.K. (1997). Item Response Theory: a brief history, common models, and extensions. In R.K. Hambleton, & W.J. van der Linden (Eds.). Handbook of modern Item Response Theory (pp. 1-31). New York: Springer. Some characteristics measured by psychological and educational tests include academic abilities, school performance, intelligence, motivation, etc. and these tests are frequently used to make decisions that have significant consequences on individuals or groups of individuals. Achieving measurement quality standards, such as test validity, is one of the most important objectives for psychologists and educators.Embretson, S.E. (1999). Issues in the measurement of cognitive abilities. In S.E. Embretson, & S.L. Hershberger (Eds.). The new rules of measurement (pp. 1-15). Mahwah: Lawrence Erlbaum Associates. AIG is an approach to test development which can be used to maintain and improve test quality economically in the contemporary environment where computerized testing has increased the need for large numbers of test items. Benefits AIG reduces the cost of producing standardized tests,Rudner, L. (2010). Implementing the graduate management admission test computerized adaptive test. In W.J. van der Linden, and C.A.W. Glas (Eds.). Elements of adaptive testing (pp. 151-165). DOI: 10.1007/978-0-387-85461-8_15. as algorithms can generate many more items in a given amount of time than a human test specialist. It can quickly and easily create parallel test forms, which allow for different test takers to be exposed to different groups of test items with the same level of complexity or difficulty, thus enhancing test security. When combined with computerized adaptive testing, AIG can generate new items or select which already-generated items should be administered next based on the test taker's ability during the administration of the test. AIG is also expected to produce items with a wide range of difficulty, fewer errors in construction, and is expected to permit higher comparability of items due to a more systematic definition of the prototypical item model.Irvine, S. (2002). The foundations of item generation for mass testing. In S.H. Irvine, & P.C. Kyllonen (Eds.). Item generation for test development (pp. 3-34). Mahwah: Lawrence Erlbaum Associates.Lai, H., Alves, C., & Gierl, M.J. (2009). Using automatic item generation to address item demands for CAT. In D.J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Web: www.psych.umn.edu/psylabs/CATCentral. Test development (including AIG) can be enriched if it is based on any cognitive theory. Cognitive processes taken from a given theory are often matched with item features during their construction. The purpose of this is to predetermine a given psychometric parameter, such as item difficulty (from now on: ). Let radicals be those structural elements that significantly affect item parameters and provide the item with certain cognitive requirements. One or more radicals of the item model can be manipulated in order to produce parent item models with different parameters (e.g., ) levels. Each parent can then grow its own family by manipulating other elements that Irvine called incidentals. Incidentals are surface features that suffer random variations from item to item within the same family. Items that have the same structure of radicals and only differ in incidentals are usually labeled as isomorphsBejar, I. I. (2002). Generative testing: from conception to implementation in Item Generation for Test Development, eds. S. H. Irvine and P. C. Kyllonen (Mahwah, NJ: Lawrence Erlbaum Associates), 199–217. or clones.Embretson, S.E. (1999). Generating items during testing: psychometric issues and models. Psychometrika, 64(4), 407-433.Arendasy, M. E., and Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of the high-stakes educational and occupational assessment. Learning and individual differences, 22, 112–117. doi: 10.1016/j.lindif.2011.11.005. There can be two kinds of Item Cloning: On the one hand, the item model may consist of an item with one or more open places, and cloning is done by filling each place with an element selected from a list of possibilities. On the other hand, the item model could be an intact item which is cloned by introducing transformations, for example changing the angle of an object of spatial ability tests.Glas, C. A. W., and van der Linden, W. J. (2003). Computerized adaptive testing with item cloning. Applied psychological measurement, 27, 247–261. doi: 10.1177/0146621603027004001. The variation of these items’ surface characteristics should not significantly influence the testee's responses. This is the reason why it is believed that incidentals produce only slight differences among the item parameters of the isomorphs. 50px The material was copied from this source, which is available under a Creative Commons Attribution 4.0 International License. A test of melodic discrimination developed with the aid of the computational model Rachman-Jun 2015Collins , T., Laney, R., Willis, A., & Garthwaite, P.H. (2016). Developing and evaluating computational models of music style. Artificial intelligence for engineering design, analysis , and manufacturing, 30, 16-43. DOI: 10.1017/S0890060414000687. was administered to participants in a 2017 trial. According to the data collected by P.M. Harrison et al., results demonstrate strong validity and reliability.Harrison, P.M., Collins, T., & Müllensiefen, D. (2017). Applying modern psychometric techniques to melodic discrimination testing: item response theory, computerized adaptive testing, and automatic item generation. Scientific reports, 7(3618), 1-18. Gierl and his colleaguesGierl, M.J., Lai, H., Pugh, D., Touchie, C., Boulais, A.P., & De Champlain, A. (2016). Evaluating the psychometric characteristics of generated multiple-choice test items. Applied measurement in education, 29(3), 196-210. DOI: 10.1080/08957347.2016.1171768.Lai, H., Gierl, M.J., Byrne, B.E., Spielman, A.I., & Waldschmidt, D.M. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of dental education, 80(3), 339-347.Gierl, M.J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical education, 47, 726-733. DOI: 10.1111/medu.12202.Gierl, M.J., Lai, H., & Turner, S.R. (2012). Using automatic item generation to create multiple-choice test items. Medical education, 46(8), 757-765. DOI: 10.1111/j.1365-2923.2012.04289.x. used an AIG program called the Item Generator (IGORGierl, M.J., Zhou, J., & Alves, C. (2008). Developing a taxonomy of item mode types to promote assessment engineering. J technol learn assess, 7(2), 1-51.) to create multiple-choice items that test medical knowledge. IGOR-generated items, even when compared to manually-designed items, showed good psychometric properties. Holling, Bertling, and ZeuchHolling, H., Bertling, J.P., & Zeuch, N. (2009). Automatic item generation of probability word problems. Studies in educational evaluation, 35(2-3), 71-76. used probability theory to automatically generate mathematical word problems with expected difficulties. They achieved a RaschRasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press. model fit and item difficulties could be explained by the Linear Logistic Test Model (LLTMFischer, G.H. (1973). The linear logistic test model as an instrument of educational research. Acta Psychological, 37, 359-374. DOI: 10.1016/0001-6918(73)90003-6.), as well as by the Random-Effects LLTM. Holling, Blank, Kuchenbäcker, and KuhnHolling, H., Blank, H., Kuchenbäcker, K., & Kuhn, J.T. (2008). Rule-based item design of statistical word problems: a review and first implementation. Psychology science quarterly, 50(3), 363-378. made a similar study with statistical word problems but without using AIG. Arendasy and his colleaguesArendasy, M.E., Sommer, M., Gittler, G., & Hergovich, A. (2006). Automatic generation of quantitative reasoning items. A pilot study. Journal of individual differences, 27(1), 2-14. DOI: 10.1027/1614-0001.27.1.2.Arendasy, M.E., & Sommer, M. (2007). Using psychometric technology in educational assessment: the case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items. Learning and individual differences, 17(4), 366-383. DOI: 10.1016/j.lindif.2007.03.005. presented studies on automatically generated algebra word problems and examined how a quality control framework of AIG can affect the measurement quality of items.
D
,
D
,
R
computerised
computerized
A
the
R
Racchman-Jun 2015fCollins
Rachman-Jun 2015Collins
A
,
R
computerised
computerized
R
modelling
modeling
D
,