Using automated programmes & approaches for test development or assessing productive skills.
Scott Crossley, Georgia State University and Arizona State University, sacrossley(a)gmail.com
Danielle McNamara, Arizona State University, dsmcnamara1(a)gmail.com
Intended learning outcomes:
Participants will become familiar with the structure and applications of the Coh-Metrix tool, the Test Easability Assessor (TEA), the Writing Assessment Tool (WAT), and the Simple NLP (SiNLP) tool to include the number and variety of indices available in each tool, the interpretation of the indices, and common methods for analyzing assessment data using the indices.
The workshop will cover the development and design of the Coh-Metrix tool, the Test Easability Assessor (TEA), the Writing Assessment Tool (WAT), and the Simple NLP (SiNLP) tool and the tools’ applications in test development and the assessment of productive skills (both reading and writing skills). Specifically, this workshop will demonstrate how these tools can be used to assess text difficulty during test development and how these tools can be used to automatically examine speaking and writing proficiency in testing situations.
Participants will learn the basics of natural language processing (NLP) and how to code a simple NLP tool (SiNLP) using the Python programming language. The participants will then scale up to an introduction of the basic programming features underlying Coh-Metrix, TEA, and WAT. Once the participants have a full understanding of how these tools calculate linguistic features, they will be given practical exercises involving the use these tools in testing situations (to include assessing text readability and assessing students’ written and spoken responses). These exercises will include basic approaches for conducting statistical analyses with the tools (using the open-source software package R) and more advanced machine learning techniques such as discriminant analysis, naïve Bayes algorithms, and linear regressions (using the WEKA software package). The participants will also be introduced to techniques to increase the credibility and reliability of statistical analyses including the use of training, validation, and test sets as well as cross-validation.
Specific background or prior knowledge:
Basic statistical knowledge (the interpretation of means, standard deviations, p values) and familiarity with designing quantitative studies.
Please bring your laptop.
Participants should read the following articles
Crossley, S. A., Varner, L., & McNamara, D. S. (under review). Analyzing discourse processing using a simple natural language processing tool (SiNLP). Discourse Processes.
Graesser, A.C., McNamara, D.S., & Kulikowich, J.M. (2011). Coh Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223-234.
McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural Language Processing in an Intelligent Writing Strategy Tutoring System. Behavior Research Methods, 45 (2), 499-515.
McNamara, D., & Graesser, A. (2012). Coh-Metrix: An automated tool for theoretical and applied natural language processing. In P. M. McCarthy & C. Boonthum (Eds.), Applied natural language processing and content analysis: Identification, investigation, and resolution. Hershey, PA: IGI Global.
Bio-data of Workshop Facilitators:
Please bring your laptop.
Registration is on a first-come first-serve basis, so please book early to avoid disappointment.
We will accept 30 participants.
To register click here.
Warwick Conferences will hold rooms for you at a special price. A link for booking will be provided in due course.