Jana Sukkarieh from the Educational Testing Service will discuss the scoring of free-text responses at this seminar.

Ongoing progress in natural language processing and a renewed interest in open-ended test questions prompted researchers and assessment organizations to consider the development of automatic scoring technologies.

In this talk, I will concentrate on discussing automatic content scoring technologies. In particular, I will emphasize the kind of content these technologies are trying to measure and the natural language understanding skills that, ideally, such technology needs to possess. I will then describe several techniques for these types of scoring tasks, focusing on the use of text analytics and mining for short free-text responses written in English as well as those written in French, Spanish, and Japanese.

I will also present a methodology for question design in view of the fact that building scoring technologies cannot be seen independently from the question design process – in fact, one drives the other. I will demonstrate how some test question attributes, e.g., the cognitive skills required to respond to a question, can be predictive of the technology’s performance. In addition, I will report on two types of evaluation: one using open-ended test questions and another using a suite that can be used for automated regression and performance evaluations within the technology and for a comparative evaluation across several technologies.

Since assessment can be seen as a way of enhancing student learning, I will outline the types of student feedback that can be provided by these text-understanding-based technologies. Finally, I will outline how these techniques can be used in other content-based text management applications such as the clustering of textual online discussions and the generation of automatic alerts for newly published articles.

Open to all members of the university community.

If you are coming from outside the Edlab, please RSVP by March 26th by sending a message to Edlab at http://ednode.com/profiles/edlabtc/about (You must have/create an Ednode account and click on the Contact button to send a message).

EdLab Seminars take place on the 5th floor of Russell Hall at Teachers College Columbia University (525 W. 120th Street, NY, NY). For detailed directions within the College visit TC Route Finder: http://maps.tc-library.org