Related Talk Titles
The topic of assessment covers several areas of learning but primarily had the goal of automating feedback systems. For many of the researchers at the conference, data mining is best when it advances student learning. The innovative approach being explored in several talks at the conference was automated assessments of open ended, questions. This included short answer questions, programming questions, and even essay questions.
The research in this topic looks to address a need for new learning assessment materials beyond simple multiple choice systems. Additionally these systems should be able to provide students individualized feedback based on their individual responses or behaviors. For open response questions the primary limitation to current assessment practices are that they need to be done manually; As such, assessment accuracy may not be uniform, subject to the educators' fatigue, misinterpretation of rubrics and other human factors.
Natural Language processing, semantic analysis and other machine learning methods are frequently used to analyze open response short answer, or essay responses or text. Currently, some of the existing tools include (SIDE) Summarization Integrated Development Environment, SPSS text analysis, and EvoGrader. There is currently no automated essay, or short answer assessment tool which works well enough to be used widely in education.
The authors of the research Automated Assessment for Scientific Explanations in On-line Science Inquiry, developed their own online system for science education called Inqits which provides real time science assessment, feedback, and tutoring.
Using knowledge tracing they can provide students with individualized feedback from a virtual coach. Knowledge tracing is a method of modeling a learner's knowledge or acquired skills. The Inqits system tracks the real time user behavior in online "labs" and the system is able to provide feedback on what skills the user may need more work on.
In Evaluation of a data driven feedback algorithm for open ended programming the researchers similarly developed a system for automatic feedback on programming questions called SourceCheck. The system compares responses in real-time to existing solutions and by measuring the "distance" between the two, it can suggest moves such as insertions, deletions, renaming, movements, or re-orderings.
In all of these studies there must be an analysis as to the accuracy of the systems and how they compare to human educators. Both of these studies showed promising results.
Providing real-time feedback to students requires real-time data which for an online platform may include click-stream data, or other data related to their interactions on the platform.
Additionally for the researchers and developers there is an additional step to take the results of their data, measure the learning outcomes, and modify or adapt the learning platform to address any deficiencies and optimize the system.
In Can a few hundred parameters outperform a few hundred thousand? the authors showed a model which by using knowledge tracing techniques could predict which questions in an assessment a student would be able to answer based on their skills. This is certainly an interesting study additionally because they have shown that a method with a few parameters can generate more accurate predictions than other more complex methods.
Closing the loop: Automated data-driven cognitive model discoveries lead to improved instruction and learning gains, interestingly looked to automate the process or loop of deploying a learning technology, data collection, system re-design. There should be an additional step of interpretation between data collection and system re-design. Again they are using knowledge tracing on an online platform to generate a map of the learners knowledge. With this map, an automated tutor is able to select the questions based on the state of the learner and what skills or knowledge they have demonstrated. They experimented on students who worked with a control automated tutor and a re-designed automated tutor and found learning improvements in the re-designed tutor.
This last study is relevant to our systems at EdLab as we may want to think about recommendation systems, and self-directed learning. What do you think about automated assessment systems?