Paper Reading #21: Human model evaluation in interactive supervised learning

Authors
Rebecca Fiebrink, Perry Cook, Daniel Trueman

Occupation

Rebecca Fiebrink is currently an assistant professor in Computer Science at Princeton University. She holds a PhD from Princeton and was a postdoc for most of 2011 at the University of Washington.
Perry R. Cook is a professor emeritus at Princeton University in Computer Science and the Department of Music.   He is no longer teaching, but still researches, lectures, and makes music.
Daniel Trueman is a musician, primarily with the fiddle and the laptop.   He currently teaches composition at Princeton University.

Location

Published in the CHI '11 Proceedings of the 2011 annual conference on Human factors in computing systems at NYC

Summary
Hypothesis
 since model evaluation is important in interactive machine learning systems, it is important to develop a good understanding of a modeling criteria that is most important to users

Methods
There was a total of three studies of people using supervised learning. In the first study focused on the design process with the seven composers with the goals to refine the Wekinator. The participants mostly spent the time to meet regularly and discuss the software in terms of its usefulness to his/her specific work and possible improvements. The second study required students to use the Wekinator for an assignment where supervised learning was necessary for music performance systems. The students were asked to use the input device to make two gesture controlled music performance systems. The last study was done with a professional musician to produce a gesture recognition system for a cello bow that was equipped with sensors. The point of the study was to produce a gesture classification for data capturing to create musically appropriate classes.

Results
The participants thought that the algorithms used for sound control were difficult to manipulate in a satisfactory manner.(GUI/ controlled procedure was tried) The second and third experiments were based on cross validation technique. The users indicate that high levels of cross validation accuracy was a good way to indicate good performance. The participants in the third experiment strongly based more of their judgement on direct validation instead of cross validation. The classification of direct validation was done into six parts- correctness, boundary shape, cost, decision, label confidence, complexity, and unexpectedness

Content
The authors focused on how users evaluate and use supervised learning system. They look at what criteria can be used during evaluation and look into different techniques applied- cross validation, direct validation. The purpose is to make decisions in algorithm performance and improvements in more effective training data

Discussion
The paper did a good job in classifying/organizing their findings to make it easy for the readers to understand what their work was about and how their whole experimental procedure went. I believe that these kind of supervised learning system can be beneficial in application as well as it allowed better classification as well as better indicators for performance