It’s all in the face: studies on monitoring users’ experience

Abstract

We propose to observe users’ spontaneous facial expressions as a method to identify adverse-event occurrences at the user interface level. We discuss an experiment designed to investigate the association between incidents during a word processing task and users’ facial expressions monitored using electromyogram sensor devices. The results suggest that an increase of task difficulty is related to an increase in specific facial muscle activity. A second study is presented where an application offers assistance based on the users’ facial expressions. This study evaluates a social agent that reacts to the users’ spontaneous display of emotions, in an attempt to reduce the frustration caused by usability problems. The results indicate that for highly frustrated users the agent often increased their frustration, while those experiencing a moderate level of frustration stated that it somewhat reduced their frustrations.