TapSense Mobile Affective Computing


In Affective Computing, different modalities, such as speech, facial expressions, physiological properties, smartphone usage patterns, and their combinations, are applied to detect the affective states of a user. In this project, we explore different challenges of developing smartphone based emotion detection application like modality selection, influence of Experience Sampling Method (ESM) on emotion detection, simplifying self-report collection.

Keystroke analysis i.e. study of the typing behavior in desktop computer is found to be an effective modality for emotion detection because of its reliability, non-intrusiveness and low resource overhead. As smartphones proliferate, typing behavior on smartphone presents an equally powerful modality for emotion detection. It has the added advantage to run in-situ experiments with better coverage than the experiments using desktop computer keyboards. As a part of this project, we explore the efficacy of smartphone typing to detect multiple affective states. We design, develop and evaluate an Android based application TapSense to determine multiple emotion states from typing activity in smartphone. This has been developed by the researchers from Complex Network Research Group (CNeRG) of IIT Kharargpur, India in collaboration with researchers from Georgia Southern University, USA .


Related Publication | Results | Links | Contact


PUBLICATIONS [Back to top]


RESULTS [Back to top]

A. Survey Findings

We performed an online survey using Facebook among 120 participants of different age group (18 to 50 years). This survey has revealed many interesting findings about individual typing pattern in smartphone. Our study reveals that 56% of the participants spent more than half an hour daily in typing and 27% of the participants spent at least an hour daily in typing using smartphone. This reveals that among several typing intensive applications, WhatsApp is the one mostly used, while texting and browsing are the ones used rarely.

We also find that different typing cues like typing speed, typing mistakes, usage of special characters, typing duration are correlated well changes in emotion states.

crowd
(a) Daily average time spent (in minutes) in typing by the participants (b) Frequency of different typing applications used (1 indicates most frequently used, 5 indicates rarely used)
motiv_survey_3
Typing cues for emotion detection. (a) Application usage variation with emotion (b) Typing intensive application usage variation with emotion (c) Variation in typing speed with emotion (d) Variation in typing mistakes with emotion (e) Variation in special character / emoticons in specific emotion states (f) Variation of typing duration with emotion

 

B. Experimental Results

We have conducted a 3-week on-the-wild study involving 22 participants (20 male, 2 female). During this study we have collected close to 135 hours of typing data spanning across 2700 typing session. We analyse the collected data to find out how accurately TapSense can determine multiple emotion states without much resource overhead.

We obtain an average accuracy (AUCROC) of 84% (standard deviation 6%) while the maximum AUCROC is 94%. The emotion states are identified with precision between 67% and 75%, and recall rate between 57% and 80%). We observe that relaxed state is identified with highest precision, followed by stressed, happy and sad states respectively. Similarly, we observe highest recall for relaxed, followed by happy, sad and stressed states.

perf_eval
Performance evaluation of classification model deployed in TapSense

We measure the energy consumption of TapSense using a Moto G2 (Android version 6:0) smartphone. We charge the phone fully and monitor how quickly the battery is completely drained by keeping TapSense on and off. It reveals that in both cases, not only the depletion time, but also the depletion rate is similar.

energy_cons
Comparison of battery depletion (keeping TapSense on and off)

 

C. Post-study Participant Feedback

We conducted a post study survey following the Post-Study System Usability Questionnaire (PSSUQ) to gauge the system effectiveness from usability perspective. 75% participants marked TapSense as non-intrusive, while 20% ranked it as moderately intrusive. 61% of the participants reported lack of swype facility as an inconvenience. Only 24% participants expressed dicomfort with level of privacy and less than 10% participants were concerned about energy consumption.

post_study_survey
User preferences in TapSense usability survey

 


Links [Back to top]


CONTACT [Back to top]


Date modified: Dec 14, 2018