Touch180: Finger Identification on Mobile Touchscreen using Fisheye Camera and Convolutional Neural Network

Touch180: Finger Identification on Mobile Touchscreen using Fisheye Camera and Convolutional Neural Network

Insu Kim, Keunwoo Park, Youngwoo Yoon, Geehyuk Lee

UIST 2018 LBW

Abstract

We present Touch180, a computer vision based solution for identifying fingers on a mobile touchscreen with a fisheye camera and deep learning algorithm. As a proof-of-concept research, this paper focused on robustness and high accuracy of finger identification. We generated a new dataset for Touch180 configuration, which is named as Fisheye180. We trained a CNN (Convolutional Neural Network)-based network utilizing touch locations as auxiliary inputs. With our novel dataset and deep learning algorithm, finger identification result shows 98.56% accuracy with VGG16 model. Our study will serve as a step stone for finger identification on a mobile touchscreen.

Prototype

The device setting of Touch180. device

The image processing pipeline of Touch180. network

The extended abstract is available from ACM DL. You can see the poster we presented in UIST 2018 from here.

My Contributions

I contributed to the design of the deep learning pipeline.