Page turns by blink detection


Recently I got interested in displaying sheet music electronically. I finally chose ComicsReader after adjusting its screen sensitivity a bit, though it requires to convert PDF scores to raster images in a cbz container.

I still dream of some touch free control for turning pages while playing the piano or organ. Voice control could have been an option, but Android currently does not allow application developers to use the built-in speech recognition continuously (but of cource hacks exist to circumvent the limitation).

But Android features face detection via Mobile Vision (though it was unavailable just at the time I tried using it), and most tablets feature a front facing camera. Google provides a tutorial how to use a pipeline to track barcodes and faces. Sample applications are available via Github. nesterov-n also provides a “Smiley” sample application.

The multiprocessor creates a Tracker instance for each face identity detected. Methods like onNewItem(), onUpdate() and onMissing() can be used to retrieve the face objects. The objects contain facial landmarks. Values between 0 and 1 provide information about the probability of smiling or wether the eyes are open. I wanted to check whether one (xor) eye was closed for a certain amount of time and to use this information to turn pages. Thus I used a fifo buffer to collect a certain amount of samples.

I got something to work in a spike, but it still is too unreliable. On the one hand closing one eye for a couple of seconds sometimes does not exceed the thresholds I was using. On the other hand false positives appeared every now and then, accidentally turning pages while playing though the head didn’t move at all.

It’s worth being investigated further, though. I’m currently playing a piece of six pages, and hands-free page turns by eye blinks were more than welcome during the test runs.