We explore opportunities for bringing tangible input and haptic output to desktop GUI applications by addressing two challenges. First, we address the lack of tangible input devices by repurposing educational and toy robots. These inexpensive robots support connecting to PCs via Bluetooth and Wi-Fi. They contain sensors such as accelerometers and gyroscopes that enable repurposing them as input devices, as well as actuators that provide haptic feedback. Second, we address the lack of software support for tangible input and haptic output by enabling our tangible input devices to pair with existing applications without changing the underlying code, through a combination of input event emulation, GUI automation and custom application APIs.

To read more, follow the link below:

Technical Paper: GUI Robots   IRON Lab

 

Publications

Darren Guinness, Daniel Szafir, and Shaun K. Kane. 2017. GUI Robots: Using Off-the-Shelf Robots as Tangible Input and Output Devices for Unmodified GUI Applications. In Proceedings of the 2017 Conference on Designing Interactive Systems (DIS '17). ACM, New York, NY, USA, 767-778.   DOI: https://doi.org/10.1145/3064663.3064706 (Edinburgh, United Kingdom, June 10-14, 2017).