Publications

Human dexterity enables versatile and precise manipulation in everyday life. Recent advances in wearable and mobile computing have opened new ways to augment this dexterity. These efforts go beyond traditional assistive tools—whose primary function has been to restore or enhance strength—and instead aim to support more precise and complex performance across diverse domains, such as playing musical instruments or performing precision tasks at a workbench. A range of technologies has been explored, including multimodal sensory cues, adaptive learning schemes, robots, and exoskeletons. Yet individual projects remain limited in scope, often confined to a specific technology or application domain. Because dexterity is inherently multifaceted, no single perspective is likely to capture it fully—yet UIST has not seen a dedicated effort to bring these perspectives together. We therefore propose this workshop, framing the topic as a broad goal: augmenting human dexterity. We aim to bring together researchers including those from mechanical engineering, neuroscience, learning science, domain-specific expertise, to discuss what dexterity entails and how technology might support it in a more holistic way. Through interdisciplinary talks, interactive demonstrations, and collaborative discussions, we hope to map out a broad research agenda in this direction.

Overeating and emotional eating are common health issues that affect people even without an eating disorder. The vagus nerve plays a critical role in the gut-brain axis, and implanted vagus nerve stimulators have been associated with reduced appetite. In this paper, we propose transcutaneous cervical vagus nerve stimulation (tcVNS) as a ubiquitous system to provide immediate, low-effort intervention during an eating episode. In a study with 24 participants, we evaluated a mobile, handheld tcVNS device during a single episode of distracted snacking. We found that participants ate 9.6% less and 23.6% more slowly during vagus nerve stimulation than during sham stimulation. Post-snacking satiety was the same in both conditions, while heart rate was lower during vagus nerve stimulation. The stimulation was described as subtle and barely noticeable. Overall, these results provide evidence for the feasibility of non-invasive vagus nerve stimulation as a low-attention intervention for managing eating behavior – one that can be packaged inside ubiquitous interactive systems. As such, we extend the design space of implicit interfaces toward physiological intervention, motivating future ubiquitous systems that pair eating-related sensing with low-attention interventions.

Grip force is commonly used as an overall health indicator in older adults and is valuable for tracking progress in physical training and rehabilitation. Existing methods for wearable grip force measurement are cumbersome and user-dependent, making them insufficient for practical, continuous grip force measurement. We introduce EchoForce, a novel wristband using acoustic sensing for low-cost, non-contact measurement of grip force. EchoForce captures acoustic signals reflected from subtle skin deformations by flexor muscles on the forearm. In a user study with 11 participants, EchoForce achieved a fine-tuned user-dependent mean error rate of 9.08% and a user-independent mean error rate of 12.3% using a foundation model. Our system remained accurate between sessions, hand orientations, and users, overcoming a significant limitation of past force sensing systems. EchoForce makes continuous grip force measurement practical, providing an effective tool for health monitoring and novel interaction techniques.

Human senses are fundamental to how we interpret and interact with the world. Computing devices are increasingly coupled with the human sensory system through interfaces such as smart glasses, earbuds, and wristbands. This opens up opportunities to dynamically mediate, modify, and augment perceptual experiences and physiological processes through multisensory stimulation. These devices go beyond assistive technologies designed for individuals with sensory impairments (e.g., hearing aids) and are now available for everyday use. Applications range from enriching immersive entertainment experiences to supporting well-being through multisensory interventions. The UIST community has been a key venue for introducing many proof-of-concept prototypes in multisensory stimulation. However, gaps remain in systematically understanding how such technologies can be designed, studied, and contextualized in long-term, everyday use. This workshop will examine barriers to transitioning prototypes from proof-of-concepts into systems for real-world use. The session will feature keynote talks, demo sessions, and an interactive device-swap activity where participants exchange and wear different devices during the afternoon session, and conclude with an open discussion to develop implementation frameworks.

Autonomic processes are responsible for maintaining the most vital bodily functions, yet physiological interfaces have had limited success in continuously augmenting these functions. A new wave of wearable and ubiquitous devices have begun to sense and actuate autonomic processes of the human body while requiring little deliberate input from the user. In this position paper, we explore three examples of such devices from our work: An airflow breathing guide, a nerve stimulator for modulating satiety and a haptic glove for implicitly learning piano. The design and experience of these devices provide a unique perspective on control and awareness in interaction. We synthesize common trends for the design of “autonomic interfaces,” sensorimotor devices that augment users’ abilities implicitly by mirroring the mechanisms of autonomic physiological processes.

The Tenth International Brain-Computer Interface (BCI) Meeting was held June 6-9, 2023 in the Sonian Forest in Brussels, Belgium. At that meeting, 21 master classes, organized by the BCI Society' s Postdoc & Student Committee, supported the Society' s goal of fostering learning opportunities and meaningful interactions for trainees in BCI-related fields. Master classes provide an informal environment where senior researchers can give constructive feedback to the trainee on their chosen and specific pursuit. The topics of the master classes span the whole gamut of BCI research and techniques. These include data acquisition, neural decoding and analysis, invasive and noninvasive stimulation, and ethical and transitional considerations. Additionally, master classes spotlight innovations in BCI research. Herein, we discuss what was presented within the master classes by highlighting each trainee and expert researcher, providing relevant background information and results from each presentation, and summarizing discussion and references for further study.

Workplace stress contributes to poor performance and adverse health outcomes, yet current stress management tools often fall short in the fast-paced modern workforce. Guided slow breathing is a promising intervention for stress and anxiety, with peripheral breathing guides being explored for concurrent task use. However, their need for explicit user engagement underscores the need for more seamless, implicit interventions optimized for workplaces. In this mixed-method, controlled study, we examined the feasibility and effects of BreathePulse, a laptop-mounted device that delivers pulsing airflow to the nostrils as an implicit cue, on stress, anxiety, affect, and workload during two levels of a memory (N-Back) task with 23 participants. We found that BreathePulse, the first airflow-only breathing guide, effectively promoted slow breathing, particularly during the easy memory task. Participants' breathing rates aligned with BreathePulse's guidance across tasks, with the longest maintenance of slow breathing – over 40% of the time – during the easy task. Although BreathePulse increased workload and had little impact on stress, it promoted mindfulness, indicating its potential for stress management in the workplace.

Passive haptic learning (PHL) is a method for training motor skills via intensive repetition of haptic stimuli while a user is focused on other tasks. For the practical application of PHL to music education, we propose passive haptic rehearsal (PHR) where PHL is combined with deliberate active practice. We designed a piano teaching system that includes haptic gloves compatible with daily wear, a Casio keyboard with light-up keys, and an online learning portal that enables users to track performance, choose lessons, and connect with the gloves and keyboard. We conducted a longitudinal two-week study in the wild, where 36 participants with musical experience learned to play two piano songs with and without PHR. For 20 participants with complete and valid data, we found that PHR boosted the learning rate for the matching accuracy by 49.7% but did not have a significant effect on learning the notes' rhythm. Participants across all skill levels in the study require approximately two days less to reach mastery on the songs practiced when using PHR. We also confirmed that PHR boosts recall between active practice sessions. We hope that our results and system will enable the deployment of PHL beyond the laboratory.

Camera-based text entry using American Sign Language (ASL) fingerspelling has become more feasible due to recent advancements in recognition technology. However, there are numerous situations where camera-based text entry may not be ideal or acceptable. To address this, we present FingerSpeller, a solution that enables camera-free text entry using smart rings. FingerSpeller utilizes accelerometers embedded in five smart rings from TapStrap, a commercially available wearable keyboard, to track finger motion and recognize fingerspelling. A Hidden Markov Model (HMM) based backend with continuous Gaussian modeling facilitates accurate recognition as evaluated in a real-world deployment. In offline isolated word recognition experiments conducted on a 1,164-word dictionary, FingerSpeller achieves an average character accuracy of 91% and word accuracy of 87% across three participants. Furthermore, we demonstrate that the system can be downsized to only two rings while maintaining an accuracy level of approximately 90% compared to the original configuration. This reduction in form factor enhances user comfort and significantly improves the overall usability of the system.

Mouth-based interfaces are a promising new approach enabling silent, hands-free and eyes-free interaction with wearable devices. However, interfaces sensing mouth movements are traditionally custom-designed and placed near or within the mouth. TongueTap synchronizes multimodal EEG, PPG, IMU, eye tracking and head tracking data from two commercial headsets to facilitate tongue gesture recognition using only off-the-shelf devices on the upper face. We classified eight closed-mouth tongue gestures with 94% accuracy, offering an invisible and inaudible method for discreet control of head-worn devices. Moreover, we found that the IMU alone differentiates eight gestures with 80% accuracy and a subset of four gestures with 92% accuracy. We built a dataset of 48,000 gesture trials across 16 participants, allowing TongueTap to perform user-independent classification. Our findings suggest tongue gestures can be a viable interaction technique for VR/AR headsets and earables without requiring novel hardware.

Gaze tracking allows hands-free and voice-free interaction with computers, and has gained more use recently in virtual and augmented reality headsets. However, it traditionally uses dwell time for selection tasks, which suffers from the Midas Touch problem. Tongue gestures are subtle, accessible and can be sensed non-intrusively using an IMU at the back of the ear, PPG and EEG. We demonstrate a novel interaction method combining gaze tracking with tongue gestures for gaze-based selection faster than dwell time and multiple selection options. We showcase its usage as a point-and-click interface in three hands-free games and a musical instrument.

BCIs using imagined or executed movement enable subjects to communicate by performing gestures in sequential patterns. Conventional interaction methods have a one-to-one mapping between movements and commands but new methods such as BrainBraille have instead used a pseudo-binary encoding where multiple body parts are tensed simultaneously. However, non-invasive BCI modalities such as EEG and fNIRS have limited spatial specificity, and have difficulty distinguishing simultaneous movements. We propose a new method using transitions in gesture sequences to combinatorially increase possible commands without simultaneous movements. We demonstrate the efficacy of transitional gestures in a pilot fNIRS study where accuracy increased from 81% to 92% when distinguishing transitions of two movements instead of two movements independently. We calculate ITR for a potential transitional version of BrainBraille, where ITR would increase from 143bpm to 218bpm.

Head worn displays are often used in situations where users’ hands may be occupied or otherwise unusable due to permanent or situational movement impairments. Hands-free interaction methods like voice recognition and gaze tracking allow accessible interaction with reduced limitations for user ability and environment. Tongue gestures offer an alternative method of private, hands-free and accessible interaction. However, past tongue gesture interfaces come in intrusive or otherwise inconvenient form factors preventing their implementation in head worn displays. We present a multimodal tongue gesture interface using existing commercial headsets and sensors only located in the upper face. We consider design factors for choosing robust and usable tongue gestures, introduce eight gestures based on the criteria and discuss early work towards tongue gesture recognition with the system.

Passive haptic learning (PHL) is a method for learning piano pieces through repetition of haptic stimuli while a user is focused on other daily tasks. In combination with active practice techniques, this method is a powerful tool for users to learn piano pieces with less time spent in active practice while also reducing cognitive effort and increasing retention. We propose a demo combining these two learning methods, in which attendees will engage in a short active practice session followed by a passive practice session using vibrotactile haptic gloves. Attendees will be able to experience the effects of passive haptic learning for themselves as well as gauge their mastery of the piece with a final performance at the end of the demo.

Speech is inappropriate in many situations, limiting when voice control can be used. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. Using a dental retainer with capacitive touch sensors, SilentSpeller tracks tongue movement, enabling users to type by spelling words without voicing. SilentSpeller achieves an average 97% character accuracy in offline isolated word testing on a 1164-word dictionary. Walking has little effect on accuracy; average offline character accuracy was roughly equivalent on 107 phrases entered while walking (97.5%) or seated (96.5%). To demonstrate extensibility, the system was tested on 100 unseen words, leading to an average 94% accuracy. Live text entry speeds for seven participants averaged 37 words per minute at 87% accuracy. Comparing silent spelling to current practice suggests that SilentSpeller may be a viable alternative for silent mobile text entry.

Passive haptic learning (PHL) uses vibrotactile stimulation to train piano songs using repetition, even when the recipient of stimulation is focused on other tasks. However, many of the benefits of playing piano cannot be acquired without actively playing the instrument. In this position paper, we posit that passive haptic rehearsal, where active piano practice is assisted by separate sessions of passive stimulation, is of greater everyday use than solely PHL. We propose a study to examine the effects of passive haptic rehearsal for self-paced piano learners and consider how to incorporate passive rehearsal into everyday practice.

Voice control provides hands-free access to computing, but there are many situations where audible speech is not appropriate. Most unvoiced speech text entry systems can not be used while on-the-go due to movement artifacts. SilentSpeller enables mobile silent texting using a dental retainer with capacitive touch sensors to track tongue movement. Users type by spelling words without voicing. In offline isolated word testing on a 1164-word dictionary, SilentSpeller achieves an average 97% character accuracy. 97% offline accuracy is also achieved on phrases recorded while walking or seated. Live text entry achieves up to 53 words per minute and 90% accuracy, which is competitive with expert text entry on mini-QWERTY keyboards without encumbering the hands.