Moles in your living room: using augmented reality in neurorehabilitation

Melvyn focuses on augmented reality (AR). “This is different from virtual reality (VR), where you put on the glasses and no longer see your surroundings,” he says. “AR also works with glasses, but you see the world as it really is and enrich it with digital content. We use AR for physiotherapy exercises. Often, people don't do these exercises when the therapist prescribes them, but because we've turned them into a game, it becomes fun to do them. With the glasses, for example, people see moles coming out of the ground in their living room, which they have to push back into the ground with a squat. The therapist prescribes these exercises this remotely, freeing up time to provide one-on-one guidance to patients with more complex conditions. For people with Parkinson's and other progressive diseases, it is also very important to remain physically active. These games help with that.”

During his PhD research, Melvyn investigated how people synchronise movement with auditory rhythms, for example by tapping to the beat of a metronome. “I always had trouble explaining to my grandmother why it was important to research this,” he says. “I personally felt it was important not only to gather knowledge, but also to look at what you can do with that knowledge in practical terms. That tapping research ultimately led to the very first AR rehabilitation treadmill. This allows patients who have had a stroke to learn to walk symmetrically again. Later, we added a projector that could display a virtual walking pattern or obstacles. And that is now a standard rehabilitation product worldwide. Every now and then, we’d receive an email from someone who had practised on it and said that it had helped them a lot. That gave me a lot of fulfilment. The social impact and practical value are very important to me, that's why I do this.”

Text continues underneath the photo.

Photo: an augmented reality mole.

As a senior researcher, Melvyn currently leads a team of four PhD candidates within the NavigAIt project, part of an EIC Accelerator grant. “NavigAit is an AR application we are developing as a walking aid for people with Parkinson's,” he explains. “One in three patients develops freezing of gait: you want to walk, but your feet feel glued to the ground. You can no longer take a step, which often leads to falling. We have developed AR cues for this. These are, for example, bars or lines that people see on the floor through the AR glasses. Patients can adjust their footsteps in a targeted manner using these cues to prevent or escape freezing.”

With these cues, NavigAit can also map out a safe walking path. “The application understands the environment and the movements of the wearer,” explains Melvyn. “NavigAit knows exactly where the floor is, but also where obstacles are, such as a threshold or table. It then uses cues to create a path that goes around the table, rather than through it. The application also takes into account the walking speed and stride length of the wearer. So there's a lot of data science behind it, but we also work together with patients to see how they want to see and activate the cues.”

Melvyn is currently investigating whether AR exercises for home use can also be used for fall prevention in older people. “This means you don't have to work one-on-one or in a group with a physiotherapist, but can do a lot independently at home,” he explains. “Innovations like this are needed to solve the healthcare crisis. We can no longer provide care as we have done for years because there are too few people to provide that care. It really needs to be supported by technology and take place more at home. Many older people notice that they are becoming more unsteady and want to do something about it themselves, so it's more preventive than curative. That's also an important transition that technology can really help with.”

Want to know more about Melvyn's research? During the NUTRIM symposium 'Bits & Bites' on 19 November 2025, he will talk about AR and how it can be used effectively in the healthcare landscape.

Text: Joëlle van Wissen

Did you enjoy reading this article? Follow us on Instagram and LinkedIn for more.