Input Device for Disabled - Sense Mouth Open

John,

Welcome to element14.

Your question reminded me of few medical innovations.

1. One device that reads lips, "The speechless speak again"

2. The medical tattoo that can sense muscle and electrical impulse changes, "Tattoo electronics and possibility"

3. An input for those who can not move at all, "The paralyzed walk, virtually, with a brain-computer interface"

Since these may cost a little more than the average person will want to pay, I would start by considering the Tattoo option or some home made variant. Combine this with voice recognition, you may have something.

Alternatively, you could try looking into Microsoft Kinect for computer control. Uses visual tracking and voice already. "Control the web and Windows 7 with Kinect."

Cabe

Cabe,

Thanks very much for taking the time to respond!

The Read Lips device seems like it would  provide quite a varied array of inputs however it looks at this stage of development to be more invasive than what Liz, our daughter might be able to tolerate.

The Tattoo is quite on the money.  A low power (only a meter or two distance between transmitter/receiver) Bluetooth wireless version, with lithium powered substrate layer and a thin film strain gauge might detect accurately the skin movements on her face when she opens her mouth.  I graduated from the University of Illinois with a degree in Engineering way back in 1968.  Followup in this area looks promising for Liz.  Liz is cognitively OK.  She reads, has excellent memory storage and retrieval and is currently motivated to improve her expressive language.

The third item, we have tried similar BCI Type P300 devices with no success.  Getting the electrodes positioned correctly is difficult with a normal person, but with Liz almost impossible because she has a lot of involuntary head movement.  The signal to noise ratio is poor unless you find a hot spot.  Implanted electrodes are often much more effective but require surgery and can become infected.

Thanks again so much for the suggestions.

I think the tattoo option, if it becomes more accurate, is the best for your case.

The importance of computer connection for Liz is entirely up to you. If you want to try the internal EEG option there have been some advancements. Though, none of which seem at all useful at the moment.

A sensor mesh placed on the brain is allowing people to move cursors already, "Direct contact brain sensors move cursor"

In full brain contact is not an option, a company is looking into a EEG contact that will not get infected. "Bolt in the head reads neural signals"

I still think a combination of facial tracking, voice, simpler touch inputs would be the easiest to implement.

What are Liz's abilities at the moment? You mentioned jaw control. Does she have complete control? If so, track the position of the jaw in 3D space. For instance, move your jaw forward, now to the side. You could use that to move the cursor to the upper right.

Eye tracking a possibility?

Any other extremity have better control?

Breathing? (breathing could suggest direction or other inputs)

Cabe

Liz is severely involved in the motor area.  She is non verbal but can make a variety of sounds. She has great difficulty initiating a controlled motor movement in response to a visual or auditory input.  The harder she focuses, the more her body freezes up.  About 25 years ago I wrote a single switch scanning program for her to communicate with educational staff as a prerequisite for attending main stream classes.  They were unable to access her cognitively without a measurement tool.  This required having Liz evaluated by therapists who observed her various hand, arm, leg, etc. movements.  We discovered she could reliably open her mouth and stick her tongue out.  This gave rise to her "yes" and "No" response to questions being structured to a binary answer.  The mouth open became the selection motor response in the single switch scanning system.  If she miss fired during scanning she would thrust her tongue to alert the aid of her motor miss.   Liz is trapped in her body and at times as she gets older becomes very frustrated.  Combining multiple motor movements for Liz would be very stressful and perhaps beyond her physical capability.

We have tried the eye gaze tracking system but it requires limited head movement.  Her jaw/mouth  movements have been conditioned over many years and at this time are the direction we feel has the best chance for success.  I built a voice/sound system years ago for her and also developed a pattern recognition program using Fourier transforms to determine different facial expressions.  The microphone system was somewhat successful, but the imagining system was defeated with her constant head movements.

My approach today is to allow Liz the freedom to see or hear her possible choices and respond with a sensor attached to her body/head wirelessly.  This allows her to make whatever random movements she must to respond correctly.  Hence my interest in the thin film technology on a polymer as a possible course.  It could be portable and noninvasive (no bolts in the head).

The ideal augmentative communication system for Liz would collect brain signals that are her thoughts and convert them to language.  Man/machine interfaces are improving ever so slowly in this area.  We hope the gap will shrink.  We never give up!

Thanks again!

John

Liz is severely involved in the motor area.  She is non verbal but can make a variety of sounds. She has great difficulty initiating a controlled motor movement in response to a visual or auditory input.  The harder she focuses, the more her body freezes up.  About 25 years ago I wrote a single switch scanning program for her to communicate with educational staff as a prerequisite for attending main stream classes.  They were unable to access her cognitively without a measurement tool.  This required having Liz evaluated by therapists who observed her various hand, arm, leg, etc. movements.  We discovered she could reliably open her mouth and stick her tongue out.  This gave rise to her "yes" and "No" response to questions being structured to a binary answer.  The mouth open became the selection motor response in the single switch scanning system.  If she miss fired during scanning she would thrust her tongue to alert the aid of her motor miss.   Liz is trapped in her body and at times as she gets older becomes very frustrated.  Combining multiple motor movements for Liz would be very stressful and perhaps beyond her physical capability.

We have tried the eye gaze tracking system but it requires limited head movement.  Her jaw/mouth  movements have been conditioned over many years and at this time are the direction we feel has the best chance for success.  I built a voice/sound system years ago for her and also developed a pattern recognition program using Fourier transforms to determine different facial expressions.  The microphone system was somewhat successful, but the imagining system was defeated with her constant head movements.

My approach today is to allow Liz the freedom to see or hear her possible choices and respond with a sensor attached to her body/head wirelessly.  This allows her to make whatever random movements she must to respond correctly.  Hence my interest in the thin film technology on a polymer as a possible course.  It could be portable and noninvasive (no bolts in the head).

The ideal augmentative communication system for Liz would collect brain signals that are her thoughts and convert them to language.  Man/machine interfaces are improving ever so slowly in this area.  We hope the gap will shrink.  We never give up!

Thanks again!

John

Upgrading your previous jaw/mouth button and tongue sensor with modern fabricating methods is a great stopgap solution until something better comes along.

How do you envision the sensor working? How do you see it attach?

Here are my thoughts: I would start by trying to build the device into the mold of a dental guard. Like what some people wear a night to avoid grinding of the teeth.

What I would like to try is molding all the electronics needed into the guard itself. A handfull of buttons molded inside the guard itself, along with a microcontroller, wireless transmitter, battery, and a coil to use for induction charging.

All these parts are off the shelf, near plug and play.

The uC immediately sends the data (which button) on every press. Simple as that.

If you are willing to give it a try, I (and hopefully the community) will help in this effort.

Cabe

Have started building a device and I am finding the wireless transmitter, battery, and a coil to use for induction charging are all too physically large. Is there a package that is the size of a microcircuit or smaller?  I am looking for wireless communication. I don't need a battery.  Just need to interrogate the sensor(s) to get the data.  What are the options with battery and no battery?

Thanks much

John

As human body has higher temperature inside, perhaps it is possible to detect mouth opening remotely using thermal camera or array of thermopiles?

Using Kinect I think is also promising. There is a "Kinect Accelerator" program with technical help from MS, but for companies only:

http://www.element14.com/community/community/news/blog/2011/11/30/microsoft-kinect-contest--and-the-3d-topographical-display

I am trying to sense directly on Liz's body.  Camera not as good.

John,

Aside from being great for mobile identification, RFID for a power source from sensors and microcontrollers (uC) is still on the fringe of research. If you can break that field, by all means.

If you are having issues, then external battery power may be the only solution.

Place the battery into the patchwork. With low power components, you might be able to do a lot.

Many of the links I would like to show you are down on element14. What I would show is just examples of how doing what you propose is possible using standard materials and tech.

A recent development might not show jaw movement, but it can accomplish everything else you want. See the pajama that is a complete baby monitoring system. This could be attached to the body and give you all the data you mentioned.

However, only sensing jaw movement will make the patch simple.

Something has to sense particular jaw muscle signal firing. From here data could be sent via wires to a body harness, like the pajama system above. You would have the best of all worlds, an off-the-shelf monitoring system with your cheek patch communication system.

To go wireless, you will have to integrate a transmitter and battery system in the patch. Finding an off-the-shelf  product to do all this is going to be impossible. I suggest we start by finding the lowest power transmitter possible and a small form factor uC to pair with it.

Cabe