Input Device for Disabled - Sense Mouth Open

John,

Welcome to element14.

Your question reminded me of few medical innovations.

1. One device that reads lips, "The speechless speak again"

2. The medical tattoo that can sense muscle and electrical impulse changes, "Tattoo electronics and possibility"

3. An input for those who can not move at all, "The paralyzed walk, virtually, with a brain-computer interface"

Since these may cost a little more than the average person will want to pay, I would start by considering the Tattoo option or some home made variant. Combine this with voice recognition, you may have something.

Alternatively, you could try looking into Microsoft Kinect for computer control. Uses visual tracking and voice already. "Control the web and Windows 7 with Kinect."

Cabe

Cabe,

Thanks very much for taking the time to respond!

The Read Lips device seems like it would  provide quite a varied array of inputs however it looks at this stage of development to be more invasive than what Liz, our daughter might be able to tolerate.

The Tattoo is quite on the money.  A low power (only a meter or two distance between transmitter/receiver) Bluetooth wireless version, with lithium powered substrate layer and a thin film strain gauge might detect accurately the skin movements on her face when she opens her mouth.  I graduated from the University of Illinois with a degree in Engineering way back in 1968.  Followup in this area looks promising for Liz.  Liz is cognitively OK.  She reads, has excellent memory storage and retrieval and is currently motivated to improve her expressive language.

The third item, we have tried similar BCI Type P300 devices with no success.  Getting the electrodes positioned correctly is difficult with a normal person, but with Liz almost impossible because she has a lot of involuntary head movement.  The signal to noise ratio is poor unless you find a hot spot.  Implanted electrodes are often much more effective but require surgery and can become infected.

Thanks again so much for the suggestions.

I think the tattoo option, if it becomes more accurate, is the best for your case.

The importance of computer connection for Liz is entirely up to you. If you want to try the internal EEG option there have been some advancements. Though, none of which seem at all useful at the moment.

A sensor mesh placed on the brain is allowing people to move cursors already, "Direct contact brain sensors move cursor"

In full brain contact is not an option, a company is looking into a EEG contact that will not get infected. "Bolt in the head reads neural signals"

I still think a combination of facial tracking, voice, simpler touch inputs would be the easiest to implement.

What are Liz's abilities at the moment? You mentioned jaw control. Does she have complete control? If so, track the position of the jaw in 3D space. For instance, move your jaw forward, now to the side. You could use that to move the cursor to the upper right.

Eye tracking a possibility?

Any other extremity have better control?

Breathing? (breathing could suggest direction or other inputs)

Cabe

Liz is severely involved in the motor area.  She is non verbal but can make a variety of sounds. She has great difficulty initiating a controlled motor movement in response to a visual or auditory input.  The harder she focuses, the more her body freezes up.  About 25 years ago I wrote a single switch scanning program for her to communicate with educational staff as a prerequisite for attending main stream classes.  They were unable to access her cognitively without a measurement tool.  This required having Liz evaluated by therapists who observed her various hand, arm, leg, etc. movements.  We discovered she could reliably open her mouth and stick her tongue out.  This gave rise to her "yes" and "No" response to questions being structured to a binary answer.  The mouth open became the selection motor response in the single switch scanning system.  If she miss fired during scanning she would thrust her tongue to alert the aid of her motor miss.   Liz is trapped in her body and at times as she gets older becomes very frustrated.  Combining multiple motor movements for Liz would be very stressful and perhaps beyond her physical capability.

We have tried the eye gaze tracking system but it requires limited head movement.  Her jaw/mouth  movements have been conditioned over many years and at this time are the direction we feel has the best chance for success.  I built a voice/sound system years ago for her and also developed a pattern recognition program using Fourier transforms to determine different facial expressions.  The microphone system was somewhat successful, but the imagining system was defeated with her constant head movements.

My approach today is to allow Liz the freedom to see or hear her possible choices and respond with a sensor attached to her body/head wirelessly.  This allows her to make whatever random movements she must to respond correctly.  Hence my interest in the thin film technology on a polymer as a possible course.  It could be portable and noninvasive (no bolts in the head).

The ideal augmentative communication system for Liz would collect brain signals that are her thoughts and convert them to language.  Man/machine interfaces are improving ever so slowly in this area.  We hope the gap will shrink.  We never give up!

Thanks again!

John

Upgrading your previous jaw/mouth button and tongue sensor with modern fabricating methods is a great stopgap solution until something better comes along.

How do you envision the sensor working? How do you see it attach?

Here are my thoughts: I would start by trying to build the device into the mold of a dental guard. Like what some people wear a night to avoid grinding of the teeth.

What I would like to try is molding all the electronics needed into the guard itself. A handfull of buttons molded inside the guard itself, along with a microcontroller, wireless transmitter, battery, and a coil to use for induction charging.

All these parts are off the shelf, near plug and play.

The uC immediately sends the data (which button) on every press. Simple as that.

If you are willing to give it a try, I (and hopefully the community) will help in this effort.

Cabe

I'm sorry for the late reply.

If you have already accepted accessories (chin strap, cap), would a pair of accelerometers work? The differential signal would indicate an opening or closing of the mouth. A nod of the head would be ignored without a difference. Some filtering would be required.

Both signals could be fed into a low power microcontroller and a low power radio, running from a coin battery or some small rechargeable.

The reason I mention this option is to allow you a choice of avoiding a long-term (even if only used sometimes) contact with any kind of electrode. There are not many materials, particularly conductive ones, that you can leave in long term contact with the skin without some kind of adverse affect.

Tim

Have started building a device and I am finding the wireless transmitter, battery, and a coil to use for induction charging are all too physically large. Is there a package that is the size of a microcircuit or smaller?  I am looking for wireless communication. I don't need a battery.  Just need to interrogate the sensor(s) to get the data.  What are the options with battery and no battery?

Thanks much

John

this is not in Any Way meant as an Answer to your/her challenge, just a few ideas.

I was thinking perhaps something along the lines of an RFID type stick-on tag that would require no power but would activate a Theramin type of receiver, also can she control her breathing? perhaps a small blow-pipe to activate or puncuate or even comunicate using?

I`m also thinking something along the lines of a Kinect type of non intrusive/intimate contact reader as well, I have no idea How one would code this, and all would certainly need discriminator type filtering for random movements.

all the best and Gods blessings to you both

Message was edited by: YT 2095 to add: can you say a little bit more about the Type of modulation of her communication in these movements, does she have Fine controllable skills on top of Large movements or is it Large (Macro) Movements overlaying finer random movements, this May help us elicit cogent data/communications from her and sensor types.

As human body has higher temperature inside, perhaps it is possible to detect mouth opening remotely using thermal camera or array of thermopiles?

Using Kinect I think is also promising. There is a "Kinect Accelerator" program with technical help from MS, but for companies only:

http://www.element14.com/community/community/news/blog/2011/11/30/microsoft-kinect-contest--and-the-3d-topographical-display