After reading about the DeeChee language acquisition experiment robot, I talked to Dr. Caroline Lyon by e-mail about how the details of how the robot and experiment worked.
There are many difficulties associated with a acquiring language, especially for a machine. For example, of the 50 most common words in English, 20 are homophones. Homophones are common in many languages. There may be an evolutionary reason for human languages to develop homophones, but machines would certainly do better with one word corresponding to one meaning.
Dr. Lyon’s research focuses on a particular issue of machine language acquisition: forming words by connecting vowels and consonants. The research is not interested in word meaning, only learning to speak the words. How the robot learns to speak the words could shed light on this one aspect of human language development.
The DeeChee robot works by running a Linguistically Enabled Synthetic Agent (LESA) program to generate babble. The babble consists of English language sounds spoken randomly with the same frequency with which they occur in English speech. A human who is not knowledgeable about how the software works is instructed to teach the robot a list of words. Eventually by chance the robot will say part of a word on the list. The human "teacher" is instructed to repeat the word and praise the robot when this happens. The human audio is put through Microsoft SAPI 5.4 speech recognizer modified to output a stream of language sounds instead of words. When the software detects a multi-syllabic utterance and a reinforcement phrase, it adds the pattern of sounds to its lexicon. The software makes no attempt to work out if this is part of a word or a phrase consisting of more than one word.
The DeeChee robot tracks the humans’ movements and follows them with its eyes. This gives the illusion that the robot is looking at the objects and learning their names. Despite the humans’ unawareness of how the software works, DeeChee is able to acquire words after only minutes of teaching. This is surprising because there are many roadblocks to success:
- The modified Microsoft phonetic transcription software often misses sounds. The software in its unmodified form would be able to work out words based on context without hearing every sound, similar to human speech recognition. The software used in these experiments sometimes detected about half of the raw speech sounds correctly.
- The human subjects were not very good at identifying words among LESA’s babbling. This was surprising to me considering humans’ tendency for pareidolia, the phenomenon which causes us to see significant images and random forms. It requires great concentration, though, for a human to listen for any of a list of words in a stream of random babble.
- In early experiments, the microphone picked up the robot’s own babble and treated it as if it were reinforcement from a human teacher. Setting up facial expressions to indicate when it was the humans’ turn to talk helped but turned out to be a distraction from the task of identifying salient words among the babble.
- Long sentences spoken by the teacher such as, “Do you remember the smile shape? It’s like your mouth.”, contain many non-salient words. This results in slower learning. In an early experiment, a human “teacher” knowledgeable about computers but not this particular software did better than someone knowledgeable about teaching children.
- Some humans were better “teachers” than others. Speaking too few words or being to generous with reinforcement phrases resulted in slower learning and mistakenly learning non-words.
The amazing part of the research is how well language acquisition occurs naturally despite these roadblocks. An example of ancient investigation in this area was a cruel experiment in 450 BCE in which two infants were locked up along together and cared for but not exposed to language. According to the ancient Greek historian Herodotus, the children independently developed the word “bekos” for food. Some elements of language appear to develop easily.
Another interesting aspect to this research is the wide interest in it. The robot appears to be right on the edge of the uncanny valley. For me it’s still in the cute zone, but many commenters find it “creepy”. Objects right on the boundary of the uncanny valley draw our attention.
DeeChee only babbles randomly and repeats sounds it hears. If computers continue developing at the rate they have been, we may see similar robots with cameras that can associate words with images. Within decades we may have machines doing this, behaving like self-aware humans but with different sets of abilities. They may be able to work math quickly but struggle to identify faces. Imagine an aspie engineer and take him a step further. DeeChee is nothing like this, but its child-like anthropomorphic behavior catches our attention because we know before long we may be working with machines with traits and capabilities formerly reserved for human beings.
For Further Reading
For the latest of Dr. Lyon’s research with DeeChee:
Lyon C, Nehaniv CL, Saunders J (2012)Interactive Language Learning by Robots: The Transition from Babbling to Word Forms.PLoS ONE 7(6):e38236.doi:10.1371/journal.pone.0038236
For details on how the software and hardware work:
Rothwell A, Lyon C, Nehaniv CL, Saunders J (2011) From babbling towards first words: the emergence of speech in a robot in real-time interaction. In: IEEE Symposium on Artificial Life (IEEE Alife 2010). pp. 86-91.
On the general difficulties of acquiring language:
Lyon C, Sato Y, Saunders J, Nehaniv CL (2009) What is needed for a robot to acquire grammar? Some underlying primitive mechanisms for the synthesis of linguistic ability. IEEE Transactions on Autonomous Mental Development 1 (3): 187-195.
On Dr. Lyon’s early research before introducing reinforcement phrases for praise:
Lyon C, Nehaniv CL, Saunders J (2010) Preparing to Talk: Interaction between a Linguistically Enabled Agent and a Human Teacher. In: AAAI Fall Symposium Series: Dialog with Robots. AAAI Press, Dialog with Robots, FS-10-05, pp. 56-61.
