Context, context, context

For its exact meaning, A word depends on the context in which it is used. This is so for ambiguous words, words with 2 or more different meanings. Examples for English are ‘bank’ (river bank, money bank), ‘break’ (coffee break, to break something into pieces), etc.

But even words that have but a single meaning depend on the specifics of that meaning on context. People who use sign language, know this very well. ‘Small’ in ‘a small elephant’ is a different kind of small, than in ‘a small ladybug’. ‘My dog’ can be a miniature dog or a big great Dane.

Context is even more important for lipreaders, because so many words look alike, or are only partially visible. For lipreaders, most of the words they see are ‘fuzzy’ patterns: patterns that do not have a direct link to one single word or meaning in their brain.

In lipreading, you will get some of the information from the speaker’s mouth, but the pattern that you see may activate 2 - 6 words in your brain, or maybe more. There rarely is a clear 1-1 link to a single word. So you have to use context, to select the correct alternative. At first, this may require conscious attention: Did he say ‘ball’? But that doesn’t make sense, so it must have been ‘mall’!

Later, you will probably be able to choose the correct alternative automatically, because you will not be lipreading word-by-word anymore, but in larger ‘chunks’. The words before and after will help your brains decide what the fuzzy bit in the middle was.

But there’s more than 1 context! There are (at least!) four:

• Sentence context. "’m tired, I’m going ….". "I want a pizza with ....."  If you get the first words of the sentence right, you need very little information from the lips to get the last words too. Sentence context helps – if you know the language, and if the speaker speaks grammatically correct English!
• Topic context. In a conversation about a football match, you can expect football words. In a talk at a bus-stop in England, expect weather words. “Third rainy day in a row, isn’t it” Topic context is partly determined by the situation: shop, bus-stop, office, pub. Especially with strangers, we use fixed phrases and ‘routines’ in standard situation. Any real conversation of course is less dependent on context: with friends in a pub, you may discuss work, relationships, health, football, your new car, or plans for the weekend. Once you know the topic of the conversation, this will help you recognize the words. It’s one of the reasons why lipreaders don’t appreciate speakers who jump from one topic to the next. And why it’s always good to ask a speaker what he or she is talking about: “I missed that, are you still talking about Sunday’s match?”
  It's also why it's so difficult to lipread a speaker who talks about a topic that you know little about. 

 Two trickier kinds of context:

• The speaker’s intentions. This is where mindreading helps. What is he or she trying to tell you, and why? Is it a joke? A warning? Instructions? A question? Body language and facial expression may give you some cues. The better you know the speaker, the easier it will be to guess what’s on someone’s mind, and what you can expect him or her to talk about.
• Your own intentions and expectations. Included in these are all of the 3 contexts above, as well as your emotions, personality, life experience and whatever else you have stored in your brain. If you expect someone to speak English, you will not be able to lipread a speaker of German. If you expect someone to talk about the weather, you may not recognize a question about  bus-times. If you expect someone to talk about football, you may miss a message about a missing pet. If you consider yourself a bad lipreader, you may not be able to get a single word no matter how predictable it is. If you are afraid to fail, you may always use your Mona Lisa smile and respond, let alone contradict someone, out of fear that you mis-read what the person said. Or: you will dominate each and every conversation because when you speak, you don’t have to listen!

Context really is a double-edged sword. It can help, and it can kill. No, not persons, but definitely conversations!

This does explain why good lipreaders are such very good listeners: they may not actually hear what the other person is saying, but they will read between the lines, they will get the underlying message. Whereas many persons with good hearing may hear all the words, but will interpret them in the context of their own expectations: they only hear what they want to, or expect to hear.

So yes, always use context. And be aware that it is is a double-edged sword.

Lipreading at work

Deborah Tannen, author of “You just don’t understand, Women and Men in Conversation” (1991) about mis-communication between men and women, also wrote “Talking from 9 to 5. How women’s and men’s conversational styles affect who gets heard, who gets credit, and what gets done at work” (1994). The book describes the different communication styles of men versus women, in working environments. Although her data were mostly collected in offices in the USA, some of her observations may have a wider validity and may explain some of the problems that lipreaders and people with hearing problems in general, experience in work situations.

Tannen’s main message is that conversation is a ritual, with  unwritten rules and subconscious expectations. When someone does not play according to these unwritten rules, we experience cognitive dissonance.  My word, not Tannen’s. What happens is not what you expect to happen. So you have to do some thinking. Were your expectations wrong, is the other  person wrong, or is there something else that can explain what just happened? Very often, our response is emotional:  “This is not fair!” Or, in Tannen’s terms who describes conversations at work mostly in game-like terms: it’s a one-down. Usually, feelings are hurt. The result: consciously or subconsciously the person  affected will try to get even, and/or will try to avoid further conversations with the person who caused the cognitive dissonance.

This happens between men and women at work. Different rules, different expectations, resulting in different subgroups. Different subgroups, each with their own conversational rituals.

This happens in meetings and in employee-employer talks. But as Tannen writes “On the job, the meat of the work that has to be done is held together, made pleasant and possible, by the ketchup, relish, and bun of conversational rituals.” (page 43)

“Talk at work is not confined to talk about work. Many moments are spent in casual chat that establishes the friendly working environment that is the necessary backdrop to getting work done. (..) Both women and men know that their small talk is just that – “small” compared to the “big” talk about work – but differences in small-talk habits can become very big when they get in the way of the easy day-to-day working relationships that make us feel comfortable at work and keep the lines of communication open for the big topics when they arise.” (page 64)

And that’s exactly where lipreaders, and people with hearing problems in general, will get in trouble. They may manage in 1-1 "big-talk", factual communication about work, when context, speaker, and conversational rituals are familiar and fairly predictable. They get the hamburger, but not  the ketchup, relish or bun. Colleagues meet in groups, near the coffee machine, smoking outside, in the cafeteria. The lipreader who tries to blend in there notices too late who is speaking, doesn’t know the context, misses the clue of the joke and laughs too late. Or: asks someone to repeat what was said. Or: responds with a remark that doesn’t fit the ritual. Or: tries to monopolize the conversation, because when you talk, you don’t have to listen. In all cases: rituals broken, cognitive dissonance, hurt feelings.

The lipreader may blame the others: they forget that I’m hard-of-hearing, they never take my needs into account, why can’t they speak one at a time, take their cigarettes out of their mouths, turn down the background music. Or: depending on the lipreader’s personality and/or frame of mind at the time, the lipreaders blames him-/herself: Stupid me! Next time, I will stay at my desk. Or at home.

One of the problems is that these rituals are automatic, subconscious. The other person may remember that the lipreader has ‘special needs’ for a week, or a day, or 2 minutes, but – especially in conversational speech – will quickly go back on automatic pilot, and forget to speak clearly.

Lipreaders shouldn’t be surprised. It’s what they do themselves. You’d expect lipreaders to be very good lipspeakers. Because they know first-hand how important it is to speak clearly, one at a time, in well organized messages. Yes, dear lipreader, you may think that that is what you do. But please ask another lipreader for more objective feedback

And yes, people with hearing problems are partly to blame, themselves. They hide their hearing aids, they get upset when someone speaks very – slowly – or – VERY LOUDLY especially for them, because they so desperately want to be seen as ‘normal’. Whatever that is.

Men AND women, hearing AND not hearing: don’t let your emotions take over when someone breaks a ritual! Cognitive dissonance is good! It makes  you – and the other person – switch off the automatic pilot, it makes you a more aware participant. Real-life differences feed your brain!

PS: Small-talk solutions?

• Befriend your colleagues on Facebook and regularly check their pages;
•  Become the editor of a weekly or monthly company newsletter or company bulletin board and ask everyone to mail you their small and big news;
• Find a buddy who will keep you updated;
• Decide - and explain to your colleagues – that, because of your hearing problems and/or personality, you don’t do small talk. Several deaf people who were finally able to hear "small-talk" after receiving a Cochlear Implant report their disappointment: "Is that all hearing people talk about?"  Yes, it is. Small-talk is mostly a feel-good ritual. It's not hamburger, it's ketchup. 

Lipreading Myth 3: only 30% ?

Almost all publications about lipreading say that lipreaders 'get' only 30% of the information that hearing people do. Lipreaders have to guess the missing 70%, which makes lipreading so very difficult and demanding. 

The bad news: yes, lipreading is very difficult and demanding.

The good news: the 30% rule may be true in (some) experimental conditions, but has no predictive value for real life.

Where does the 30% come from? 

Spoken English uses approximately 44 different phonemes (http://en.wikipedia.org/wiki/Phonemes_of_English). When you present these in isolation or in short (meaningless or meaningful) words without sound, subjects can discriminate appr. 12 different 'visemes' = phoneme groups.

For the lipreader, some phonemes are indistinguishable. The usual example is 'b - p - m': 3 different phonemes that look exactly the same to the lipreader. Words like 'pat', 'bat', 'mat' cannot be distinguished by lipreading.

Other phonemes invisible and cannot be seen at all: 'cate', 'gate', 'hate' all look like 'ate'. 

Depending on the research conditions, the speaker, and the subjects, a lipreader of English can see 5 different vowel groups, and 7 different consonant groups. These groups are sometimes called 'visemes'. English has 44 different auditory distinguishable phonemes, and only appr. 12 visually distinguishable visemes (but numbers and groupings vary, depending on the methods, speakers, etc.). So: lipreaders 'do it' with 30% of the information. 

But!

In everyday life, people do not speak in single, one syllable words. Well OK, some people do: 'Hi', 'yes', 'nope', 'wow', 'bye'.

But not without context! Someone is leaving and says 'bye'. The lipreader sees something that could be 'pie', 'my', or 'bye'. Does he or she have to guess 3 times, before s/he gets the right answer? Eh... no? 

Plus

• Longer words are easier to lipread than one syllbale words, because longer words have fewer 'look-a-likes'. Without context, it will be difficult to guess 'bye' correctly. 'See you later' (not one word? OK, not in writing, but in spoken language it looks like one word!) has fewer look-a-likes and is easy to recognize, especially in the context of someone leaving. Invisible or indistinguishable phonemes can be identified 100% correct in the context of a meaningful word. Context helps! 
• Some speakers articulate so clearly, that you CAN see the difference between 'P' and 'B'. Other speakers on the other hand, barely open their mouths. All you see is 'open - closed - open - closed'. With a good 'lipspeakers', a lipreader may get 100% of the information. A bad lipspeakers: 0% + a lot of frustration! 
• You don't need all the phonemes, to get all the information! Even hearing people don't hear all phonemes correctly, their brains automatically fill in the gaps. Texters don't need all letters to understand each other, they can use abbeviations like FBOW (for better or worse), ATB (all the best), and many others. Texters get only ... 20% of the information that old fashioned writers get? No. They get maybe 20% of the letters, but experienced texters can get 100% of the information! 
• In the context of a sentence, even invisible ..... can be guessed! Of course, to be able to use context, you need foreknowledge of the language, the speaker and the topic...  

The moral of the story:

Yes, lipreading is difficult. Yes, lipreaders have to do it with information that has many more 'gaps', than hearing people do. But foreknowledge and context can help you fill in many of the gaps!

And no, lipreaders should not be afraid to make mistakes. Few speakers are 100% predictable. And those who are, are often boring... 

Lipreading Myth 2: 93% of all communication is nonverbal

You may have heard (or said yourself!) that research has shown that communication ( getting a message across) depends mainly on nonverbal cues or body language. 

To be exact: that 93% of a message is transmitted by means of nonverbal cues or body language, and that only 7% depends on spoken words. 

Not true of course, because this would make lipreaders much, much better communicators than blind people; it would make telephone and radio pretty much as useless for hearing people as is it is for hard-of-hearing and deaf people. And it would take all the fun out of whispering in the dark.

And what about reading: no non-verbal cues whatsoever! If you believe that 93% of communication is non-verbal, you may as well stop reading now.


The research quoted – actually: mis-quoted - in the statements about the importance of nonverbal communication, was done by Albert Mehrabian in the 1960’s. Only two studies, with a number of limitations.

Most importantly: Mehrabian wasn’t looking at ‘communication’ in general, he was looking at inconsistent communication of attitudes. Basically: if someone says a positive word (for instance: ‘dear’) with a negative tone of voice, what do you believe? The word, or the tone of voice? The word, or a picture of a facial expression? 

In that specific experimental context – a tape recording of a single word spoken by an unknown speaker, body language shown as photographs of facial expressions and  female subjects, tested in a laboratory setting - most subjects based their judgement (like – dislike) on the actual word in 7% of the cases, on tone of voice in 38% of the cases, and on facial expression in 55% of the cases. 38 + 55 = 93. So: nonverbal wins in 93% of the cases!

Yes, true, if you’re marketing nonverbal. But for the rest of us, communicating fairly consistently and not just about attitudes: the message really, truly is in the words!

Of course, nonverbal cues can help. IF the nonverbal cues are consistent with the message. Which they will usually be, when a person speaks the truth. Even then, you have to take into account what are sometimes (USA only?) called the 3 C’s of nonverbal communication:

• Context: when you interpret a nonverbal cue, you have to take the context (situation, topic) into account. Shivering may mean that someone is cold. Or scared. Or antsy. Looking away may mean that someone is bored. Or uncomfortable. Or: that he or she heard a door open, or a phone ringing.
• Clusters: you have to look at groups (clusters) of nonverbal cues. A single nonverbal cue in isolation can mean many different things. So actually, this is context too: interpret nonverbal cues in the context of other nonverbal cues. Someone is shivering? Any other cues that the person is cold, or scared, or antsy? A person looks away: any other cues that he or she is bored, or uncomfortable, or heard a sound that  you missed?
• Congruence: nonverbal cues have to be consistent with each other AND with the verbal message. So basically, that’s context too.  If the person says that he or she is cold, or scared or antsy AND shivers, AND you are pretty sure the person is not lying, trust the nonverbal cues! 

If you are now thinking that 'reading' nonverbal cues resembles lipreading: it does! The lipreader also has to take into account the 3 C's: context, clusters, and congruence. Actually, these 3 C's are important for ALL communication. 

Mehrabian’s female subjects in the ‘60s mostly believed the nonverbal cues, but they didn’t have the 3 C’s: no context, no clusters, and no congruence. And they may well have guessed wrong. 

In real life: yes, use nonverbal cues to help you understand a speaker. But when cues are inconsistent or don’t match: don’t try and guess.  Ask. 

If you're not convinced, watch this video from YouTube. First without sound or subtitles, then with. Words carry more than 7% of the message, don't they? 

Video: Busting the Mehrabian myth

Lipreading Myth 1: Lipreaders can do it

A number of times a year I receive e-mails asking me for the name of a lipreading expert. In one case, someone had recorded video at a wedding, all guests had said something .... yes, and that was the problem: what did the guests say? The sound was very bad, and no-one could hear what the guests said. So can I please caption the video for them?

No.

There's a YouTube video of the new pope meeting the Belgian king and queen. Some Royalty TV programme mailed me: can I please tell them what the Belgian king said to the pope? There was no sound, they weren't even sure what language was being spoken. Please send us the text before tonight's progamme.

Sorry, can't do.

Someone involved in a lawsuit. They had video of the suspects, shot from far away. Can I please tell them what the suspects were saying? 

No.

I know that - in the UK and the USA - there are specially trained 'forensic' lipreaders. For more info about this, see Wikipedia: 
http://en.wikipedia.org/wiki/Forensic_speechreading

But really, it is a myth that an expert lipreader can tell you what is being said by a speaker on video. There is too much uncertainty. 

To be able to lipread what someone says, you have to be familiar with 1. the language, 2. the speaker, 3. the topic, and 4. the context. The face and mouth-movements of the speaker have to be very clearly visible. And even then, an expert lipreader will not be able to tell you with 100% certainty what was said.

Recognition, or matching, on the other hand, is quite easy. Once you know what a person says, you can recognize the visual patterns and check whether what you see matches the words that you are expecting. 

That's probably why many hearing people think that lipreading is easy: "Oh, yes! Of course, now I can see what the pope / suspect says!"

In many cases, however, many different words / phrases will match what you see. 

For funny examples, watch the Bad Lip-Reading videos on Youtube

http://www.youtube.com/user/BadLipReading 

"I'm proud to say yo' mama took a Cosby sweater"

This particular video of Obama's inauguration:
http://www.youtube.com/watch?v=fpwhA-LdOHo&feature=c4-overview-vl&list=PLF396D3CEF0169A35

Research!

I just stumbled across a recent report by Ruth Campbell, the UK (world?) expert on lipreading/speechreading:

"This report aims to clarify what is and is not possible in relation to speechreading, and to the development of speechreading skills. It has been designed to be used by agencies which may wish to make use of speechreading for a variety of reasons, but it focuses on requirements in relation to understanding silent speech for information gathering purposes."


Highly recommended! 

http://www.ucl.ac.uk/dcal/research/research-projects/images/speechreading

Lipread.eu becomes Lipreading.eu

Unfortunately, the European LipRead project came to an early end. Some of the activities started under the LipRead umbrella, however, will be continued on the www.lipreading.eu website. 

I'll continue to use this blog for my 'thinking in progress' about lipreading, learning to lipread, teaching lipreading.

Lipreading Olympics

Watching the gymnasts doing amazing things on bars and beams during the Olympic Games, I mainly wonder why people want to do these things.

But I also listen to the commentators, who tell me what is good or bad about certain performances. Those commentators see things, that I don’t. They recognize specific elements in a performance, and can somehow see that a knee was bent, a hand was misplaced, while I just see one stream of movements.

Same as with expert and beginner lipreaders? Experts see things, that beginners don’t?

On TV, they help me by re-playing exercises in slow motion and telling me what I’m supposed to see. Ah, now I see it! Yes, that knee was bent, that hand was misplaced!

Does that mean that I will see it myself, the next time? Probably not at normal speed, but once I know what to look out for, I expect I will be able to see it in slow motion, the next time. And after a lot of practice, I will get as good as the judges or the commentators! That’s how they learned, too, didn't they? Hundreds (thousands?) of hours of watching, and being told what to look out for?

I wonder if it helps judges and commentators to see the small differences between one performance and the next, if they can ‘do’ (or could do) these exercises, themselves. Does it make it easier to judge these performances, to see the small differences, if you can do (or could do, or have tried to do) these jumps and flicks and twists, yourself?  

Beginner lipreaders see just one stream of mouth-movements. Experts see words and sentences. So for the beginners, we play videos of speakers in slow-motion, and tell them what to look out for. Again and again. First in slow motion, then at normal speed.

And maybe we ask beginners to pronounce the words and phrases, themselves. So that they know what to look out for, when  someone else uses these words or phrases. That’s why most lipreading teachers tell lipreaders to practice in front of a mirror. 

Instead of a mirror, we can use a webcam. We can ask the lipreader to record him/herself and to play the video in slow-motion, to learn to see and feel the sometimes tiny differences between one pattern, one word or phrase, and the next.

Will that help? Will that make expert lipreaders of us all?

Watch this space, we'll let you know! 

Why I don't want to teach visemes...

1. Hearing learners of English do NOT learn English, by listening to English phonemes and/or by learning to recognize phonemes in CVC or VCV words (C = Consonant, V = Vowel). Can you imagine? A beginner's English (or French, German, etc.) Course, where you begin by learning to recognize the 'a' sound, then practice discriminating 'a' and 'e'. Boring! And probably: useless. So why does anyone think this is how we should teach lipreading?

2. Lipreaders can learn to discriminate between visemes, and they can learn to identify some visemes (not phonemes!) in a standardised context: this particular speaker, this particular pattern, and maybe even: this particular video-recording. However, this doesn't help them one bit, when they try to lipread connected text, or even words. Visemes 'change shape' in the context of other visemes! The 'a' that you've learned to discriminate from the 'o', suddenly looks very different in another word or phrase. Because speech is dynamic, and because each viseme is influenced by the visemes before and after. In the same way that phonemes are influenced by the phonemic context. This is called co-articulation, see wikipedia http://en.wikipedia.org/wiki/Coarticulation. 

3. Visemes may change shape and/or be more or less visible, depending on their location in a word or sentence. Stressed words are easier to recognize (they usually last longer!), than unstressed words. Stressed vowels are more easy to recognize, then unstressed vowels. Visemes in unstressed pre- or suffixes may be totally invisible. 

4. Visemes are not standardised. There are major differences between speakers, and within speakers.

5. Mouthpatterns - and therefore visemes - change depending on the speaking style of the speaker, the speaking rate, emotions, etc. etc. The 'a' that you learned to discriminate, suddenly looks very different when the speaker is smiling! Or in a hurry. Or tired. Or angry. Or shouting, because you still don't understand ;-( 

6. Many visemes are ambiguous, many phonemes are invisible.The lipreader needs to use context to predict what a certain mouthpattern can be. Context: the situation, the speaker, the language, the topic, the sentence, the word. This is what we have to teach lipreaders from the very start: how they can use context to disambiguate the unfortunately very ambiguous visual speech signal. 

Don't shout, whisper?

Most speech-therapists and lipreading tutors will use voiceless speech (silent speech) in their lipreading training.

I've asked them why they do this. The answers so far:

1. In lipreading groups, some participants may have some hearing. If the tutor uses normal speech, the participants with some hearing have an advantage over the deaf participants. Using silent speech makes for a more equal playing field.
2. Many therapists / tutors include normally hearing partners in their groups. By using silent speech, they normally hearing partners experience what it is like, to be deaf.

But many (all?) speech therapists / tutors who train lipreading on a 1-1 basis, also use silent speech. Why?

I don't know.

Maybe because many clients (still) have some hearing, and exercises would be too easy if they could hear & see what the therapist / tutor says?

On the other hand, it makes the activities more difficult, more frustrating, and also: less realistic. In real life, people WILL use their residual hearing!

From my perspective, therapists should teach lipreading learners how they can combine the two channels (ears, eyes). Some things can be heard, some things can be seen.
If exercises are too easy for a particular learner, when the therapist / tutor uses his/her voice, then the therapist can add background noise or white noise. Or s/he can ask the learner to turn down the volume of the hearing aids or CIs. But I would not recommend to use voiceless speech!

A related question:

• How do voiceless and whispered speech compare to normal speech, visually? Are the mouthpatterns the same? No, because in voiceless speech, the vocal cords don't vibrate. Are there other differences?
  When whispering, does the speaker move lips, tongue and jaw more actively, to compensate for the lack of volume?
  I've tried to find research on the 'lipreadability' of normal, whispered and voiceless speech. So far, no luck. I've also looked for the visual characteristics of normal, whispered and voiceless speech. Still searching. It does seem that whispered speech is spoken more slowly, and that consonants last a bit longer compared to normal speech. 

All books, websites, flyers etc. about communication with hard-of-hearing and deaf people repeat the advice that the speaker shouldn't shout. Shouting changes mouthpatterns and makes lipreading more difficult.

Maybe whispering makes lipreading easier? Maybe we should have buttons, t-shirts and banners that say: 

"I'm hard-of-hearing / deaf. Don't shout, please whisper!"

True? Not true? 

Anyone who has data on this, or an opinion: please share it with me and the other readers of this blog!

PS: I tried to find a picture on the internet to show 'whispering'. In all pictures, people whisper directly into someone's ear. Sometimes, they cover their mouth with their hand to reduce visibility. That is NOT what I mean when I use the word 'whispering'! For me, whispering includes full visibility of the face of the speaker!

Visemes or Morphemes?

Most lipreading tutors and books will start by teaching visemes. Visemes are for lipreaders, what phonemes are for listeners: the smallest 'standardized' building blocks of words. Phonemes are the basic 'sounds' of a language, visemes are the basic 'mouth-patterns'. I use 'mouth-patterns' instead of 'lip-patterns', because to recognize a viseme, you have to watch lips, tongue, cheeks, and to see voicing: the throat of the speaker.
OK, so I should call them 'speech-patterns'... to include the throat? For the time being: mouth-patterns.

For LipRead, I tried to make comprehensive lists of the phonemes and visemes of the project languages: English, German, Dutch, Norwegian and Turkish. For phonemes, that was relatively easy, because I could use Jacques Koreman's L1L2 Map: http://calst.hf.ntnu.no/.
Of course, the L1L2 map doesn't show 'allophones', the different variations of each phoneme, but it's a start.

For visemes, it was much more difficult. Pretty much everyone seems to make his/her own set of visemes for a language. Some based on research, some on articulatory features, some on ... I don't know. There is no 'standardized' set of visemes, within or across languages.

Charlotta Engström looked at Swedish visemes for her Master Degree Project in Speech Communication. One of her conclusions:

"Evidently the classification of phonemes into visemes can be done
differently with respect to factors like language, speaker, listener, speech
situation, lighting conditions and, as will be shown below, phonemic
context. The better the viewing conditions are, the more contrastive
categories can be discerned. In other words, visemes are not constant units."

see: http://www.speech.kth.se/prod/publications/files/1644.pdf

The main thing everyone seems to agree on, is that there are far fewer visemes than phonemes. Many phonemes are invisible, many others are ambiguous: several phonemes share the same mouthpattern. The usual example is 'b-m-p'. Three phonemes - one viseme.

BUT!
Even if we could agree on the number of visemes, teaching single visemes is only a very first step. Visemes change shape, in the context of other letters! A viseme that is perfectly visible when pronounced on its own, may be completely invisible when it's in the company of other letters / phonemes. Or it will look very different. Visible speech is not a sequence of still pictures, it's dynamic. The articulators have to go from one location to the next, and often what you see is not the end-location, but the movement!

Two examples - although I'm not very good at finding examples for English, so please correct me if I'm wrong or if you have better examples:

• 'food' versus 'foot': indistinguishable because both words share the same visemes. But if you say the words side by side, people will see the difference. Because of the end 'd' in food, the 'oo' in 'food' is much longer in duration, than in 'foot'. So we add another viseme: long 'oo' versus short 'oo'? 
• 'bend' - 'lend' - 'blend': in many speakers, the 'l' is quite visible. But when it's preceded with a pretty dominant 'b', it may completely disappear from sight!

This doesn't only happen within words, it happens in all connected speech: phrases, sentences, etc.

Step 2 in teaching lipreading, would have to be to teach 'blends', or 'co-articulation'. Of course most tutors will do that, as soon as they move on from simple CVC (consonant - vowel - consonant) words. Then they will have to explain that yes, sorry, there are zillions of exceptions to the visemes that you've just learned.

OR
We teach morphemes, instead of visemes? Morphemes are the next 'basic part' in words. Not single letters / phonemes, but short words, or syllables. The good thing about morphemes is that they are meaningful. The bad news is that there are many more morphemes in a language, than phonemes.
As for the examples above: each of the words is a separate morpheme: food, foot, bend, lend, blend.

What I don't know, but I'm sure that it's been researched: there is more co-articulation within, than between morphemes?

Morphemes show up in research about teaching children to read. E.g. Improving Literacy by Teaching Morphemes, by T. Nunes and P. Bryant, 2006.

The moral of the story: I really really think we can Improve Lipreading by Teaching Morphemes! 

Wikipedia on Morphemes: http://en.wikipedia.org/wiki/Morpheme

Phonics vs. whole language

If lipreading is - a bit - like reading, maybe we can learn something from all the research that has been done to find the best way to teach young children to read print.

In a nutshell, there are two approaches to teaching reading: phonics and 'whole language'.

This is what Wikipedia says about them:

"Phonics" emphasizes the alphabetic principle – the idea that letters represent the sounds of speech, and that there are systematic and predictable relationships between written letters and spoken words, which is specific to the alphabetic writing system. Children learn letter sounds (b = the first sound in "bat" and "ball") first and then blend them (bl = the first two sounds in "blue") to form words. Children also learn how to segment and chunk letter sounds together in order to blend them to form words (trap = /t/, /r/, /a/, /p/ or /tr/, /ap/).

"Whole language" is a method of teaching reading that emphasizes literature and text comprehension. Students are taught to use critical thinking strategies and to use context to "guess" words that they do not recognize. In the younger grades, children use invented spelling to write their own stories.

Comparable to the analytic approach of teaching lipreading, and the synthetic approach. Right?

[Oh.. I wasn't sure if it was 'analytic' approach, or 'analytical' approach so I asked Google. And stumbled on an article that I hadn't seen before: "Teaching lip-reading: The efficacy of lessons on video", by Barbara Dodd¹, Geoff Plant² and Mark Gregory¹. In the British Journal of Audiology 1989, Vol. 23, No. 3 , Pages 229-238. I will download it, read it, and tell you about it, later.
I do know now that it is the analytic approach vs. the synthetic approach.]

The confusing bit is that in reading, they also have 'analytical phonics' and 'synthetic phonics', and these are just the opposite of what you'd expect:

• analytical phonics, also known as the Whole Word approach, is an approach to the teaching of reading in which the phonemes associated with particular graphemes are not pronounced in isolation.
• synthetic phonics involves the development of phonemic awareness from the outset. As part of the decoding process, the reader learns up to 44 phonemes (the smallest units of sound) and their related graphemes (the written symbols for the phoneme).

In Dutch, we call this 'hakken en plakken', which Google translates into 'cut and paste'... which isn't too bad.   Cutting: = cutting a word into smaller bits. Pasting = pasting smaller bits together to make a word. Analytical = cutting. Synthetic = pasting. 

For now, the main thing is that there are two approaches to teaching reading: from letters (graphemes) to words to sentences to stories. And: from stories to sentences to words.

Also called bottom-up versus top-down processing:

• bottom-up: from the letters on the page, up through your eyes, to the visual, then language, then cognitive processing centers in your brain, and finally: to awareness.
• top down: from your expectations of the text (awareness) through your eyes, to the words on the page: does it say what I expect it to say?

Although the reading experts go back and forth about the best way to teach reading, the solution is obvious. In fluent reading, we do both, it's an interactive process. We predict, then we check if our predictions are correct. If they are not, we look more closely to see what the text really says, and feed our brain the new information.  So in teaching reading, we have to teach both. 

In teaching lipreading, we see the same two approaches. Some tutors / therapists start by teaching visemes, then short words, then words in sentences: the analytic approach.

Others start by teaching learners to use their world knowledge and their knowledge of the language, to predict what a speaker may say: the synthetic approach.

And most lipreading tutors and therapists of course teach both!

Many older books and websites, however, seem to favor the analytic approach: they start by teaching visemes. But I think that is mainly because all they had, were photographs of lip-patterns. Static lip-patterns, of the kind that you rarely see in real speech. 

Because we don't speak in single phonemes / visemes, we speak in sentences. In words and sentences, phonemes / visemes change their sound / shape, depending on their position in a word, and the letters /phonemes before and after! 

But if all you have are photographs, all you can do is show single visemes. Those photographs may make lipreaders aware of the things people do with their mouths when they speak. But they don't teach lipreading.

Speech is dynamic, it is a temporal pattern. You cannot cut speech up into single phonemes, well, you can, but then you'll have a hard time understanding what is being said. In the same way, you can't cut up speech into single visemes, because you lose the dynamics and the temporal pattern: the changes over time. If you do cut up speech into visemes, you'll have a very hard time understanding what is being said!

We read, and lipread, interactively - by predicting, then recognizing whole words and even phrases, and checking our expectations against what we see. 

Thanks to video, we can now use 'moving pictures' to show 'real-life', living and moving visemes in their natural context: connected speech!

Two other things that we can learn from reading:

• When we read, we convert the words on the page into sound. First we read out loud; later, we hear the words that we read, with our mind's 'inner ear'. This is called 'phonological coding', we convert visual patterns (written words) into sound patterns (spoken and/or heard words). In the same way, many lipreaders recode what they see, into a 'phonological' code: they watch someone speak, but with their inner ear they 'hear' his/her voice. Is this a requirement for good lipreading? Should we teach it? I'm not sure.
• Some reading teachers say that children become better readers, if we teach them to recognize morphemes. I think that would work for lipreading, too: we should teach morphemes, instead of visemes.
  

... to be continued!
  

Lipreading is like reading...

Lipreading is like reading a handwritten text, on a moving, dynamic display that shows only two to three letters at a time.

Handwritten, so very personal. Some people have clear handwriting, other people's handwriting is sloppy. Some write large letters, others tiny. Even within a person, there are differences. If you're in a hurry, your handwriting deteriorates. If you write a text while you focus on the content - on what you want to write - instead of on the act of handwriting: your handwriting deteriorates.

In handwritten text, many letters are ambiguous when seen in isolation. Even whole words can be hard to recognize when taken out of context. Yet when you can see the entire sentence, letters and words are easy to read.

In printed text, letters are standardised - once you've got used to a certain font, each a looks the same, as does each b, c, and the rest of the alphabet. There's a recognizable space to indicate the end of one word, and the beginning of the next, and punctuation marks to indicate breaks and endings. In handwritten text, the letters are variable in form; their shapes are not standard, but influenced by the location in a word (beginning, middle, or ending) and the letters before and after. And words may run into each other.


To read handwritten text, we prefer good lighting, good paper, a good pen, and good contrast between paper and pen.

Last but not least: the content must be interesting, or we don't even bother to try.  As an example, a picture of a handwritten note, that I found on the internet.

All of this applies to a lipreader trying to lipread a speaker: speech is personal, ambiguous, variable, and we lipreaders need all the help we can get - or we give up before we've even started.

What makes lipreading even more difficult than reading handwritten text, is that the display (the speakers head) moves. Very few speakers will look the lipreader in the eye for more than a few sentences. People look away because they are embarrassed by the lipreader's attention, because something catches their eye, because they have to think, or because they're just used to moving their head, body and hands while they speak.

Worst of all, though, is that the lipreader sees only two to three letters at a time. I'll try and make an animation to show what that looks like for reading handwritten text. For now, try and imagine reading the handwritten note above, through a moving window that lets you see just two or three letters at a time. Difficult? Yes!
Now increase the speed to 14 - 25  letters per second. Impossible? Yes!

Yet that's what lipreaders do.

Somehow.

About this blog

January 1, 2012 was the starting date of the LipRead project, a EU project about lipreading.

Interested readers can find the official project info on the website: www.lipread.eu . Some of the info on the website has been translated into Dutch and Turkish. German translations are being made as I write.

The website is for the official info. The partners in the consortium use e-mail, for all internal discussions. We will send out a newsletter to interested outsiders, every now and then (people can register for the newsletter, on the website).

I am now starting this blog as yet another way to keep interested people informed of what we are doing. It will be a personal blog - what you read here, are my personal thoughts. Other partners in the consortium may not agree with me. And it's very much: thinking in progress.

I want to keep track somewhere of all the things that I read, hear, or think up myself. I will use a public blog to do this, because I hope to find as many people as possible, who can help us try and understand what lipreading is all about, and how it can or should be taught.

I have been e-mailing with a number of people about these things since January: partners in the consortium, people in Europe, and even someone from 'down-under'. I want to use this blog to share their thoughts (with their permission!) with others, and vice versa.
Very much 'crowd-sourcing', even though we are a pretty small crowd, spread thinly across the globe ;-)

I will write this blog in English, even though Dutch is my first language, to try and connect to as many people as possible, around the world. So please apologize my "Denglish" or "Eurish"!
I've asked the German partner in the project, Tanja Hubert, to start her own LipRead blog in German, for the colleagues in Germany. If she does, it will be her personal log with her thoughts about the project. But of course we will share and transfer and translate, whenever we want everyone's feedback.

I will not be able to work on this blog, every day, and the blog will not be in a diary format. The plan is to start a new page for each question that we are trying to answer (or deal with) in the LipRead project.

For instance:

• is lipreading like learning a new language, or is it different? 
• what tools or methods or tests can we use, to evaluate the effectiveness of our materials?  
• is lipreading English easier, or more difficult than lipreading Dutch? 

I'll add questions, as we go along. I'll add reports of my 'thinking in progress' to every page/question, whenever I've found new information, talked to people, or have another one of my 'eureka' experiences.

NB: I have these quite regularly, most are dudds, so they come with a warning: most have a very limited shelf-life ;-(


And yes: please, comment! In English if possible, in any other written language if necessary. I can use Google Translate to help me understand. 


Liesbeth

www.lipread.eu
www.liplezer.nl 
www.pragmaprojecten.nl

PS: for readers who are new to Blogger: you can respond to this blog (anonymously if you want) by clicking on the '0 comments' link at the bottom of each post:

this is just a picture, you'll find the real link further down