Infants generally begin to recognize emotional expressions after 6 months of age; some reports have even observed recognition in neonates. Emotional recognition is important for social referencing, in which infants “read” their parents' expressions to understand events. Three methods for studying how infants come to recognize emotion are described in this article: peek-a-boo, matching faces and voices, and multimodal presentation.
- emotional recognition
- social referencing
- multimodal information
- amodal information
- facial expression
- vocal expression
Imagine the following scenario:
A 14-month-old infant plays on the floor with her mother when an unfamiliar person enters. The mother stands up, smiles, extends her hand, and walks over to the stranger. Observing this, the infant loses interest and goes back to her toys.
The infant has shown a skill—social referencing—that illustrates her ability to use expressions and actions to understand events.1 Had the mother responded with fear or anger in her face, voice, or gestures, the child would have acted quite differently. Infants learn to read and understand expressions and to use that information to guide their actions. This skill seems to develop rapidly, starting with a sensitivity to emotions, leading later to an understanding of those emotions. The as-yet unresolved questions about this process are: What information do younger infants detect in expressions? Can they discriminate various expressions? When can they understand the meanings of those expressions?
To date, most researchers agree that infants begin to recognize emotional expressions at approximately about 7 months of age.2,,3 In some cases, even the 5-month-old shows limited understanding, and there are scattered reports of neonates' responding differentially to expressions.4,,5 But true recognition seems to appear after 6 months of age. Using emotional recognition for social referencing, as described in the scenario, begins at approximately 8 to 10 months.1
In learning to discriminate and understand expressions, infants rely on contextual cues. Broadly defined, these cues include familiar settings, familiar persons, and multimodal (ie, sight and sound and touch) and amodal (ie, rhythm, intensity, rate, shape) information. By looking closely at the developing child's responsiveness to emotional expressions in a number of contexts, we can gain understanding of the separate and combined influences of these contexts. The experiments described in the following article have been completed recently, but data analysis and interpretation still are underway.
FAMILIARITY OF SITUATION: PEEK-A-BOO
The peek-a-boo game was used to examine young infants' perception of other people's expressions.6 Peek-a-boo provides a unique opportunity to examine infants' responses to expressions for the following reasons:
Parents and infants are familiar with the game.
Infants are attentive and enjoy peek-a-boo.
Infants have specific expectations about how the game is played.
Exaggerated expressions can be presented in a familiar context.
Peek-a-boo allows presentation of dynamic expressions to infants.
Infant responses to changes in expressions can be measured.
Forty 4.5-month-old infants were enrolled and randomly assigned to one of four “emotion change” groups. These groups were sad, anger, fear, and, as the control, consistent happy/surprise. The infants were shown three typical happy/surprise peek-a-boos followed by a fourth, in which the test expression varied according to group assignment. Each block of trials concluded with a typical happy/surprise (Table 1).
In each trial, the experimenter covered her face with a cloth for 3 seconds and called the child's name. She then reappeared for ∼7 seconds with the target facial expression and said “peek-a-boo” in an affectively matching tone. The experimenter's facial and vocal expressions and the infants' behaviors were filmed for later analysis. To ensure consistency and accuracy of facial expressions, the investigator used Ekman and Friesen's technique for portraying facial expressions using specific muscle movements.7 In addition, the experimenter had a small mirror behind the cloth and another above the infant to monitor her own expressions
The infants' total looking time (TLT) was calculated for each trial. The Maximally Discriminative Facial Movement Coding System also was used to identify the infants' facial expressions.8
Infants in the control group of consistent happy/surprise peek-a-boos increased their TLT slightly over the first three to four trials, and then diminished (Table 2). This finding is consistent with typical visual habituation studies. Infants in the sad group decreased their TLT from that trial forward. Infants in the angry group increased their TLT from that trial forward, including all the typical peek-a-boos that followed. Infants in the fear group increased their TLT the largest extent, but TLT decreased immediately with subsequent typical peek-a-boos and continued to decrease until the second fear face. Ongoing statistical analysis confirmed these results.
The infants' expressions, using the MAX system, generally showed interest/surprise throughout the experiment. There were very few full-fledged expressions of other discrete emotions. Overall, the mean frequency of interest/surprise increased both times that infants saw the sad face. The frequency increased only the first time infants saw fear or anger and decreased consistently for the control infants. In addition, facial coding for any emotional change increased for sad, fear (to a far lesser degree), and anger. For the control group, the incidence of facial movements decreased.
In general, infants responded differentially to facial/vocal expressions presented in the context of a peek-a-boo game. Looking time increased for anger and fear portrayals and decreased for sad or the control happy/surprise expression. Interest expressions, however, were highest for sad portrayals, and the incidence of facial movement was most apparent for sad portrayals.
A follow-up experiment was conducted in which the peek-a-boo face was held for 14 seconds to give infants more time to respond. The results were similar to those for the first study: decreased looking at consistent happy/surprise peek-a-boos over time; decreased looking at sadness (particularly on block 2); increased looking at fear on the first exposure, followed by decreases; and increased looking at anger on both trials. Observers also completed global coding of the infants' facial expressions. Infants in the consistent happy/surprise group became more positive throughout the session. In the sad group, the majority became more negative when they saw the sad face. Infants in the anger and fear groups had variable facial expressions. Additional analysis of their expressions is ongoing.
Results from these two experiments provide evidence that 4-month-old infants can discriminate facial/vocal expressions of anger, fear, sadness, and happy/surprise when presented in a familiar context, and that infants' reactions to these expressions are specific. The looking time data confirm previous results for discrimination of static facial expressions and vocal expressions,9–11 but the infants' own expressions were different from those they were exposed to. At this time, it is unclear whether infants were responding to the facial expressions, vocal expressions, or both.
Familiarity of Person: Matching Faces and Voices
Infants at 5 to 7 months of age can detect common meanings in facial and vocal expressions that allows them to match a happy face with a happy voice or a sad face with a sad voice.4Matching experiments often record infants' looking time when they are presented with videotapes of paired facial expressions (eg, sad and happy, angry and happy, happy and neutral) along with an appropriate sound or silence.12 (For review, see Walker-Andrews).13 In general, 2-month-old infants look at happy expressions regardless of the sounds they hear—they looked at the happy face when it was accompanied by sound 81% of the time, compared with 77% of the time when it was projected silently. However, 5- and 7-month-old infants increase their looking time to any facial expression (happy, sad, angry, neutral) when it is sound-specified. Even when the synchrony relations are disrupted (by delaying the soundtrack by 5 seconds, or by occluding the mouth area of the face), infants look proportionately longer to the film that is sound-specified. Moreover, when the facial expressions are presented in either of two orientations (upside down or upright) with a single soundtrack, 7-month-olds only responded to the vocal match when facial expressions are shown in the upright orientation.
Infants as young as 3 months of age were presented videotape of either their mother or a stranger depicting happy and sad facial and vocal expressions.14 During the experiment, infants saw two facial expressions simultaneously for four 25-second trials (the expressions were shown alternately on the right and left sides), accompanied synchronously by a single soundtrack matching one of the facial expressions. TLT for each infant on the two expressions was recorded and analyzed.
When infants were presented with videotapes of their mother, they generally looked longer at an expression when it was sound-specific (57%) than when it was silent (50%). They looked at the happy expression 67% of the time when it was sound-specific and 52% of the time when it was silent. With sound, the sad face was looked at 48% of the time, but only 33% of the time when silent.
In contrast, infants who were presented with videotapes of the stranger's face and voice did not show a preference. The average looking time was 50%, regardless of whether the expression was accompanied by sound. Individual values for happy were 54% (sound) and 53% (silent); values for sad were 46% (sound) and 47% (silent). Furthermore, infants who were shown their own mothers demonstrated an overall preference for the happy expression.
These data show that infants as young as 3 months of age are able to detect the correspondence between facial and vocal expressions, suggesting that they can understand something about the affective displays of their mothers. There are several potential explanations, none entirely satisfying. First, infants could be aroused differentially by adult expressions and match their arousal level with what they see and hear; however, infants failed to make intermodal matches between the faces and voices of strangers. They also may have used the temporal synchrony between face and voice; but even with synchronized sound and expressions, infants preferred their mothers over strangers and preferred right-side-up faces to upside-down faces. Ongoing studies of mothers' facial expressions with asynchronous voices among 3-month-olds currently indicates that infants look longer at matching facial/vocal expressions, even when the voice is out of synch.
Multimodal Information for Expressions
Young infants probably require the “whole” emotional expression to appreciate its meaning.3 That is, infants may first recognize affective expressions as part of a unified multimodal event that has a unique communicative meaning. Faces and voices typically are experienced together and as part of an event that also includes touch and smell. Yet, infants learn to discriminate auditory and visual information and to detect abstract invariants that specify the same emotional meaning. As noted by Flavell15:
In the extralaboratory world, people do not present themselves to infants as voiceless or faceless voices. … Moreover, the face and voice are unified in space and time: The voice and the face share the same spatial location, and the face's mouth movements are temporally synchronized. In addition, certain specific faces always co-occur with certain specific voices. … Finally, how each face looks and acts on a given occasion is highly correlated with how its voice sounds; for instance, happy and sad voices usually accompany happy and sad faces, respectively.
In this model, infants detect unimodal information that has the potential to specify the meaning of an expression. They detect acoustic parameters, such as timbre and frequency, that provide information for affect. Even neonates can detect the fundamental frequencies of two different voices and discriminate between them, but there is no evidence that this information specifies an emotional nuance to the infant. Likewise, facial feature differences in expressions can be detected at several months postnatally. However, this modality-specific information does not allow for recognition of emotions until somewhat later.
Recognition probably occurs first in multimodal contexts. The critical information specifying an emotion is found in the overall dynamic flow, particularly in the invariant patterns of movement and change undergone by facial musculature, body, and voice. Therefore, for infants, dynamic, naturalistic, and multimodal presentations may be the optimal stimuli.16
Sixty 5-month-old infants were visually habituated to a single videotape of a compound stimulus (either a facial/vocal expression of happy or angry or a dot-light face accompanied by a happy or angry vocal expression), and their TLT recorded. Once the infant reached a criterion of habituation, one of several changes was made.
Group 1: Face and Voice Change
These infants were visually habituated to a happy or angry facial expression accompanied by its characteristic, but asynchronous, vocal expression. Once the infant became habituated, she was presented with two posttests in which both the face and voice were changed.
Group 2: Face-only Change
These infants saw a happy or angry facial expression accompanied by a characteristic synchronous vocal expression. When habituated, the facial expression was switched, but the original voice continued.
Group 3: Voice-only Change
These infants saw a happy or angry facial expression accompanied by its characteristic synchronous vocal expression. When habituated, the voice was switched.
Group 4: Dot-light Face, Voice-only Change
This group of infants saw a dot-light rendition of a happy or angry facial expression accompanied by its characteristic vocal expression. (Infants will treat a dot-light face as if it were a face.17 Research with 7-month-olds indicates that infants will show intermodal matching for such facial and vocal expressions.18) After the habituation criterion was reached, the vocal expression was switched from happy to angry or vice versa.
The length of looking time when habituation was broken by the new stimulus was as follows (greatest to least):
Both face and voice changed
Voice-only change, accompanied by upside-down, dot-light face
Voice-only change, accompanied by dot-light face
Face-only change; voice remained constant
Voice-only change; face remained constant
(tie with 5) Control—no change in face or voice.
An analysis of variance revealed that the overall between-groups effect was significant, and more detailed analysis is being performed.
It appears that infants can discriminate changes in face and voice together; changes in vocal expressions when accompanied by dot-light faces; and changes in facial expressions alone. However, they did not discriminate changes in vocal expression when accompanied by a normal fully illuminated face.18 This difference may be inherent in the study design that could have biased infants' attention to the face or voice. For example, visual attention was used as an index of the infants' looking and listening behavior. Also, the face and voice were asynchronous in all trials, a condition that should have been obvious when infants were presented a fully illuminated facial expression for habituation. Additionally, the peculiar sight of the dot-light faces may have led to enhanced attention to the vocal expressions. These factors may have combined in interesting ways to produce the infants' patterns of fixation on the dishabitation trials. Additional research to clarify these contributions is ongoing.
In two exploratory studies, 3- to 4-month-old infants seem better able to discriminate and perhaps recognize facial and vocal expressions in familiar contexts. In the first, presenting the expressions to the infants in a familiar situation (the peek-a-boo game), and in the second, presenting the expression in the context of a familiar person (the child's own mother) have led to greater understanding of the infant's sensitivity to emotional expressions.
Studying infants' perceptions of facial and vocal expressions are important methods to learn how infants develop a sense of meaning to the world. Our focus has been on the information available to infants and to determine whether they merely detect that information, discriminate instances of it, and/or respond to it in meaningful ways. In these projects, we are beginning to look at what might be called contextual information, as provided by familiar situations, familiar people, and “redundant” presentations (multimodal renditions of emotional expressions). My suggestion is that as infants' perceptual systems mature, they gain experience with the emotional expressions of those around them; they then may use contextual information to discriminate and understand the meaning of those expressions as social signals for their own behavior.19
In these three studies, we have begun accruing evidence for young infants' discrimination of emotional expressions in familiar contexts, for familiar persons, and when facial or vocal information are available. By using converging evidence, we can make progress in determining how infants recognize emotional expressions as meaningful. That is, when infants respond consistently to specific expressions and when they can detect correspondences between visual and acoustic information, they may begin to use emotional information to guide their own actions.
I thank the families whose infants participated in these studies as well as the research assistants who aided in data collection.
- Received August 2, 1998.
- Accepted September 29, 1998.
- Address correspondence to John G. Warhol, PhD, The Warhol Institute, 225 First Ave, Atlantic Highlands, NJ 07716.
This research was supported by National Science Foundation Grant SBR-9408993.
- TLT =
- total looking time
- ↵Klinnert M, Campos JJ, Source J, et al. Emotions as behavior regulators: social referencing in infancy. In: Plutchik R, Kellerman H, eds. Emotions in Early Development. New York, NY: Academic Press; 1983;2:57–86
- Nelson CA
- ↵Everhart V, Henry S. Indicators of early empathy: newborns learning the meaning of changes in facial expression of emotion within 5 minutes. Presented at the International Conference on Infant Studies; 1992; Miami, FL
- ↵Montague D. Infants' discrimination of emotional expressions in a peekaboo interaction. Presented at the Society for Research in Child Development; 1995; Indianapolis, IN
- ↵Ekman P, Friesen W. Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press; 1978
- ↵Izard CE. The Maximally Discriminative Facial Movement Coding System (MAX). Newark, DE: University of Delaware; 1979
- Young-Browne G,
- Rosenfeld HM,
- Horowitz FD
- ↵Walker-Andrews AS. Infants' perception of the affordances of expressive behaviors. In: Rovee-Collier CK, ed. Advances in Infancy Research. Norwood, NJ: Ablex; 1988;5:173–221
- ↵Kahana-Kalman R. Contextual Effects in Intermodal Perception of Emotional Expressions by Young Infants. New Brunswick, NJ: Rutgers University; 1997. Dissertation
- ↵Flavell JH. Cognitive Development. 2nd ed. Englewood Cliffs, NJ: Prentice-Hall; 1985
- ↵Dickson L, Walker-Andrews AS. Discrimination of vocal expressions: the effects of context. Presented at the Society for Research in Child Development; April 1997; Washington, DC
- ↵Oster H. “Recognition” of emotional expression in infancy. In: Lamb ME, Sherrod LR, eds. Infant Social Cognition. Empirical and Theoretical Considerations. Hillsdale, NJ: Erlbaum; 1981;85–125
- Copyright © 1998 American Academy of Pediatrics