Facial Action Coding System

Muscles of head and neck

The Facial Action Coding System (FACS) is a system to taxonomize human facial movements by their appearance on the face, based on a system originally developed by a Swedish anatomist named Carl-Herman Hjortsjö.^[1] It was later adopted by Paul Ekman and Wallace V. Friesen, and published in 1978.^[2] Ekman, Friesen, and Joseph C. Hager published a significant update to FACS in 2002.^[3] Movements of individual facial muscles are encoded by the FACS from slight different instant changes in facial appearance. It has proven useful to psychologists and to animators.

Background

Blind athlete expressing joy in athletic competition. The fact that unsighted persons use the same expressions as sighted people shows that expressions are innate.

In 2009, a study was conducted to study spontaneous facial expressions in sighted and blind judo athletes. They discovered that many facial expressions are innate and not visually learned.^[4]

Method

Using the FACS^[5] human coders can manually code nearly any anatomically possible facial expression, deconstructing it into the specific "action units" (AU) and their temporal segments that produced the expression. As AUs are independent of any interpretation, they can be used for any higher order decision making process including recognition of basic emotions, or pre-programmed commands for an ambient intelligent environment. The FACS manual is over 500 pages in length and provides the AUs, as well as Ekman's interpretation of their meanings.

The FACS defines AUs, as contractions or relaxations of one or more muscles. It also defines a number of "action descriptors", which differ from AUs in that the authors of the FACS have not specified the muscular basis for the action and have not distinguished specific behaviors as precisely as they have for the AUs.

For example, the FACS can be used to distinguish two types of smiles as follows:^[6]

• the insincere and voluntary Pan-Am smile: contraction of zygomatic major alone
• the sincere and involuntary Duchenne smile: contraction of zygomatic major and inferior part of orbicularis oculi.

The FACS is designed to be self-instructional. People can learn the technique from a number of sources including manuals and workshops,^[7] and obtain certification through testing.^[8]

Although the labeling of expressions currently requires trained experts, researchers have had some success in using computers to automatically identify the FACS codes.^[9] One obstacle to automatic FACS code recognition is a shortage of manually coded ground truth data.^[10]

Uses

Use in medicine

The use of the FACS has been proposed for use in the analysis of depression,^[11] and the measurement of pain in patients unable to express themselves verbally.^[12]

Cross-species applications

The original FACS has been modified to analyze facial movements in several non-human primates, namely chimpanzees,^[13] rhesus macaques,^[14] gibbons and siamangs,^[15] and orangutans.^[16] More recently, it was developed also for domestic species, including dogs,^[17] horses^[18] and cats.^[19] Similarly to the human FACS, the animal FACS has manuals available online for each species with the respective certification tests.^[20]

Thus, the FACS can be used to compare facial repertoires across species due to its anatomical basis. A study conducted by Vick and others (2006) suggests that the FACS can be modified by taking differences in underlying morphology into account. Such considerations enable a comparison of the homologous facial movements present in humans and chimpanzees, to show that the facial expressions of both species result from extremely notable appearance changes. The development of FACS tools for different species allows the objective and anatomical study of facial expressions in communicative and emotional contexts. Furthermore, a cross-species analysis of facial expressions can help to answer interesting questions, such as which emotions are uniquely human.^[21]

The Emotional Facial Action Coding System (EMFACS)^[22] and the Facial Action Coding System Affect Interpretation Dictionary (FACSAID)^[23] consider only emotion-related facial actions. Examples of these are:

Emotion	Action units
Happiness	6+12
Sadness	1+4+15
Surprise	1+2+5B+26
Fear	1+2+4+5+7+20+26
Anger	4+5+7+23
Disgust	9+15+17
Contempt	R12A+R14A

Computer-generated imagery

FACS coding is also used extensively in computer animation, in particular for computer facial animation, with facial expressions being expressed as vector graphics of AUs.^[24] FACS vectors are used as weights for blend shapes corresponding to each AU, with the resulting face mesh then being used to render the finished face.^[25] Deep learning techniques can be used to determine the FACS vectors from face images obtained during motion capture acting, facial motion capture or other performances.^[26]

Codes for action units

See also: List of muscles in the human body § The muscles of the head

For clarification, the FACS is an index of facial expressions, but does not actually provide any bio-mechanical information about the degree of muscle activation. Though muscle activation is not part of the FACS, the main muscles involved in the facial expression have been added here.

Action units (AUs) are the fundamental actions of individual muscles or groups of muscles.

Action descriptors (ADs) are unitary movements that may involve the actions of several muscle groups (e.g., a forward‐thrusting movement of the jaw). The muscular basis for these actions has not been specified and specific behaviors have not been distinguished as precisely as for the AUs.

For the most accurate annotation, the FACS suggests agreement from at least two independent certified FACS encoders.

Intensity scoring

Intensities of the FACS are annotated by appending letters A–E (for minimal-maximal intensity) to the action unit number (e.g. AU 1A is the weakest trace of AU 1 and AU 1E is the maximum intensity possible for the individual person).

• A Trace
• B Slight
• C Marked or pronounced
• D Severe or extreme
• E Maximum

Other letter modifiers

There are other modifiers present in FACS codes for emotional expressions, such as "R" which represents an action that occurs on the right side of the face and "L" for actions which occur on the left. An action which is unilateral (occurs on only one side of the face) but has no specific side is indicated with a "U" and an action which is bilateral but has a stronger side is indicated with an "A" for "asymmetric".

List of AUs and ADs (with underlying facial muscles)

Main codes

AU number	FACS name	Muscular basis
0	Neutral face
1	Inner brow raiser	frontalis (pars medialis)
2	Outer brow raiser	frontalis (pars lateralis)
4	Brow lowerer	depressor glabellae, depressor supercilii, corrugator supercilii
5	Upper lid raiser	levator palpebrae superioris, superior tarsal muscle
6	Cheek raiser	orbicularis oculi (pars orbitalis)
7	Lid tightener	orbicularis oculi (pars palpebralis)
8	Lips toward each other	orbicularis oris
9	Nose wrinkler	levator labii superioris alaeque nasi
10	Upper lip raiser	levator labii superioris, caput infraorbitalis
11	Nasolabial deepener	zygomaticus minor
12	Lip corner puller	zygomaticus major
13	Sharp lip puller	levator anguli oris (also known as caninus)
14	Dimpler	buccinator
15	Lip corner depressor	depressor anguli oris (also known as triangularis)
16	Lower lip depressor	depressor labii inferioris
17	Chin raiser	mentalis
18	Lip pucker	incisivii labii superioris and incisivii labii inferioris
19	Tongue show
20	Lip stretcher	risorius with platysma
21	Neck tightener	platysma]
22	Lip funneler	orbicularis oris
23	Lip tightener	orbicularis oris
24	Lip pressor	orbicularis oris
25	Lips part	depressor labii inferioris, or relaxation of mentalis or orbicularis oris
26	Jaw drop	masseter; relaxed temporalis and internal pterygoid
27	Mouth stretch	pterygoids, digastric
28	Lip suck	orbicularis oris

Head movement codes

AU number	FACS name	Action
51	Head turn left
52	Head turn right
53	Head up
54	Head down
55	Head tilt left
M55	Head tilt left	The onset of the symmetrical 14 is immediately preceded or accompanied by a head tilt to the left.
56	Head tilt right
M56	Head tilt right	The onset of the symmetrical 14 is immediately preceded or accompanied by a head tilt to the right.
57	Head forward
M57	Head thrust forward	The onset of 17+24 is immediately preceded, accompanied, or followed by a head thrust forward.
58	Head back
M59	Head shake up and down	The onset of 17+24 is immediately preceded, accompanied, or followed by an up-down head shake (nod).
M60	Head shake side to side	The onset of 17+24 is immediately preceded, accompanied, or followed by a side to side head shake.
M83	Head upward and to the side	The onset of the symmetrical 14 is immediately preceded or accompanied by a movement of the head, upward and turned or tilted to either the left or right.

Eye movement codes

AU number	FACS name	Action
61	Eyes turn left
M61	Eyes left	The onset of the symmetrical 14 is immediately preceded or accompanied by eye movement to the left.
62	Eyes turn right
M62	Eyes right	The onset of the symmetrical 14 is immediately preceded or accompanied by eye movement to the right.
63	Eyes up
64	Eyes down
65	Walleye
66	Cross-eye
M68	Upward rolling of eyes	The onset of the symmetrical 14 is immediately preceded or accompanied by an upward rolling of the eyes.
69	Eyes positioned to look at other person	The 4, 5, or 7, alone or in combination, occurs while the eye position is fixed on the other person in the conversation.
M69	Head or eyes look at other person	The onset of the symmetrical 14 or AUs 4, 5, and 7, alone or in combination, is immediately preceded or accompanied by a movement of the eyes or of the head and eyes to look at the other person in the conversation.

Visibility codes

AU number	FACS name
70	Brows and forehead not visible
71	Eyes not visible
72	Lower face not visible
73	Entire face not visible
74	Unscorable

Gross behavior codes

These codes are reserved for recording information about gross behaviors that may be relevant to the facial actions that are scored.

AU number	FACS name	Muscular basis
29	Jaw thrust
30	Jaw sideways
31	Jaw clencher	masseter
32	[Lip] bite
33	[Cheek] blow
34	[Cheek] puff
35	[Cheek] suck
36	[Tongue] bulge
37	Lip wipe
38	Nostril dilator	nasalis (pars alaris)
39	Nostril compressor	nasalis (pars transversa) and depressor septi nasi
40	Sniff
41	Lid droop	levator palpebrae superioris (relaxation)
42	Slit	orbicularis oculi muscle
43	Eyes closed	relaxation of levator palpebrae superioris
44	Squint	corrugator supercilii and orbicularis oculi muscle
45	Blink	relaxation of levator palpebrae superioris; contraction of orbicularis oculi (pars palpebralis)
46	Wink	orbicularis oculi
50	Speech
80	Swallow
81	Chewing
82	Shoulder shrug
84	Head shake back and forth
85	Head nod up and down
91	Flash
92	Partial flash
97*	Shiver/tremble
98*	Fast up-down look

See also

• Computer facial animation
• Computer processing of body language
• Emotion classification
• Facial electromyography
• Facial feedback hypothesis
• Facial muscles
• Microexpression

References

1. Hjortsjö CH (1969). Man's face and mimic language. free download: Carl-Herman Hjortsjö, Man's face and mimic language" Archived 2022-08-06 at the Wayback Machine
2. Ekman P, Friesen W (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Palo Alto: Consulting Psychologists Press.
3. Ekman P, Friesen WV, Hager JC (2002). Facial Action Coding System: The Manual on CD ROM. Salt Lake City: A Human Face.
4. Matsumoto, D., & Willingham, B. (2009). "Spontaneous facial expressions of emotion of blind individuals". Journal of Personality and Social Psychology, 96(1), 1-10
5. Freitas-Magalhães (2012). "Microexpression and macroexpression". In Ramachandran VS (ed.). Encyclopedia of Human Behavior. Vol. 2. Oxford: Elsevier/Academic Press. pp. 173–183. ISBN 978-0-12-375000-6.
6. Del Giudice M, Colle L (May 2007). "Differences between children and adults in the recognition of enjoyment smiles". Developmental Psychology. 43 (3): 796–803. doi:10.1037/0012-1649.43.3.796. PMID 17484588.
7. Rosenberg EL. "Example and web site of one teaching professional". Archived from the original on 2009-02-06. Retrieved 2009-02-04.
8. "Facial Action Coding System". Paul Ekman Group. Retrieved 2019-10-23.
9. Facial Action Coding System. Retrieved July 21, 2007.
10. Song, Juan; Liu, Zhilei (10 Mar 2023). "Self-supervised Facial Action Unit Detection with Region and Relation Learning". arXiv:2303.05708 [cs.CV].
11. Reed LI, Sayette MA, Cohn JF (November 2007). "Impact of depression on response to comedy: a dynamic facial coding analysis". Journal of Abnormal Psychology. 116 (4): 804–9. CiteSeerX 10.1.1.307.6950. doi:10.1037/0021-843X.116.4.804. PMID 18020726.
12. Lints-Martindale AC, Hadjistavropoulos T, Barber B, Gibson SJ (2007). "A psychophysical investigation of the facial action coding system as an index of pain variability among older adults with and without Alzheimer's disease". Pain Medicine. 8 (8): 678–89. doi:10.1111/j.1526-4637.2007.00358.x. PMID 18028046.
13. Parr LA, Waller BM, Vick SJ, Bard KA (February 2007). "Classifying chimpanzee facial expressions using muscle action". Emotion. 7 (1): 172–81. doi:10.1037/1528-3542.7.1.172. PMC 2826116. PMID 17352572.
14. Parr LA, Waller BM, Burrows AM, Gothard KM, Vick SJ (December 2010). "Brief communication: MaqFACS: A muscle-based facial movement coding system for the rhesus macaque". American Journal of Physical Anthropology. 143 (4): 625–30. doi:10.1002/ajpa.21401. PMC 2988871. PMID 20872742.
15. Waller BM, Lembeck M, Kuchenbuch P, Burrows AM, Liebal K (2012). "GibbonFACS: A Muscle-Based Facial Movement Coding System for Hylobatids". International Journal of Primatology. 33 (4): 809–821. doi:10.1007/s10764-012-9611-6. S2CID 18321096.
16. Caeiro CC, Waller BM, Zimmermann E, Burrows AM, Davila-Ross M (2012). "OrangFACS: A Muscle-Based Facial Movement Coding System for Orangutans (Pongo spp.)" (PDF). International Journal of Primatology. 34: 115–129. doi:10.1007/s10764-012-9652-x. S2CID 17612028.
17. Waller BM, Peirce K, Caeiro CC, Scheider L, Burrows AM, McCune S, Kaminski J (2013). "Paedomorphic facial expressions give dogs a selective advantage". PLOS ONE. 8 (12): e82686. Bibcode:2013PLoSO...882686W. doi:10.1371/journal.pone.0082686. PMC 3873274. PMID 24386109.
18. Wathan J, Burrows AM, Waller BM, McComb K (2015-08-05). "EquiFACS: The Equine Facial Action Coding System". PLOS ONE. 10 (8): e0131738. Bibcode:2015PLoSO..1031738W. doi:10.1371/journal.pone.0131738. PMC 4526551. PMID 26244573.
19. Caeiro CC, Burrows AM, Waller BM (2017-04-01). "Development and application of CatFACS: Are human cat adopters influenced by cat facial expressions?" (PDF). Applied Animal Behaviour Science. 189: 66–78. doi:10.1016/j.applanim.2017.01.005. ISSN 0168-1591.
20. "Home". animalfacs.com. Retrieved 2019-10-23.
21. Vick SJ, Waller BM, Parr LA, Smith Pasqualini MC, Bard KA (March 2007). "A Cross-species Comparison of Facial Morphology and Movement in Humans and Chimpanzees Using the Facial Action Coding System (FACS)". Journal of Nonverbal Behavior. 31 (1): 1–20. doi:10.1007/s10919-006-0017-z. PMC 3008553. PMID 21188285.
22. Friesen W, Ekman P (1983), EMFACS-7: Emotional Facial Action Coding System. Unpublished manuscript, vol. 2, University of California at San Francisco, p. 1
23. "Facial Action Coding System Affect Interpretation Dictionary (FACSAID)". Archived from the original on 2011-05-20. Retrieved 2011-02-23.
24. Walsh, Joseph (2016-12-16). "Rogue One: the CGI resurrection of Peter Cushing is thrilling – but is it right?". The Guardian. ISSN 0261-3077. Retrieved 2023-10-23.
25. "Discover how to create FACS facial blendshapes in Maya | CG Channel". Retrieved 2023-10-23.
26. Gudi, Amogh; Tasli, H. Emrah; Den Uyl, Tim M.; Maroulis, Andreas (2015). "Deep learning based FACS Action Unit occurrence and intensity estimation". 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). pp. 1–5. doi:10.1109/FG.2015.7284873. ISBN 978-1-4799-6026-2. S2CID 6283665. Retrieved 2023-10-23.

External links

• Paul Ekman's articles relating to FACS
• Paul Ekman's Facial Action Coding System (FACS)
• More information on the different animal FACS projects
• New Yorker article discussing FACS
• Details from 1978 edition of FACS
• Site at WPI
• download of Carl-Herman Hjortsjö, Man's face and mimic language" Archived 2022-08-06 at the Wayback Machine (the original Swedish title of the book is: "Människans ansikte och mimiska språket". The correct translation would be: "Man's face and facial language")