As an increasing number of industries, including aviation, are taking steps to incorporate behaviour detection within their security programmes, regulators are in search of academically-substantiated proof that the techniques being offered can indeed identify those with negative intent. For decades, many have promulgated the case for the identification of micro-expressions, primarily relying on the 1970s body of work produced by Paul Ekman, since immortalised by the TV series, ‘Lie to Me’. Yet Ekman’s work has never been published, the convenient excuse being that it was, and seemingly still is, too security sensitive to share! Recent academic studies, which have been published, actually show more evidence opposing Ekman’s claims – that micro-expressions cannot be detected by real-time human observation and that non-verbal cues cannot definitively diagnose deception.
Few question that micro-expressions exist, but the danger is that people buy into programmes that provide perfect conference acts and that are, like ‘Lie to Me’, great entertainment – every television network seems to love having its ‘human lie detector’ demonstrate how those found guilty by the courts were ‘obviously’ hiding something, whilst being conveniently unwilling to comment on cases awaiting a verdict. In this vein, Louise Jupe examines what we do know about micro-expressions and whether they serve any benefit in helping us identify criminal intent or whether they are just flickers that could indicate stress which is, after all, commonplace at airports. So, in training the next generation of behaviour detection officers, this article cites what we do know works and serves up a reminder that, stage act aside, there is no such thing as a human lie detector.
In the fight against international terrorism and other major security threats, the allure of ‘failsafe’ systems to detect whether or not an individual has malintent has become central to many media and academic debates. Approximately 213,000 passengers land or depart from London’s Heathrow Airport every day; the total number of daily passengers across the globe is much more difficult to quantify. With terrorist activity often occurring after international flights, aviation security plays a pivotal role in the prevention of such international security threats. In light of the failings of the Transportation Security Administration to use scientifically supported methods, exemplified in their implementation of the Screening Passengers by Observation Techniques (SPOT), let’s examine whether micro-expressions are true indicators of those who are lying.
Micro-expressions have become well known, not only amongst the scientific community but within industry and lay individuals. Micro-expressions’ notoriety stems predominantly from the fictional American TV show, Lie to Me. The show was based loosely on the work of Dr Paul Ekman and his work on facial expressions and their links with internal feelings, i.e. the notion that emotional states in humans manifest on individuals’ faces. Micro-expressions are emotional displays that are quick or fragmented and are typically thought to indicate a mismatch/disharmony between what is being said by an individual, and what is felt..
Micro-expressions have stemmed from initial observations by Charles Darwin on the notion of ‘leakage’, i.e. that observable signs of emotion cannot be suppressed and may leak out in the form of overt facial expressions. Paul Ekman aimed to study which muscles of the face we can control and which ‘leak’ and therefore cannot be controlled. He based his empirical work on a system entitled ‘Emotion Facial Action Coding System (EMFACS)’, created in the 1980s. Movements of the face were tracked, and inferences were drawn as to the emotional state of the individual. However, Ekman’s work has been subjected to widespread scrutiny, especially in light of his decision not to publish his findings on some of his projects. Our overall understanding is that the accuracy of deception detection based on the observation of non-verbal cues is close to the level of chance (DePaulo et al., 2003). Basing deception detection on cues that may not even exist is not only naïve, misinformed and misleading, but it is also a waste of public science funding.
“…stage act aside, there is no such thing as a human lie detector…”
Professor Barret first explored Ekman’s work as an undergraduate. She wanted to find objective measures of emotion. Barret quickly realised that the work of Ekman had serious limitations; in particular, his methodology involved giving participants a list of emotions and instructing them to match the faces he used to those emotions. She believed that by doing so, Ekman had ‘primed’ his participants, i.e. influenced outcomes by exposing participants to particular stimuli that would affect their responses. When she replicated his study without providing participants with a list of emotions, she found that their ability to recognise emotions was considerably reduced (Barret, 2014). Barret has since developed her own theory of emotions and has stated that there is no clear area of the brain that prioritises the recognition of emotion in humans (Barrett, 2017). Not only do micro-expressions have little to no supporting empirical evidence, but the body of theoretical and experiential evidence is founded upon a theory of emotions developed using a significantly flawed methodology.
If Dr Lightman used them, why can’t we?
Micro-expression training has been the core of Ekman’s work for some time, and his training system is available online. His practice is said to teach people to detect micro-expressions in real-time. Despite depictions in Lie to Me, it is not possible to identify micro-expressions in real-time via simple human observation. Therefore, the process used by Dr Cal Lightman (depicted by the actor Tim Roth) in Lie to Me and Ekman’s training system are not only inaccurate, but likely to be (to the best of my current knowledge) almost impossible. The hype surrounding such micro-expression research appears to be generated more by publications in popular media than science.
“…the US National Research Council who were unequivocal in their position that there is no theoretical justification as to why psychological states of emotion would be significantly different between truth tellers and liars…”
Overall, there has been a somewhat simplistic understanding that there are six basic emotions, a theory which forms a core part of Ekman’s online training. The ‘basic’ emotions are anger, disgust, fear, happiness, sadness and surprise. While Ekman initially claimed that these emotions were observed cross-culturally, he has since noted that not all cultures recognise the six ‘basic’ emotions (Ekman & Friesen, 1986). Despite this retraction, Ekman continues to offer micro-expression training based on this theory around the globe. I will not devote too much space to this aspect but will explain why the notion of six basic emotions is flawed. Essentially, the six previously mentioned basic emotions are what we call ‘coarse-grain’ emotions or collectives, and within each collective is what is known as ‘fine-grain’ expressions. For example, surprise can include, but is not limited to, astonishment, amazement, shock, disbelief, distress and wonder. This is a clear indicator as to why human emotions/expressions are multifaceted and complicated. Their understanding (or recognition) requires more than a basic (coarse grain) basis as to why any one individual would feel such a unique emotion at any one time.
Do micro-expressions identify liars?
Ekman’s training system claims that it can teach you to detect these six basic emotions via micro-expressions on the face. But how could this be implemented in a lie detection setting? The basic notion is that when individuals lie, their micro-expressions will conflict with what they say (e.g. their face expresses disgust or shame while their speech expresses happiness or sincerity) because they are hiding or ‘masking’ emotional arousal. Therefore, this is known as an anxiety-based protocol, which is based on the idea that liars will be more anxious than truth tellers. Ultimately, deception researchers have been keen to demonstrate that anxiety is not a ‘one size fits all’ signifier of deception. The idea that anxiety is synonymous with deception is the same reason many deception scholars have little to no faith in the polygraph. This includes the US National Research Council who were unequivocal in their position that there is no theoretical justification as to why psychological states of emotion would be significantly different between truth tellers and liars. Therefore, the same criticism applies to micro-expressions.
“…some of the proponents of micro-expressions and deception detection were to go on to call their work ‘pseudoscientific’…”
One of the major pitfalls of using micro-expressions to detect deception is that not all individuals display micro-expressions upon exposure to certain stimuli (Porter & ten Brinke, 2008). In their study, Porter and ten Brinke first examined expressions in genuine and deceptive facial expressions. They found that 100% of their participants showed internal emotional inconsistency (i.e. they did not show the same expression repeatedly, despite being shown images of the same nature). The discrepancy shown was related to the reactions to pictures of assumed disgusting, sad, frightening, happy, and neutral photos, responding to each with a genuine or deceptive (simulated, neutralised, or masked) expression. They also found that only 21.95% of participants showed micro-expressions. Some of the proponents of micro-expressions and deception detection were to go on to call their work ‘pseudoscientific’ (Frank & Svetieva, 2015), demonstrating further misunderstanding (or attempted manipulation of understanding) of what science is and what it is not.
“…individuals who watched Lie to Me had lower accuracy scores in a lie detection test than those who did not watch the show…”
Further studies also show problems with micro-expressions. In an empirical study, individuals who watched Lie to Me had lower accuracy scores in a lie detection test than those who did not watch the show (Su & Levine, 2016). I would suggest that there is a fundamental issue with the concept of deeming individuals’ facial expressions as ‘masked’ or ‘genuine’ since facial muscle reactions are not consistent in or among individuals, or even between cultures.
More recently, Sarah Jordan and colleagues looked at the efficacy of Paul Ekman’s online micro-expressions training tool (METT) (Jordan et al., 2019). Participants in their study were split into three groups. One group received the METT training, the second received bogus training created by the researchers and the third had no training at all. Afterwards, they were shown video recordings of people expressing truthful and deceptive statements and asked to make a judgement on whether or not they thought they were lies or truths. Their findings showed that METT training is not effective at teaching individuals to detect deception. Those in the METT condition only showed an accuracy rate of 46.30%, while the bogus group showed almost similar accuracy rates at 47.30%. Essentially, their accuracy rates were just below chance (i.e. equating to the same possibility of predicting the head or tail landing when flipping a coin). No security measure with an accuracy rate of 46.3% would be considered viable in aviation security setting.
What about artificial intelligence and micro-expressions?
If we can’t detect micro-expressions in real-time, can an automated system do the hard work for us? And how are individuals proposing that micro-expressions may hold the key to catching the liars from the truth-tellers in an aviation context? Many computational and deception researchers are looking at ways of automating the reading of micro-expressions. Since 2019, an Artificial Intelligence (AI) based system called ‘iBorderCtrl’ has been tested within various international airports in the European Union (EU), including Hungary, Latvia, and Greece. The system is referred to as an Automatic Deception Detection System (ADDS). The predominant aspect of the system consists of an artificial ‘agent’ that analyses micro-expressions. It is deemed a non-invasive psychological profiling system, which stems from a system developed in computational academia called ‘Silent Talker’ (see Rothwell, Bandar, O’Shea, & McLean, 2006). Initially, passengers pre-register information from home (similar to online check-in) and then upon arrival at the airport, a series of biometrics are taken from them (e.g. fingerprints and vein pattern analysis). Passengers then approach a virtual ‘agent’ who asks a variety of security-based questions (e.g. “What is your surname?”, “What is your citizenship and the purpose of your trip?”). After this, an individual is scored out of 100 (i.e. their ‘truthfulness’ score) and given a QR code to give to border control agents. Sounds flawless, right? Unfortunately, it’s not.
One criticism of ‘iBorderCtrl’ is that it is based on the theory that response behaviour is consistent across individuals. Evidence has already shown that not all individuals display micro-expressions and that individuals have inconsistent facial reactions to the same stimulus. There is no evidence to suggest that all humans experience the same somatovisceral changes leading to similar and identifiable facial expressions when feeling the same way. Furthermore, the use of images in training systems ¬– which feeds into the development of automated systems – suffers from a fundamental flaw. How do we know what these individuals are actually feeling? Introspective reflection is a difficult endeavour. It is subject to its own examination within the psychological domain and thus the data being fed into artificial systems are already known to be dubious.
“…evidence has already shown that not all individuals display micro-expressions and that individuals have inconsistent facial reactions to the same stimulus…”
A macro-web of limitations
I don’t want to jump into some of the more complex issues of using AI in lie detection, as that in itself has a long and convoluted series of limitations. In an article I wrote with Dr Keatley regarding the use of AI in aviation security, the AI limitations we identified related to weaknesses in the current state of psychological science; that is that the knowledge we have as deception researchers does not have enough support to be used in an automated system. Of those we identified, these were due to the use of, but not limited to: a lack of micro-expression evidence and deception; the use of biometric faking; iBorderCtrl’s confusion of ‘biomarkers’ and ‘biometrics’; unacceptable accuracy rates, and the final decision regarding a passenger’s viability to travel being made by a border agent.
In our article, we also discussed the ‘artificial crisis’, in which scientists are turning to AI as we fail to find answers to some of the most crucial security questions. ‘iBorderCtrl’ reported lab accuracy rates of up to 79%. Closer inspection of the data also shows that initial trials yielded deception accuracy rates of 54% (Rothwell et al., 2006, p. 769). However, some of their tests showed accuracy rates as low as 43% (Rothwell et al., 2007, p 332). If only 0.5% of travellers were deemed to be deceptive or have malintent, this equates to 1,065 individuals (based upon the aforementioned daily Heathrow Airport statistics). Of these 1,065 individuals, if we are using systems which only perform at an accuracy rate of 43%, then there is potential for 458 of those with malintent to slip through the system due to inadequate detection systems. If this only relates to Heathrow Airport in a single 24-hour period, then calculating the total potential number of individuals that would be missed by such systems globally is not only an arduous task but one which clearly indicates the danger of using methods with such accuracy rates.
Using an automated system comes with further issues, which are equal to those associated with evaluation carried out by humans, including what we call ‘false rejections’ and ‘misses’. A false rejection is when a truth teller is identified as deceptive. Not only does this impact on security personnel’s time management, it is also a human dignity issue. Facial abnormalities may ‘alert’ the border control operative that the individual passing through has failed the initial ADDS system. It is our human nature to make ‘snap’ judgements. Being given a low score via a QR code is likely to feed into these biases by creating snap judgements based upon an entirely flawed methodology. However, more important are the ‘misses’: when a deceptive individual is identified as being truthful. This no longer becomes a time or economic issue; it becomes a possible life or death situation.
So, what now?
Despite my evident uncertainty regarding the use of micro-expressions in aviation security, be this face-to-face or automated, I do not believe we are facing a grave future. My current belief is that due to the global fear of catastrophic malintent, we are rushing to implement under-developed solutions due to our belief in our scientific abilities. Dr Keatley and I have outlined where we think AI may have a future in lie detection. In essence, we have stated that we are currently at a point where linguistic cues have higher (but still not acceptable) accuracy rates. Examples of this include ‘Reality Monitoring’ (Masip, Sporer, Garrido, & Herrero, 2005), ‘Assessment Criteria Indicative of Deception’ (ACID) (Colwell, Hiscock-Anisman, Memon, Taylor, & Prewett, 2007) and the ‘Verifiability Approach’ (Nahari, 2018). Therefore, using AI as a way of predicting future linguistic changes, we may be able to understand how language changes over time, and therefore stay one step ahead in terms of linguistic analysis. This is only one suggestion. However, despite numerous scientists’ (including my own) desire to find the one golden cue, we still need to admit we are at a ‘work in progress’ stage of deception detection.
Louise Marie Jupe is a final year doctoral student at The University of Portsmouth. Her thesis examines ways if identifying those who are lying about their identity. Louise also has interests in pseudoscience in practice, AI in deception detection and ethics within psychological research.
References
Barrett F, L. (2014). Opinion | What Faces Can’t Tell Us. Retrieved March 8, 2019, from https://www.nytimes.com/2014/03/02/opinion/sunday/what-faces-cant-tell-us.html
Barrett F, L. (2017a). How Emotions are Made:The Secret Life of the Brain. Boston, MA: Houghton Mifflin Harcourt.
Colwell, K., Hiscock-Anisman, C. K., Memon, A., Taylor, L., & Prewett, J. (2007). Assessment Criteria Indicative of Deception (ACID): an integrated system of investigative interviewing and detecting deception. Journal of Investigative Psychology and Offender Profiling, 4(3), 167–180. https://doi.org/10.1002/jip.73
DePaulo, B. M., Malone, B. E., Lindsay, J. J., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74–118. https://doi.org/10.1037//0033-2909.129.1.74
Ekman, Paul, & Friesen, W. V. (1986). A New Pan-Cultural Facial Expression of Emotion. Motivation and Emotion, 10(2), 159–168.
Frank, M. G., & Svetieva, E. (2015). Microexpressions and Deception. In Understanding Facial Expressions in Communication: Cross-Cultural and Multidisciplinary Perspectives (pp. 227–242). Springer India. https://doi.org/10.1007/978-81-322-1934-7_11
Jordan, S., Brimbal, L., Wallace, D. B., Kassin, S. M., Hartwig, M., & Street, C. N. H. (2019). A test of the micro‐expressions training tool: Does it improve lie detection? Journal of Investigative Psychology and Offender Profiling, jip.1532. https://doi.org/10.1002/jip.1532
Masip, J., Sporer, S. L., Garrido, E., & Herrero, C. (2005). The detection of deception with the reality monitoring approach: a review of the empirical evidence. Psychology, Crime & Law, 11(1), 99–122. https://doi.org/10.1080/10683160410001726356
Nahari, G. (2018). The Applicability of the Verifiability Approach to the Real World. In P. R. J (Ed.), Detecting Concealed Information and Deception: Recent Developments (pp. 329–349). https://doi.org/10.1016/B978-0-12-812729-2.00014-8
Porter, S., & Brinke, L. Ten. (2008). Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19(5), 508–514. https://doi.org/10.1111/j.1467-9280.2008.02116.x
Rothwell, J., Bandar, Z., O’Shea, J., & McLean, D. (2006). Silent talker: A new computer-based system for the analysis of facial cues to deception. Applied Cognitive Psychology, 20(6), 757–777. https://doi.org/10.1002/acp.1204
Rothwell, J., Bandar, Z., O’Shea, J., & McLean, D. (2007). Charting the behavioural state of a person using a backpropagation neural network. Neural Computing and Applications, 16(4–5), 327–339. https://doi.org/10.1007/s00521-006-0055-9
Su, L., & Levine, M. (2016). Does “lie to me” lie to you? An evaluation of facial clues to high-stakes deception. Computer Vision and Image Understanding, 147, 52–68. https://doi.org/10.1016/j.cviu.2016.01.009