Machine learning can be used for a multitude of tasks in the payments industry. Things like fraud detection and customer retention are great examples. Recently had the opportunity to talk with George Pliev the CEO of Neurodata Lab about how he and his team are using machine learning to interpret the emotional state of people.
Does the machine recognize mood with voice, facial features, text?
We use multimodal approach to emotion recognition. We developed a cloud-based technology that can recognize emotions by analyzing human data from several channels, including voice, facial expressions, body movements, and psychophysiological parameters. Thus we work
To train a machine learning algorithm, for instance, a neural network, to recognize emotions, a dataset is needed – these are photos, videos or audio data in which people express different emotions. For instance, talk shows or specific research datasets like our RAMAS and Emotion Miner Data Corpus. To train algorithms we use data that has been previously analysed, ‘marked up’, by a large number of people. Based on what kind of emotion most people saw in a particular video fragment, the algorithm will be trained to detect this emotion in new videos that it has not seen before. Neural networks thus learn how to associate a particular set of emotion cues coming from different channels (face, voice, body) with a particular set of emotions and even cognitive states.
How accurate it is?
When analyzing audiovisual content, the algorithm can be some percent sure about an emotional expression. For instance, algorithm can predict with some probability that this person expresses 96% happiness, 3.6% surprise and 0.4% mix of other emotions. The more data the system analyzes, the better it gets at predictions.
In some cases the algorithm can give false predictions, but the potential number of those is tiny. At the same time, multimodal emotion recognition systems are more accurate than unimodal, meaning affective data coming from the face will be confirmed by that received from the voice.
Who is using this type of tech?
We collaborate with both big corporations (Rosbank, Société General Group, Microsoft, Samsung) and start-ups (Promobot, a robotics company with whom we are going to CES). They represent the industries where emotional analysis is of big interest and can be used for a number of solutions:
- Natural interaction for human robotics, virtual assistants, chatbots
- Predictive analytics for HR/recruitment
- Customer Experience Management
- In-cabin analytics of driver’s state and solutions for self-driving cars
At some point companies ran into the idea that the decision about a purchase is massively influenced not only by what consumers think about the product, but what they actually feel about it. Objective emotional analytics can be an invaluable tool. Neuromarketing instruments used for tiny focus groups were able to provide some cues about how customers feel about products. Emotion analytics open a new era for companies to recognize the emotions of each customer, right at the time of purchase.
Same with video recruitment – HR people want to have a full picture about a candidate, and emotion-related analytics can say a lot about a person. For instance, the style of speaking reflects valuable information about one’s personality. Studies have shown that nonverbal behavior is as important as the verbal response in job interviews.
Emotions have also been playing an increasingly important role in human-machine interactions. You might have heard about Vector the cute robot by Anki. It is a direct proof human robots are moving towards the Emotion AI. The same happens in service robotics – Emotion AI-enhanced robots are more attractive for people. Empathy is an important element of communication, even in human-to-human communication. If the interlocutors pay attention to the feelings and emotions of each other, the effectiveness of their communication increases by many times.
The second direction is about wellness and healthcare. Emotion recognition can help people with autism and impaired social skills understand true moods of the people around. At the same time, people can use emotion cues based for instance on the analysis of their physiology to manage stress and prevent the episodes of depression.
Emotion AI can also be applied in social and political contexts. Last week we finished a project with the Swiss biggest media holding — we analyzed the emotions in 800 videos of speeches of the Swiss parliament members. For each politician we then created an emotional profile. This may change the rules of the game in certain circumstances.
Where do you see this going in the next 5 years?
In business, Emotion AI-based systems will be used for constant, objective and correct evaluations of customer service. AI systems for different sort of analysis of employee work and customer experience will be spreading across the stores, customer service offices and call-centres. These systems will allow not only to analyze how many customers there were in the store at a certain day or what was their journey through the store, but automatically analyze their attitude towards the product/service, while simultaneously estimating how polite is the employee towards the customer. This will set a new standard in customer experience and employee performance management.
This is especially relevant for retail banking, where high quality customer service means success and revenue. No wonder that in professional CX communities and associations members are mostly represented by CX people from financial services.
Second obvious trend are human-computer interfaces. These are Emotion AI-enhanced robots, personal digital assistants, chatbots. If we delve further into possible applications of Emotion AI, we will come to medicine. It’s one thing when a device simply picks up, “understands” your mood and according to it switches on music, adjusts lights or makes coffee. The other is when it evaluates the degree of fatigue or determines any deviations from the norm by a respected type. Or a disease. In short, emotional technologies today and further will be in demand in biometrics and security systems, in robotics and the gaming industry, AR/VR, intelligent transport systems (unmanned vehicles).
Are there issues with different speech patterns or facial features in different parts of the world?
Several decades ago Paul Ekman’s universal theory of emotions was very wide-spread. He thought that as far as facial expression is concerned, people display and recognize some emotions, which Ekman called ‘basic’, in universal ways. No matter where we are and whom we are talking to, we will always recognize when our interlocutor is expressing five emotions: anger, fear, disgust, happiness, sadness.
Today this theory is broadly criticized, for example by James Russell, Beatrice de Gelder, or most famously Lisa Barrett. They claimed that emotional expression differs from culture to culture, from person to person. To train AI-algorithms to correctly recognize emotions it is crucial to use specific affective data and take into account the culture, language, gender and even age when trying to determine what emotions are expressed.
