UTA Researchers Building App to Predict Emotions

Digitizing Emotion

Imagine if domestic violence offenders received a warning and coping tools before they had angry outbursts. Imagine the impact this would have on the 3 to 6 million domestic violence victims in the United States.

Someday, an interdisciplinary team, comprised of researchers from the University of Texas at Arlington, hope to have an app for that. They aim to discover an objective way to predict anger in the brain before a person acts on it and develop a smart training tool for those who want to control their emotions.  

The cross-collaborative effort integrates researchers from UTA’s Colleges of Engineering, Science, Education and School of Social Work.

A volunteer wears a skull cap with electrodes. Photo by Alex Lepe

An unplanned formation of this diverse team began more than a year ago at a meeting between Electrical Engineering Professor Dr. J.-C. Chiao and School of Social Work Professor Dr. Peter Lehmann.

“Dr. Lehmann was working on a domestic violence case,” Chiao said. “He was wondering, since all of the methods they use are subjective, if there is any way they could assess a domestic violence offender in an objective way–one that a therapist or judge can use to determine how this person is making progress dealing with their emotion. At that time, we started to brainstorm.”

Dr. Yuan Bo Peng

Emotions are difficult to analyze quantitatively, Chiao said, so he reached out to other faculty. The first was Dr. Yuan Bo Peng, Professor of Psychology, College of Science, and a medical doctor with research expertise in neuroscience. Peng and Chiao have worked many years on the study of neural pathways for chronic pain. They decided to analyze emotions noninvasively from the brain signals and invited Dr. Jodi Tommerdahl, Professor of Curriculum and Instruction, College of Education to join the team. Her expertise is the utilization of the electroencephalogram (EEG) recording to analyze how humans learn. The team realized that this might become an opportunity to combine engineering and neuroscience to study human behaviors. Lehmann introduced Assistant Professor of Social Work Dr. Anne Nordberg to the team. Her primary interest is criminal justice reform, principally innovative ways to reduce reliance on incarceration. Nordberg educated the team on how humans go through emotions and process anger, which helped the researchers understand how to test human emotions and recruit research volunteers.

At that point, the team hit a snag with a major issue. “The brain signals contain a lot of data,” Chiao said. “If the signals from multiple parts of the brain are broken down into every second to analyze, we are facing a huge amount of data to analyze. With the conventional way of comparing waveforms, it is very difficult to comprehend the meaning of data in real time.”

So, the team tapped Dr. Shouyi Wang, Assistant Professor, Department of Industrial, Manufacturing and Systems Engineering. His expertise is big data analysis. Wang uses data-mining techniques to correlate data that seem very massive and random into classified and organized features. 

Dr. Shouyi Wang

UTA awarded a funding grant, which the team used to recruit volunteers. Their project is aligned well with the university’s strategic themes of data-driven discovery, health and the human condition, and sustainable urban communities.

During the laboratory testing, a volunteer wears a skull cap with electrodes. They are given visual stimuli to see how the volunteers' emotions go through different stages. Some of the stimuli are calming, neutral, or joyful. Some stir frustration or possibly anger. At the same time, the EEG is recorded. “The goal is to determine if the physiological signatures in a human who goes through emotion changes can be identified,” Chiao explained. “Then maybe the team can develop something a person can wear that provides feedback to the person when emotions change. Very often, we get angry for no reason at all so sometimes we don’t know what triggered the anger,” he continued. “Sometimes the anger or frustration happens very quickly, so what the team wants to do is to design some sort of wearable for the person that they can detect and know what is about to happen and when frustration is about to build up. It would send a message to the person to alert that this emotion is happening.”

An EEG screen digitizes emotions detected through the skull cap. Photo by Alex Lepe

Tommerdahl serves as the neuroscientist on the project. She typically works in the neuroscience of language and reasoning. At UTA, she has an EEG lab, where the team can measure the neuro-signals coming from the brain. Because she has the equipment and the expertise, she was a natural member of the project to gather data. “The way the EEG works is you need to provide some kind of a stimulus at a time zero, and then what an EEG does is to measure the electrical signals that the brain produces in the first two seconds afterward,” Tommerdahl said. “We’re not looking at the development of emotions over time for several minutes. We provide stimuli, and we measure the neural reaction over short periods of time. The volunteer gets the stimuli repeated several times, so then we can average them all together and start getting some meaningful data of how the brain really reacts. We are trying to understand exactly what specific emotions look like on the EEG screen.” 

In their preliminary design, an app will pop up on a smart phone or smart watch to inform the person that he or she is about to get frustrated or angry. Then, it might follow with some music, messages, or a protocol like counting from one to 10, or ‘take a deep breath.’ Responses will be designed for that particular person. This system also can be used as a training tool that people with emotional issues can wear to train themselves to calm down or to analyze what types of situations make them angry.

This will be beneficial not only to that individual but also to family or friends.

20 people per minute are physically abused by an intimate partner in the United States

Chiao and his team are recruiting a group of volunteers without knowing if those individuals have anger issues or not as a double-blind study. When they finish the study with conclusive results, those will be published in research journals.

The team is not working with offenders at this time. In the future, they hope to work directly with people who have anger outburst issues. That will require collaboration with the courts with new research protocols. At the same time, the team is developing a training tool for people who may want to make themselves better people by controlling their emotions and be happier, Chiao said.

“Many of the people we serve {in social work} as practitioners and researchers are struggling with mental health disorders,” Nordberg said. “The more we can learn about the emotion regulation and dysregulation that is the hallmark of many of those disorders, the better that we can serve our clients. For instance, we know that emotion regulation plays a part in all kinds of psychiatric conditions, so the more we can arm ourselves with knowledge and a variety of interventions, the more possibilities we have to help clients,” she said. “Being a part of this research team, for me, is a chance to get in on the research at a stage that social work normally does not start at, which is trying to sort out what is happening in people’s brains when it comes to emotion regulation and investigating what future interventions could be.”

Nordberg works with a population of former offenders. “I try to avoid that term because it is offensive,” she said. “With those folks, anger and negative emotions get them into all kinds of trouble. In the wrong setting, they end up right back in jail. Among that group of people that I am so familiar with, they don’t want to have these negative emotions that are causing them all kinds of problems. If you give people tools to help control and manage anger, not everyone will use the tools, but some of them might. That’s true of any intervention that we see in a social work setting.”

Anne Nordberg

“Domestic violence is based in issues of power and control, and until those issues are addressed, offender recidivism will continue to plague the families we see at SafeHaven,” said Kathryn Jacob, CEO and President SafeHaven of Tarrant County.

“SafeHaven is the largest and most comprehensive agency in Tarrant County providing services at no cost to domestic violence victims at two shelters—one in Arlington and one in Fort Worth.

“While we are not a part of the research project at UTA, we do support and encourage any study that would deepen the conversation about offenders in cases of intimate partner violence.”

“Our brain works in a very mysterious way,” Chiao said. “If you are exposed to a positive stimulus, you will start to develop the neuropathways in your brain to become more positive. If we have a tool to make a person realize their own emotions quantitatively – either positive or negative—then maybe we can use this to train the individuals, or the individuals can train themselves to become happier and calmer people. That is the bigger scope of what the team is trying to accomplish.”

Detecting emotion is extremely difficult, and sometimes even the most outgoing person will consciously or subconsciously hide his or her own feelings, Chiao said.

Only 34% of people who are injured by intimate partners received medical care for their injuries.

“We have to see beyond that. That is why we tell people that we only have a preliminary conclusion,” he said. “Does our conclusion apply for everybody? We don’t really know, and we are trying to figure out in the next several years what will be the most objective way to analyze this. At this moment, we want to move forward to make the research a larger scale, so we are trying to figure out from what we already have how we can widen our scope to validate our hypothesis. If we are detecting the emotion response, how do we blend that from the neuroscience point of view from what we understand about the human brain? Dr. Yuan Bo Peng and Dr. Jodi Tommerdahl have years of experience in understanding how neuropathway works, so they are explaining from the human physiology, trying to figure out why the human responds in a certain way.”

The team discovered that emotional response is individual. Older volunteers react differently than their younger group. The older volunteers try to control the feeling. “This is an excellent finding for us because now we know what kind of feature we will be trying to look at for people who have the training to control their anger and who don’t have the training to control their anger,” Chiao said. “Our next step is trying to figure out how we prioritize different factors so we can identify the signature we are looking for.”

72% of all murder-suicides involve an intimate partner. 94% of the victims of these murder-suicides are female.

It is difficult to conclude this kind of neuroscience project in a short time, Chiao said. “The team is aiming to discover features from preliminary studies to help the engineer to develop hardware and software that can be used in a more comfortable and convenient way for a larger population of people in order to collect statistically meaningful data,” he said.

Chiao envisions the hardware as something light that a person can put on easily and wear all day. “Our team has been working on flexible devices and better electrodes to detect the signals in a convenient and comfortable way,” he said.

The team also is working on the software development to a prototype stage so the brain signal analysis can be done in a smart portable device.

1 in 15 children are exposed to intimate partner violence yearly

It turned out that the diverse team members have not only similar research methodologies but also similar outgoing personalities.

“Our research discussion meetings often are full of laughs and jokes, making a tough research project fun and progress quickly,” Chiao said. “Also, we have regular research meetings, and our students from engineering, education, psychology, and social work working in the same lab. This became an opportunity to fuse multi-disciplinary fields together. So the social work students are learning about high-tech tools; psychology students are learning about big data analysis; and education students are learning about the brain. It became another benefit.

“I think the next generation of engineers, educators, social workers, and psychologists will need to know about the other sides,” he said. 

* Facts courtesy of the National Coalition Against Domestic Violence