Abstract:

In this project, we try to determine the emotion of a person based on their speech. Generally, this is done via signal processing on the audio features like pitch, mel-frequency cepstral coefficients, the intensity of the speech signal, and the speech rate of recorded speeches. For our project, the typical features used for the processing of the audio are the pitch and the sound pressure level (SPL). Along with the audio we are also using the text spoken to determine the sentiment being portrayed. Different classification techniques based like k-means clustering, Naïve Bayes, perceptron and k-nearest neighbor were used to try and obtain the optimal classification.

Introduction:

An emotion is a particular feeling that characterizes a state of mind, such as joy, anger, love, fear etc. Understanding emotions of people is a big factor in our day-to-day social life. The emotional information conveyed through speech is an important factor in human–human interaction and communication. Humans feel most natural communicating with other humans because the extra information represented in their emotional expressions can be recognized, processed, and reflected. Hence, when humans interact with computer systems, there is a gap between the information conveyed and the information perceived.The automatic identification of emotions in speech & texts is important for many applications. Most important of these areas are emotionally intelligent systems in call-centers or automated telephone systems. Call-centers often have a difficult task of managing customer disputes, where ineffective resolution of these disputes can often lead to customer discontent, and loss of business. Recognition of emotion in the spoken dialogue can help process calls according to perceived urgency. If a caller is detected as being angry or confused in the automated system, their call can be switched over to a human operator for assistance. Some other areas where it can be useful are: market analysis, effective computing and e-learning environments, including educational games.

Data Collection:

Speech using simulated emotional expression was obtained from the Toronto Emotional Speech Set (TESS) from the University of Toronto. In their experiment a set of 200 target words were spoken in the carrier phrase “Say the word _____” by two actresses (aged 26 and 64 years) and recordings were made of the set portraying of seven different emotions viz., Happy, Sad, Surprise, Fear, Neutral, Angry and Disgust. There are 2800 stimuli in total.

To classify the sentiment of transcribed speech, a dataset of 30000 movie reviews provided for an assignment in the NLP class was used to train our model. Along with the reviews, a lexicon of words marked as positive and negative were added to increase the accuracy for the Averaged perceptron algorithm.

Flow Diagram:

Classification Process Flow

The process of the emotion detection is done in two parts:
1. The Sentiment Analysis on the text uses Perceptron classifier and returns a tag (POS/NEG) which is added to the feature vector.
2. Using kNN classifier to classify the modified feature vector and find the emotion.

Feature Extraction:

Praat (the Dutch word for "talk" or "speak") is a free scientific computer software package for the analysis of speech in phonetics. It was designed, and continues to be developed, by Paul Boersma and David Weenink of the University of Amsterdam. It is a very flexible tool to do speech analysis. It offers a wide range of standard and non-standard procedures, including spectrographic analysis, articulatory synthesis, and neural networks. The two main features of a human speech are pitch and sound pressure level (SPL). These features are very likely to get influenced by the human emotions. Emotions such as Happy, Sad, Fear, Angry and Surprise, have different permutations of values of pitch and SPL. To make use of this heuristic knowledge, the Praat tool was used to extract these features out of the above mentioned datasets. To be more precise, minimum Pitch and SPL, maximum Pitch and SPL, mean SPL and standard deviation of SPL were extracted. Another important feature is the sentiment of the words being spoken in a conversation. The sentiment of the words spoken help classify the emotions more precisely. This feature was obtained by transcribing speech to text using the Speech Recognition library in python as a first step, and then classifying the sentence by its sentiment, Positive or Negative.

Feature Analysis:

To understand the feature set obtained from extraction using Praat, plots of the data were made using the programming language R used for statistical analysis. It is difficult to visualize any data beyond 3 dimensions, consequently, a number of plots were made using R. Some of the plots that were considered are as follows:

Minimum Pitch vs Samples

Maximum Pitch vs Samples

Maximum SPL vs Samples

Mean SPL vs Samples

Classification Techniques Analysis:

Analysis of features was performed using WEKA (Waikato Environment for Knowledge Analysis). WEKA is a data mining workbench that allows comparison between many different machine learning algorithms. In addition, it also has functionality for feature selection, data pre-processing, and data visualization.The selection of classifiers was done by evaluating several algorithms over the data set and selecting the top performers. It is this distinction that affords the selection of certain parameters to be obtained using only the feature set extracted from the audio parsing. For sentiment analysis, two different algorithms were evaluated, namely, Naive Bayes and Averaged Perceptron.

Text Classification:-

Averaged Perceptron

	Precision	Recall	F-Score
Positive	0.963	0.925	0.944
Negative	0.954	0.978	0.996

Naive Bayes

	Precision	Recall	F-Score
Positive	0.898	0.876	0.887
Negative	0.927	0.941	0.934

It was observed that the perceptron gave a better F-score than Naïve Bayes due to which we used Perceptron. The reason for the increase in F-score of the Negative tag is due to the higher number of files with that tag in the training set than the other, giving the model more data to analyze.

Emotion Classification:-
Initially the K-means clustering algorithm was used to try and classify the data, but due to high overlap of different values of different features across classes the use of the classifier was discarded. The analysis of running the dataset over multiple classifier algorithms using Weka is as shown below. The F-scores obtained in WEKA are as follows:

Emotions	Classifiers
	Naive Bayes	Multilayered Perceptron	Decision Trees	K-Nearest Neighbors
Angry	0.703	0.748	0.72	0.747
Disgust	0.628	0.781	0.779	0.778
Sad	0.673	0.901	0.903	0.918
Fear	0.34	0.645	0.618	0.659
Happy	0.671	0.895	0.895	0.89
Surprise	0.777	0.884	0.869	0.884
Neutral	0.976	0.989	0.995	1.0

As per the above analysis using Weka, k-Nearest Neighbor classifier was selected as the optimal classifier.

For kNN algorithm, to evaluate the performance of the classifier, 10-fold cross-validation accuracy was calculated. 10-fold cross-validation was performed with different values of k where the optimal k was found to be 5.

10-fold Cross Validation Accuracy

This figure below shows the accuracy over 5 iterations for different values of k (1, 2, 3, 4, 5, 7, 10).

Average 10-fold Cross Validation Accuracy

This figure after the following one shows the average accuracy over 5 iterations for different values of  k (1, 2, 3, 4, 5, 7, 10)

Future Work:

As per observation, there are further improvements that can be done with either classifier. The training set used for the sentiment analysis could be a rather more conversational data, in order to improve detection of the sentiment based on the text. Similarly, more features like jitter, rate of words spoken and energy entropy block can be used to better classify the emotions. Also, a way to go about it is to keep appending/adding data to the training set while testing the classifier.

Key Engineers:

- Shreya Jhajharia (sjhajhar@usc.edu)
- Yatharth Pandey (ypandey@usc.edu)
- Kedar Prabhu (kprabhu@usc.edu)