{"id":7397,"date":"2021-01-27T13:15:39","date_gmt":"2021-01-27T13:15:39","guid":{"rendered":"http:\/\/ismiletechnologies.com\/?p=7397"},"modified":"2022-12-09T00:43:46","modified_gmt":"2022-12-08T19:13:46","slug":"facial-emotion-recognition-in-neuromarketing","status":"publish","type":"post","link":"https:\/\/ismiletechnologies.com\/en_us\/dataops\/facial-emotion-recognition-in-neuromarketing\/","title":{"rendered":"Facial Emotion Recognition in Neuromarketing"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Traditional marketing research tools like surveys and focus groups<\/span>\u00a0are\u00a0<\/span>used to evaluate customer response<\/span>s<\/span>\u00a0on various aspects of the product,\u00a0<\/span>but\u00a0<\/span>they\u00a0<\/span>do not fully capture what’s going on in the customer’s mind. Responses are not always true in s<\/span>ome<\/span>\u00a0cases. This is where the idea of Neuromarketing\u00a0<\/span>becomes<\/span>\u00a0use<\/span>ful<\/span>. Using non-verbal consumer responses and brain signals\u00a0<\/span>is a<\/span>\u00a0more reliable\u00a0<\/span>method\u00a0<\/span>as they are spontaneous and aren’t altered by the conscious mind. The widely used methods are – eye tracking, facial emotion detection, EEG data, and fMRI signals.\u00a0<\/span>\u00a0<\/span><\/p>

In this blog, we’ll talk about\u00a0<\/span>Facial Emotion Detection.<\/span><\/b>\u00a0<\/span><\/p>

Facial Emotion Recognition:<\/span><\/b> <\/b>Facial expressions and emotions are the most prominent of non-verbal communication. A lot of things that cannot be conveyed by words can easily be expressed using the face.\u00a0<\/span>\u00a0<\/span>The most prominent 7 emotions are<\/span>\u00a0–\u00a0<\/span>ang<\/span>er<\/span>, disgust, fear, happiness, sad<\/span>ness<\/span>, surprise<\/span>d<\/span>, and neutral.\u00a0<\/span>We experimented with 2 of the most famous machine learning models: CNN and SVM.<\/span>\u00a0<\/span><\/p>

  1. CNN<\/span><\/b> : <\/span>Convolutional Neural Networks are known to work well for image analysis tasks. Before building the model, we need to prepare the data for the network.\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Dataset:<\/span><\/b> \u00a0<\/span>The dataset is acquired from Kaggle and consists of the 7 classes (0=Angry, 1=Disgust, 2=Fear, 3=Happy, 4=Sad, 5=Surprise, 6=Neutral)\u00a0<\/span>mentioned above. There are 13,690 \u2018350×350\u2019\u00a0<\/span>pixel-colored<\/span> images of faces exhibiting 7 different expressions.<\/span><\/li><\/ol>

    Distribution of images amongst the classes –<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    N<\/span>otice that the dataset has\u00a0<\/span>several<\/span>\u00a0more images for the happiness and neutral class<\/span>es<\/span>.\u00a0\u00a0<\/span>\u00a0<\/span><\/p>

    Sample image from the dataset –<\/span><\/p>

    Pre-Processing:<\/span><\/b>\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0
    <\/span>The following pre-processing is performed on the dataset:<\/span>\u00a0<\/span><\/p>

    1. The facial emotion is independent of the color of the image.\u00a0<\/span>So,<\/span> we can change the image from RGB to grayscale thus reducing the number of channels. This results in an overall reduction in data by a fraction of 3.<\/span><\/li>
    2. The extra background can add noise and misleading information to the input, so extract only the facial part using Cascade Classifier (CV API) and set the image to a fixed size of 48X48 pixels to maintain consistency.<\/span><\/li><\/ol>

      Our emotion data is categorical. We perform one-hot encoding which converts the labels into vectors containing 0<\/span><\/span>\u2019<\/span><\/span>s and a 1 in the index position of the class<\/span><\/span>.<\/span><\/span>\u00a0<\/span>
      <\/span>\u00a0<\/span>
      <\/span>One hot encoding:<\/span><\/span>\u00a0<\/span>
      <\/span><\/span>Now, the data is ready to be used to train the model.<\/span><\/span>\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

      \n\t\t\t\t\t\t
      \n\t\t\t\t\t
      \n\t\t\t
      \n\t\t\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      Architecture:<\/span><\/b> \u00a0<\/span>The architecture is built using the\u00a0<\/span>Keras<\/span>\u00a0framework.<\/span>\u00a0<\/span><\/p>

      Input:<\/span><\/b>\u00a048×48 grayscale image<\/span>\u00a0<\/span><\/p>

      Output:<\/span><\/b>\u00a0<\/span>T<\/span>he probability of each expression class<\/span>\u00a0<\/span><\/p>

      The network architecture comprises of:<\/span>\u00a0<\/span><\/p>

      • 5 convolutional layers<\/span>\u00a0<\/span><\/li>
      • 3 sub-sampling layers and\u00a0<\/span>\u00a0<\/span><\/li>
      • 1 fully-connected layer.<\/span><\/li><\/ul>

        Layers:<\/span><\/b>\u00a0<\/span><\/p>

        1. The first layer of CNN is a convolution layer that applies a convolution kernel of 3<\/span>\u00a0<\/span>\u00d7<\/span>\u00a0<\/span>3 and outputs 64 images of 48 x 48 pixels. This is followed by a sub-sampling layer that uses max-pooling (with kernel size 3 \u00d7<\/span>\u00a0<\/span>3) to reduce the image to the third of its size.\u00a0<\/span>\u00a0<\/span><\/li><\/ol>
          1. The second convolutional layer has an output of 64 images of 16 \u00d716 pixels, followed by a sub-sampling layer with kernel size 2 \u00d7<\/span>\u00a0<\/span>2.\u00a0<\/span>\u00a0<\/span><\/li><\/ol>
            1. The third layer is the same as the second.<\/span>\u00a0<\/span><\/li><\/ol>
              1. The fourth convolutional layer outputs 128 images of size 8 \u00d7<\/span>\u00a0<\/span>8 pixels and uses max pooling with kernel 2 \u00d7<\/span>\u00a0<\/span>2.\u00a0<\/span>\u00a0<\/span><\/li><\/ol>
                1. The fifth layer is the same as the fourth.\u00a0\u00a0<\/span>\u00a0<\/span><\/li><\/ol>
                  1. The output from the last convolutional layer is given to a fully connected hidden layer. This layer maps 1024 nodes to 7 output nodes (one for each expression that outputs their probability) in a fully connected fashion.\u00a0<\/span>\u00a0<\/span><\/li><\/ol>

                    \u00a0<\/span>\u00a0<\/span>\u00a0<\/span><\/p>

                    Hyper-parameter Tuning:<\/span><\/b>\u00a0<\/span><\/p>

                    Multiple values for the hyper-parameters were tested and the following gave us the best accuracies:<\/span>\u00a0<\/span><\/p>

                    • Optimizer<\/span><\/b>\u00a0\u2013 Adam with a\u00a0<\/span>learning rate<\/span><\/b>\u00a0\u2013 1e-3 and a\u00a0<\/span>decay<\/span><\/b>\u00a0of 3.125e-5<\/span>\u00a0<\/span><\/li><\/ul>
                      • Batch size\u00a0<\/span><\/b>\u2013 32<\/span>\u00a0<\/span><\/li>
                      • Loss<\/span><\/b>– Categorical cross-entropy<\/span>\u00a0<\/span><\/li><\/ul>

                        \u00a0<\/span><\/p>

                        Results:<\/span><\/b>\u00a0<\/span><\/p>

                        The ratio of samples in the training set, validation set and test set<\/span>\u00a0is 6.8: 2.9: 0.2<\/span><\/b>\u00a0<\/span><\/p>

                        The CNN model achieved a test accuracy of<\/span>\u00a085.15%<\/span><\/b>\u00a0<\/span><\/p>

                        \u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t\t\t

                        2.\u00a0<\/span><\/span>SVM<\/span><\/span>\u00a0<\/span>
                        <\/span>\u00a0<\/span>
                        <\/span>A Support Vector Machine (SVM) finds hyperplanes in high-dimensional space to classify the data points.<\/span><\/span>\u00a0<\/span><\/p>

                        \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 <\/span><\/span>Here H3 is the best hyperplane<\/span><\/span>\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t\t\t

                        Dataset:<\/span><\/b>\u00a0<\/span><\/p>

                        We use another dataset for our SVM model.\u00a0<\/span>There are 1071 images in the dataset which are divided into 803 and 268 images for training and testing dataset. It has 6<\/span>\u00a0classes (<\/span>Anger, Disgust, Happiness, Sadness, Surprise<\/span>d<\/span>, Neutral)<\/span>.<\/span>\u00a0<\/span><\/p>

                        The previous dataset was\u00a0<\/span>very\u00a0<\/span>skewed which could cause biases in prediction. We chose a much lesser skewed dataset.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t\t\t

                        Pre-processing:<\/span><\/b>\u00a0<\/span><\/p>

                        In deciding the facial expression, the points on the eyebrows, eyes, nose, lips, and jawline are of the utmost importance. All these points are extracted and are interpolated from the nose-tip. The distance of these points (green lines) from the nose-tips are used as features for the SVM model.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t\t\t

                        Model architecture<\/span><\/b>\u00a0<\/span><\/p>

                        The API Scikit-learn is used to implement SVM with C=1 and linear kernel.<\/span>\u00a0<\/span><\/p>

                        Results:<\/span><\/b>\u00a0<\/span><\/p>

                        The test accuracy was found to be 61.69<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t\t\t

                        Conclusion:<\/span><\/b>\u00a0<\/span><\/p>

                        CNN gives us a better accuracy of 85.15%. It is worth noting that it is also more complex and computationally expansive than the SVM model.<\/span>\u00a0<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t
                        \n\t\t\t
                        \n\t\t\t\t\t\t
                        \n\t\t\t\t\t\t\t
                        \n\t\t\t\n\t\t\t
                        \n\t\t\t\t\t\t\t\t\t
                        \n\t\t\t\t\t\t

                        \n\t\t\t\t\t\t\tSneha Bahl\t\t\t\t\t\t<\/h4>\n\t\t\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\t\t\t\t\t
                        \n\t\t\t\t\t\t

                        Data Scientist Intern<\/p>\n\t\t\t\t\t<\/div>\n\t\t\t\t\n\t\t\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

                        Traditional marketing research tools like surveys and focus groups\u00a0are\u00a0used to evaluate customer responses\u00a0on various aspects of the product,\u00a0but\u00a0they\u00a0do not fully capture what’s going on in the customer’s mind. Responses are not always true in some\u00a0cases. This is where the idea of Neuromarketing\u00a0becomes\u00a0useful. Using non-verbal consumer responses and brain signals\u00a0is a\u00a0more reliable\u00a0method\u00a0as they are spontaneous and […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[249],"tags":[95],"class_list":["post-7397","post","type-post","status-publish","format-standard","hentry","category-dataops","tag-cloud"],"_links":{"self":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/7397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/comments?post=7397"}],"version-history":[{"count":4,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/7397\/revisions"}],"predecessor-version":[{"id":36049,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/posts\/7397\/revisions\/36049"}],"wp:attachment":[{"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/media?parent=7397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/categories?post=7397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ismiletechnologies.com\/en_us\/wp-json\/wp\/v2\/tags?post=7397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}