Gumball Emotion Machine
(GEM)
Roles
Main Developer
Secondary Designer
Tools
P5js
Arduino
Teachable Machine
Dates
Aug 2021 -Dec 2021
&
May 2022 - Aug 2022
Partner
Overview
Purpose: Despite being a class assignment, my partner and I wanted to make something meaningful and fun. He wanted to do something with machine learning and I wanted to connect to the user on a more personal level.
Everyone has a "comfort food" when they're feeling down and a snack they look forward to when they are happy. Thus the Gumball Emotion Machine, GEM for short, was born to help figure out one's emotional state and determine if one should indulge in a snack.
.png)
The current state of GEM: the system only identifies the "Happy" state of the user. Face recognition is a bit unreliable and needs improvement. The overall mechanism and code logic works as expected.
The end goal of GEM: have the system be fully capable of differing between "Happy" and "Sad" states and provide the appropriate candy based on the user's preferences. Also, create a completely independent system that can be used commercially in storefronts.
GEM utilizes the serial communication between P5js and Arduino:
Through facial recognition, Teachable Machine analyzes the emotional state
P5js translates the facial data into commands sent to Arduino.
Arduino instructs the machine to dispense candy if the user was determined to be "Happy".
Initiate
.png)
Sense
.png)
Dispense
.png)
Setup
.png)
.png)
.png)
​Mechanism Explanation:
-
The laptop displays the video feed. On the video feed is a virtual representation of the machine.
​
-
The circuit diagram (top image) hooks up a circuit board, a servo motor, and a NeoPixel light display.
​
-
The machine consists of:
-
compartment (top right) to hold the candy
-
a base
-
internal mechanism that rotates on a servo motor (bottom right image)
-
NeoPixel Light display
-
How It Works:
When a person is shown to be "Happy" in front of the camera that is represented synchronously by the bar on the virtual machine and by the pixel lights on the physical machine. Candy is dispensed when the bar reaches the smiley face. The servo motor is activated when this condition is met rotating the plate to an opening that allows the candy to fall through.
Issue Faced:
​
The mechanism would jam due to the low quality of the parts used and the candy getting stuck trying to fall through the opening. So the servo motor would get stuck and not reset after activating.
Teachable Machine
How It Works:
​
Teachable Machine is hosted by Google. Basically, how it works is that you provide the system with image samples for different labeled categories. For GEM, two main categories were set corresponding to a state of being "Happy" and a state of being "Sad". The system is then trained on those image samples using to an already established pool of big data as a comparison. When the training is complete, the system returns a model that is then connected to P5js to be used.
​
​
Issue Faced:
​
The system can easily distinguish between distinctive objects. However, the system has trouble identifying the minute differences in facial expressions. Therefore, it was hard for GEM to pick up traces of the "Happy" state being under certain conditions. More diverse image samples will be needed to improve the facial recognition system.
​