︎ “Haunting Eyes: A Gesture-Triggered ESP32 Animatronic Display”

Abstract

Haunting Eyes brings life to two LCD eyes powered by an ESP32-S3. Using gesture control and sound effects, the eyes blink, move, and react to nearby motion—creating a creepy yet fascinating interactive display.

️ Haunting Eyes: A Gesture-Triggered ESP32 Animatronic Display

Why I Made This Project

I wanted to build something that’s both fun and a little spooky — a pair of eyes that feel alive. Using the ESP32-S3, round GC9A01A displays, and a gesture sensor, I designed a small animatronic setup that follows movement and reacts with eerie sound effects.

️ Components Used

Component	Description
ESP32-S3 Maker Feather AIoT S3	Main controller board
2 × GC9A01A 240×240 Round TFT Displays	Displays for the eyes
APDS9960 Gesture & Proximity Sensor	Detects hand movement
WT2605C Serial MP3 Player	Plays sound effects
Micro SD Card	Stores MP3 audio files
5V Power Source	Supplies power to all modules
Jumper Wires & Breadboard	For prototyping connections

Circuit Connections

Module	Pin	ESP32-S3 Pin
Left Eye (TFT)	CS	14
Right Eye (TFT)	CS	21
Both Eyes	DC	9
Both Eyes	RST	6
Gesture Sensor (APDS9960)	SDA	42
Gesture Sensor (APDS9960)	SCL	41
MP3 Player	RX	40
MP3 Player	TX	39

All modules share common GND and 3.3V lines.

️ Wiring Diagram

Since both the lcd are sharing the same pins, except for the CS pin, I have created simple connection board that can connect both the lcd.

And also i have make simple interfacing board to connect the Maker Feather ESP32 to the Seeed Grove MP3 module and LCD module. The another grove port just the extra port.

Programming the ESP32-S3

Required Libraries

Adafruit_GFX
Adafruit_GC9A01A
Adafruit_APDS9960
Seeed Serial MP3 Player (WT2605C)

Main Features

Realistic eye animation — smooth blinking, dilation, and random movement
Gesture-triggered “fast eye” reaction
MP3 sound playback using WT2605C
Automatic sound fade-out after inactivity

Code

#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_GC9A01A.h>
#include <Adafruit_APDS9960.h>
#include <math.h>
#include "WT2605C_Player.h"

// ==== Pin configuration (ESP32-S3 Maker Feather AIoT S3 defaults) ====
#define TFT_DC    9
#define TFT_RST   6
#define TFT_CS_L 14
#define TFT_CS_R 21

// I2C pins for ESP32-S3
#define I2C_SDA  42
#define I2C_SCL  41

// MP3 module pins (WT2605C)
#define MP3_RX 40   // ESP32 RX <- MP3 TX
#define MP3_TX 39  // ESP32 TX -> MP3 RX
#define MP3_BAUD 115200

Adafruit_GC9A01A tftL(TFT_CS_L, TFT_DC, TFT_RST);
Adafruit_GC9A01A tftR(TFT_CS_R, TFT_DC, TFT_RST);
Adafruit_APDS9960 apds;
GFXcanvas16 canvas(240, 240);

// ==== Eye geometry ====
#define EYE_CENTER_X 120
#define EYE_CENTER_Y 120
#define EYE_RADIUS   105
#define IRIS_RADIUS  50
#define PUPIL_RADIUS 28
#define MOVE_X       25
#define MOVE_Y       18

// ==== Eye state ====
float pupilX = 0, pupilY = 0, targetX = 0, targetY = 0;
float pupilScale = 1.0;
bool  dilating = true, blinking = false;
uint32_t blinkStart = 0, nextBlink = 0;
const int blinkDuration = 320;
uint32_t lastMove = 0;
int moveInterval = 1200;

// ==== Shimmer & glint ====
float shimmerPhase = 0.0f;
const float shimmerSpeed = 0.035f;
const uint8_t ringCount = 5;

// ==== Gesture + proximity ====
uint32_t lastGestureMs = 0;
const uint16_t GESTURE_DEBOUNCE_MS = 120;
const uint8_t  PROX_BLINK_THRESHOLD = 60;

// ==== MP3 Player ====
#define COMSerial Serial1
#define ShowSerial Serial
WT2605C<HardwareSerial> Mp3Player;
bool isPlaying = false;
uint8_t currentVolume = 20;
uint32_t lastGestureDetected = 0;
const uint32_t GESTURE_TIMEOUT_MS = 4000; // stop music after 4s inactivity
bool isFadingOut = false;
uint32_t fadeStartTime = 0;
const uint32_t FADE_DURATION_MS = 2000; // fade-out over 2 seconds

// ---- Helper ----
static inline uint16_t modulateRed(uint8_t baseR, float gain01) {
  gain01 = constrain(gain01, 0, 1);
  uint8_t r = (uint8_t)(baseR * gain01);
  return ((r & 0xF8) << 8);
}

static inline float clampf(float v, float lo, float hi) {
  return (v < lo) ? lo : (v > hi) ? hi : v;
}

// ---- Draw single eye ----
void drawEye(GFXcanvas16 &gfx, float offsetX, float offsetY, float blinkFactor) {
  gfx.fillScreen(GC9A01A_BLACK);
  gfx.fillCircle(EYE_CENTER_X, EYE_CENTER_Y, EYE_RADIUS, GC9A01A_WHITE);
  gfx.fillCircle(EYE_CENTER_X + offsetX, EYE_CENTER_Y + offsetY, IRIS_RADIUS, GC9A01A_RED);

  for (uint8_t i = 0; i < ringCount; i++) {
    float t = shimmerPhase + i * 0.6f;
    float gain = 0.9f + 0.1f * (0.5f + 0.5f * sinf(t * 2.0f));
    uint16_t c = modulateRed(255, gain);
    int r = IRIS_RADIUS - 2 - i * 4;
    if (r > 4) gfx.drawCircle(EYE_CENTER_X + offsetX, EYE_CENTER_Y + offsetY, r, c);
  }

  gfx.fillCircle(EYE_CENTER_X + offsetX, EYE_CENTER_Y + offsetY,
                 (int)(PUPIL_RADIUS * pupilScale), GC9A01A_BLACK);
  gfx.fillCircle(EYE_CENTER_X + offsetX - 12, EYE_CENTER_Y + offsetY - 12, 5, GC9A01A_WHITE);

  float glintPhase = shimmerPhase * 0.55f;
  float gx = EYE_CENTER_X + offsetX + cosf(glintPhase) * (IRIS_RADIUS - 6);
  float gy = EYE_CENTER_Y + offsetY + sinf(glintPhase) * (IRIS_RADIUS - 6);
  gfx.fillCircle((int)gx, (int)gy, 3, GC9A01A_WHITE);

  if (blinkFactor > 0.0f) {
    int h = (int)((EYE_RADIUS + 30) * blinkFactor);
    if (h > 240) h = 240;
    gfx.fillRect(0, 0, 240, h, GC9A01A_BLACK);
    gfx.fillRect(0, 240 - h, 240, h, GC9A01A_BLACK);
  }
}

// ---- MP3 control ----
void playGestureSound() {
  if (!isPlaying) {
    Mp3Player.playSDRootSong(1);  // start track 0001.mp3
    Mp3Player.volume(currentVolume);
    isPlaying = true;
    isFadingOut = false;
    ShowSerial.println("🎵 Gesture detected → Playing sound continuously");
  }
  lastGestureDetected = millis();
}

void handleFadeOut() {
  if (!isFadingOut) {
    isFadingOut = true;
    fadeStartTime = millis();
  }

  uint32_t elapsed = millis() - fadeStartTime;
  if (elapsed < FADE_DURATION_MS) {
    // compute fading volume
    float fadeProgress = 1.0f - (elapsed / (float)FADE_DURATION_MS);
    uint8_t newVol = (uint8_t)(currentVolume * fadeProgress);
    Mp3Player.volume(newVol);
  } else {
    // fully faded out
    Mp3Player.pause_or_play(); // pause/stop
    Mp3Player.volume(currentVolume); // restore volume for next time
    isPlaying = false;
    isFadingOut = false;
    ShowSerial.println("🔇 Fade-out complete, music stopped");
  }
}

void stopGestureSoundIfIdle() {
  if (isPlaying && (millis() - lastGestureDetected > GESTURE_TIMEOUT_MS)) {
    handleFadeOut();
  }
}

// ---- Setup ----
void setup() {
  Serial.begin(115200);
  delay(50);
  Serial.println("Scary Eyes + Gesture + WT2605C MP3 (with fade-out)");

  tftL.begin(); tftR.begin();
  tftL.setRotation(2); tftR.setRotation(2);
  tftL.fillScreen(GC9A01A_BLACK);
  tftR.fillScreen(GC9A01A_BLACK);

  // I2C setup for APDS9960
  Wire.begin(I2C_SDA, I2C_SCL, 400000);
  if (!apds.begin(10, APDS9960_AGAIN_4X, 0x39, &Wire)) {
    Serial.println("❌ APDS9960 init failed");
  } else {
    Serial.println("✅ APDS9960 ready");
    apds.enableProximity(true);
    apds.setProxGain(APDS9960_PGAIN_2X);
    apds.enableGesture(true);
  }

  // MP3 setup
  COMSerial.begin(MP3_BAUD, SERIAL_8N1, MP3_RX, MP3_TX);
  Mp3Player.init(COMSerial);
  Mp3Player.volume(currentVolume);
  Serial.println("✅ WT2605C ready");

  randomSeed(analogRead(0));
  nextBlink = millis() + random(2500, 5000);
}

// ---- Main Loop ----
void loop() {
  uint32_t now = millis();

  // Gesture handling
  uint8_t g = apds.readGesture();
  if (g && (now - lastGestureMs > GESTURE_DEBOUNCE_MS)) {
    lastGestureMs = now;
    playGestureSound(); // continuously plays during gesture activity

    switch (g) {
      case APDS9960_LEFT:  targetX -= 6; break;
      case APDS9960_RIGHT: targetX += 6; break;
      case APDS9960_UP:    targetY -= 5; break;
      case APDS9960_DOWN:  targetY += 5; break;
    }
    targetX = clampf(targetX, -MOVE_X, MOVE_X);
    targetY = clampf(targetY, -MOVE_Y, MOVE_Y);
    moveInterval = 900;
    lastMove = now;
  } else {
    stopGestureSoundIfIdle();
  }

  // Proximity-triggered blink
  uint8_t prox = apds.readProximity();
  if (!blinking && prox >= PROX_BLINK_THRESHOLD) {
    blinking = true;
    blinkStart = now;
  }

  // Random gaze
  if (now - lastMove > moveInterval) {
    targetX = random(-MOVE_X, MOVE_X);
    targetY = random(-MOVE_Y, MOVE_Y);
    moveInterval = random(900, 1500);
    lastMove = now;
  }

  // Smooth motion
  pupilX += (targetX - pupilX) * 0.08f;
  pupilY += (targetY - pupilY) * 0.08f;

  // Dilation
  pupilScale += (dilating ? 0.006f : -0.006f);
  if (pupilScale > 1.25f) { pupilScale = 1.25f; dilating = false; }
  if (pupilScale < 0.80f) { pupilScale = 0.80f; dilating = true;  }

  // Blinking
  float blinkFactor = 0.0f;
  if (!blinking && now > nextBlink) {
    blinking = true; blinkStart = now;
  }
  if (blinking) {
    float t = (now - blinkStart) / (float)blinkDuration;
    if (t >= 1.0f) {
      blinking = false;
      nextBlink = now + random(3000, 6000);
    } else blinkFactor = sinf(t * M_PI);
  }

  shimmerPhase += shimmerSpeed;

  // Draw eyes
  drawEye(canvas, pupilX, pupilY, blinkFactor);
  tftL.drawRGBBitmap(0, 0, canvas.getBuffer(), 240, 240);
  drawEye(canvas, -pupilX, pupilY, blinkFactor);
  tftR.drawRGBBitmap(0, 0, canvas.getBuffer(), 240, 240);

  delay(16);
}


Prior for the complete code, each of the devices are tested separately to ensure that everything is working fine.

How It Works

When powered on, the eyes come alive — blinking, dilating, and looking around randomly. When you move your hand near the APDS9960 sensor, the ESP32 instantly:

Speeds up eye movement — giving the illusion of awareness ️
Plays a spooky sound via the MP3 module
Gradually fades out the sound when no gesture is detected

The result is an interactive, lifelike pair of eyes that seem to watch you in real time!

Step-by-Step Build Instructions

1. 3D Printing
The first step that i did in building this project is deciding the model of the skeleton. I decided to 3D print the skeleton. I have used the design from SF3DPrints. The model can be downloaded from here.

Below are some of the pictures and videos of the 3D printed skeleton

{gallery}3D Printed Skeleton

Next, i removed all the supports and check if the 3D printed model looks fine. Next make the cut the skull to half.

2. Assembling the part
First i try to fix the lcd to the eye socket. I have to make some cutouts before i can fix the lcd to eye socket and i have use hot glue gun to hold the lcd at its place

Next i have place the APDS9960 sensor inside the nose hole as can be seen in the picture

Once it is in place i have use the hot glue gun as well to hold it in place. And i have placed all the components inside the skull

3. Upload the sketch.

Select ESP32S3 Maker Feather and correct COM port, then upload your .ino file. I have few testing on the display and the sensor

4. Insert the SD card.
Place your spooky sound file as 0001.mp3 in the SD card root. I have downloaded the sound from here. There a lots of sound affect available here.

5. Power it up.
The eyes will begin blinking and scanning. Wave your hand — the motion triggers the sound and rapid movement!