escudero / brandpose

# BrandPose: where shoppers look, from an ordinary security camera
🌐 **English** · Português (Brasil)
> Personal project by **Claudio Escudero** (linkedin.com/in/claudioescudero). Not a product.
The question behind the project: **can you estimate where each shopper looks in a store using only an ordinary security camera and open source AI models?**
The starting point was this video from CactusXR (Auki). In it, store cameras produce an attention heatmap over the shelves. This repository tries to reproduce the idea from scratch and documents the path, including what worked and what went wrong.
> 📹 **Video used in the demo:** public demo of the ACTi E43A security camera in a US convenience store (YouTube, 60 s, 1080p, fixed camera). All processing below was done on this video. Footage © ACTi Corporation.
## Result
🎬 **Demo video (42 s):** watch on YouTube · download from the release
| Gaze estimated frame by frame | The scene in 3D (from a single image) | Attention report |
|---|---|---|
| | | |
Over 60 s from a real security camera in a convenience store:
| Zone | Attention | % | People |
|---|---|---|---|
| Candy and chocolate | 16.1 s | 32% | 8 |
| Sunglasses and accessories | 15.8 s | 31% | 10 |
| Back-wall snacks | 7.2 s | 14% | 7 |
| Coffee and juice | 2.8 s | 5% | 6 |
| Endcap | 2.5 s | 5% | 2 |
That is 797 estimated gazes, 17 shoppers and 9 zones with attention. These numbers are **model estimates**, with no manual ground truth (see the limitations).
## How it works
 [code] 
1. **People:** YOLO11-pose finds each person, the 17 body keypoints and an ID that follows the person frame by frame (ByteTrack, on every frame).
2. **Gaze:** Gaze-LLE (CVPR 2025) takes a **square crop around each head** and returns a heatmap of where the person is looking, 10 times per second.
3. **Rules:** the gaze point is smoothed over time and assigned to a store zone. Gazes that are not shelf attention are discarded (see below).
4. **3D:** Depth Anything V2 estimates the depth of one frame, which becomes a point cloud of about 500 thousand points.
5. **Video:** OpenCV, NumPy and Pillow draw each frame, FFmpeg encodes it, and the soundtrack is synthesized in code.
Details of each stage, with libraries, versions, formats and timings, are in `docs/arquitetura.md` (in Portuguese).
## What I learned along the way
| Finding | Evidence | What changed |
|---|---|---|
| Squeezing the whole frame into 448×448 blinds the model | A 58 px head reached the model at about 13×23 px | Per-person crop: head 3.5 times larger and undistorted |
| The tracker lost people when running at 3 frames per second | At t=24 s, 4 people were detected but only 2 tracked | Tracking on every frame |
| **Gaze leaving the image sticks to the edge** | 34% of peaks were less than 24 px from the edge | A peak less than 32 px from the edge counts as "out of frame" |
| **Gaze landing on other people became "shelf attention"** | 25% of gazes fell on someone's silhouette | A target on a person counts toward