Mushroom
Rush

The Brief

Mushroom Rush is an interactive desktop experience that brings Latvia’s autumn mushroom foraging tradition to life using machine learning and custom animation. In Latvia, mushroom foraging is recognized as national intangible cultural heritage, where families rise before dawn to head into the forest every autumn weekend.

Instead of just explaining the tradition, I wanted users to feel it: the anticipation, the patience, and the rush of discovery.

This project was built primarily as a technical showcase. All visual assets like illustrations and character art were AI-generated so I could focus entirely on the code: training and integrating ML5 models, choreographing GSAP animation sequences, and building a six-state web application from scratch.

An animated grandmother character guides the user through the whole journey, covering cultural storytelling, mushroom identification, a classification quiz, and a real-time hand-gesture foraging hunt.

The Process

From Concept to Storyboard

Before opening Figma, I sketched the entire six-state experience by hand. Every scene transition, ScrollTrigger point, and GSAP plugin, down to Physics2D bouncing mechanics and MorphSVG animations — was mapped directly onto paper. Having that visual roadmap made designing the layout and writing the code seamless as the project shifted from scrollytelling into live AI hand tracking.

Testing the Hand-Gesture Model

Building the interactive hunt required controlled testing with ML5’s HandPose model. I ran tests across three different lighting conditions, four cursor-smoothing algorithms, and four pinch-detection thresholds (20px, 30px, 40px, and 50px).

Each threshold was tested with 20 intentional pinches to measure false positives versus missed inputs. I landed on a 30px pinch distance paired with a 2-second hover timer to confirm selection. This setup avoided accidental triggers while keeping the hand tracking feeling responsive and forgiving.

Training the Classifier

For the identification quiz, I trained a custom Teachable Machine model using real photos across three categories: common edible, rare edible, and poisonous mushrooms. I loaded the trained model using Teachable Machine’s tmImage library to run real-time predictions in the browser as users examine each mushroom.

Driving Animation via ML Data

The core concept of this build was letting machine learning outputs directly drive animation states instead of just printing text on screen.

For instance, picking a rare mushroom triggers a golden particle burst and +20 points. Picking an edible mushroom triggers green sparkles and +10 points, while picking a poisonous mushroom fires red warning particles, screen shake, and a 10-point penalty.

I used a similar visual approach for the scrollytelling section. As the user scrolls, a chanterelle morphs into a death cap using GSAP’s ScrollTrigger and MorphSVG. This animation coordinates seven separately timed elements including the cap shape, color, gills, stem, and five staggered warning callouts. The danger of mistaking a poisonous mushroom for an edible one is shown visually at the exact pace the user scrolls.

Technical Error Handling

Running a live webcam feed and ML tracking in the browser meant building for edge cases directly inside the render loop. I created dynamic, auto-dismissing notifications to guide the player without interrupting gameplay:

Each notification uses a 3-second timeout that automatically clears the alert once tracking stabilizes.

State management also required explicit memory cleanup across all six application states. When transitioning states, my cleanup routine stops hand detection loops, kills active media streams, and clears leftover canvas elements. It also kills every active GSAP timeline and resets global ScrollTriggers. Clearing these explicitly fixed early performance bugs where animation loops and event listeners doubled up across scene changes.

Final Result

Mushroom Rush is a six-state interactive experience combining cultural heritage with browser-based machine learning. It takes users from narrative storytelling into interactive mushroom education, an AI classification quiz, and a live hand-gesture foraging hunt.

What Worked

Connecting ML data outputs directly to feedback mechanics turned a standard classification task into a playful game. Tuning the HandPose model through physical testing paid off, making the 30px pinch threshold and camera controls feel natural without requiring tutorial screens.

What I’d Improve

All visual assets were generated with AI for speed. Given more time, I would illustrate the grandmother character and mushroom assets by hand so the visual polish matches the technical effort. I would also expand the classifier’s training dataset beyond three categories to reflect the true diversity of wild mushrooms.

The full experience is hosted online — try it live using the sidebar link. See the button below to try it live.