In the second phase of his project research, our intern, Atharva set out to push the boundaries of human-computer interaction by leveraging MediaPipe, Google’s advanced computer vision framework. His project aimed to explore how technology can seamlessly integrate with fitness to encourage physical activity among adults. By combining motion tracking and interactive design, the research focused on creating engaging, tech-driven experiences that inspire movement and redefine the relationship between technology and well-being.

LOCATION:
We considered various locations, including parks and gyms, but chose the Equilibrium Climbing Station in Anjuna for its dynamic climbing environment, which suited our interactive setup.

THE SETUP:
The setup was minimal, using just a computer, projector, and HD webcam.

CONCEPTS:
We created concepts focused on keeping climbing central, with engaging visuals and games to challenge climbers. Our goal was real-time interactivity, making the experience fun, challenging, and shared.

DEVELOPMENT:

Route Plexus:
We created a visual representation of climbers' paths on the wall, called Route Plexus. It served two purposes: helping climbers easily navigate to the nearest hold and visualizing the growing network with each climb.

To map the holds, we used Python's OpenCV library for contour detection, capturing an image of the area and marking each hold. The data was then output in a table format with IDs for each hold. It was further connected with skeletal data from MediaPipe.

Climb-Vision
Climbers often lost focus and energy while navigating routes due to fading colour-coded holds. To address this, we introduced real-time projected markers to the climbing routes, providing a visual aid that helped climbers navigate more efficiently.

C-Limbs
Using MediaPipe’s skeletal structure visualization, we projected climbers' real-time skeletons beside them. This fun visual also served as a tool for analyzing their body positions and movement.

PRELIMINARY FACTORS:

Now that we had conducted several experiments, these specific factors were considered before developing new things further.

GAMIFYING THE CLIMB:

Pingpong – Multiplayer: Climbers race to keep up and protect their side of the wall, testing speed and focus.
Gravity Grab – Single-player: Climbers collect falling balls while ascending, testing agility and focus. Both games motivate climbers to push their limits and return for more.

These games provide a sense of accomplishment with every climb, motivating participants to challenge their limits and return for more.

DEVELOPMENT:

The setup for the games was simple, using a projector and an HD webcam paired with TouchDesigner. To configure the input, the canvas size was determined based on the wall size and projector placement during the recce. MediaPipe tracked body positions through the camera feed, outputting position data on a scale of 0 to 1.

The Pingpong development process involved creating an interactive climbing game by mapping real-world movements to virtual gameplay. Starting with the precise positioning of the ball in the virtual space, we translated hand movements using MediaPipe’s skeletal tracking. A Python script detected hand-ball intersections, triggering gameplay events. Ball physics were simulated to match the climbers’ ascent, and adjustments were made to prevent deflection issues. The climbing field was divided into player areas based on hip positions, and barriers were added to create dynamic win conditions. The final system integrated these elements, providing an engaging, interactive experience that combined climbing and gaming seamlessly.

The Gravity Grab development used a similar method to the Pingpong game for visualizing hands in screenspace. Elements were spawned using four timers, with each cycle determining the time it took for elements to reach the bottom. The spawn locations along the x-axis were randomized for suspense. To ensure elements were within reach of the climber, hip position data was used to adjust the spawn range based on their handspan. The score updated whenever a player’s hand intersected with a falling element. This created an engaging and dynamic climbing experience by combining real-time hand tracking with interactive, randomized gameplay.