Week 05 | Tools & Technology #2 | Body Tracking | Working with MediaPipe

Monday, February 5, 2024

Room H003
Note that we are in a completely different classroom today! You can access the room by going through either the Mechatronics (G014) or Laser Cutter (H001) Workshops.

Inspiration #

This week will be our first introduction to working with tracking people and their bodily movements in a space. These are some examples I have been working on previously.

The projects below are from a time before the Kinect existed and these kinds of things were much harder to do. Couple of them use the basic blob tracking technique we will build today.

Animoitu Liike, 2009 #

Mortimer, 2009 #

Mortimer includes some scenes where we used real-time dynamic projection mapping. This was achieved using infrared camera and LEDs.

Mortimer Full Performance

Giants of the Hoods, 2010 #

Giants of the Hoods was a project where we went to different neighborhoods in Helsinki and Espoo and projected real-time animations on the facades of buildings. The characters were controlled by people dancing

People were also able to have themselves photographed and they could decide which part of their body they would lioke to donate to the “Giant” projected on the wall. The character basically ended up being a collage of the people who participated.

This was based on a similar setup used in the Reverse Shadow Theatre and used the Animata software.

{<vimeo 12910372>}}

Animoitu Liike, 2011 #

After the original Kinect was released, new experimental projects popped into existence everyday. This is my update on the Animoitu Liike project.

AHNE, 2011 #

Ahne was a research project created in the SOPI research group. It started out as an exploration to create spatial interface that would not have any visual feedback. The only feedback would be sound and haptics. The user would wear a glove that had a haptic feedback motor and wireless sensors.

Body Tracking (Pose Detection) #

Kinect (Azure) #

Pros:

Works in complete darkness since it uses infrared
Works directly in TouchDesigner
You also get depth image and point cloud

Cons:

Expensive sensor
Windows only
Has been discontinued

Although the Azure Kinect is discontinued by Microsoft, there are a couple of options that are almost direct replacements. Orbbec makes two cameras that use the samer Body Tracking models as the Kinect.

Femto Mega

Femto Bolt

In TouchDesigner, you can find the Kinect Azure TOP and CHOP that give you the image and tracking data from the Kincet Azure sensor.

ZED #

Pros:

Actively maintained and supported
Works directly in TouchDesigner
You also get depth image and point cloud

Cons:

Expensive sensor
Windows only

MediaPipe #

MediaPipe is the simplest option in the sense that it doesn’t require any special hardware and works with different operating systems.

Pros:

No special hardware needed (any webcam that is able to output a 720p image will work)
Does also much more beyond the pose tracking (face tracking, hand tracking, object tracking, image classification, image segmentation)
Multiplatform

Cons:

No true 3D information. The Z-coordinate is a relative coordinate based on the center-of-mass
The TouchDesigner plugin is sometimes buggy and doesn’t load the camera. Especially with the Aalto computers.

Workaround to get the camera image working with Aalto computers #

Some of the computers we have do not for some reason load the camera inputs correctly with the MediaPipe GPU plugin. If that is the case with you, use this workaround:

Open OBS Studio

Change the video settings to 1280x720/60 fps

Add a Video Capture Device into as the Source

Change the settings of the webcam to also be 1270x720/60fps

Scale the source image to the output by selecting it and pressing Ctrl+F (Fit to screen)

Turn on the Virtual Camera by pressing the Start Virtual Camera button.

You can also dowload the profile and scene files by clicking here. Once you have downloaded them, do the following:

Unzip the files

Select Profile -> Import

Select the mediapipeobs folder

Select Profile -> mediapipe

Select Scene Collection -> Import

Click the … to open the filebrowser

Select the mediapipe.json file that you downloaded

Select Scene Collection -> mediapipe