Week 05 | Tools & Technology #1 | Setting Up MediaPipe

Week 05 | Tools & Technology #1 | Setting Up MediaPipe

Monday, February 3, 2025

Room 3420 (Marsio)

This week, we are going to dive into the tools and technology we have available for actually creating embodied experiences.

Using TouchDesigner with the Aalto licenses #

You can download and use the non-commercial version on your own computer, but I would highly recommend working on the Aalto computers. We have a total 20 educational licenses available. The license is allocated to you when you launch the software. It is not possible to get the Aalto license on your own computer.

Sometimes the license manager does not work and it cannot connect to the the server. In those cases, it seems that restarting the computer usually seems to fix the issue.

Using the physical computers at Väre #

These classrooms have TouchDesigner installed. You can use any of them when there is space.

Using vdi.aalto.fi to connect to the Aalto computers remotely #

Log in to vdi.aalto.fi with your Aalto account and choose any of the rooms listed above and you should have access to a computer with TD installed.

This actually works really well and it’s very convenient! Can be used with a low-power machine like a laptop with much better performance. Also great for using on your Mac to access some of the Windows-only features.

Setting up MediaPipe GPU with TouchDesigner #

During the second half of the class, we will setup the MediaPipe GPU plugin for your TouchDesigner installations.

Torin Blankensmith and Dom Scott have created an excellent plugin for TouchDesigner that enables many of the tracking posibilities of MediaPipe straight in a very TD friendly formatting.

Add the MediaPipe files to Your TouchDesigner Palette #

Do the following in order to have an easy access to the components throughout the rest of the course:

  1. Download the plugin from the GitHub repository. Click Releases and download the latest release.zip file(v0.3.3 when writing this).
  2. Unzip the .zip archive somewhere on your computer.
  3. Open TouchDesigner.
  4. Find the Palette on the left side of the interface and scroll down to My Components.
  5. Right-click on My Components and choose Add Folder. Name it MediaPipe GPU.
  6. Right-click on this new folder you created and choose Browse Folder. It opens your File Explorer (Windows) or Finder(Mac) in a location where your custom paletette components are stored.
  7. Copy all of the .tox files from the toxes folder into the MediaPipe GPU folder you had created earlier.
  8. Go back to TouchDesigner. Right-click on top of the MediaPipe GPU folder in the Palette and choose Refresh Folder.
  9. You can open the MediaPipe TouchDesigner.toe file to see an example on how to to use some of this data. We will make our own experimentations next week.

Using MediaPipe in your own projects #

After you have added the components to your Palette, you have access to the .tox files easily. Just make sure you do the following step to not make your file size too large.

  1. Drag the main MediaPipe component to your project from the Palette
  2. Select the component and go to the Common page of the properties.
  3. Turn on Enable External .tox
  4. Add the path of the component to the External .tox Path to point to the location where your Palette folder is. Easisest way to do this is to go to the Palette and find the folder you created earlier (MediaPipe GPU), right click and select Browse Folder, drag the MediaPipe.tox into the text box for External .tox Path

Doing this makes sure that the component is properly loaded and you need to only have one copy of the component on your hard drive.

Workaround to get the camera image working with Aalto computers #

Some of the computers we have do not for some reason load the camera inputs correctly with the MediaPipe GPU plugin. If that is the case with you, use this workaround:

  1. Open OBS Studio
  2. Change the video settings to 1280x720/60 fps
  3. Add a Video Capture Device into as the Source
  4. Change the settings of the webcam to also be 1270x720/60fps
  5. Scale the source image to the output by selecting it and pressing Ctrl+F (Fit to screen)
  6. Turn on the Virtual Camera by pressing the Start Virtual Camera button.

You can also dowload the profile and scene files by clicking here. Once you have downloaded them, do the following:

  • Unzip the files
  • Select Profile -> Import
  • Select the mediapipeobs folder
  • Select Profile -> mediapipe
  • Select Scene Collection -> Import
  • Click the … to open the filebrowser
  • Select the mediapipe.json file that you downloaded
  • Select Scene Collection -> mediapipe

Information About MediaPipe #

More information:

Artists #

Homework #

Watch #

Continue the video lectures by Jelle van Dijk

Experiment: Gesture Interaction #

Experiment with gestural interactions using MediaPipe or some other tool of your choice. Create a simple interactive sketch that does the following:

  • It should use some form of tracking with live video (hands, face, body, object tracking, classification etc.)
  • Use the data to control one output graphics, video, animation, sound, lights, mechanical movement etc.
  • Record a short video of it and submit to MyCourses
  • Try to do something that is not just one-to-one mapping of your movements to the movements of a virtual avatar. Think about interactions and different types of controls or conversions from input to output.
  • Keep it simple, no need to do anything beyond one type of interaction. This is not yet your final project.

I recommend experimenting with TouchDesigner and MediaPipe, but you are free to use some other tools as well, such as:

Deadline: Tuesday, February 25, 23:59