Internship Report

Nvidia jetson realsense depth map in touchdesigner

3 min read

While working on VJing setup in touchdesigner with MIDI controller Jakob and I started talking about depth cameras, and he was very interested if I got that to work in Touchdesigner, while I didn’t have enough time to integrate it in my VJing setup before I had the party. I did manage to make it work.

Firstly I tried hooking the Realsense Depth Camera up to my macbook, and installing the SDK (for using the depth-sensing capabilities), but it turned out there was no official support for M1 macs. I’m guessing because Realsense is made by Intel. And Apple by moving to their own chips drastically cut into the sales of Intel processors, so I understand that they don’t want to develop for their competitor. I also tried the Microsoft Azure Kinect, but it being from Microsoft, meant it only had support for Windows.

I tried my hand at compiling software from source code for the first time. The people at Lightbuzz wrote a really nice guide, but I kept getting compilation errors for LibUSB (a library to interface with usb ports via C++).

Jakob suggested I’d use an in-between-PC running Linux instead, use that to run the SDK, and then to send that data over to my macbook running TouchDesigner. So I started using the Nvidia Jetson Nano.

Setting up the Jetson was quite a hassle because the Operating System that came with it was 4 years out of date. So I had to update the thing.

After that I wrote a little script to get the depth data from Realsense and display it in an image using OpenCV Mat. And then I discovered OSC (more about that in the link).

I first send just the depth data, before I send the entire depth map as a test:

At first I was thinking of sending the full resolution image to my macbook, but the OSC packets became way too big. I made some small experiments sending binary blobs over OSC.

Then I needed to convert the OpenCV Mat into one big string, to send it as one blob, because by OpenCV Mats are a big spreadsheet of pixel values, and strings should be only one line. More on that here.

the end result is getting depth data into touch designer, I’m actually now using this whole setup to help Joëlle de Jong the technicals of her body-reactive-typography graduation project.

Graph View

Backlinks