Pose2Art: low cost video pose capture for interactive spaces10/19/2022 at 00:00 • 0 comments
Jerry Isdale, MIOAT.com
Maui Institute of Art and Technology
notes started oct 12 2022
This Page is now supersceded by the Project
Create a low cost, (mostly) open source 'smart' camera system to capture human pose and use it to drive interactive immersive art installations. Yes, its kinda like the Microsoft Kinect/Azure 'product', but DIY open to upgrading.
- use one or more smart cameras to capture Human Pose from video stream
- stream that data (multicast?) – pose points (OSC), raw frames, skeleton overlay (video), outline, etc etc
- receive stream to drive CGI Rendering Engine using skeleton data etc
- project that stream on wall (or use all above streams as input to video switcher/overlay
- Edge Computing: Raspberry Pi 4 and Nvidia Jetson Nano are target platforms i have. Google Coral may be a better low cost alternative to the raspberry pi 4.
- Camera: Does not need to be high resolution. a usb webcam or CSI interface (ribbon cable, rPi camera, arducam, etc.)
- Network: Wired Ethernet is preferred over WiFi for installations to avoid interference. A single cable can connect the edge device with PC. Configuration of software is a bit tricky
- Rendering Engine: Decently powerful computer with high graphics card running TouchDesigner, Unity, Unreal or similar visual software
- Display: either a video wall or projection video setup
Multiple cameras can be used to create 3d pose tracking
Stream video from edge cameras to rendering engine. Not sure of usable protocol
Tracking multiple people, in contact with each other (dancing, acro-yoga etc)
Depth Camera: cameras that give Point Cloud depth data could be used
This very much a work in process (with uneven progress).
18Nov: I have gotten the Camera/Pose etc working and feeding Points over network to PC via OSC which feeds data into TouchDesigner
Currently i;m note taking on both pi and pc (with multiple boot sd cards for different OS on Pi)
Example Art Installations
(insert links to still/video of pose tracking in interactive environments)
10 steps in Pose2Art process
(make a graphic of this flow)
- image capture
- pose extract
- pose render (optional)
- stream send pose data, video optionally
- physical send ( transport)
- physical rcv
- stream receive
- stream process
Project Plan and this Page
- This doc:survey's tech options for each of the 10 stages listed
- survey existing solutions, focus on newer ones with multi-person options
- find one that runs on my rPi4
- build out a demo using rPi4 and TouchDesigner for rendering.
Nov 20 status:
The QEngineering Raspberry Pi image comes with TensorFlowLite properly installed, along with a C++ demo of pose capture. Adding Libosc++ got it emitting OSC data. Fair bit of mucking around with network static ip, routes, and firewalls was required, but finally got it working with PC. Found at least 1 TouchDesigner example of reading OSC pose data and got it working. Looking into other demos, like a Kinect driving TD Therimin simulator.
OSC (OpenSoundControl) currently chosen as data transport. It is VERY much user defined messages, and I have yet to see any 'standards' for how to name the Pose data. Kinect tracked point names might be useful.
Survey of System Demos
Web searching turned up a LOT of links on pose estimation using machine learning. Some include source code repositories and documentation, others are academic papers or other non-replicable demos. This section is a summary of some. Hopefully one will be found to actually work?
30 oct 2022: links below this update
Attempting to run demos has been Interesting, with lots of classic dependency issues. Some Python pose examples were made to work, but alas Very slowly. The QEngineering rPi example...Read more »
shoots just seeing this like 5 yr later. I found my 2 powergloves stashed in a box the other day. Other than mounting on a MyHistory wall (which i dont have), I dont know what to do with em.