Brick uses a combination of custom hardware and software to track images (also known as “fiducials”) and fingers on its tabletop.
Hardware
A camera underneath the screen tracks special images (fiducials) and fingers as they are pressed against the screen area and moved around. Because we are using a projector to project the images up onto the screen, we need to avoid doing our tracking in the visible light spectrum. Instead, we have used the tried method of tracking in infrared light. We built custom LED arrays of IR LED’s that emit diffuse infrared light across the screen. We use an IR bandpass filter on our camera to block out the unwanted visible light, and only let the IR that we want pass.
Software
Doing this, we are able to track the fiducials and fingers on the table using open source softwares reacTIVision and tBeta. After tracking with reacTIVision, we can then send that data into our own custom software (Weather Report uses Max/MSP/Jitter and Reaktor) over OSC & the TUIO protocol. Our second installation, Roots uses the Processing programming environment to handle all of the tracking data and visual elements, and sends the data to custom software written in the ChucK programming environment to create and control the audio/music side of things.
Our custom software synthesizes that data, both aurally and visually, and sends that back through headphones, speakers, and visually onto the screen for an interactive experience.
