Building a ReacTable

16. Project Status #4

2009-08-12T23:59:00.000-07:00

I've replaced my temporary fix for those clicking noises with a more elegant, permanent fix.

The next features in the pipeline:

Rotating controllers changes frequency (rotating generators already does this)
New controller: Envelope generator
New controller: Sequencer

At the suggestion of a friend, I got Foundations of Computer Music from my local library, but I have to admit it's slow going. I'm definitely missing some prerequisite knowledge.

15. Project Status #3

2008-12-20T00:53:00.000-08:00

Much has happened since my last update. In December, the camera arrived, and I wrote the connection manager, integrating the basic synthesizer I had written. By the end of December I was making the same funny noises, but now moving objects around on the tabletop. Then, around Christmas-time I ran into a tricky problem. I added a rotation action to my generators, which changes frequency. (ie. you turn the object to change it's pitch) However, rotating caused terrible clicking noises. Despite some friendly offers of help, I lost interest in the project. Just tonight, three months later, I sat down for a few hours and came up with a workaround!

So, I'm happy to report that I'm not done with this project yet.

14. First generated sounds

2008-12-02T22:08:00.000-08:00

I've added support for generators to have multiple controllers. I recorded some simple sounds to share with you.

First, a sinewave generator with 4 sinewave controllers of various frequencies.

http://jaredbeck.com/pub/permanent/2008-12-02/5sines.mp3

Second, a mix of a bunch of different waveforms. There's a sawtooth controller in there, one or two square wave controllers, and I think the generator was square wave also.

http://jaredbeck.com/pub/permanent/2008-12-02/crazy.mp3

I know, these are really trivial examples of synthesis, but I'm just getting started. Maybe I'll write a step sequencer next. Then I can actually make some beats.

13. Endianness Conversion

2008-11-25T21:31:00.000-08:00

The following are two common tasks I have been dealing with, except I haven't been using the appropriate bitwise operators for efficiency. These examples were a big help:

How do I convert short (16 bit) samples to bytes to store them in a byte array?

Example: Store 16-bit sample in a little-endian byte array:

short sample = ...;
byte[] buffer = ...;
int offset = ...;
// low byte
buffer[offset + 0] = (byte) (sample & 0xFF);
// high byte
buffer[offset + 1] = (byte) (sample >> 8) & 0xFF;

(citation)

How can I reconstruct sample values from a byte array?

Example: Reconstruct a 16-bit sample from signed PCM into a signed int (32-bit):

int sample = (buffer[offset + 0] & 0xFF) | 
(buffer[offset + 1] << 8);

(citation)

Note the bitwise inclusive OR (|) and the signed left shift operator
(<<). The bitwise AND (&) I was already using to mask, as above,
but the other two I wasn't using yet. I was using integer arithmetic
instead of these bitwise operators, which is a little embarrassing.

12. Endianness

2008-11-25T20:12:00.000-08:00

Endianness of components in my project:

The Java virtual machine is big-endian. (citation needed)
My Intel-based mac is little-endian.
"PCI busses are, by design, little endian" (citation) (Not applicable in my situation because my audio is on-board, but interesting to know.)
My macbook's audio device is probably this: "Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 02)" which I assume is little-endian, because it is from Intel. (This is someone else's output from lspci, which I don't have installed)
The examples I've seen so far at jsresources.org are little-endian

I am using 16-bit samples, but if you are only using 8-bit, you can ignore endianness:

For Java Sound, endianess matters if the size of samples (as given by AudioFormat.getSampleSizeInBits()) is greater than 8 bit. For 8 bit data, while the endianess still has to be specified in an AudioFormat object, it has no significance.

(citation)

Conclusion:

I can perform audio synthesis calculations in big-endian, because that is what the JVM speaks, but my final output should be little-endian, because that is what all of my Intel hardware speaks. Obviously, I should only convert once, for the final output.

11. Project Status #2

2008-11-24T23:10:00.000-08:00

Working from the other end of the project, so to speak, I've written a proof-of-concept tree. So far I have a generator and controller.

So, instead of making the beeeeeeeeeeeeeeeeeeep of the generator, now I can make a beep beep beep. (with a square wave controller) I can also use a sine or triangle controller to make a woo woo woo, or a sawtooth controller to make a chirp chirp chirp. It's a big step for me!

The next step is to support more than one controller, so I can make a beep beep beep BEEP beep beep beep BEEP.

Before I can do that, I need to clean up my code, fix a bug in my two's complement math, and get some sleep.

10. Building tabletop and cabinet

2008-11-18T19:36:00.001-08:00

I got a glass table top for free. Woot. I traced the table top onto paper. Drew arcs. These arcs will be patterns for cutting the plywood. Actually, I will be making two rings out of these arcs. The bottom ring will be a lip for the glass to rest on. You'll see.

I cut out the arc patterns. The left over paper circle below the glass shows usable area. Cell phone shown for scale.

Getting ready to glue arcs into rings. See how there are two rings? The rings overlap for strength.

Gluing arcs. Yay clamps!

Planning the cabinet ..

Simple cabinet out of 2x2s and 1x4s. It's actually quite sturdy, and lightweight. I'm satisfied.

To make the table more portable, I want to be able to take the top off. I use bolts, eye straps, and wingnuts.

9. Camera and Lighting

2008-11-16T20:31:00.000-08:00

Until now, I have been using the built-in camera on my laptop to play around with the ReacTIVision software. As I get closer to having a real table to work with, I have been researching the requirements for a camera.

Background

From the official site:

For the tracking, the objects need to be properly illuminated, so the camera and thus the computer vision application can see them correctly. For the projection onto a table, the surface needs to be dark though, so the user can see the projected image well enough. Since these two necessary steps logically exclude each other, the solution is to operate in two different spectra:

The projection has to be visible to the user, so the computer vision component needs to operate in a different, invisible spectrum such as infrared. Most CCD cameras are perfectly sensitive within the near IR spectrum

(Near IR refers to light with a wavelength of 700 to 900 nm) See this WP article on Infrared Photography.

IR Lamps

therefore infrared LED lamps can be used to illuminate the table.

I haven't found a good cheap near-IR lamp yet. The cheapest I've found yet are about $50 and look overpowered (50-100 LEDs) I'll post back here when I've researched this further.

Filtering out visible light

All light from the visible spectrum needs to be filtered in the camera, so the computer vision algorithm is not disturbed by the projection.

I have had some trouble finding filters that block visible light. Edmund Optics is the only supplier I've found so far. Look for their "R-72 UV / VIS-Cut Filter" Unfortunately, this filter is probably too expensive for me.

UPDATE: I just found these unmounted filters which are cheap and sound OK. They look kind of opaque, so I asked my optics guy "Do you think the image quality will be OK?" and he said "no problem at all. Only opaque at visible wavelengths. For IR, it's like a clear window."

IR-blocking lens coatings

Eventually an existing infrared blocker needs to be removed from the camera sensor.

Many cameras, especially delicate color CCDs, have an IR-blocking coating. Unibrain sells lenses without this coating, and suggests that you use these lenses with their B/W cameras, not the color. (Which is fine for our purpose)

CCD vs. CMOS

In general cameras with CCD sensors should perform better than those with a CMOS chip. Additionally you should make sure that the camera has an acceptable lens. These two criterias basically exclude cameras below the 50$ range.

Yeah, no kidding.

Avoid Interlaced Mode

DV cameras supporting full-frame mode are suitable, while those with interlaced mode only, will not work at all.

I think that interlacing is a technique of alternating odd and even lines to maintain image quality while reducing bandwidth. I don't understand why this would be a problem, because I'd expect the camera drivers to abstract this out, protecting the programmer. But, the ReacTable guys know what they're talking about, so we'll avoid interlaced mode.

Official recommendation

For lowest latency and best performance we recommend firewire cameras from the top range, such as industrial cameras with a high framerate, resolution and sensor size. These cameras usually also come with high quality lenses.

Right, but we don't all have grant money and University budgets :-) The ReacTable guys use an AVT Guppy. I haven't asked a salesman for a quote yet, but I assume they are pretty pricey.

UPDATE: An anonymous post on the reactivision forum (at sourceforge) says that the Guppy is about $1000. That was in 2007

Conclusion

Cheaper firewire cameras, such as the unibrain fire-i also allow optional wide-angle lenses.

Unibrain is the best vendor I've found yet for this project. They sell a "firewire board camera" (a CCD camera on a PCB board) with either color or B/W CCD, and a mini-lens mount. They also sell lenses without an IR-blocking coating, as I mentioned. They are also affordable. Mini-lenses are $30-50 and the camera is $150 (for B/W CCD) through 1394store.com (link)

8. ReacTable music

2008-11-13T23:54:00.000-08:00

Actual music

7. Other uses for tangible tabletop interfaces

2008-11-13T23:38:00.000-08:00

These people use their table to create fractals. The music in the video does not come from their table, of course.

6. TuioDemo - A starting point

2008-11-13T23:01:00.000-08:00

Martin Kaltenbrunner, of the ReacTable team, has written a TUIO client / demo in Java called, aptly, TuioDemo. This has been my starting point so far.

I've done some research (mostly playing around) with the Java Sound API, and it seems adequate for building a synthesizer (the most important requirement being that it is capable of handling PCM-encoded audio composed on the fly) See also: jsresources.org, a site by two of the guys who wrote the Java Sound API.

So, being satisfied so far with Java audio and Kaltenbrunner's TuioDemo, I will begin designing my own synthesizer with the concepts from TuioDemo as a starting point.

Being a bit of a stickler for formal software engineering, I've created a class diagram of TuioDemo (link, 300k PDF) to get me started. It will be useful as a reference point.

5. Project Status #1

2008-11-13T12:17:00.000-08:00

Software

I have printed off the fiducial markers, and I can wave them in front of my laptop's built-in camera. ReacTIVision recognizes them and talks to my TUIO client. (I am using the example client TUIO_JAVA.zip from the official software page)

Using the Java Sound API (javax.sound.*) I've added some basic sound effects.

Adding a marker starts a sound looping
Removing a marker stops the loop
Rotating a marker changes the volume

Hardware

So far, I have my laptop and a round glass tabletop. I need an external camera, a projector, and I need to build a stand for the tabletop. I've done some research on the camera, and I'll put that into its own blog post.

4. So, you want to build a ReacTable?

2008-11-11T23:21:00.000-08:00

The great majority of the work has already been done for you, and yet the work that remains is daunting.

As of November 2008, these things are free:

The computer vision library, ReacTIVision
The protocol for communicating events, TUIO
Sample TUIO-client code in many languages
All of the research published by the ReacTable team

These things you must do yourself:

Research - You had better get a binder, because you need to research modular synthesis, infrared videography, TUIO, PCM-encoded audio. The official site is a good start, but I have yet to find a single comprehensive resource. I hope to contribute to the body of knowledge with this blog.
Buy some hardware - A round glass tabletop, a camera (more to come on specifics), a projector
Write the connection manager software - Basically, a TUIO client that listens for events and maintains a n-ary tree of synthesis objects
Write the synthesizer software - Given a tree from the connection manager, create sound
Write the graphics software - Given a tree from the connection manager, draw the graphics

If you can't write software, or don't want to, I recommend trying the MIDI route (see previous post)

3. I will not use MIDI

2008-11-11T23:11:00.000-08:00

ReacTIVision can communicate events in MIDI or in TUIO. I will be using TUIO.

According to the software page on the official site:

Keep in mind though that MIDI has less bandwidth and data resolution compared to Open Sound Contol, so the new MIDI feature is meant as an convenient alternative in some cases, but TUIO still will be the primary messaging layer.

If you want to use MIDI, you could check this out:

A user at the reaktor forum posted this brief tutorial on how to use reacTIVision in conjunction with commercial audio software such as reaktor.

2. Video Demo

2008-11-11T23:06:00.000-08:00

Courtesy of Marcos Alonso, of the ReacTable team

1. The ReacTable - Introduction

2008-11-11T21:39:00.000-08:00

The ReacTable is a musical instrument. Specifically, it is a multi-player digital modular synthesizer with a tangible table-top interface. A picture might help:

The ReacTable
Creative Commons Attribution ShareAlike 2.0 License

Let's break that down:

Synthesizer - An electronic instrument capable of producing a variety of sounds by generating and combining signals of different frequencies
Modular Synthesizer - A synthesizer composed of interchangeable, easily re-arranged parts
Digital - The alternative to analog
Tangible table-top interface - To make music, you just move around blocks on a table top.