15. Project Status #3

Much has happened since my last update. In December, the camera arrived, and I wrote the connection manager, integrating the basic synthesizer I had written. By the end of December I was making the same funny noises, but now moving objects around on the tabletop. Then, around Christmas-time I ran into a tricky problem. I added a rotation action to my generators, which changes frequency. (ie. you turn the object to change it's pitch) However, rotating caused terrible clicking noises. Despite some friendly offers of help, I lost interest in the project. Just tonight, three months later, I sat down for a few hours and came up with a workaround!

So, I'm happy to report that I'm not done with this project yet.

14. First generated sounds

I've added support for generators to have multiple controllers. I recorded some simple sounds to share with you.

First, a sinewave generator with 4 sinewave controllers of various frequencies.

http://jaredbeck.com/pub/permanent/2008-12-02/5sines.mp3

Second, a mix of a bunch of different waveforms. There's a sawtooth controller in there, one or two square wave controllers, and I think the generator was square wave also.

http://jaredbeck.com/pub/permanent/2008-12-02/crazy.mp3

I know, these are really trivial examples of synthesis, but I'm just getting started. Maybe I'll write a step sequencer next. Then I can actually make some beats.

13. Endianness Conversion

The following are two common tasks I have been dealing with, except I haven't been using the appropriate bitwise operators for efficiency. These examples were a big help:

How do I convert short (16 bit) samples to bytes to store them in a byte array?

Example: Store 16-bit sample in a little-endian byte array:

short sample = ...;
byte[] buffer = ...;
int offset = ...;
// low byte
buffer[offset + 0] = (byte) (sample & 0xFF);
// high byte
buffer[offset + 1] = (byte) (sample >> 8) & 0xFF;

(citation)

How can I reconstruct sample values from a byte array?

Example: Reconstruct a 16-bit sample from signed PCM into a signed int (32-bit):

int sample = (buffer[offset + 0] & 0xFF) | 
(buffer[offset + 1] << 8);

(citation)

Note the bitwise inclusive OR (|) and the signed left shift operator
(<<). The bitwise AND (&) I was already using to mask, as above,
but the other two I wasn't using yet. I was using integer arithmetic
instead of these bitwise operators, which is a little embarrassing.

12. Endianness

Endianness of components in my project:

1. The Java virtual machine is big-endian. (citation needed)
2. My Intel-based mac is little-endian.
3. "PCI busses are, by design, little endian" (citation) (Not applicable in my situation because my audio is on-board, but interesting to know.)
4. My macbook's audio device is probably this: "Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 02)" which I assume is little-endian, because it is from Intel. (This is someone else's output from lspci, which I don't have installed)
   
5. The examples I've seen so far at jsresources.org are little-endian

I am using 16-bit samples, but if you are only using 8-bit, you can ignore endianness:

For Java Sound, endianess matters if the size of samples (as given by AudioFormat.getSampleSizeInBits()) is greater than 8 bit. For 8 bit data, while the endianess still has to be specified in an AudioFormat object, it has no significance.

(citation)

Conclusion:

I can perform audio synthesis calculations in big-endian, because that is what the JVM speaks, but my final output should be little-endian, because that is what all of my Intel hardware speaks. Obviously, I should only convert once, for the final output.

11. Project Status #2

Working from the other end of the project, so to speak, I've written a proof-of-concept tree. So far I have a generator and controller.

So, instead of making the beeeeeeeeeeeeeeeeeeep of the generator, now I can make a beep beep beep. (with a square wave controller) I can also use a sine or triangle controller to make a woo woo woo, or a sawtooth controller to make a chirp chirp chirp. It's a big step for me!

The next step is to support more than one controller, so I can make a beep beep beep BEEP beep beep beep BEEP.

Before I can do that, I need to clean up my code, fix a bug in my two's complement math, and get some sleep.

10. Building tabletop and cabinet

I got a glass table top for free. Woot. I traced the table top onto paper. Drew arcs. These arcs will be patterns for cutting the plywood. Actually, I will be making two rings out of these arcs. The bottom ring will be a lip for the glass to rest on. You'll see.


I cut out the arc patterns. The left over paper circle below the glass shows usable area. Cell phone shown for scale.


Getting ready to glue arcs into rings. See how there are two rings? The rings overlap for strength.


Gluing arcs. Yay clamps!


Planning the cabinet ..


Simple cabinet out of 2x2s and 1x4s. It's actually quite sturdy, and lightweight. I'm satisfied.


To make the table more portable, I want to be able to take the top off. I use bolts, eye straps, and wingnuts.

9. Camera and Lighting

Until now, I have been using the built-in camera on my laptop to play around with the ReacTIVision software. As I get closer to having a real table to work with, I have been researching the requirements for a camera.

Background

From the official site:

For the tracking, the objects need to be properly illuminated, so the camera and thus the computer vision application can see them correctly. For the projection onto a table, the surface needs to be dark though, so the user can see the projected image well enough. Since these two necessary steps logically exclude each other, the solution is to operate in two different spectra:

The projection has to be visible to the user, so the computer vision component needs to operate in a different, invisible spectrum such as infrared. Most CCD cameras are perfectly sensitive within the near IR spectrum

(Near IR refers to light with a wavelength of 700 to 900 nm) See this WP article on Infrared Photography.


IR Lamps

therefore infrared LED lamps can be used to illuminate the table.

I haven't found a good cheap near-IR lamp yet. The cheapest I've found yet are about $50 and look overpowered (50-100 LEDs) I'll post back here when I've researched this further.


Filtering out visible light

All light from the visible spectrum needs to be filtered in the camera, so the computer vision algorithm is not disturbed by the projection.

I have had some trouble finding filters that block visible light. Edmund Optics is the only supplier I've found so far. Look for their "R-72 UV / VIS-Cut Filter" Unfortunately, this filter is probably too expensive for me.

UPDATE: I just found these unmounted filters which are cheap and sound OK. They look kind of opaque, so I asked my optics guy "Do you think the image quality will be OK?" and he said "no problem at all. Only opaque at visible wavelengths. For IR, it's like a clear window."


IR-blocking lens coatings

Eventually an existing infrared blocker needs to be removed from the camera sensor.

Many cameras, especially delicate color CCDs, have an IR-blocking coating. Unibrain sells lenses without this coating, and suggests that you use these lenses with their B/W cameras, not the color. (Which is fine for our purpose)


CCD vs. CMOS

In general cameras with CCD sensors should perform better than those with a CMOS chip. Additionally you should make sure that the camera has an acceptable lens. These two criterias basically exclude cameras below the 50$ range.

Yeah, no kidding.


Avoid Interlaced Mode

DV cameras supporting full-frame mode are suitable, while those with interlaced mode only, will not work at all.

I think that interlacing is a technique of alternating odd and even lines to maintain image quality while reducing bandwidth. I don't understand why this would be a problem, because I'd expect the camera drivers to abstract this out, protecting the programmer. But, the ReacTable guys know what they're talking about, so we'll avoid interlaced mode.


Official recommendation

For lowest latency and best performance we recommend firewire cameras from the top range, such as industrial cameras with a high framerate, resolution and sensor size. These cameras usually also come with high quality lenses.

Right, but we don't all have grant money and University budgets :-) The ReacTable guys use an AVT Guppy. I haven't asked a salesman for a quote yet, but I assume they are pretty pricey.

UPDATE: An anonymous post on the reactivision forum (at sourceforge) says that the Guppy is about $1000. That was in 2007


Conclusion

Cheaper firewire cameras, such as the unibrain fire-i also allow optional wide-angle lenses.

Unibrain is the best vendor I've found yet for this project. They sell a "firewire board camera" (a CCD camera on a PCB board) with either color or B/W CCD, and a mini-lens mount. They also sell lenses without an IR-blocking coating, as I mentioned. They are also affordable. Mini-lenses are $30-50 and the camera is $150 (for B/W CCD) through 1394store.com (link)