Capturing to the screen with QuickTime for Java

by Chris Adamson

Related link:

Back when I wrote QuickTime for Java: A Developer's Notebook, it was assumed that media capture was broken in QTJ. This dates back to a huge disruptive change from Carbon to Cocoa between Apple's java 1.3 and 1.4 implementations, with Cocoa not supporting some stuff that QTJ relied on. When they got QTJ running with java 1.4, they did so with a scaled-down API that didn't feature everything from the old version (for details, see this blog on the breakage and this article on the resolution). One of the things left out was a means of getting an on-screen preview of your video capture device.

Because of this, a lot of people assumed that all capture was broken. But most of the capture stuff was written QuickTime API's that were straight C and not related to Carbon or Cocoa (after all, they got ported to Windows!). I was able to show this in the book, with sections on capturing and previewing audio, saving it to disk, capturing and saving video to disk, and capturing and saving audio and video and saving them into the same file. Plenty of functionality there.

Still, it bugged me that Apple didn't satisfy the very obvious need for an onscreen preview component. In the book, I tried to fake it with a very slow means of grabbing Picts from the SequenceGrabber and converting those into Java images. It works, but the frame rate was awful, and I closed the chapter with:

I didn't split that out as its own lab because the performance is pathologically bad (one frame per second -- at best), and because it's an awkward workaround in lieu of a better way of getting a component from a SequenceGrabber. Presumably, someday there will be a proper call to get a QTComponent from a SequenceGrabber -- maybe another overload of QTFactory.makeQTComponent( ) -- and kludgery like this won't be necessary.

Good news - this is officially not necessary anymore. There's a better way.

The story starts with an Apple demo called QDCocoaComponent, which unassumingly bills itself as "shows how to use the Quicktime Sequence Grabber and to display video in a QuickDraw port (Sub class of NSQuickDrawView) in a Java canvas running on Mac OS X." Since this draws the camera output into a Java space, this got some people thinking that it might be a means of solving our need for a video capture preview component, provided that it could work with QTJ's SequenceGrabber (Apple's sample creates the SequenceGrabber in native code and doesn't actually involve QTJ).

But the magic here isn't necessarily the use of a QDCocoaComponent; it's the use of a "data proc", which in the native API is a sort of callback that you can register for whenever the grabber gets some data.

Jochen Broz, on the quicktime-java list used that to put the key pieces together. In his message Example for using Sequence Grabbers DataProc to display video in java he sets up the QTJ equivalent, an SGDataProc to get the callbacks

OK, so he's getting called back. Now what? Here's the slick part - he manages to get the bytes on the screen entirely with Java. Key to the trick is the fact that the capture device is (or assumed to be) running in 32-bit RGB, a perfectly reasonable color-space for both QuickDraw and Java2D. Here's the code that figures out a suitably large transfer buffer and creates a Java2D image. Notice how the buffer array gets wrapped by the DataBuffer, and how the WriteableRaster arguments that mask off the red, green, and blue bits in each int:

// Setting up the buffered image
int size = gWorld.getPixMap().getPixelData().getSize();
int intsPerRow =
size = intsPerRow*cameraImageSize.getHeight();
final int[] pixelData = new int[size];
DataBuffer db = new DataBufferInt(pixelData, size);
ColorModel colorModel =
new DirectColorModel(32, 0x00ff0000, 0x0000ff00, 0x000000ff);
int[] masks= {0x00ff0000, 0x0000ff00, 0x000000ff};
WritableRaster raster =
intsPerRow, masks, null);
final BufferedImage image =
new BufferedImage(colorModel, raster, false, null);

Then he defines a Java AWT Component whose only job in life is to drawImage the image on every call to paint().

Now, remember what I said about setting up a "transfer buffer"? His callback execute() method looks to see if it's getting called with video data (it would also get calls for every audio capture timeslice), and if so, it decompresses the frame into a GWorld (actually called a QDGraphics, but everyone in the know casually refers to them by their native name). He then gets to copy the pixels from the GWorld right into the Java2D BufferedImage's buffer - and array copies, while they do move a lot of data, are native and generally heavily optimized calls.

Then he just has a tight loop that idle()s the SequenceGrabber (i.e., manually gives it time to run), which will call back to the SGDataProc, which will decompress the image, copy bytes, and repaint.

And here's the result:


So how's the performance? Granted, it pegs the CPU, and the transition from QuickDraw to Java2D is something you'd want not to do if possible. Still, the println's I threw in show it's not bad, once you pardon the startup lurches:

[chrisg5:~/dev/qtjtests/dataproc-sg] cadamson% java DataProcTest2
1 frames/sec
1 frames/sec
33 frames/sec
31 frames/sec
31 frames/sec

I got equivalent results running with the new J2SE 5.0 release 3, by the way.

One other note: in the book, I suggested using a Matrix with your video captures to impose a mirror image effect. iChat does this with its preview window, presumably because it's more natural to see yourself in a mirror image than to see yourself as the camera does. To do that, add the following lines right after the cameraImageSize is defined:

final Matrix mirrorMatrix = new Matrix();
mirrorMatrix.setSx (-1.0f);
mirrorMatrix.setTx (cameraImageSize.getWidthF());

This creates a Matrix which scales pixels into negative space, effectively flipping them right-to-left, then moves them back to positive coordinates. Replace the idMatrix in the frame-decompressing DSequence's constructor with this mirrorMatrix to get the effect.

In conclusion, my deep thanks to Jochen -- this is a big problem that we've been waiting a long time for someone to solve. There's more that can be done with this approach, not the least of which would be to roll it into a general-purpose, easy-to-use component. Jochen's code is copyrighted and doesn't specify a license, so you'll want to ask him first.

What do you think of this all-Java/QTJ appoach?


Amit Zohar
2006-02-20 12:49:27
So how do I capture video and audio in Java and save it into a movie file while allowing for a preview as well?