Draw or Discard: A Java Media Application Considered

by Chris Adamson


Here's a potential media application. Can Flash or one of the Java media libraries handle it?


11 Comments

James Ward
2007-01-17 21:52:15
Hi Chris. Cool idea! I started building something recently with the free Flex SDK that could be used as a basis for this. I wanted to control a web-app by putting colored dots on my fingers and then have the webcam record my fingers and figure out my finger gestures. I got about halfway done before my flight reached it's destination. Here is some basic Flex code to get you started:


<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" creationComplete="initApp()">


<mx:Script>
<![CDATA[


[Bindable] public var cam:Camera;


private function initApp():void
{
cam = Camera.getCamera();
cam.setMode(160,120,30);
v.attachCamera(cam);
}


private function pause():void
{
var bd:BitmapData = new BitmapData(160, 120);
bd.draw(v); // copies the webcam's pixels to a BitmapData object


// do something with the BitmapData now
}
]]>
</mx:Script>


<mx:VideoDisplay id="v" width="160" height="120"/>


<mx:Button label="Take Picture" click="pause()"/>


</mx:Application>



With the BitmapData object you can get the pixels in a variety of different ways. Check out the API documentation for BitmapData:
http://livedocs.macromedia.com/flex/201/langref/flash/display/BitmapData.html


I'd be interested in helping you figure out the algorithm to find the bar code and decipher it. Please email me if you want to work on this together.


-James Ward
james@jamesward.org

Roland Schaer
2007-01-18 00:30:31
As an Ex-Flash Developer (currently working as Java Developer) i would probably prefer Flash over Java.
One of the major advantages Flash has, is the ability to create awesome User Interfaces. The Media Support, as you pointed out, lacks on both platforms but Adobe was always a step ahead i guess :-/
Steve Loughran
2007-01-19 07:40:27
Chris, I've been following your java media work for a while, and it is very poignant. Java was positioned as applets, "life for web pages", and in fact flash does that to the level of hosting interesting games for kids.


desktop wise, the only mainstream java app I've seen, other than Java IDEs for server-side code, has to be azureus. And its big.


In a past life, I did work on software that got bundled with add on hardware for PCS. C++ and COM only. No .NET because it wasnt universal, so you couldn't rely on it being there. If you redistributed it you'd field all the support calls when it didnt work. So we stuck to C++, applying java dev practises: unit testing, cruise control, to the process. It mostly worked. We did check our com interfaces worked with .NET though, for the future.


the product still ships, and now there is a linux runtime. Behind the COM API were C++ classes, and they were STL+unicode across the board. It leaves the team with a cross compile, test and support problems, but its the best way to integrate with the OS.


And Java? I've never got drag and drop to work reliably on any of the linux desktops, even though its coming up to 25 years since the Mac launched.

Tim O'Brien
2007-01-19 14:23:10
Chris, I think you are right about NetBeans.... After all, they hired two brilliant Java developers with the potential to change the whole Java environment (JRuby dudes), and they put them on the netBeans team. huh?


But, the idea strikes me as a great one. how come no one has done this yet - using a camera as a barcode reader?

cooper
2007-01-19 17:49:08
is that after a point, I expect the reader to take on faith my claim that Web 2.0's read/write nature will eventually express itself in media, and that we'll need media frameworks that offer more than just playback. So, I'd like to propose an application and go about seeing how feasible it is.


Take it on faith? Odeo and YouTube both offer "record your *cast directly from your browser" functionality now. I don't think you need to take it on faith when you see people doing it with Flash now. :P


As to your app, I don't know if it matters that Flash can't get at the pixels on the client. You can, and maybe *should* do that on the server -- at least for your CCG app. You would want a record of the full pixels on the server to add a level of auditability. Of course, if every card had a UUID, presumably it is already in a database somewhere. You would want some kind of "active agree" before a card could be "traded" in the database -- so someone couldn't shank your cards-- and in terms of initial registration, you would do better simply scanning a UUID on the "pack" that was purchased.


In terms of the bar code idea, though. It would be impressive to see things like AllConsuming or even Amazon Reviews supporting this.


Chris, I think you are right about NetBeans.... After all, they hired two brilliant Java developers with the potential to change the whole Java environment (JRuby dudes), and they put them on the netBeans team. huh?

Well, I guess it depends on priorities. Ruby isn't sweeping the world as a desktop development language either. Chris and I chatted about this on AIM a while back, and I would say his estimation here is probably right. However, I am not 100% that is necessarily a bad thing. One of the longstanding problems with the Java APIs was their mediocre support for tooling. Matisse is a pretty good example of where things can go when you work on both at the same time. All that said, media is just one part of the problem with desktop Java. Lack of any kind of modern HTML/Web support is a huge one up and down the line. Poor (if getting better) abilities to support modern desktop app metaphors is another. Poor base widget sets and the difficulty of things like Drag and Drop support stand out as other problems. The whole NetBeans RCP starts to address some problems, but the real lacking that Java has lies below the RCP level. I think if the focus on NetBeans has any real drawbacks, it is the denial that a lot of Sun technologies would see wider developer acceptance if they were not marketed as NetBeans-coupled, even when they aren't.
Chris Adamson
2007-01-19 18:22:59
Hi Chris. Cool idea! I started building something recently with the free Flex SDK that could be used as a basis for this. I wanted to control a web-app by putting colored dots on my fingers and then have the webcam record my fingers and figure out my finger gestures. I got about halfway done before my flight reached it's destination. Here is some basic Flex code to get you started:

James, that's awesome. Thanks for the unblocker. The ability to draw into the offscreen buffer is the missing link. I guess the next step for you from here would be to look at the pixels... might be simpler (if more expensive) to convert to HSV and see whether you can find a sufficient percentage of sufficiently green or red or whatever pixels (hue between specific values, with brightness and saturation over certain threshholds). This kind of reminds me of Lisa Lippincott's "Scroll Plate" from MacHack, when she held up styrofoam plates with colored arrows to her iSight to scroll the front-most window. Anyways, your hack is definitely doable, and figuring out bar-code scanning is just a more complex application of the same ideas. I have no damn time, but I've downloaded Flex in hopes of trying to dig into this. Thanks again.

Brendon J. Wilson
2007-01-22 20:55:18
Actually, you *can* get at pixel data from Flash using the BitmapData class - you can draw a Video object framegrab to the BitmapData object using BitmapData object's draw() method. It can also be done in Java (I actually wrote the code originally in Java and ported to Flash in an evening), although Flash is much nicer from a cross-platform compatibility (and this is from a true-blue Java guy).


I don't know if you're interested, but I actually just released a demo of a Flash-based barcode reader. See the details at:


http://www.brendonwilson.com/blog/2007/01/22/barcode-recognition-in-flash/


The code is still in its early form, but it provides a workable demonstration of what is possible. I hope to expand this code in the coming month or so to work with fixed focus webcams, such as those on the MacBook Pro, as many laptops are now shipping with webcams as standard integrated peripherals.


I'd really be interested in hearing what things people would like to build with this.

James Ward
2007-01-23 21:02:07
Actually, you *can* get at pixel data from Flash using the BitmapData class - you can draw a Video object framegrab to the BitmapData object using BitmapData object's draw() method.


That's exactly what the code I posted does. :)


Have you posted the ActionScript code you used to read the bar code from the BitmapData?


Thanks.


-James

Brendon J. Wilson
2007-01-25 18:59:10
Not yet - currently still a work in progress, as I'm trying to work through the problem of dealing with fixed focus lenses. I'm still trying to decide what to do with the library once it's done. I could release it as open source, or I could use it to develop a service built on top of it.
max jb
2007-04-19 01:36:26
do u no how to unblock this page at a school computer if not a unblocking web search page thanks.
jacob halencak
2007-08-26 16:42:50
you have the right idea but think more about a sort of board like in yu-gi-oh! they use the dual disc's think something like that but usb with mic and not a bar code but more of a small chip in the bottom right corner of the cards that way your not have to scan it but just place it down on the board the board also that way depending on what TCG you play you can put your cards into defence mode or like MTG you have to tap your cards all the time and you can with a board hooked up to the computer by usb. yeah selling the packs and booster packs with the chip well bring money in but people would in joy a more of actual interaction because you need to still use your head when placing the right cards on that board.