Telerobotics Using Lego Mindstorms and Java

by Ming Chow

At last, I finally have the sanity (or insanity) to delve back into working with my Lego Mindstorms set with Java. This time, I want to use Java and Lego Mindstorms for telerobotics.

I moved to a new home this past summer, and I have settled nicely in the new place. I also graduated back in May with my Master Degree in Computer Science. Thus, life has slowed down, and I have more free time to spare. So what am I going to do now with my free time and the upcoming "dead-time" in Massachusetts, the start of a predictably long and cold-harsh winter, besides watching college hockey?

I am not eager to shell-out money for a new hobby. I like to be resourceful, and use what I have. I also know that there will not be many opportunities for me to utilize much of what I learned and accomplished in my studies in Computer Science.

Then I thought about my Lego Mindstorms set. I was one of the lucky few that received one of the very first sets for Christmas nearly six years ago. I haven't touched it since my Senior Design Project at Tufts, where my project was an improved bowling computer scoring system with pin tracking. I prototyped the pin tracking component using the Lego Mindstorms RCX brick with a light sensor, programmed using Java.

Ah yes, I programmed the Lego Mindstorms RCX brick using Java. More specifically, I used leJOS, the firmware replacement (a 32 K Java Virtual Machine) for the Lego RCX brick that enables me to program Lego Mindstorms robots in Java. Software such as RCX Code (which comes with the Lego Mindstorms set) and ROBOLAB were great for fun and education, but they never provided me a whole lot of flexibility, considering my programming experience. Furthermore, I could never have the ability receive or send commands to the RCX brick directly from my PC. When the leJOS project was released, it was a blessing for me, and it gave my Lego Mindstorms some new life. My pin tracking prototype was simple: one light sensor beneath a pin. I would press the RUN button on the RCX brick to read a value for the light (low value for dark, bowling pin standing; high value for bright, bowling pin down), and then send the information to my scoring system interface. You see the issue here is that I have to press a button on the RCX brick to trigger an action, instead of triggering a command on the scoring system to the RCX brick to read the light value. But for some reason, I could never get the latter capability to work --telerobotics, commanding a robot from remote distances (the most famous examples are used in medical operations, and by NASA to explore Mars).

I thought more about the problem. If I could successfully resolve the dilemma, then it would open up a world of technical and professional development opportunities for me. I never really had the opportunity to work with microprocessors or hardware projects during my studies in college. The RCX brick is essentially a microprocessor. This endeavor would also be an opportunity to utilize many facets of Computer Science and the physical sciences including Human-Computer Interactions/User Interfaces, Artificial Intelligence, Algorithms, and Physics.

First things first, I bought new batteries for the RCX (requires 6 AA batteries) and for the IR tower (requires one 9-volt battery). I have not touched both devices for over a year. I downloaded and installed the latest version of leJOS onto my computer (Win32 version), and I downloaded a fresh new version of the leJOS firmware onto the RCX (via serial communication port). One of the major differences for me now is my experience with the Eclipse IDE, which makes creating, managing, and debugging Java applications significantly easier. The last time I worked with leJOS, I did not have a satisfactory Java integrated development environment to suit my needs and resources.

In particular, I am interested in creating a Java client application using AWT and Swing that will allow me to control the RCX. For now, I am not interested in creating an interface on the web that will allow me to control the RCX because it will involve time setting up a web server and among other things. For a simple test, I wanted to create an interface that will allow me to run and stop a motor connected to the RCX. Thus, the interface will consist of two components: a client program with one button to run and stop the motor, and a program on the RCX to listen for inputs from the client program on the PC.

I created a new Java application project in Eclipse and imported the leJOS class files into my project (external JAR files in the .../lejos/lib directory). I tried using some leJOS plugins for Eclipse and found them to be cumbersome (e.g. valid statements that references the RCXBean class not recognized). I create a client program with one button to start and stop a motor on the RCX, and a program for the RCX to listen to the actions from the client program. In the client program, I used the RCXBean class to communicate with the RCX, which was trivial to do.

Test setup

I connected a motor with a wheel to the C motor port on the RCX. I turned on the RCX, compiled and downloaded the RCX program to it, and ran the program by pressing the RUN button. The motor is idle and the RCX is waiting for an input. I compiled and ran the client program via Eclipse. A window with one button labeled Start Motor appeared. When I pressed the button, the motor ran with the wheel spinning, and the label of the button on my client program changed to Stop Motor. A few seconds later, I pressed the newly labeled button, and the motor stopped! I repeated the process several more times, and my interface worked without any problems.

I skipped through some technical details above. Thanks to my time, and after looking into the leJOS framework much further, I can now create Java client programs, and even web applications, to communicate with my RCX brick. This past weekend, I also created an interface that allows a user to press a touch sensor connected to the RCX, and the signal is sent to a PC client program. The source code for the motor interaction example above (see, and for the touch sensor interface (see, are available on my personal website. I am excited about this development, and it opens many possibilities. Certainly, I can create even more complex user interfaces to communicate with the RCX. For example, a control system for fictitious cable car or transportation system. Or how about a fictitious energy management control system (solar panels)? Or a small-scale surveillance system that measures environmental information (e.g. temperature) and returns data to a client program on the PC? The educational benefits from using Lego Mindstorms with leJOS and Eclipse are tremendous. For cost-conscious curricula, the software involved is open-source and can be downloaded for free. This can be a great advanced learning experience for those who have mastered the traditional drag-and-drop Lego Mindstorms applications. There are possibilities for educators to build small-scale fictitious systems to illustrate real-world scenarios. I am sure I will have more news and developments soon.

UPDATE (November 25, 2007): Due to the continuing interest on this project, I created a project on Google Code at Source code examples are available there.