Asynchronous job execution

by Dejan Bosanac

In software development we are often encouraged to decouple things in order to achieve greater flexibility in our code. So we decouple model objects from views, interfaces from their implementation, etc.

Here, I would like to write about different kind of decoupling, front-end from back-end. In many cases, projects tend to depend on resources that are either unreliable or overloaded. For example, besides your "main" database, you may need to write data to some LDAP server that is overloaded or you need to provide some data to the resources over the unreliable network connection. In most of these use cases, these operations are triggered by users through the user interface (web or stand-alone application). It is totally unnecessary (and usually unacceptable) to have your users wait a few minutes for operation to complete or in worse case to have dysfunctional system just because one resource is unavailable.

8 Comments

Julien
2007-07-18 01:32:58
I described such infraastructure a while ago on my blog:


http://www.jroller.com/jdelfosse/

Bri
2007-07-18 04:18:39
Change "retires" to "retries" in 7th paragraph.
Dejan Bosanac
2007-07-18 04:40:18
Fixed, thanks
Gabriel
2007-07-18 04:47:47
Hi!


Great idea,


just a typo - before anything intelligent : "You can also, define a number of retries" instead of "...retires".

Ciprian
2007-07-19 02:49:08
It is easier and it makes sense to decouple job execution by using a blocking queue (or a pool of threads) instead of a quartz scheduler or a heavyweight JMS server. The scheduler should be used just to run a job at some moment in time or recurring. The persistence mechanism behind the quartz or activemq would guarantee appropriate recovery only using distributed transactions involving the JMS server and the client database, otherwise the recovery from the client could also take care of the entire recovery without the need of a persistent queue (JMS). The JMS based notification and the whole idea of building a mechanism of job submission are fine. JBI offers interesting solutions also.
Anand Muthu
2007-07-27 05:29:00
Yuck! Using of a Brokering System for a BG Jobs would be very costly. Since, many Background job would be processing a input files, Parsing , CRUD functions , Reportgeneration and so on.. As you said, Quartz uses the DB for these operation and it will be a low cost operation compared to MQ complexity. System <--> MQ Communication would be more complex in this JQR.
Dejan Bosanac
2007-07-27 06:22:32
Quartz and blocking queues are excellent all-Java, "all in one application" solutions. I've even made "web service" interface to such systems in order to expose them to other applications, but still I think the full MQ support has some benefits:


* You can use existing MQ broker infrastructure for job queues
* Queues and workers are decoupled, so you can have multiple workers (multiple machines) handling the same queue, which can be helpful in heavy loaded environments
* MQ brokers (ActiveMQ in this case) are designed to be clusterable and easily set up for high-availability, so again it help a bit in environments where such requests are necessary.


As for the complexity, it sure is more complex underneath, but I try to make things as simple as possible for "application developer".


Thanks for your comments.

Anand Muthu
2007-07-27 10:56:06
Everything can be decoupled easily , But Coupling needs a pragmatic approach of extensive Programming :D


Anyway ! Thanks a lot bloging about your New Approach which can be used to Communicate with different System. Even , a grain of web service will make your *project* a Grant Successful one.


Thanks Dejan :)