Moves On Rails: Background processing with WorkerQueue

Background processing with WorkerQueue

bart.tenbrinke@movesonrails.com (Bart ten Brinke) — Wed, 13 Aug 2008 21:08:00 +0200

Yes, I know there are a lot of background runners available for rails (I tried most of them), and they all seemed to have their downsides:

BackgroundRB -> Major memory leaks when performing large tasks and bloaty.
Spawn -> I really do not want to fork on every request. I use merb for that.
Starling -> Memcache and only does sequential processing.
ActiveMessaging -> Round hole Square peg.

Also non of these can handle big file uploads nicely and reliable. That is when I decided to create my own: WorkerQueue.

Main features:

Simple
Fast
Works via a rake task and prefers cron
Can handle large amounts of data
Sequential and Parallel execution on demand
Simple error reporting
Offload tasks to other or multiple machines

You can clone a copy here:

  git clone git://github.com/barttenbrinke/worker_queue.git vendor/plugins/worker_queue

Or visit github.

UPDATE: WorkerQueue 1.0 Stable is now available on Github.

"Background processing with WorkerQueue" by Bart ten Brinke

Wed, 20 Aug 2008 10:56:05 +0200

Background RB is a great tool, but not for the things I wanted to use background processing for. Combined with the (ruby) memory leaks, this is te sollution I came up with. So far this is working a lot better in production then our old backtroundrb setup.

"Background processing with WorkerQueue" by hemant

Tue, 19 Aug 2008 20:54:37 +0200

well, i have fixed all known memory leaks in backgroundrb git version. There is nothing there that leaks (except for the fact that, ruby itself leaks in many cases).