Background processing with WorkerQueue

Posted by Bart ten Brinke Wed, 13 Aug 2008 19:25:48 GMT

Yes, I know there are a lot of background runners available for rails (I tried most of them), and they all seemed to have their downsides:

  • BackgroundRB -> Major memory leaks when performing large tasks and bloaty.
  • Spawn -> I really do not want to fork on every request. I use merb for that.
  • Starling -> Memcache and only does sequential processing.
  • ActiveMessaging -> Round hole Square peg.

Also non of these can handle big file uploads nicely and reliable. That is when I decided to create my own: WorkerQueue.

Main features:

  • Simple
  • Fast
  • Works via a rake task and prefers cron
  • Can handle large amounts of data
  • Sequential and Parallel execution on demand
  • Simple error reporting
  • Offload tasks to other or multiple machines

You can clone a copy here:

  git clone git://github.com/barttenbrinke/worker_queue.git vendor/plugins/worker_queue

Or visit github.

UPDATE: WorkerQueue 1.0 Stable is now available on Github.

Comments

Leave a response

  1. hemant said 6 days later:
    well, i have fixed all known memory leaks in backgroundrb git version. There is nothing there that leaks (except for the fact that, ruby itself leaks in many cases).
  2. Bart ten Brinke said 7 days later:
    Background RB is a great tool, but not for the things I wanted to use background processing for. Combined with the (ruby) memory leaks, this is te sollution I came up with. So far this is working a lot better in production then our old backtroundrb setup.

RSS feed for this post trackback uri

(leave url/email »)

   Preview comment