Dynamic Queue Assignment for Resque Jobs

Ariejan de vroom

Ariejan de Vroom – 16 March 2010
202 words in about 1 minute

Resque is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Sounds great! Let’s dive in directly:

1
2
3
4
5
6
7
8
9
10
class Archive
  @queue = :file_serve

  def self.perform(repo_id, branch = 'master')
    repo = Repository.find(repo_id)
    repo.create_archive(branch)
  end
end

Resque.enqueue(Archive, @repo.id)

This example was taken from the Resque README. It works great, but there’s a problem: you hard-code the queue to use. For an app I’m currently working on this is unwanted behaviour. So, how do you make Resque pick the right queue based on the data you feed it? In the following example I’ll show you how to pick the right queue based on the user that requests the job.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class StreamWorker  
  def self.enqueue(resource)
    Resque::Job.create(select_queue(resource), self, resource.id)
  end    

  def self.select_queue(resource)
    resource.user.use_priority_queue? ? :stream_high : :stream_low
  end

  def self.perform(resource_id)
    resource = Resource.find_by_id(resource_id)
    return false if resource.nil?
    resource.process!
  end
end

Instead of using Resque’s convenience method Resque.enqueue we create a Resque::Job ourselves. The select_queue determines which queue is used. Creating a StreamWorker job in the right queue is as easy as running:

1
StreamWorker.enqueue(@resource)

In case you’re wondering, the User#use_priority_queue? method returns a boolean based on the role the user has. Happy queueing!

Ariejan de vroom

Ariejan de Vroom

Software Engineer • CodeRetreat Facilitator • Ruby, Go and C Programmer • Electronics Apprentice

At Kabisa, privacy is of the greatest importance. We think it is important that the data our visitors leave behind is handled with care. For example, you will not find tracking cookies from third parties such as Facebook, Hotjar or Hubspot on our website. Only cookies from Google and Vimeo are used in order to improve the user experience of our visitors. These cookies also ensure that relevant advertisements are displayed. Read more about the use of cookies in our privacy statement.