Webrat Story Steps

Posted by Andre Foeken Tue, 19 Aug 2008 12:19:26 GMT

We have just released a plugin for Rails making it much easier to use Rspec Stories for your integration tests!

Clone it here from github.

No longer will you have to figure out your own grammar. We have done it for you. Now you are able to write stuff like this out of the box:

    Given a user with username 'Dummy' exists
    And it has password 'monkey'

Even relationships (has many and belongs to are supported!)

    Given the administrator is logged in
    And a user with username 'A' exists
    And the user with username 'A' has a responsibility for / 
    the role with name 'Administrator'

Naturally we also support a lot of clicking, selecting and checking!

    When he clicks the 'Admin' link
    And he clicks the 'Roles' link
    And he clicks the 'Add a new role' link
    When he selects 'Planner'
    And he clicks the button

Even simple Javascript popups!

   Then he should see a popup with the message 'Are you sure?'
   When he confirms the popup

And it's simple to extend. We extended it to allow us to check our importer.

    Given the latest version of iO Connect is used
    And there are no addresses in the system
    When a client with objectId '1' is created in iO
    And an address with street '1st' is created in iO
    And the importer has been run
Then an imported address with street '1st' should exist in Moves

no comments

Rails log analyzer

Posted by Bart ten Brinke Thu, 14 Aug 2008 21:06:12 GMT

You've probably all been there: your application is running slow, but why? What views or actions are clogging up the mongrels? Or are the mongrels just waiting for the database?

Rails log analyzer is a simple but very powerful command-line analysis tool to quickly determine what is taking time, on all kinds of different levels. At the moment it can tell you the following statistics:

  • Top 10 most requested actions
  • Top 10 actions by time - cumulative
  • Top 10 actions by time - per request mean
  • Top 10 worst DB offenders - cumulative time
  • Top 10 worst DB offenders - mean time
  • Mongrel process blockers (> 1.0 seconds) - frequency
  • Requests graph - requests per hour

For an example run, or the analyzer take a look at the github. http://github.com/wvanbergen/rails-log-analyzer/

Posted in ,  | Tags , , ,  | no comments

Background processing with WorkerQueue

Posted by Bart ten Brinke Wed, 13 Aug 2008 19:25:48 GMT

Yes, I know there are a lot of background runners available for rails (I tried most of them), and they all seemed to have their downsides:

  • BackgroundRB -> Major memory leaks when performing large tasks and bloaty.
  • Spawn -> I really do not want to fork on every request. I use merb for that.
  • Starling -> Memcache and only does sequential processing.
  • ActiveMessaging -> Round hole Square peg.

Also non of these can handle big file uploads nicely and reliable. That is when I decided to create my own: WorkerQueue.

Main features:

  • Simple
  • Fast
  • Works via a rake task and prefers cron
  • Can handle large amounts of data
  • Sequential and Parallel execution on demand
  • Simple error reporting
  • Offload tasks to other or multiple machines

You can clone a copy here:

  git clone git://github.com/barttenbrinke/worker_queue.git vendor/plugins/worker_queue

Or visit github.

UPDATE: WorkerQueue 1.0 Stable is now available on Github.

Posted in  | Tags , , ,  | 2 comments

The power of simplicity

Posted by Bart ten Brinke Mon, 28 Jul 2008 09:23:05 GMT

Our colleagues from iO made this really nice clip about the power of simplicity in IT solutions.

This is iO - YouTube

Don't digitize your problems, automate them :)!

Posted in  | Tags , ,  | no comments

Testing ActiveResource based models: HTTPmock and parameters

Posted by Pieter Bos Mon, 14 Jul 2008 15:54:04 GMT

Some of you probably have written and written tests for applications with activeresource. You might know that you can use httpmock to test applications. However, you cannot properly use parameters, only a path wich contains them in a fixed order. For example, if you want to use the parameters city and country, you have to define the following two entries:

"/visits/2008-01-15?country=Netherlands&City=Groenlo"
"/visits/2008-01-15?city=Groenlos&Country=Netherlands"

Of course, this is no use. A http mock should be able to respond to all parameters, regardless of the order.

So with the adapted http mock, you can just do this:

    ActiveResource::HttpMock.respond_to do |mock|            
      mock.get '/visits/2008-01-15.xml?country=Netherlands&city=Groenlo'', 
            {}, "<visit><data>somecontent</content></weather>", 
            200, {}
    end 

Now write a test containing something like:

Visits.find("2008-01-15'" :params => {
            :country => "Netherlands", 
            :city => "Groenlo").data.should == "somecontent"

And it'll work :).

Download the patch

Also, a request in httpmock defined the following function:

def == (other)
  hash == other.hash 
end

which is fixed as well.

no comments

Nested include has major memory leak (Rails 2.0.1).

Posted by Bart ten Brinke Wed, 02 Jul 2008 14:00:33 GMT

As our mongrels were using up quite a lot of memory, so I tried to figure out what was causing this.

When running the app locally I found out that one certain page caused the mongrel to grow from 60 to 190 megabytes. A whopping 130 megabytes!

After commenting out some of the code, I realized that a single line was causing all of the memory usage

contracts = Contract.find( :all, 
  :conditions => ['contracts.employee_id IN (?) ', employees ],
  :include => [:expertise_profile => :qualifications ] )

Ouch! The nested include of rails somehow leaks a large amount of memory. The fix was a piece of cake.

contracts = Contract.find( :all, 
  :conditions => ['contracts.employee_id IN (?) ', employees ],
  :include => :expertise_profile )

Now my mongrel stays at 60 megabytes. I don't know if this issue persists in the new 2.1 Rails, but I'll check that soon!

Posted in  | Tags , , , ,  | 4 comments

libXML and activeresource revisited

Posted by Pieter Bos Thu, 26 Jun 2008 08:32:58 GMT

A while ago, an article was posted on this blog by Bart which showed how activeresource can be sped up with libXML. However, this patch was not complete and broken in rails 2.1. So Brian Guthrie made a new patch. However, this patch also has an issue:

If you parse an xml file containing the google weather api for example, you get something like:

<weather> <current_conditions></current_conditions> <forecast_conditions><low data="10"/></forecast_conditions> <forecast_conditions><low data="12"/></forecast_conditions> </weather>

to an object with a single forecast_conditions object with the low temperature at 12 degrees.

So, i created a simple patch:

   def to_hash(hash={})
      #if there is already an entry with the given name, switch 
      if text?
        hash[CONTENT_ROOT] = content
      else            
        sub_hash = insert_name_into_hash(hash, name)            
        attributes_to_hash(sub_hash)
        if array?
          children_array_to_hash(sub_hash)
        else
          children_to_hash(sub_hash)
        end
      end
      hash
    end

    protected

    def insert_name_into_hash(hash, name)
      sub_hash = {}
      if hash[name]                        
        if !hash[name].kind_of? Array 
          hash[name] = [hash[name]]              
        end
        hash[name] << sub_hash
      else
        hash[name] = sub_hash
      end   
      sub_hash
    end

This replaces the line that just overwrites the previous hash with one that creates an array if required. Also, libxml doesn't like empty files. So much that it just segmentation faults on empty input. So we add to the from_xml method:

        def from_xml(xml)              
          xml.gsub!(/\s*\n\s*/, '')
          if(xml.blank?)
            return {}
          else              
            typecast_xml_value(undasherize_keys(XML::Parser.string(xml).parse.to_hash))
          end
        end

Download the patch!

1 comment

Campfire Notifier for cruisecontrol.rb

Posted by Andre Foeken Thu, 17 Apr 2008 11:39:36 GMT

We added a campfire notifier to our build of cruisecontrol. Click on the link to download the builder plugin. It uses tinder as an API.

Posted in  | 2 comments

Optimizing math

Posted by Andre Foeken Thu, 10 Apr 2008 21:19:19 GMT

As you might have guessed from Bart's previous article we've been looking at ways to speed up our Rails app. We've been profiling and query optimizing but at some point we reached a dead end.

Our app needs to calculate a lot of distances between geo locations. Until now we've been happy using a home-grown Ruby method to calculate these distances but our profiling showed that it was (as one may suspect) horribly slow.

We now have several options:

  • We could try to built a faster Ruby method (but that would be hard since it's pure math and not really a lot can be done here)

  • We could use the database (mysql in our case) to calculate our distances (A lot more db connections in our case, since we need distances between lots of points. Not the standard stuff that gems like acts_as_mappable can handle)

  • Use RubyInline to create a faster C based method

We decided to look at RubyInline. A gem that enabled C code to be used right inside a Ruby script. We rewrote the method in C. A simple benchmark proved that our inline C method was 2.3 times faster!

require 'inline'
inline do |builder|
    builder.include '<math.h>'
    builder.c "double calc_distance_between(...) { ... }"
end

Although this result is very good, it does complicate your app and makes it less readable. These inline methods have to be used with care. But in our simple (and very localized) case we decided to keep the C method in favor of the pure Ruby call.

Posted in ,  | Tags , , , , ,  | 1 comment

Optimizing the queries of your rails app

Posted by Bart ten Brinke Wed, 09 Apr 2008 19:51:20 GMT

When you are developing your application, you should allways look for the following line in your development console.

Processing EmployeesController#index[GET]
Employee Load (0.055003) 
SELECT * FROM `people` WHERE `people`.`type` = 'Employee'

 select_type | key_len | type | Extra       |
---------------------------------------------  =>
 SIMPLE      |         | ALL  | Using where |

| id | possible_keys | rows | table  | ref | key
--------------------------------------------------
| 1  |               | 6965 | people |     |

The type ALL means that you are preforming a full table scan in a query. This is usually not a problem when you are in development mode, but what is you have millions of people in your database?

Usually it is pretty straight forward to find out where you were calling this from, as you will probably remember what the request was you did. If you are having a hard time, install the query_trace plugin. This gives the following result:

vendor/plugins/query_analyzer/lib/query_analyzer.rb:38:in `select'
app/controllers/employees_controller.rb:71:in `find'
app/controllers/employees_controller.rb:71:in `index'
vendor/plugins/browser-prof/lib/browser-prof.rb:32:in `process'

Looking at line 71 of the employees controller is a good idea here as you might be doing something stupid. As line 71 just reads: @employees = Employee.find(:all)) we have to turn to your database.

mysql -u root --database myapp_development

mysql> EXPLAIN SELECT * FROM `people` WHERE `people`.`type` = 'Employee';
+----+-------------+--------+------+------------------+
| id | select_type | table  | type | possible_keys    |
+----+-------------+--------+------+------------------+ =>
|  1 | SIMPLE      | people | ALL  |                  |
+----+-------------+--------+------+------------------+

+------+---------+------+------+-------------+
| key  | key_len | ref  | rows | Extra       |
+------+---------+------+------+-------------+
| NULL | NULL    | NULL | 6873 | Using where |
+---------+------+------+------+-------------+
1 row in set (0.00 sec)

As you can see we are not hitting any indexes. Lets try adding an index.

mysql> create index people_type_test on people (type);
Query OK, 6715 rows affected (1.38 sec)
Records: 6715  Duplicates: 0  Warnings: 0

Now we run the explain again:

mysql> EXPLAIN SELECT * FROM `people` WHERE `people`.`type` = 'Employee' ;
+----+-------------+--------+------+------------------+
| id | select_type | table  | type | possible_keys    |
+----+-------------+--------+------+------------------+ =>
|  1 | SIMPLE      | people |range | people_type_test |
+----+-------------+--------+------+------------------+

+------------------+---------+-------+------+-------------+
| key              | key_len | ref   | rows | Extra       |
+------------------+---------+-------+------+-------------+
| people_type_test | 768     | const | 2496 | Using where |
+------------------+---------+-------+------+-------------+
1 row in set (0.00 sec)

Thats more like it, now we need to add this to our app trough a migration.

class CreatePeopleIndices < ActiveRecord::Migration
  def self.up
    add_index :people, :type
  end

  def self.down
    remove_index :people, :type    
  end
end

After a db:migrate and a restart of the server, we now see the following in the development console:

Employee Load (0.027666)
SELECT * FROM `people` WHERE `people`.`type` = 'Employee'
Analyzing Employee Load

 select_type | key_len | type | Extra       |
---------------------------------------------  =>
 SIMPLE      | 768     | ref  | Using where |

| id | possible_keys        | rows | table  | ref   | key
----------------------------------------------------------------------
| 1  | index_people_on_type | 2496 | people | const | people_type_test

Success! Also note that the load on the database has been cut in half.

Posted in  | Tags , , , ,  | 2 comments

Older posts: 1 2 3 ... 7