Posts Tagged Ruby

Experimental Resolution of Gem Dependencies with DNS

This week Jeremy Hinegardner (aka @copiousfreetime) was in Montpellier to visit. In addition to enjoying some good meals, good wine and time in the pool we also took this opportunity to work on an experimental gem dependency resolution system using DNS. I’ve written about this before and the latest gist has some new details in it as the result of our work this week.

First of all we’ve outlined how to deal with some of the dependency issues we had not yet dealt with, such as less-than, less-than-or-equals-to and not-equals. Additionally we outlined how to deal with situations where a dependency defines 2 or more requirements (such as ["> 1.0.0", "<= 2.0.0"]). We’ve also added in support for listing all known versions of a specific gem by moving the notion of the latest gem version so it uses a prefix of latest and then putting one PTR for each version of the gem on the gem domain name.

In addition to working on the details of the spec we also got some data loaded! Thanks to the work Jeremy did previously on his Gemology project a lot of the code had already been written. I spent some time working on moving the code we needed into the rubygems-dns project while Jeremy worked on exporting the existing specs we needed from his database and writing an import script. While the implementation is far from complete I’m happy to say that you can actually query the DNS server now and answer questions with it.

First a query to list all of the versions of a gem. We’ll use the Sinatra gem as an example:


dig @ns8.dnsimple.com sinatra.index.rubygems.org PTR

The result will be an unordered list of PTR records, with each record representing a version with the version number in reverse order:


sinatra.index.rubygems.org. 86400 IN PTR 1.3.0.sinatra.index.rubygems.org.
sinatra.index.rubygems.org. 86400 IN PTR 0.1.0.sinatra.index.rubygems.org.
sinatra.index.rubygems.org. 86400 IN PTR 5.1.0.sinatra.index.rubygems.org.
sinatra.index.rubygems.org. 86400 IN PTR 6.1.0.sinatra.index.rubygems.org.
sinatra.index.rubygems.org. 86400 IN PTR 7.1.0.sinatra.index.rubygems.org.
sinatra.index.rubygems.org. 86400 IN PTR 0.2.0.sinatra.index.rubygems.org.
...snip...

So the first line indicates that there was a 0.3.1 version of Sinatra and line 2 indicates there was a 0.1.0 version of Sinatra.

Next, let’s get the latest version:


dig @ns8.dnsimple.com latest.sinatra.index.rubygems.org CNAME


latest.sinatra.index.rubygems.org. 600 IN CNAME e.0.3.1.sinatra.index.rubygems.org.

This shows us that 1.3.0.e is the latest sinatra version published (as of the snapshot).

Next, we can query for PTR records for that same name:


dig @ns8.dnsimple.com latest.sinatra.index.rubygems.org PTR


latest.sinatra.index.rubygems.org. 600 IN CNAME e.0.3.1.sinatra.index.rubygems.org.
e.0.3.1.sinatra.index.rubygems.org. 86400 IN PTR 1.tilt.index.rubygems.org.
e.0.3.1.sinatra.index.rubygems.org. 86400 IN PTR 1.rack.index.rubygems.org.

This shows us that the version 1.3.0.e has two runtime dependencies, tilt and rack, and that they require the latest version of the 1.x releases for each of those. We can then use:


dig @ns8.dnsimple.com 1.tilt.index.rubygems.org PTR

To show us that the latest version:


1.tilt.index.rubygems.org. 600 IN CNAME 2.3.1.tilt.index.rubygems.org

That’s it for now. There is more work to do, like encoding the <, <=, != and dependencies with 2 or more requirements, but this will at least give an example how this all might actually work.

, ,

1 Comment

Testing REST APIs with Cucumber and Rack::Test

Corey Haines has been staying here in South France with me for the last few days and we had been meaning to pair up on some coding tasks, but hadn’t been able to until today. I had in my mind already what I wanted to work on and he was happy to oblige. Our task was to set up some cucumber features that would describe the DNSimple REST API properly, allowing all of the available end points to be tested. I had worked on this in the past trying to use Webrat, but since moving the DNSimple code to Rails 3 I’ve switched to Capybara none of the old code worked. Additionally the old code was only able to handle GET requests and never properly handled POST, PUT and DELETE, nor did it handle the various content types supported by the API.

This evening Corey and I sat down and started looking into how we could make the tests live again. First we attempted to use Capybara directly, but ran into issues when trying to set HTTP headers. The DNSimple API uses Basic HTTP Authentication and thus we needed to set the header for that. We also need to set the Content-Type and Accept headers to ensure that Rails handles the input and output correctly. After peeling back some layers of the Capybara onion we landed upon Rack::Test. When using Rack, Capybara delegates request and response handling down to Rack::Test. At this point we decided to see if we could use Rack::Test directly in our step definitions, and it worked like a charm.

Rack::Test has a module called Rack::Test::Methods that can be mixed into a class to provide it with methods for get, post, put, delete as well as last_request, last_response, header and more. We mixed Rack::Test::Methods into the Cucumber world at the top of our API steps file like so:

Then our steps could simply call the Rack::Test methods directly. For example, here’s how to send the HTTP Basic Auth header:

And to set content type and accept headers:

And finally to actually send requests and test the responses. The entire api_steps.rb as it is right now looks like this:

And here is an example of a Cucumber feature that uses the API steps:

I doubt we’ve ended up with the ultimate final set of steps that can be used repeatedly for testing RESTful APIs, however I think it is an excellent start. I’m still on the fence about how to check the response body, specifically whether it is better to match the response body as a string or to actually parse it and write custom steps for each of the different types of objects, but time will tell.

As usual pairing with Corey resulted in some good code, a usable solution and something that I think others will benefit from as well.

, , , , ,

12 Comments

Cucumber and Resque Jobs

Recently I’ve begun using Resque on more and more of my projects. Resque is a very well-implemented queueing system for Rails background jobs. I also use Cucumber to describe the expected behavior of the applications I write. Cucumber expects a sequence of steps that are executed synchronously so mixing in an asynchronous task is a bit of a problem. The good news is that I’ve come up with an easy way to take those background tasks and force them to run immediately as a synchronous job with very little code.

This file goes in features/support and opens up the Resque module to replace the enqueue method with one that delegates immediately to the worker. This is a simple and effective technique and works like a charm.

, , ,

1 Comment

Stubbing Before Everything with RSpec

Let’s say I want to always stub some external service client before every model and controller spec. For example, if I want to stub RSpreedly (a nice client interface to the Spreedly recurring payment processor), I put the following in spec/support/stubs.rb:

Now RSpreedly::Subscriber instances will always be stubbed. Simple and sweet.

, , ,

No Comments

XSS and Building HTML in JavaScript

In my recent article on Rails, JQuery, Unobtrusive JS and Graceful Degradation I implemented the Ajax logic for adding a new entry using JSON returned from the server to build the HTML in JavaScript. An attentive reader brought to my attention the cross-site scripting vulnerability that is possible with this implementation and I think this would be a good time to address the issue and open the comments to other options.

First of all let’s look at the most obvious attack vector, which is pretty simple: a malicious user would enter a script into the person name field and save that and then it would execute the script when the person is rendered in the person list. First I verified that this attack vector worked, and yes it did work, so then I considered where the value entered by the user needed to be sanitized. It’s pretty straightforward to sanitize the rendering when Rails is involved, just add the h() helper before the name output in the listing, so line 4 of index.html.erb becomes:

This is a fairly standard idiom in Rails (not necessarily the best way to sanitize user input, but the most common way to do it). This fix only fixes the problem when the page is rendered by Rails, but it does not address the issue for items that are added via Ajax, and here is where things get a bit tricky.

You’d think that sanitizing user input in JavaScript would be fairly easy, and to be clear you can do it with the JavaScript escape() function, however this results in some pretty ugly percent-encoded content (which may or may not matter to you. I was curious though whether there was something similar to the h() helper in Rails available in JavaScript. There is but finding it proved to be a bit of a challenge.

What I ended up doing what searching Stack Overflow for solutions that others had come up with and eventually landed upon this bit of code: http://code.google.com/p/google-caja/source/browse/trunk/src/com/google/caja/plugin/html-sanitizer.js. Unfortunately this bit of code by itself doesn’t work unless you have another JS library (html4) and that library requires building with Ant. Fortunately additional searching led me to a minified version with everything already compiled. I’ve included this minified library as html-sanitizer-minified.js and added a call to html_santize before rendering the person name in the JavaScript as such:

..and it works. I’ve deployed this fix to http://rails-jquery-unobtrusive.heroku.com/ and have updated the github repo so you can see the changes in action.

So is this the best solution for handling user input and avoiding XSS when you want to build HTML in JavaScript? I’m not sure. It’s probably not the only solution so feel free to add yours in the comments below. Also, consider this a warning if you are developing JavaScript code that builds and renders HTML: tainted user input is a threat and you need to deal with it.

, , , ,

1 Comment

Rails, JQuery, Unobtrusive JS and Graceful Degradation

I’m always on the lookout for ways to improve my web applications. To that end I wanted to see if I could come up with a way to marry jQuery and Rails in a way that was both unobtrusive and a way that would degrade gracefully in a Rails 2.3.5 application. I’m looking forward to seeing how UJS with jQuery in Rails looks as of Rails 3.0, however in the meantime the method I describe below may prove to be useful.

If you want to go straight to the code it is available on Github: http://github.com/aeden/unobtrusive-jquery-example. I have also put a live demo on this up on Heroku: http://rails-jquery-unobtrusive.heroku.com.

Rather than going through all of the steps needed to set up a Rails app, the best thing to do is to follow along in the code from the example above.

First let’s take a look at the PeopleController (app/controllers/people_controller.rb):

This looks like a fairly normal RESTful controller. The one exception is the delete method, which will be used by browsers that have JavaScript turned off to provide a page that 1.) confirms that the record should be deleted and 2.) has a form that can post to the destroy method.

Next, let’s take a look at the index view:

There is a list of people with each person having a delete link next to their name. The delete link points to /person/:id/delete which will execute the delete action that I mentioned above (I’ll show you how to remove the “/delete” part of that link with JavaScript later). There is also a form for adding a new person. Note that there is no JavaScript in this page, just normal HTML elements with classes.

While we’re looking at views, let’s look at the only other view, the delete view:

This is just a simple form that confirms that they want to delete the user and provides a cancel link back to the people_url if they don’t. Note that the form tag uses the REST-ful URL for accessing the resource and sets ‘_method’ to ‘delete’ in the hidden field tag.

Finally, let’s take a look at the most complicated bit, the jQuery JavaScript. To make things easier to follow I’ll go through it piece by piece, starting with the document ready handler:

This is the jQuery way for executing code when the document has finished loading. All it does is call two methods, one for connecting up the add form and one for connecting up the delete links. To keep things organized and namespaced a bit I placed these two methods onto a JS object called ‘Actions’. First, the connectAddForm implementation:

This function finds the form element with the class “people” and attaches an event handler to the submit event. When the form is submitted an AJAX POST is made. The data type is JSON, which will call the ‘create’ action in the people controller with the accepts type set to a value that triggers Rails to render JSON as the response. The form is serialized and passed as data and when a successful response is received a bit of HTML is appended to the people list and the person name field is cleared.

Now for the connect delete links JavaScript:

Here each element that is in the people list that matches the selector a.delete and adds a click handler to it. Notice that I used the jQuery live method here, the benefit being that as new records are added, and new elements that match the selector expression are added automatically get the click handler that I define here. This saves me from the trouble of having to add the handler each time that a record is added to the page. The click handler first finds the li that it may eventually delete, displays a confirmation message to the viewer and if they confirm that they want to delete the record it will post via AJAX to the /person/:id URL with the method ‘delete’. The URL is pulled from the anchor’s href attribute and the ‘/delete’ is removed using a simple regular expression replace. Upon a successful delete the li that was found earlier is hidden using the slideToggle() function.

That’s it. You can try this in a browser either with JavaScript turned on, in which case it’ll use all of the JavaScript goodness, or you can turn of JavaScript and see the magic of graceful degradation. I hope you’ve enjoyed reading this little tutorial and if you have any suggestions on how to improve the code in the project please feel free to fork the github project, make your changes and send me a pull request. I’ll add any cool mods into the tutorial as I can.

Update: I’ve published a follow-on article that talks about changes to the code for this tutorial to handle sanitizing user input both in Rails and in JavaScript. I suggest reading it to understand XSS vulnerabilities that you need to deal with when accepting user input.

, , , ,

4 Comments

Aloha on Rails – Technical Debt Slides

Thanks to everyone who came out to Aloha on Rails and a huge mahalo to Seth for putting so much into it. It was awesome.

Here are the slides for my talk: Technical Debt . I’ve left the presenter notes in so you can see what I was thinking when I put the slide together (since most of the slides are just pictures anyhow.) Enjoy!

, , , , ,

2 Comments

SDC Hawaii LLC – Agile Web Development in Paradise

Over the last year-and-a-half (maybe even longer), we at SDC Hawaii LLC have been working hard on building dotMP, chi.mp and other projects for a Saipan-based company. We’ve decided that the time is right to start taking on new contracts, so as of Friday SDC Hawaii LLC is available for custom web application development. We’ve even rolled out a brand new SDC Hawaii LLC web site.

So what do you get if you contract the SDC Hawaii team? How about this: you get 4 top notch Ruby developers, including yours truly, along with one awesome designer/UX expert, who are ready to bend over backwards to make sure that your web application is designed, tested and implemented with great care. We are all passionate about web application development and put our hearts and souls into everything we create. Our team is well-versed in agile development, with daily stand-ups, pair programming and test-driven development. And finally we have extensive experience with Amazon’s Web Services including S3, EC2, SQS and Elastic Map Reduce.

So come on over to the SDC Hawaii web site and contact us if you’re interested in having our team develop your next web application.

, , , , ,

No Comments

Capistrano, Rails and Amazon CloudFront

Amazon CloudFront (ACF), the content delivery network that is now part of Amazon’s Web Services provides a simple means of deploying content to a fast, low-cost global content delivery network. As part of our deployment process for chi.mp we now deploy our static assets to ACF (well, technically this will be deployed as part of the next release). Here’s how we do it: http://gist.github.com/95255.

Let’s go through this piece by piece:


# get the previous timestamp
old_timestamp = File.read("config/deploy_timestamp").to_i rescue 0

The first step is to get the previous timestamp. This will be saved in memory and compared against file modification times to determine if a put or a copy should be used.

# generate timestamp into config/deploy_timestamp
timestamp = Time.now.to_i
File.open("config/deploy_timestamp", 'w') do |f|
  f.write(timestamp)
end

The next step is to update the time stamp file and store the new time stamp in memory.

# generate minified JS and CSS
system('rake asset:packager:build_all')

This portion uses the asset packager plugin for Rails to package up JS and CSS files into bundled assets.

# sync local public/ directory to S3 bucket
# the S3 bucket directory should be the
#  timestamp generated above
require 'right_aws'
s3 = RightAws::S3.new(access_key_id, secret_access_key)
bucket = s3.bucket('a-bucket')

Here we connect to S3 using RightScale’s RightAws library. You will need to put your access key ID and secret access key into the variables provided. The bucket name should be a bucket that has already been set up as an Amazon CloudFront S3 source. Documentation on how to set up a CloudFront configuration can be found in the ACF Getting Started Guide.

put_count = 0
copy_count = 0
Dir.glob('public/**/*').each do |f|
  next if File.directory?(f)
  key = "#{timestamp}/#{f.gsub(/public\//, '')}"
  if File.new(f).mtime.to_i > old_timestamp
    puts "putting #{f} into S3 as #{key}"
    bucket.put(key, File.read(f), {}, 'public-read')
    put_count += 1
  else
    old_key = bucket.key(
      "#{old_timestamp}/#{f.gsub(/public\//, '')}"
    )
    if old_key.exists?
      puts "copying #{old_key} to #{key}"
      old_key.copy(key)

      acl = s3.interface.get_acl(bucket.name, old_key.name)
      s3.interface.put_acl(bucket.name, new_key.name, acl[:object])

      copy_count += 1
    else
      puts "putting #{f} into S3 as #{key}"
      bucket.put(key, File.read(f), {}, 'public-read')
      put_count += 1
    end
  end
end
puts "done. #{put_count} files uploaded,
  #{copy_count} keys copied"

This code loops through all of the files and directories in the public directory and for any file it first checks to see if the file modification time is after the previous timestamp. If it is then the data from the file will be pushed to S3 using a key that is the file path prefixed with the time stamp. If the modification time is before the previous time stamp then the file hasn’t changed. In this case the script will look up the old key in S3. If the old key exists then key.copy() is used to make a copy of the resource (reducing the time required to process large files) and set the ACL to the old key’s ACL. If the key does not exist then the file data will be put into S3.

The S3 key is always prefixed with the time stamp. This is done because Amazon CloudFront will cache files for a minimum of 24 hours. If you were to release a new version of an asset and overwrite the old asset in S3 it could be quite some time before ACF would pick up the change. By prefixing the key with the time stamp you will always get the latest version as long as the asset host is configured properly in your Rails application (more on that in a bit).

# add and commit the config/deploy_timestamp file
system('git add config/deploy_timestamp')
system('git commit -m "deploy_assets complete,
   updating timestamp"')
system('git push')

This last little bit of code commits the updated deploy_timestamp file into the git repository and pushes it to the remote repo. This works for us because we use Github. If you do not then you’d want to adjust these lines to push the file into your source code repository.

The last piece of the puzzle is to set the asset host in Rails. Here’s what that might look like:

ts_file = File.join(RAILS_ROOT, "config/deploy_timestamp")
ts = File.read(ts_file).to_i
config.action_controller.asset_host = Proc.new { |source, request|
  if request.ssl?
    "https://yoursite.com"
  else
    "http://cdn.yoursite.com/#{ts}"
  end
}

This code can go either in your config block in config/environment.rb or in specific environment configs. You’ll also need to set up a CNAME record pointing cdn.yoursite.com to your Amazon CDN host. If you aren’t using Rails you’d still need some way to indicate that all images should be originating from Amazon CloudFront. Note that resources requested from SSL encrypted pages must still go to your SSL-enabled servers since CloudFront does not support SSL at this time.

One final caveat: if you are using images that are specified in stylesheets you’ll need to ensure that you use relative paths to those images.

Update 1 It turns out that when you copy a key in S3 the original key ACL is not retained. This is unfortunate since it means copied assets will now be marked private. Furthermore it appears that RightAWS does not support URI-based group identifiers for S3 ACLs, which means there is no way to change the permissions on a copied key to public-read. It seems likely that I can switch to another S3 lib to get this functionality, but that’ll have to wait. More updates will be forthcoming when I fix this.

Update 2 Thanks to Alex’s comment I’ve re-enabled the key copy in the example code. It’s a bit slow to copy the ACL, but for large files it will still perform significantly better.

, , ,

2 Comments

S3 in Moneta

Sometimes you just have to shave a yak, mostly because it’s fun.

I am working on a project where I need access to a distributed key/value store, in my case S3. I could have just gone to S3 directly but since Yehuda Katz recently announced Moneta, which is a generic key/value store interface I figured I’d give a shot at building an S3 implementation. The result is in my fork on github.

Some points of interest. I refactored the specs so that specs which had dependencies (such as the memcache implementation) are in their own spec files. This makes it much easier to run only a single spec for one implementation at a time, highly useful since some of the dependencies are not things I really want or need on my dev machine, nor do I really want to run all of the specs every time.

The Moneta interface includes the ability to specify :expires_in with a number of seconds as a common means for setting expiration of the key in the store. This is not something S3 supports out of the box. Initially I tried to make it work with HTTP Expires header to no avail. I settled on using a meta header (non-standard HTTP headers used by S3 for client info) to store the expiration date and then I check for the expiration when retrieving the S3 key object. This actually worked quite well even though it did require jumping through some implementation hoops and resulted in more code than I was originally hoping for.

I also had to raise the expiration times just a smidge in the specs since it was quite possible that the test operation would not complete before 1 second would pass and it was causing random behavior. Once I got the expiration working it was golden and all of the specs were passing.

Side note: I don’t usually use RSpec, but the ability to useĀ it_should_behave_like “some spec” is quite elegant.

That’s all for now. Enjoy!

, , , ,

4 Comments