Sometimes you just have to shave a yak, mostly because it’s fun.
I am working on a project where I need access to a distributed key/value store, in my case S3. I could have just gone to S3 directly but since Yehuda Katz recently announced Moneta, which is a generic key/value store interface I figured I’d give a shot at building an S3 implementation. The result is in my fork on github.
Some points of interest. I refactored the specs so that specs which had dependencies (such as the memcache implementation) are in their own spec files. This makes it much easier to run only a single spec for one implementation at a time, highly useful since some of the dependencies are not things I really want or need on my dev machine, nor do I really want to run all of the specs every time.
The Moneta interface includes the ability to specify :expires_in with a number of seconds as a common means for setting expiration of the key in the store. This is not something S3 supports out of the box. Initially I tried to make it work with HTTP Expires header to no avail. I settled on using a meta header (non-standard HTTP headers used by S3 for client info) to store the expiration date and then I check for the expiration when retrieving the S3 key object. This actually worked quite well even though it did require jumping through some implementation hoops and resulted in more code than I was originally hoping for.
I also had to raise the expiration times just a smidge in the specs since it was quite possible that the test operation would not complete before 1 second would pass and it was causing random behavior. Once I got the expiration working it was golden and all of the specs were passing.
Side note: I don’t usually use RSpec, but the ability to use it_should_behave_like “some spec” is quite elegant.
That’s all for now. Enjoy!
#1 by Peter Kieltyka on March 4, 2009 - 3:18 pm
Nice work! How come you decided to use the RightAWS library instead of AWS-S3? RightAWS comes with a lot of other features (EC2, SQS, etc.) that are not necessary for your S3 key-value store. Plus, the AWS-S3 code base looks a lot cleaner.
#2 by anthonyeden on March 4, 2009 - 6:56 pm
I used RightAWS because that’s what chi.mp uses and this is one step towards a much larger goal, one which involves several pieces of AWS. So I guess you could say it’s purely selfish.
#3 by EMK on March 6, 2009 - 5:45 pm
We use RightAWS because Paperclip uses it for S3 backend to file attachments. Once you start using one Amazon web service, you will probably be tempted to use more in the future anyway. For instance RightAWS supports CloudFront, which is a natural next step for many S3 uses. Another plus is that RightAWS sits on top of a robust HTTP layer supporting retries and some other stuff. I don’t know if AWS-S3 has this or not.