Writing multi-threaded applications is quite a challenging task, but perhaps even more challenging is debugging them when they don’t work as you expect. These past couple of weeks I’ve been working hard to troubleshoot issues with a distributed processing system built on Refinery. When running with a relatively small number of threads things work fine – processing goes about its merry way at a level of throughput that is expected. As soon as we increase the number of threads though, all hell breaks loose.

As part of the troubleshooting process we decided to hook up a profiler. I started out looking at the profiler that’s built into JRuby (use the –sample option) and got a little bit of data. Urged on by @dje I started looking at Java profilers. It’s been a while since I’ve dug into the world of Java so I started with open source tools to get my bearings. Suffice to say I did not find what I was looking for there, so I did what any self-respecting Internet-savvy being would do in my place: I twittered it.

Thanks to @sethladd I downloaded YourKit, got it hooked up in about 30 minutes with the production system and away I went. The results so far have been promising. Some of the issues that could be fixed have, however we are basically at a point where we’ve run into a wall with Ruby’s Net::HTTP implementation. The long term solution is probably to replace the Net::HTTP library with something based on the Reactor pattern (see my jruby-http-reactor project for the initial baby-steps in that direction) but the bottom line is without a profiler to help us get headed on the right path we probably would have spent a lot more time “fixing” things that weren’t broken. If you’re considering using JRuby consider that you will have access to a wide variety of Java tools that can be quite helpful when troubleshooting.