Wednesday, August 27th, 2008

IronRuby vs JRuby vs MRI. Performance Mini-Shootout.

One of the things I went over in my eRubyCon talk was the relative performance of IronRuby compared to both JRuby  (which I consider to be a cousin of sorts to IronRuby) and MRI. I did this by running the same tests that Antonio Cangiano ran in December.

There were a few test results that really stood out.  The lists test was by far the worst for IR, where it was 56 times as slow as MRI and 3 times as slow as JRuby.  But this wasn’t the only sore spot, there were other places that show room for improvement.  Namely, vm1_blocks in which IronRuby was almost 9 times as slow as MRI.  Also so_exception where IronRuby brought up the tail by being 10 times as slow as MRI.

The nitty gritty is that IronRuby is about twice as slow as MRI with JRuby being slightly faster, overall.  As the below graph illustrates:


There are a few high points in this test though.  Number one, is that IronRuby ran the majority of the tests.  There were only four failures out of forty, for a 90% success rate.  For comparison JRuby had 95% success rate. Number two, there were actually a few tests in which IronRuby beat JRuby and MRI.  No small feat for such a young implementation.

I think this mini-shootout illustrates that there is a lot of room for improvement for IronRuby.  However, a little perspective goes a long way.  JRuby was announced in 2001, IronRuby was announced last April.  The JRuby team has had 7 years to get to where they are today, whereas IronRuby has only had almost a year and a half! As the DLR improves IronRuby will receive those performance gains for free, and as compatibility becomes more stable the core team can focus on performance as well. The raw data can be downloaded here.

6 Comments on “IronRuby vs JRuby vs MRI. Performance Mini-Shootout.”

  1. A bit more perspective: JRuby’s probably been completely rewritten a half-dozen times. The current core runtime shares little in common with the code from 2002. We’ve also only had full-timers on it since late 2006, shortly before John Lam went to Microsoft; before then it was a pure spare-time OSS project. Since most of JRuby core has been rewritten in the intervening two years (and we’ll give it another 6 months, since we really ramped up efforts a little before Tom and I were hired by Sun) it’s probably safer to compare IronRuby’s 1.5 years with the 2.5 years JRuby’s been under intensive work. Anything we gained before those 2.5 years would be akin to what IronRuby gained from work on Ruby.NET, IronPython, and DLR. And it’s safe to say that the runtime portions of the DLR are not just enabling technologies, they’re absolutely essential technologies for implementing dynamic languages on the CLR, since it does not do the same sorts of dynamic optimizations the JVM does. DLR, then, simply gives dynlangs on CLR a fighting change to compare performance-wise with the JVM as-is.

    What am I trying to say? I guess that IronRuby is about on schedule in comparison to JRuby, all tings considered. And neither the DLR (great though it may be) nor the IronRuby developers (skilled though they are) make it any easier to implement Ruby on the CLR than on the JVM. And I doubt the current IronRuby team would claim any differently. Ruby is hard.

  2. Michael

    Thanks for the clarification Charles. I would be curious if there’s any historical JRuby performance data just as way of comparison.

  3. Nothing published, but all the past JRuby versions are available out there.

  4. Thanks for doing the investigation. The only perf-related stuff that we’re looking at right now is startup time and straight-line code throughput. In the short term, we’re going to write an interpreter to speed up startup. In the medium-long term, we’re going to look at cached precompilation.

  5. […] 19-12-2008 JRuby access to C libraries using FFI Saved by flyingkaratechop on Fri 19-12-2008 IronRuby vs JRuby vs MRI. Performance Mini-Shootout. Saved by jimmyandsammy on Mon 15-12-2008 ruby-plsql gem now supports JRuby and Oracle JDBC driver […]

Leave a Reply