Wednesday, August 27th, 2008

IronRuby vs JRuby vs MRI. Performance Mini-Shootout.

One of the things I went over in my eRubyCon talk was the relative performance of IronRuby compared to both JRuby  (which I consider to be a cousin of sorts to IronRuby) and MRI. I did this by running the same tests that Antonio Cangiano ran in December.

There were a few test results that really stood out.  The lists test was by far the worst for IR, where it was 56 times as slow as MRI and 3 times as slow as JRuby.  But this wasn’t the only sore spot, there were other places that show room for improvement.  Namely, vm1_blocks in which IronRuby was almost 9 times as slow as MRI.  Also so_exception where IronRuby brought up the tail by being 10 times as slow as MRI.

The nitty gritty is that IronRuby is about twice as slow as MRI with JRuby being slightly faster, overall.  As the below graph illustrates:


There are a few high points in this test though.  Number one, is that IronRuby ran the majority of the tests.  There were only four failures out of forty, for a 90% success rate.  For comparison JRuby had 95% success rate. Number two, there were actually a few tests in which IronRuby beat JRuby and MRI.  No small feat for such a young implementation.

I think this mini-shootout illustrates that there is a lot of room for improvement for IronRuby.  However, a little perspective goes a long way.  JRuby was announced in 2001, IronRuby was announced last April.  The JRuby team has had 7 years to get to where they are today, whereas IronRuby has only had almost a year and a half! As the DLR improves IronRuby will receive those performance gains for free, and as compatibility becomes more stable the core team can focus on performance as well. The raw data can be downloaded here.

7 Comments on “IronRuby vs JRuby vs MRI. Performance Mini-Shootout.”

  1. A bit more perspective: JRuby’s probably been completely rewritten a half-dozen times. The current core runtime shares little in common with the code from 2002. We’ve also only had full-timers on it since late 2006, shortly before John Lam went to Microsoft; before then it was a pure spare-time OSS project. Since most of JRuby core has been rewritten in the intervening two years (and we’ll give it another 6 months, since we really ramped up efforts a little before Tom and I were hired by Sun) it’s probably safer to compare IronRuby’s 1.5 years with the 2.5 years JRuby’s been under intensive work. Anything we gained before those 2.5 years would be akin to what IronRuby gained from work on Ruby.NET, IronPython, and DLR. And it’s safe to say that the runtime portions of the DLR are not just enabling technologies, they’re absolutely essential technologies for implementing dynamic languages on the CLR, since it does not do the same sorts of dynamic optimizations the JVM does. DLR, then, simply gives dynlangs on CLR a fighting change to compare performance-wise with the JVM as-is.

    What am I trying to say? I guess that IronRuby is about on schedule in comparison to JRuby, all tings considered. And neither the DLR (great though it may be) nor the IronRuby developers (skilled though they are) make it any easier to implement Ruby on the CLR than on the JVM. And I doubt the current IronRuby team would claim any differently. Ruby is hard.

  2. Michael

    Thanks for the clarification Charles. I would be curious if there’s any historical JRuby performance data just as way of comparison.

  3. Nothing published, but all the past JRuby versions are available out there.

  4. Thanks for doing the investigation. The only perf-related stuff that we’re looking at right now is startup time and straight-line code throughput. In the short term, we’re going to write an interpreter to speed up startup. In the medium-long term, we’re going to look at cached precompilation.

  5. […] 19-12-2008 JRuby access to C libraries using FFI Saved by flyingkaratechop on Fri 19-12-2008 IronRuby vs JRuby vs MRI. Performance Mini-Shootout. Saved by jimmyandsammy on Mon 15-12-2008 ruby-plsql gem now supports JRuby and Oracle JDBC driver […]

  6. Hi Brice,I got a copy of your .jar file afterall. It was alwyas there, but in the rhino extras .zip, so I boneheadly missed it. But there is a real problem that appear when I try to use the .jar file in an Android java project. I am developing using the Android plugin for Eclipse. And in Eclipse, the .jar file appears to be empty. The problem is that your .jar file has already been converted for use with Android so all the .class files have been replaces with a .dex file. This makes total sense for the ASE repository, since those .jar file will just be copied directly to the phone. But it causes some problems when the .jar file is referenced for use in a java application.When referencing a library in Eclipse, the Android plugin automatically tries to convert the .class files in the referenced .jar file to the classes.dex file, and this process fails with the error “Error generating final archive: duplicate entry: classes.dex” if the .jar file is already converted. Hence your .jar file ends up looking empty in Eclipse and Eclipse will not complie.Would it be possible to get an unconverted copy of your .jar file?I’m really excited to use this stuff because I have some really cool ideas I want to implement and it is a bit silly that the Eclipse plugin for Android can’t use .jar files already converted for Android, but I guess that’s life. :)Thanks!

Leave a Reply