Python vs. Perl vs. Java vs. C++ Runtimes

source:
http://www.flat222.org/mac/bench/

This page is a long overdue response to the python/java comparison 'A subjective analysis of two high-level, object-oriented languages ' by Glyph Lefkowitz (now shortened to [1]) (update 6 June 2002: I did email Glyph expecting a link from his page,as it states he would, but never got a reply- until a few days ago when he was kind enough to email me the Klez worm).

The code has been somewhat changed, mainly to make the Java code more natural. when I read the original code I was often thinking "That's not the (simplest/most natural) way to do it". I didn't feel that for the Python code, which I suspect says Good Things about the simplicity and clarity of Python. I've also implemented the code in C++ for comparison.

It's worth noting that both the original tests, and these ones derived from them are extremely naive; most of the tests stand a chance of being no-op'd by a clever compiler (or run time). It would be far more interesting to see them slightly expanded (to things like matrix manipulation, balanced tree generation, etc), and I may tackle that (in my copious free time) in the future.
Update 2002-05-13: Came across Doug Bagley's benchmarks here, this is exactly what I was thinking of, so now I dont have to bother doing it.

Note that these tests all include the startup time for the JVM. If you are willing to discount startup time and focus on the time spent running the benchmark, you can deduct about a second from the Java scores (and almost nothing for Pytohn and C++). This is particularly relevant for the 'no', 'speed' and 'native' benchmarks.

All the times are an average of 3 executions, platform information is at the bottom of the page. I increased the number of iterations for some of the benchmarks from [1], for the sole reason to make the graph look better.

The framework that builds and runs the benchmarks, constructs the graph and produces this page from a template is written in python.

Personal bias disclaimer: I really like Python, Java and C++. I love the freedom and flexibility of Python (but I hate 'self'). I love the raw speed of C++, but hate the arcane development process and debugging it. I like Java for the fact that although it's a bit slower than C++, it's much easier to work with (oh, and large portable libraries too).

Update 10th August 2002, updated the benchmarks on a new machine, updated java versions to 1.4.0_01 as the baseline, and 1.4.1 BETA for something to compare against, got a mail pointing out inconsistant use of range/xrange in the python code which is now fixed. Also added perl for fun - note I'm not a perl programmer, I just hacked these up. Corrections are welcome.

Update 26th August 2002, Got a mail from Jay Soffian who pointed out that Perl has a range operator. Updated the results again to see much better Perl performance - even sneaks past C++ in the console test!

Results Table

Console

The console test is pretty standard, it just dumps data out. The console was redirected to /dev/null to minimise the impact of the efficiency of the terminal. The end result is that everything is well behind C++, but Java is especially bad.

for x in xrange(1000000):
    print x

for $i (0 .. 1000000-1) {
  print "$i/n"
}

public class test {
    public static void main(String[] args) {
        for (int i = 0; i < 1000000; i++) {
            System.out.println(i);
        }
    }
}

#include <iostream.h>


int main(int argc,char *argv[]) {

  for(int i=0;i<1000000;i++) {
    std::cout<<i<<std::endl;
  }

}

Hash

The hash test primarily measures the efficiency of the default hash implementation. For Python this is a dictionary, for Perl an associative array, for Java a HashMap, and for C++ a map. Java suffers an extra penalty here due to the "primitives are not Objects" problem. Thus, Java needs to allocate extra Objectified versions of the integers to install into the map. This "Everything is an Object, except for the fundamental things" dichotomy is my pet hate with Java. This may or may not be relevant. If you commonly want to put primitive types into containers (as I did when doing a lot of Java), this is a fair test showing up a legitimate Java problem. OTOH, if you normally already have real Objects that you want to use, you can expect Java to be faster than is shown here.

Before any perl hackers write to me to point out that you could use a list instead of a hashtable for this particular bit of code - I know.

for i in range(6000):
    x={}
    for j in range(1000):
        x[j]=i
        x[j]

for $i (0 .. 6000-1) {
  %x=();
  for $j (0 .. 1000-1) {
    $x{$j}=$i;
    $x{$j};
  }
}

import java.util.*;

public class test {
    public static void main(String[] args) {
        for (int i = 0; i < 6000; i++) {
            Map x = new HashMap();
            for (int j = 0; j < 1000; j++) {
		Integer I=new Integer(i);
		Integer J=new Integer(j);
                x.put(I,J);
                x.get(I);
            }
        }
    }
}
    

#include <iostream.h>
#include <map>

int main(int argc,char *argv[]) {

  for (int i = 0; i < 6000; i++) {
    map<int,int> x;
    for (int j = 0; j < 1000; j++) {
      x[i]=j;
      x[i];
    }
  }
  
  
}

IO

These results were surpising, as I expected all the systems to be much closer. Java's IO system really is as odd as it looks here (although trivial to wrap).

f=open('/tmp/scratch','wb')
for i in xrange(1000000):
    f.write(str(i))
f.close()
    

open($F,">/tmp/scratch");
for  $i (0 .. 1000000-1) {
  print $F "$i/n";
}
close($F);

import java.io.*;

public class test
{
    public static void main(String[] args) {
        try {
            File f = new File("/tmp/scratch");
	    PrintWriter pw= new PrintWriter(
			    new BufferedWriter(
			    new FileWriter(f)));
            for (int i = 0; i < 1000000; i++) {
                pw.print(i);
            }
            pw.close();
        }
        catch(IOException ioe) {
            ioe.printStackTrace();
        }
    }
}

    

#include <iostream.h>
#include <fstream>
using namespace std;


int main(int argc,char *argv[]) {

  ofstream out;
  out.open("/tmp/scratch");
  for(int i=0;i<1000000;i++) {
    out<<i;
  }
  out.close();
  
}

List

The pattern continues: Python slow, Java not fast, C++ blazing. Note: I cheated on the C++ and let it put 'int's in the intial list because I couldn't be bothered to create a new type. Maybe this helps the speed for C++.

for i in range(3000):
    v=['a','b','c','d','e','f','g']
    for j in range(1000):
        v.append(j)
        v[j]

for $i (0 .. 3000-1) {
  @v=('a','b','c','d','e','f','g');
  for $j (0 .. 1000-1) {
    push(@v,$j);
    $v[j];
  }
}

import java.util.*;

public class test {
    public static void main(String[] args) {
	List initial = new ArrayList();
	initial.add("a");
	initial.add("b");
	initial.add("c");
	initial.add("d");
	initial.add("e");
	initial.add("f");
	initial.add("g");

        for (int i = 0; i < 3000; i++) {
            List v = new ArrayList(initial);
            for (int j = 0; j < 1000; j++) {
                v.add(new Integer(j));
                v.get(j);
            }
        }
    }
}
    

#include <iostream.h>
#include <vector>

int main(int argc,char *argv[]) {

  vector<int> initial;
  initial.push_back(1);
  initial.push_back(2);
  initial.push_back(3);
  initial.push_back(4);
  initial.push_back(5);
  initial.push_back(6);
  initial.push_back(7);

  for (int i = 0; i < 3000; i++) {
    vector<int> v(initial);
    for (int j = 0; j < 1000; j++) {
      v.push_back(j);
      v[i];
    }
  }
  
  
}

No

My favorite of the silly benchmarks. This does nothing, just calls the program. About all it tells you is the overhead for short running tasks (ie 'ls' type utilities). Java has a big hit of about a second, whilst Python and C++ are much better.

public class test {
    public static void main(String[] args) {
    }
}

int main(int argc,char *argv[]) {
}

Speed

An empty loop iteration. Python and Perl slow, everything else fast.

for x in xrange(20000000):
    pass

for $i (0 .. 20000000-1) {
}

public class test {
    public static void main(String[] args) {
        for (int i = 0; i < 20000000; i++);
    }
}

    

int main(int argc,char *argv[]) {
  for (int i = 0; i < 20000000; i++);
}

Native

This was really shocking as I've been using SWIG for a while in order to control C++ modules from Python. I rewrote the code from [1] in order to use SWIG as I feel that this is more representative of what people do. As SWIG now supports Java, I used the same interface file for both languages. I expected the Java support, being new, to be inferior to the mature Python support, but the numbers show that JNI is really quite reasonable, at least compared to whatever the equivalent in Python is. C++ of course has a rather large advantage in this test
Update 2002-05-13: Got a nice mail from William S Fulton, the Java SWIG maintainer with an explanation of this behaviour. He has some benchmarks of the SWIG interface speed here

Perl has some really good speed here (relative to Python at least)

from pymaths import *

obj=Maths()

for i in xrange(1000000):
    obj.add(i,0)

use perlmaths;
$obj=new perlmaths::Maths();

for $i (0 .. 1000000-1) {
  $obj->add($i,0);
}

public class test {
    
    static {
	System.loadLibrary("javamaths");
    }
    

    public static void main(String[] args) {
	Maths m=new Maths();

	for(int i=0;i<1000000;i++) {
	    m.add(i,0);
	}

    }
}

#include "maths.hpp"
#include <iostream>

int main(int argc,char *argv[]) {

  Maths m=Maths();

  for(int i=0;i<1000000;i++) {
      m.add(i,0);
  }

}

Conclusions

Starting from the shaky premise that these benchmarks actually mean anything for "real world" applications, one conclusion is that Python is really really slow (unless you're doing nothing

I found the performance of Python for these benchmarks really surprising as it is at odds to my everyday experience. I've just completed a small demonstration program that does interactive scientific simulations in a VR environment. Whilst both the simulation code and the rendering system are C++, the app itself is Python, and used wxPython for the GUI. Even the eventloop was inside Python, and the app was running very fast. Python is not fast in an absolute sense - by absolute I regard C or C++ as the defacto baseline for comparing execution speed. However, it's fast enough, at least for acting as a glue layer for C++ objects. Coupled with the freedom and ease of working in Python, it really shines.

Java is getting fairly close to C++ (speedwise). I haven't taken the time to compare the speed history of the Java releases, so I dont know if the speed improvments are leveling off or if the remaining gap will be closed. From the results for the 1.4.1 beta, it looks like there is a good bit of overhead, which should

A final note on [1]. It's very old, and lots of things have changed since then. Even when it was written I wouldn't have accepted most of the Java problems reported in the second half of the page as they didn't match my experience at all. Today, they are even less accurate.

For no other reason except that I appear to be on a soapbox at the moment, here are my entirely subjective thoughts on the languages - modulo perl which I dont know enough about.

Platform

The software was run on a PC running debian woody. Information and software versions used were:

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 5
model name	: Pentium II (Deschutes)
stepping	: 2
cpu MHz		: 349.076
cache size	: 512 KB
fdiv_bug	: no
hlt_bug		: no
sep_bug		: no
f00f_bug	: no
coma_bug	: no
fpu		: yes
fpu_exception	: yes
cpuid level	: 2
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr
bogomips	: 696.32

perl -v

This is perl, v5.6.1 built for i386-linux

Copyright 1987-2001, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the
GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on
this system using `man perl' or `perldoc perl'.  If you have access to the
Internet, point your browser at http://www.perl.com/, the Perl Home Page.

swig -version

SWIG Version 1.3.13u-20020718-1718
Copyright (c) 1995-1998
University of Utah and the Regents of the University of California
Copyright (c) 1998-2001
University of Chicago

Compiled with CC
python -V
Python 2.1.3
g++ --version
2.95.4
java -version
java version "1.4.0_01"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0_01-b03)
Java HotSpot(TM) Client VM (build 1.4.0_01-b03, mixed mode)
java2 -version
./makeversions: line 14: /home/mikec/apps/j2re1.4.1/bin/java: No such file or directory

back (to my main page)
Last modified: Mon Sep 9 19:01:05 CEST 2002