Jpython Update Jim Hugunin Corporation for National Research Initiatives

JPython Update

• Jim Hugunin

• Corporation for National Research Initiatives

What’s JPython?

• The Python Language implemented in

• Adheres very closely to the standard implementation (CPython)

• Python code runs on any JVM

• JPython applet in Remote Microscope demo

• Python can use Java packages

• includes subclassing from Java
• Python classes can even be subclassed by Java

Overview

• Where it is today (20 minutes)

• What it can do
• Outstanding differences

• Where it is going (30 minutes)

• Taking more advantage of Java
• JPython-2.0 to be 10X faster that CPython-1.5?

Dejanews searches for “jpython”

• You should use JPython

• You should use Python (and it it has this nifty JPython available too)

• Maybe should compile language X to JVM

• JPython used as existence proof
• Counters bad experiences (Jacl, …)

• Embed Java instead of Perl, Tcl, or Python

• People can always use at least JPython on top

A new kind of posting

• Job Posting to comp.lang.java.corba on 10/30/98

• Responsibilities:

• ... Develop test harnesses in JPython to ensure non-regression Integration tests can be run automatically after a build is done. ...

Object Domain CASE Tool

• Commercial tool released in September

• 100% Pure Java UML Tool

• Forward and reverse engineering of Java, C++ and Python code

Java Scripting Competition

• Big three scripting languages

• Perl - Nothing
• Tcl - Jacl
• last release in February, terrible performance
• Python - JPython
• last release in October, good performance

• Other options

• Scheme - Kawa
• NetRexx - merges Rexx and Java

JPython vs. Jacl Performance

• Simple benchmark from web (had Tcl code)

• iterative factorial (using floats)
• recursive factorial (using floats)
• string manipulation test
• exec’ing process in os
• simple file i/o

• Don’t take benchmark’s too seriously!

Trivial Access to Java Packages

• Use Java packages with no wrappers

• Even better than SWIG

• Java’s design makes this possible

• int sum(double *data, int n);
• vs.
• int sum(double[] data);

from TkInter import Button

• from TkInter import Button

• def quit(): sys.exit()

• QUIT = Button(frame, text='QUIT', foreground='red', command=quit)

• from javax.swing import JButton

• def quit(event): sys.exit()

• QUIT = JButton(text='QUIT', foreground=red, actionPerformed=quit)

Outstanding Differences

• Trivial Differences

• JPython -> "1.0E20" CPython -> "1e+020”
• CPython doesn't allow 001.1, and does allow 0e

• Things that just need to be fixed

• looping over a dictionary is allowed
• printing recursive list -> StackOverflowError
• importing site at startup, command-line options
• standard exceptions are not class-based

Big Differences

• Weaker system interaction

• weak os and no select or signal modules
• no readline or signal handling in interpreter

• No C-based extensions (but Java packages)

• True garbage collection

• Better merging of types and classes

• Performance worse by 2X-10X

JPython and System Interaction

• Java lacks Python’s close system interaction

• Least common denominator choice

• os module

• much of posix is impossible without JNI

• select, signal

• fancy socket stuff

• Ctrl-C handling, readline support

Can’t use existing C-based extension modules

• Means C extensions must be rewritten in Java to be used in JPython

• I think this is much easy than writing in C...

• Often can write them in JPython as a thin wrapper around existing Java packages

• os is an example

• Might change in the future, but unlikely

Missing built-in modules

• Some surprising modules are there:

• pdb, profile, marshal

• Some are (relatively) straightforward

• operator, struct, cmath, zlib, binascii
• cPickle, cStringIO, bsddb (Finn Bock)

• Some are a lot of work

• TkInter, imp
• Numeric (Tim Hochberg)

More missing built-in modules

• Some might never be there based on JPython’s design

• rexec, dis

• Some are really hard based on Java’s design

• posix (os), select, signal

• Some are considered outdated

• regex, regsub

Lots of Extra Modules

• javax.swing

• java.sql

• com.ibm.xml

• javax.mail

• javax.media

• com.ms.com

Garbage Collection

• No reference counting at all in JPython

• Use Java’s garbage collection model instead

• Circular references no longer leak

• Finalization time is now unclear

Better merging of types/classes

• [].__class__ makes sense

• Can pass any container to exec/eval

• not just dictionaries

• Still some outstanding issues

• __finditem__
• vs.
• raise IndexError on __getitem__

Performance Issues

• CPython is 2-10X faster

• only 2-6X faster on platforms with a JIT

• Excuses, excuses...

• JPython is version 1.0 (actually 1.0.3)
• Java is version 1.1

• JPython-2.0 can be up to 2000X faster

• wait until the end of my talk

Current Design is Conservative

• Uses Java stack (but not really stack frames)

• Uses Java for bytecode

• All operations are basically method calls corresponding to Python bytecodes

• JVM stack looks a lot like PVM stack

• x + y

• x._add(y)
• frame.getlocal(1)._add(frame.getlocal(2))

Why not Java Stack Frames?

• Would Break

• locals()
• sys.settrace()

• Would make harder

• correct exception line numbers
• handling local variable name errors

Why not Java namespaces?

• Messing with other modules namespaces

• import foo
• foo.range = myrange

• Covert namespace manipulation

• foo.__dict__[‘bar’] = 42

• Compile-time vs. run-time paths

Why not Java objects on stack?

• Dynamic namespaces

• break most type inference
• can’t know function return types if you don’t know what function is actually being called

• Generally can’t know more than PyObject

jpythonc2

• Very aggressive compilation

• Using Java’s advantages whenever possible

• Requires some “assumptions”

• Whole-program analysis is the trick

• Let’s you make sure assumptions hold

• Without whole-program analysis?

• Requires programmer annotations of some form

Using Java Stack Frames

• Locals as Java local variables

• Most JIT’s use registers to hold these

• Breaks locals()

• can detect use of locals() and disable!

• Breaks sys.settrace()

• this is the price you pay

Using Java Namespaces

• None

• Three interpretations (in module foo)

• __builtin__.None
• might have been altered from original
• foo.None
• possibly both this and above
• might have been altered in various ways
• local variable None

Two solutions

• Whole-program analysis can detect

• and abort if needed

• Could add restrictions

• Can’t change __builtin__.None
• Only foo can set foo.None
• foo doesn’t use globals() or foo.__dict__

Using primitive types

• Java has primitive bytecodes in VM

• JIT’s often turn these into machine code
• Can add two ints extremely efficiently

• Need to know types to pull this off

• Overflow bounds checking

• Much more efficient if choose to disable
• Still savings from not allocating/freeing objects

Type inference

• Complete (whole-program)

• def foo(x):
• y = x+10
• foo(100)

• Partial (ML-like)

• def foo:int(x:int):
• y = x+10

The importance of Any

• x = 2
• y = x+10

• x and y are now integers

• y = “goodbye”

• y is now an Any

Fully Dynamic Implementation

• Module foo

• x = 42

• JPython-1.0

• public static PyInteger _c42 = new PyInteger(42);
• frame.setglobal(“x”, _c42);

Static Namespaces

• Module foo

• x = 42

• JPython-2.0

• public static final PyInteger _c42 = new PyInteger(42);
• public static PyObject x;
• foo.x = _c42;

Primitive Types

• Module foo

• x = 42

• JPython-2.0p

• public static int x;
• foo.x = 42;

Another Example

• Python Module

• None

• Dynamic Namespaces

• frame.getglobal(“None”);

• Static Namespaces

• __builtins__.None;

Compiler Design

• Symbolic (Partial) Evaluation

• Completely object-oriented
• Results of operations are types + code to produce

• Interesting future possibilities

• Blitz -- very efficient C++ lib for numeric
• re - compile-time optimization of regex’s
• …

Systems to Benchmark

• Complete Systems

• JPy1 - JPython-1.0.3
• CPy - CPython-1.1.5

• Aggressive compiler prototypes

• JPy2 - aggressive namespace, no primitive types
• JPy2p - Use raw ints for integers, same for strings
• Also, disable numeric bounds checking

• Hardware: P-II 233; OS: NT4.0sp3; JVM: MS

Simple Benchmarks

• def while_test(i):

• while i > 0: i = i - 1

• def for_test(i):

• y = 0

• for x in range(i): y = y + 1

• def recursive_test(i):

• if i > 0: recursive_test(i - 1)

PyStone Results

• Not the last word in benchmarks, but…

• Must support a large subset of Python

• Ident1, Ident2, … = range(6)
• from time import clock
• “Pystone(%s) time for %d passes = %g” % (__version__, LOOPS, benchtime)
• class Record: …
• map(lambda x: x[:], Array1Glob)

Disclaimer

• Handles almost nothing not in pystone

• First generation prototype

• Made one small change to pystone

• Doesn’t use default args
• Just didn’t have time to implement in jpythonc2

Where the time’s going

• Proc8 manipulates lists of ints

• Type inference system treats lists as Any’s

• Could probably infer types of list elements

• Mutable nature of lists makes this challenging
• Might be easier to include type annotations

• What if we leave this section of code out?

Complete Type Inference Limitations

• Requires whole-program analysis to work

• Can only be used with jpythonc/freeze

• Gives up advantages of typing for documentation/safety

• Solution is optional static types?

Optional Static Types

• ML-style partial type inference

• Deafult signature is Any

• Allows mixing of typed/untyped code

• Things that disable optimization

• __getattr__, getattr(), __dict__, globals(), exec, eval, ...

• Things that throw runtime exceptions

• math.pi = “foo”

Add Java to Python or Python to Java?

• How to merge Python and Java?

• Python + Optional Static Types

• Java + Syntactic Sugar + Dynamic Types

Little Things I Like About Java (Most could be added to Python)

• interfaces

• synchronized methods/blocks

• labeled breaks/continues

• block comments /**/

• assign ops (+=, *=, ...)

• boolean considered fundamental type

• never write “from StringIO import StringIO”