A Java Out-of-Memory Error involving GZIP, Typica, and SimpleDB

UPDATE

I am providing an update here to the root cause.

Overview
I ran into an interesting Out of Memory bug this week. It occurs if you use gzip to send/receive data and under-utilize your Java Heap memory. This land-mine has existed since 2004, though hopefully you will not be bitten by it.

Problem Stack
A Java process was throwing the following Out-of-Memory Error.

JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
Exception in thread "SDB WriterPool_4_rentalusers_incremental-thread-1" java.lang.OutOfMemoryError: ZIP004:OutOfMemoryError, MEM_ERROR in inflateInit2
at java.util.zip.Inflater.init(Native Method)
at java.util.zip.Inflater.<init>(Inflater.java:105) 
at java.util.zip.ZipFile.getInflater(ZipFile.java:416) 
at java.util.zip.ZipFile.getInputStream(ZipFile.java:359) 
at java.util.zip.ZipFile.getInputStream(ZipFile.java:324) 
at java.util.jar.JarFile.getInputStream(JarFile.java:467) 
at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:165) 
at java.net.URL.openStream(URL.java:1041) 
at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:455) 
at com.xerox.amazonws.common.AWSQueryConnection.<init>(AWSQueryConnection.java:102) 
at com.xerox.amazonws.sdb.Domain.<init>(Domain.java:72)
at com.xerox.amazonws.sdb.SimpleDB.getDomain(SimpleDB.java:202)
....
at java.lang.Thread.run(Thread.java:803)

Figuring it Out

This OutOfMemoryError is not java.lang.OutOfMemoryError: Java heap space. The latter is thrown when the memory occupied by all of your reachable Java objects exceeds the max heap -Xmx setting
Inflater is a JNI class used for gunzipping compressed files. Deflater is its counterpart for gzipping
In the example above, the Typica library is using the Inflater to get to a version file inside the Typica.jar. Every time a batch put or some other call is made and you create a new Domain object, Typica is mindlessly unjarring the Typica jar to get this file. (Why doesn’t Typica cache this version file once?).
There is a limited amount of native heap memory on the system. It must be shared by
1. Your Java program (JVM included)
2. Other user programs
3. Native libraries called by your Java program
The Inflater object takes little memory inside the JVM and quite a lot of memory outside the JVM in the native heap
The Inflater’s finalize() cleans up the memory outside the JVM in the native heap
As you know, finalize() is called after minor and major collections. If a major/minor cycle doesn’t happen for a while, the finalize won’t run and you risk running out of native heap memory

This is what was happening! Since we were GCing every 40-60 minutes, Inflater’s finalizers were not run often enough to clear up the native JNI memory. We ran out of native heap memory.

The bug has been around since 2004. Here it is: http://bugs.sun.com/view_bug.do?bug_id=4797189

Fix/Workaround

If you must use Typica, cache the Domain objects. Every time a Domain object is created, you are unjarring the typica.jar to get the version file. As a work-around, you can also reduce your Heap Memory so that finalizers run more often. As for me, I stopped using Typica.

Other links
Some links that I skimmed and found useful, though they are very detailed: