A Java Out-of-Memory Error involving GZIP, Typica, and SimpleDB
I am providing an update here to the root cause.
I ran into an interesting Out of Memory bug this week. It occurs if you use gzip to send/receive data and under-utilize your Java Heap memory. This land-mine has existed since 2004, though hopefully you will not be bitten by it.
A Java process was throwing the following Out-of-Memory Error.
JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
Exception in thread "SDB WriterPool_4_rentalusers_incremental-thread-1" java.lang.OutOfMemoryError: ZIP004:OutOfMemoryError, MEM_ERROR in inflateInit2
at java.util.zip.Inflater.init(Native Method)
Figuring it Out
- This OutOfMemoryError is not java.lang.OutOfMemoryError: Java heap space. The latter is thrown when the memory occupied by all of your reachable Java objects exceeds the max heap -Xmx setting
- Inflater is a JNI class used for gunzipping compressed files. Deflater is its counterpart for gzipping
- In the example above, the Typica library is using the Inflater to get to a version file inside the Typica.jar. Every time a batch put or some other call is made and you create a new Domain object, Typica is mindlessly unjarring the Typica jar to get this file. (Why doesn’t Typica cache this version file once?).
- There is a limited amount of native heap memory on the system. It must be shared by
- Your Java program (JVM included)
- Other user programs
- Native libraries called by your Java program
- The Inflater object takes little memory inside the JVM and quite a lot of memory outside the JVM in the native heap
- The Inflater’s finalize() cleans up the memory outside the JVM in the native heap
- As you know, finalize() is called after minor and major collections. If a major/minor cycle doesn’t happen for a while, the finalize won’t run and you risk running out of native heap memory
This is what was happening! Since we were GCing every 40-60 minutes, Inflater’s finalizers were not run often enough to clear up the native JNI memory. We ran out of native heap memory.
The bug has been around since 2004. Here it is: http://bugs.sun.com/view_bug.do?bug_id=4797189
If you must use Typica, cache the Domain objects. Every time a Domain object is created, you are unjarring the typica.jar to get the version file. As a work-around, you can also reduce your Heap Memory so that finalizers run more often. As for me, I stopped using Typica.
Some links that I skimmed and found useful, though they are very detailed: