http://stackoverflow.com/questions/817856/when-and-how-should-i-use-a-threadlocal-variable
Classloader leaks: the dreaded "java.lang.OutOfMemoryError: PermGen space" exception
Did you ever encounter a java.lang.OutOfMemoryError: PermGen space error when you redeployed your application to an application server? Did you curse the application server, while restarting the application server, to continue with your work thinking that this is clearly a bug in the application server. Those application server developers should get their act together, shouldn't they? Well, perhaps. But perhaps it's really your fault!
Take a look at the following example of an innocent looking servlet.
The way that this works is that each application is loaded using its own classloader. Simply put, a classloader is a special class that loads .class files from jar files. When you undeploy the application, the classloader is discarded and it and all the classes that it loaded, should be garbage collected sooner or later.
Somehow, something may hold on to the classloader however, and prevent it from being garbage collected. And that's what's causing the java.lang.OutOfMemoryError: PermGen space exception.
If there's a problem with garbage collecting classes and if you keep loading new classes, the VM will run out of space in that memory region, even if there's plenty of memory available on the heap. Setting the -Xmx parameter will not help: this parameter only specifies the size of the total heap and does not affect the size of the PermGen region.
In this picture you see the objects loaded by the application classloader in yellow, and the rest in green. You see a simplified container object that holds references to the application classloader that was created just for this application, and to the servlet instance so that the container can invoke the doGet() method on it when a web request comes in. Note that the STATICNAME object is owned by the class object. Other important things to notice:
To illustrate this, let's see what happens when the application gets undeployed: the Container object nullifies its references to the Servlet1 instance and to the AppClassloader object.
As you can see, none of the objects are reachable, so they all can be garbage collected. Now let's see what happens when we use the original example where we use the Level class:
In this picture you see something you may not have expected: the Level class holds a static member to all Level objects that were created. Here's the constructor of the Level class in the JDK:
Only the LeakServlet object can be garbage collected. Because of the reference to the CUSTOMLEVEL object from outside ofAppClassloader, the CUSTOMLEVEL anyonymous class objects (LeakServlet$1.class) cannot be garbage collected, and through that neither can the AppClassloader, and hence none of the classes that the AppClassloader loaded can be garbage collected.
Conclusion: any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak.
I did not mentiond yet the simplest recipe for disaster: a thread started by the application while the thread does not exit after the application is undeployed.
Take a look at the following example of an innocent looking servlet.
package com.stc.test; import java.io.\*; import java.util.logging.\*; import javax.servlet.\*; import javax.servlet.http.\*; public class MyServlet extends HttpServlet { protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { // Log at a custom level Level customLevel = new Level("OOPS", 555) {}; Logger.getLogger("test").log(customLevel, "doGet() called"); } }Try to redeploy this little sample a number of times. I bet this will eventually fail with the dreadedjava.lang.OutOfMemoryError: PermGen space error. If you like to understand what's happening, read on.
The problem in a nutshell
Application servers such as Glassfish allow you to write an application (.ear, .war, etc) and deploy this application with other applications on this application server. Should you feel the need to make a change to your application, you can simply make the change in your source code, compile the source, and redeploy the application without affecting the other still running applications in the application server: you don't need to restart the application server. This mechanism works fine on Glassfish and other application servers (e.g. Java CAPS Integration Server).The way that this works is that each application is loaded using its own classloader. Simply put, a classloader is a special class that loads .class files from jar files. When you undeploy the application, the classloader is discarded and it and all the classes that it loaded, should be garbage collected sooner or later.
Somehow, something may hold on to the classloader however, and prevent it from being garbage collected. And that's what's causing the java.lang.OutOfMemoryError: PermGen space exception.
PermGen space
What is PermGen space anyways? The memory in the Virtual Machine is divided into a number of regions. One of these regions is PermGen. It's an area of memory that is used to (among other things) load class files. The size of this memory region is fixed, i.e. it does not change when the VM is running. You can specify the size of this region with a commandline switch: -XX:MaxPermSize . The default is 64 Mb on the Sun VMs.If there's a problem with garbage collecting classes and if you keep loading new classes, the VM will run out of space in that memory region, even if there's plenty of memory available on the heap. Setting the -Xmx parameter will not help: this parameter only specifies the size of the total heap and does not affect the size of the PermGen region.
Garbage collecting and classloaders
When you write something silly like private void x1() {
for (;;) {
List c = new ArrayList();
}
}
you're continuously allocating objects; yet the program doesn't run out of memory: the objects that you create are garbage collected thereby freeing up space so that you can allocate another object. An object can only be garbage collected if the object is "unreachable". What this means is that there is no way to access the object from anywhere in the program. If nobody can access the object, there's no point in keeping the object, so it gets garbage collected. Let's take a look at the memory picture of the servlet example. First, let's even further simplify this example:package com.stc.test; import java.io.\*; import java.net.\*; import javax.servlet.\*; import javax.servlet.http.\*; public class Servlet1 extends HttpServlet { private static final String STATICNAME = "Simple"; protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { } }After loading the above servlet, the following objects are in memory (ofcourse limited to the relevant ones):
In this picture you see the objects loaded by the application classloader in yellow, and the rest in green. You see a simplified container object that holds references to the application classloader that was created just for this application, and to the servlet instance so that the container can invoke the doGet() method on it when a web request comes in. Note that the STATICNAME object is owned by the class object. Other important things to notice:
- Like each object, the Servlet1 instance holds a reference to its class (Servlet1.class).
- Each class object (e.g. Servlet1.class) holds a reference to the classloader that loaded it.
- Each classloader holds references to all the classes that it loaded.
To illustrate this, let's see what happens when the application gets undeployed: the Container object nullifies its references to the Servlet1 instance and to the AppClassloader object.
As you can see, none of the objects are reachable, so they all can be garbage collected. Now let's see what happens when we use the original example where we use the Level class:
package com.stc.test; import java.io.\*; import java.net.\*; import java.util.logging.\*; import javax.servlet.\*; import javax.servlet.http.\*; public class LeakServlet extends HttpServlet { private static final String STATICNAME = "This leaks!"; private static final Level CUSTOMLEVEL = new Level("test", 550) {}; // anon class! protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { Logger.getLogger("test").log(CUSTOMLEVEL, "doGet called"); } }Note that the CUSTOMLEVEL's class is an anonymous class. That is necessary because the constructor of Level is protected. Let's take a look at the memory picture of this scenario:
In this picture you see something you may not have expected: the Level class holds a static member to all Level objects that were created. Here's the constructor of the Level class in the JDK:
protected Level(String name, int value) { this.name = name; this.value = value; synchronized (Level.class) { known.add(this); } }Here known is a static ArrayList in the Level class. Now what happens if the application is undeployed?
Only the LeakServlet object can be garbage collected. Because of the reference to the CUSTOMLEVEL object from outside ofAppClassloader, the CUSTOMLEVEL anyonymous class objects (LeakServlet$1.class) cannot be garbage collected, and through that neither can the AppClassloader, and hence none of the classes that the AppClassloader loaded can be garbage collected.
Conclusion: any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak.
More sneaky problems
I don't blame you if you didn't see the problem with the Level class: it's sneaky. Last year we had some undeployment problems in our application server. My team, in particular Edward Chou, spent some time to track them all down. Next to the problem with Level, here are some other problems Edward and I encountered. For instance, if you happen to use some of the Apache Commons BeanHelper's code: there's a static cache in that code that refers to Method objects. The Methodobject holds a reference to the class the Method points to. Not a problem if the Apache Commons code is loaded in your application's classloader. However, you do have a problem if this code is also present in the classpath of the application server because those classes take precedence. As a result now you have references to classes in your application from the application server's classloader... a classloader leak!I did not mentiond yet the simplest recipe for disaster: a thread started by the application while the thread does not exit after the application is undeployed.
Detection and solution
Classloader leaks are difficult. Detecting if there's such a leak without having to deploy/undeploy a large number of times is difficult. Finding the source of a classloader leak is even trickier. This is because all the profilers that we tried at the time, did not follow links through classloaders. Therefore we resorted to writing some custom code to find the leaks from memory dump files. Since that exercise, new tools came to market in JDK 6. The next blog will outline what the easiest approach today is for tracking down a glassloader leak.ThreadLocal and memory leaks
(imported from http://blogs.codehaus.org/people/avasseur, read comments there)
Last day I realized that using ThreadLocal can be very dangerous when it comes to long running applications and garbage collections.
When you read the documentation of the ThreadLocal, you read that the object put in a ThreadLocal context is associated to the thread, and will be garbaged once the thread is dead.
Each thread holds an implicit reference to its copy of a ThreadLocal as long as the thread is alive and the ThreadLocal object is accessible; after a thread goes away, all of its copies of ThreadLocal variables are subject to garbage collection (unless other references to these copies exist).
So if you use ThreadLocal to store some object instance there is a high risk to have the object stored in the thread local never garbaged when your app runs inside an app server like WebLogic Server, which manage a pool of working thread - even when the class that created this ThreadLocal instance is garbage collected.
To solve this issue, you will probably have to check for it first with a memory profiler tool like JProfiler, and then wrap the value put in the ThreadLocal in a WeakReference.
That did not appears to be obvious while reading the javadoc. Isn't it ?
Last day I realized that using ThreadLocal can be very dangerous when it comes to long running applications and garbage collections.
When you read the documentation of the ThreadLocal, you read that the object put in a ThreadLocal context is associated to the thread, and will be garbaged once the thread is dead.
Each thread holds an implicit reference to its copy of a ThreadLocal as long as the thread is alive and the ThreadLocal object is accessible; after a thread goes away, all of its copies of ThreadLocal variables are subject to garbage collection (unless other references to these copies exist).
So if you use ThreadLocal to store some object instance there is a high risk to have the object stored in the thread local never garbaged when your app runs inside an app server like WebLogic Server, which manage a pool of working thread - even when the class that created this ThreadLocal instance is garbage collected.
To solve this issue, you will probably have to check for it first with a memory profiler tool like JProfiler, and then wrap the value put in the ThreadLocal in a WeakReference.
That did not appears to be obvious while reading the javadoc. Isn't it ?
