JDB Example: Generating a Thread Dump

Every once in a while I run into a situation where the usual means for generating a thread dump (stack trace) does not work, making it difficult to track down pesky deadlocks. This seems somewhat more common under OSX, but I’ve seen it happen under Linux as well.

Typically you can type Control-&backslash; (Control-Break on Windows), or send a process the QUIT signal (e.g., kill -QUIT ) to dump a trace of all active threads in the Java VM. Intermittently, however, I have found that this does not work for java processes that are launched via a shell script. The Control-&backslash; is sometimes simply ignored, while the QUIT signal appears to kill the script, but without dumping a stack trace or killing the java process. (This suggests that in either case, the script is probably just intercepting the signal. There’s probably a fix for that, but I haven’t explored it.)

The jdb debugger utility, included as part of the JDK, provides an alternate way to get a stack trace. First, you must pass two additional arguments to the java vm to tell it to listen for connections from the java debugger. On OSX, Linux, or UNIX, this looks like:

java -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n class 

This will cause the VM to listen for debugger connections on port 8000. You may use any usued port number that the account in which the program is running may listen. For non-priviledged accounts, this typically means any port number over 1024. For security reasons you might not want to include these options on a production system, but for testing and debugging they impose no measurable performance penalty.

As an example, here is a simple DeadlockDemo class that will produce a deadlock at some indeterminate point. It creates three threads, each of which will repeatedly try to randomly acquire two locks. When a thread successfully gets both locks, it will update a counter. Eventually two threads will try to acquire the same two locks in the opposite order, resulting in a deadlock. Deadlock situations are, of couse, not the only cases where stack traces are useful, but are probably the most common:

import java.awt.*;
import javax.swing.*;

/**
* Demo program that should (eventually) produce a deadlock
*/
public class DeadlockDemo extends JFrame implements Runnable {
   /**
    * Label with counter of iterations before deadlock
    */
   private JLabel text = null;

   /**
    * Counter of number of updates
    */
   private int count = 0;

   /**
    * Set of objects to randomly synchronize on
    */
   private Object[] locks = null;

   public DeadlockDemo() {
       setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
       text = new JLabel("Starting up...", JLabel.CENTER);
       setLayout(new BorderLayout());
       add(text, BorderLayout.CENTER);

       locks = new Object[] { new Object(), new Object(), new Object() };
   }

   public void startSomeThreads() {
       for (int i = 0; i < 3; i++) {
           Thread runner = new Thread(this);
           runner.setName("Runner Thread " + i);
           runner.start();
       }
   }

   public void run() {
       while (true) {
           // Pick two locks at random to synchronize on. Eventually
           // two threads will try to acquire the same two locks in
           // the opposite order, resulting in a deadlock.
           synchronized(locks[(int) (Math.random() * locks.length)]) {
               synchronized(locks[(int) (Math.random() * locks.length)]) {
                   count++;
                   synchronized(text) {
                       text.setText("Counter: " + count);
                   }
               }
           }
       }
   }

   public static void main(String[] args) {
       DeadlockDemo demo = new DeadlockDemo();
       demo.setSize(200, 200);
       demo.setVisible(true);
       demo.startSomeThreads();
   }
}

---

Figure 1. DeadlockDemo, a program that (eventually) deadlocks.

You can run this class from the command line thusly:

java -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n DeadlockDemo

To get a thread dump, you first need to attach the debugger. In another terminal window, type:

jdb -attach 8000

Note that the port number (8000, in this example) must match the port number that you provided when you launched the virtual machine. You will see:

Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
>

At the jdb prompt, enter “suspend” to temporarily suspend all running threads in the VM. Your program will become unresponsive after you do this. Next, enter “where all” to generate the thread dump. Here is a complete example:

Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
> suspend
All threads suspended.
> where all
DestroyJavaVM:
Runner Thread 2:
 [1] DeadlockDemo.run (DeadlockDemo.java:47)
 [2] java.lang.Thread.run (Thread.java:613)
Runner Thread 1:
 [1] DeadlockDemo.run (DeadlockDemo.java:47)
 [2] java.lang.Thread.run (Thread.java:613)
Runner Thread 0:
 [1] DeadlockDemo.run (DeadlockDemo.java:47)
 [2] java.lang.Thread.run (Thread.java:613)
AWT-EventQueue-0:
 [1] java.lang.Object.wait (native method)
 [2] java.lang.Object.wait (Object.java:474)
 [3] java.awt.EventQueue.getNextEvent (EventQueue.java:345)
 [4] java.awt.EventDispatchThread.pumpOneEventForHierarchy (EventDispatchThread.java:216)
 [5] java.awt.EventDispatchThread.pumpEventsForHierarchy (EventDispatchThread.java:190)
 [6] java.awt.EventDispatchThread.pumpEvents (EventDispatchThread.java:184)
 [7] java.awt.EventDispatchThread.pumpEvents (EventDispatchThread.java:176)
 [8] java.awt.EventDispatchThread.run (EventDispatchThread.java:110)
Java2D Disposer:
 [1] java.lang.Object.wait (native method)
 [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:116)
 [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:132)
 [4] sun.java2d.Disposer.run (Disposer.java:123)
 [5] java.lang.Thread.run (Thread.java:613)
AWT-Shutdown:
 [1] java.lang.Object.wait (native method)
 [2] java.lang.Object.wait (Object.java:474)
 [3] sun.awt.AWTAutoShutdown.run (AWTAutoShutdown.java:259)
 [4] java.lang.Thread.run (Thread.java:613)
AWT-AppKit:
Signal Dispatcher:
Finalizer:
 [1] java.lang.Object.wait (native method)
 [2] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:116)
 [3] java.lang.ref.ReferenceQueue.remove (ReferenceQueue.java:132)
 [4] java.lang.ref.Finalizer$FinalizerThread.run (Finalizer.java:159)
Reference Handler:
 [1] java.lang.Object.wait (native method)
 [2] java.lang.Object.wait (Object.java:474)
 [3] java.lang.ref.Reference$ReferenceHandler.run (Reference.java:116)

---

Figure 2. A stack trace produced within jdb.

Here we see that our three “Runner Thread n” threads are all stuck at line 47 in the run method, each waiting for a lock that one of its siblings is holding. The user interface is not blocked, as the event dispatch thread (”AWT-EventQueue-0“) is waiting for another input event. The other threads are handling various background tasks in the VM.

To exit jdb, type “quit“. Your program, with the exception of any deadlocked threads, will become responsive again. If you do not wish to exit out of jdb, you can type “resume” instead of “quit” to undo the effects of “suspend“. Note that you can only generate a thread dump when threads are suspended.

Windows

On Windows, a slightly different mechanism is used for communication between the Java virtual machine and JDB. Rather than specifying a port for the connection, you tell the VM to use shared memory. Here’s what the command would look like for the DeadlockDemo example on Windows:

java -Xdebug -Xrunjdwp:transport=dt_shmem,server=y,suspend=n DeadlockDemo

To connect jdb to the virtual machine, enter:

jdb -attach jdbconn

Once connected, the commands used within jdb (suspend, where all, and quit) are the same as on OSX and Linux.

Another Alternative:

On Windows the 'standard', ie. default, connector uses the shared memory transport.

To connect to a socket, you need to tell jdb to use the socket attaching connector, ie

jdb -connect com.sun.jdi.SocketAttach:port=9000

It is hard to remember the names and params of all the connectors. You can do
jdb -listconnectors