Tracking the elusive PASTA+ process tree

PASTA+ is a Java-based project (primarily servlets) that functions as a metadata-driven data repository. In short, metadata that is written in the Ecological Metadata Language (EML) standard is uploaded and parsed by PASTA+, which then initiates processes to download and evaluate individual data files before finally registering and archiving the data in the repository. When multiple uploads are occurring concurrently and other users are searching and downloading data, the process tree can become quite chaotic. Discovering and understanding the state of these processes can be quite daunting. The following are simple steps to gain some insight into these processes.

Finding the root Tomcat8 process

Since most of PASTA+ utilizes Tomcat8, it is simple to discover the process id of this job using the ps command:

ps aux | grep tomcat8
tomcat8  14765 75.5  4.2 3490652 172080 ?      Sl   14:56   0:24 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.util.logging.config.file=/var/lib/tomcat8/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.awt.headless=true -XX:+UseConcMarkSweepGC -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/share/tomcat8/bin/bootstrap.jar:/usr/share/tomcat8/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat8 -Dcatalina.home=/usr/share/tomcat8 -Djava.io.tmpdir=/tmp/tomcat8-tomcat8-tmp org.apache.catalina.startup.Bootstrap start

From this command, we see that the process id of tomcat8 is 14765. This process id is for all of tomcat8, not just the context running PASTA+.

Finding all child threads associated with the Tomcat8 process

We can again use the ps command to identify all child ("spawned") processes that originate from the root tomcat8 process id 14765:

ps -T -p 14765
  PID  SPID TTY          TIME CMD
14765 14765 ?        00:00:00 java
14765 14766 ?        00:00:01 java
14765 14767 ?        00:00:00 java
14765 14768 ?        00:00:00 java
14765 14769 ?        00:00:00 java
14765 14770 ?        00:00:00 VM Thread
14765 14771 ?        00:00:00 Reference Handl
14765 14773 ?        00:00:00 Finalizer
14765 14774 ?        00:00:00 Surrogate Locke
14765 14775 ?        00:00:00 Signal Dispatch
14765 14776 ?        00:00:15 C2 CompilerThre
14765 14777 ?        00:00:02 C1 CompilerThre
14765 14778 ?        00:00:00 Service Thread
14765 14779 ?        00:00:00 VM Periodic Tas
14765 14780 ?        00:00:00 AsyncFileHandle
14765 14781 ?        00:00:00 GC Daemon
14765 14783 ?        00:00:00 NioBlockingSele
14765 14788 ?        00:00:00 Thread-4
14765 14792 ?        00:00:00 ContainerBackgr
14765 14793 ?        00:00:01 http-nio-8080-e
14765 14794 ?        00:00:00 http-nio-8080-e
14765 14795 ?        00:00:00 http-nio-8080-e

We see here a number of child processes, but do not necessarily see how much system resources each are using. An easy way to see this is to use the top command:

top -H -p 14765
top - 15:44:24 up 2 days, 23:51,  3 users,  load average: 0.98, 0.34, 0.21
Threads:  38 total,   0 running,  38 sleeping,   0 stopped,   0 zombie
%Cpu(s): 26.6 us,  9.3 sy,  0.0 ni, 47.8 id, 10.8 wa,  0.0 hi,  5.5 si,  0.0 st
KiB Mem :  4039736 total,   118156 free,   370428 used,  3551152 buff/cache
KiB Swap:  1003516 total,  1001200 free,     2316 used.  3290364 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND          
15814 tomcat8   20   0 3539720 221384  18756 S 24.6  5.5   0:18.47 Thread-74        
14767 tomcat8   20   0 3539720 221384  18756 S  0.7  5.5   0:02.45 java             
14768 tomcat8   20   0 3539720 221384  18756 S  0.7  5.5   0:02.47 java             
14776 tomcat8   20   0 3539720 221384  18756 S  0.3  5.5   0:43.49 C2 CompilerThre  
14765 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 java             
14766 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:01.04 java             
14769 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:01.15 java             
14770 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:01.22 VM Thread        
14771 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.06 Reference Handl  
14773 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.08 Finalizer        
14774 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 Surrogate Locke  
14775 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 Signal Dispatch  
14777 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:06.02 C1 CompilerThre  
14778 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 Service Thread   
14779 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:01.16 VM Periodic Tas  
14780 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.12 AsyncFileHandle  
14781 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 GC Daemon        
14783 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.10 NioBlockingSele  
14788 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.00 Thread-4         
14792 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.27 ContainerBackgr  
14793 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:01.18 http-nio-8080-e  
14794 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.34 http-nio-8080-e  
14795 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.09 http-nio-8080-e  
14796 tomcat8   20   0 3539720 221384  18756 S  0.0  5.5   0:00.29 http-nio-8080-e  

Now we can see only the root tomcat8 process (denoted by the java command) and all tomcat's child (or spawned) processes. In addition, we can also see that Thread-74 is utilizing 24.6% of the system's CPU capacity.