WebLogic Troubleshooting

For Online Oracle Weblogic Admin Training, please contact through http://www.appsdbatraining.com

WebLogic Troubleshooting

WebLogic Troubleshooting – Logging

1) Server log

WebLogic server creates server log file by default under:

/<domain-name>/<server name>/<server name>.log

The location is configurable.

2) JDBC log

All SQL statements and DB related exceptions/errors.

This file is created under /<server name>/jdbc.log

3) STDout log (If the process is redirected to STDout)

4) Domain log

All domain level information is logged into this file.This is subset of server log file.

<domain name>/<domain name>.log

5) Access log

All http requests are recorded in this log file

/<server name>/access.log

6) Transaction log

All servers record transaction in the tlog file

/<server name>/<server name>.tlog

 WebLogic Troubleshooting (Server crash)

Server Crash
This implies the Weblogic java process no longer exists.
Server crash can occur only because of native code. (Java cannot cause a process to crash)
Determine all potential sources of native code used by the WebLogic Server.
• nativeIO.
• Type 2 jdbc driver.(We generally use thin driver for most applications)
• Native libraries accessed with JNI calls.
• SSL native libraries.
• JVM itself. Most of the times its from JVM.

Sometimes the JVM will produce a small log file that may contain useful information as to which library the crash has originated from (hs_err_pid*.log).

 Server Crash Analysis

When a JVM is crashed, a core file(binary image of the process) is created. Run pmap and pstack against the core file to get the library that caused the crash.
Demo to figure out offending library using existing pmap & pstack out files.
Check list:

1) hs _err_pid*.log (Look for library that caused the crash)
2) pmap core (core file created in JVM root dir)
pstack core
3) Using debugger (gdb,dbx,adb) (if above two steps does not provide any information)

WebLogic Troubleshooting (Server hang)

A server is said to be hung when:

1) Process is still alive
2) Server does not accept any requests because all the execute threads busy or stuck for some reason.
3) No reponse sent to clients.
4) java weblogic.Admin PING command doesn’t return a normal reponse.

Server Hang Analysis:
The first step is to take multiple thread dumps.
• A thread dump is a snapshot of the JVM at the particular instant.
• Multiple thread dumps are necessary to conclude that the threads are stuck and not progressing.

Procedure to take thread dumps:
Unix:
Open shell window and issue the command kill -3 <PID> where PID is java processID of weblogic. Thread dumps are logged on to STDout file.

Windows:
Do ctrl-break on command window where weblogic is running.Thread dumps are created on the same command window.

Windows Service:
Open a command prompt and issue the command(Make sure beasvc.exe is in the PATH)
c:\> beasvc -dump -svcname:service-name
Thread dumps are created in the defined log file.While creating service, we can provide log option in installservice script as: -log:”d:\bea\domains\mydomain\myserver-stdout.txt

• Before we analyze thread dumps, it is important to know the common thread states:
1) Runnable [marked as R in some VMs]:
This state indicates that the thread is either running currently or is ready to run the next time the OS thread scheduler schedules it.
2) Object.wait() [marked as CW in some VMs]:
Indicates that the thread is waiting for some condition to be fulfilled.
3)Waiting for monitor entry [marked as MW in some VMs]:
Indicates that the thread is waiting to enter a synchronized block.These threads are something to watch out because there is lock contention here. Thread is waiting for a lock on object and some other thread is holding the lock.

In case of weblogic, the main worker threads are from group weblogic.kernel.defalt:
“ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default’“….
This is the set of threads we need to look for hang/slow performance issues.
This is a snapshot of idle thread waiting for some work to be assigned.
On an idle system you would see lot of threads in the below state:
“ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default'” daemon prio=5 tid=0x031a6308 nid=0x980 in Object.wait() [2dff000..2dffd8c]
at java.lang.Object.wait(Native Method)
– waiting on <0x112cf2c0> (a weblogic.kernel.ExecuteThread)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
– locked <0x112cf2c0> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)

• As for thread dump analysis & conclusion, lets see a sample thread dump and drill into it further Demo of thread dump (Thread stuck issue on UAT)

WebLogic Troubleshooting (Server slow)

• Server performing Slow
There are lot of reasons for server performing slow.First step is to take thread dumps and see what the threads are doing. If there is nothing wrong with the threads there are other reasons why server performs slow:

• Process runs OutOfMemory:
If java heap is full, server process appears to be hung and not accepting any requests because each request needs heap for allocating objects.So if heap is full, none of the requests get served, all the requests fail with java.lang.OutOfMemory

• OutOfMemory Analysis:

OutOfMemory can occur because of real memory crunch or a memory leak causing the heap to fill with orphaned objects.
First step is to enable GC and run the server again.
(-XX:printGCDetails)
The STDout file would show the garbage collection details.If the error is because of memory leak, then we would need to use profilers like Introscope or optmizeIT to figure out the source of leak.

Process size = java heap + native memory + memory occupied by the executables and libraries.
On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. This is data bit limitation (2 pow 32), Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 – 2 GB).This is not a limitation on 64 bit machines like solaris(sparc) or windows running on Itanium (64 bit)

WebLogic Troubleshooting – Fragmentation

 

Nagulu Polagani

"We are all apprentices in a craft where no one ever becomes a master."