学步园

Even a transaction timeout will not kill or timeout any action that is done by the resources that are enlisted in this transaction. The actions will run as long as they take, without interruption. A transaction timeout will set a flag on the transaction that will mark it as rollback only, so that any subsequent request to commit this transaction will fail with a TimedOutException or RollbackException. However, as mentioned above, the long running JDBC calls can lead to blocked WebLogic Server execute threads, which can finally lead to a hanging instance, if all threads are blocked and no execute thread remains available for handling incoming requests.

More recent WebLogic Server versions have a health check functionality that regularly checks if a thread does not react for a certain period of time (the default is 600 seconds). If this happens, an error message is printed to your log file similar to following:

Top of Page

Problem Troubleshooting
Different programming techniques or JDBC connection pool configurations can lead to deadlocks or long running JDBC calls that lead to hanging WebLogic Server instances. General information about how to troubleshoot and analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

This pattern addresses JDBC calls causing a server hang and other well known JDBC-related causes for common problems leading to hanging WebLogic Server instance.  Other Support Patterns referenced in this pattern are at the WebLogic Server Support Patterns Site.

Quick Links

• Why does the problem occur?
• Analysis of a hanging WebLogic Server instance
• Tips and Tricks to optimize your JDBC code and JDBC connection pool configuration

Why does the problem occur?
The following are some different possible reasons that can cause JDBC calls to lead to a hanging WebLogic Server instance:

• Use of DriverManager.getConnection() in your JDBC code.
• SQL Queries issued to the database take unexpectedly long time to return.
• Database for which the JDBC connection pool is configured hangs and does not return from calls in a timely manner.
• A slow or overloaded network causes database calls to slow down or hang.
• A deadlock causes all execute threads to hang and wait forever.
• RefreshMinutes or TestFrequencySeconds property in the JDBC connection pool causes hang periods in WebLogic Server.
• JDBC connection pool shrinking and re-creation of database connections causes long response times.

Top of Page

Synchronized DriverManager.getConnection()
Older JDBC application code sometimes uses DriverManager.getConnection() calls to retrieve a database connection using a certain driver. This technique is not recommended as it can cause deadlocks or at least relatively low performance for your connection requests. The reason behind this is, that all DriverManager calls are class-synchronized, meaning that one DriverManager call in one thread will block all other DriverManager calls in any other thread inside one WebLogic Server instance.

In addition to that, the constructor for a SQLException makes a DriverManager call, and most drivers have DriverManager.println() calls for logging, so any of these can block all other threads that issue a DriverManager call.

DriverManager.getConnection() can take a relatively long time until it returns with the physical connection created to the database. Even if no deadlock occurs, all other calls need to wait until that one thread gets its connection. This is not a best practice in a multi-threaded system like WebLogic Server.

If you prefer to use JDBC connections in your JDBC code, you should use a WebLogic Server JDBC connection pool, define a DataSource for it, and get the connection from the DataSource. This will give you all advantages from a pool (resource sharing, connection reuse, connection refresh if a database was down, etc). It also will help you avoid the deadlocks that may happen with DriverManager calls. See detailed information on how to use JDBC connection pools, DataSources, and other JDBC objects in WebLogic Server at: http://e-docs.bea.com/wls/docs81/jdbc/intro.html#1036718 and http://e-docs.bea.com/wls/docs81/jdbc/programming.html#1054307.

A typical thread blocked in a DriverManager.getConnection() call looks like:

A thread hanging in a long running SQL call will show a very similar stack in a thread dump as the one for a hanging database. Please compare the next section for details.

Hanging Database
Good database performance is key for the performance of an application that relies on this database. Consequently, a hanging database can block many or all available execute threads in a WebLogic Server instance and finally lead to a hanging server. To diagnose this, you should take 5 to 10 thread dumps from your hanging WebLogic Server instance and check your execute threads (in the default queue or your application thread queue) to see if they are currently in SQL calls and waiting for a result from the database. A typical stack trace for a thread that currently issues a sql query could look similar to following example:

Top of Page

Slow Network
Communication between WebLogic Server and the database relies on a well-performing and reliable network in order to serve the requests in a timely manner. Slow network performance can therefore lead to hanging or blocking execute threads waiting for results of SQL queries. The related stack traces will look similar to example above in Hanging Database section. It is not possible to find the root cause of the hanging or slow SQL queries by solely analyzing the WebLogic Server thread dumps. These give the first hint that something is wrong with the performance of the SQL calls. The next step is to check if there is a database or network problem that causes poorly performing SQL calls.

Deadlock
Both an application level deadlock as well as a deadlock on the database level can lead to hanging threads. You should check your thread dumps to see if there is an application level deadlock. Information on how to do this is provided in Server Hang – Application Deadlock Pattern. A database deadlock can be detected either in the database log or by the SQL Exception that can be found in the WebLogic Server log file. An example for a related SQL Exception is:

RefreshMinutes or TestFrequencySeconds
If you see recurring periods of low database performance, slow SQL calls, or connection peaks, the setting of the RefreshMinutes or TestFrequencySeconds configuration property in your JDBC connection pools could be the reason. This is described in detail in Investigating JDBC Problems Pattern. Unless you do not have a firewall between your WebLogic Server instance and your database, you should disable this functionality.

Pool Shrinking
Physical connections to a database are resources that should be opened once and kept open as long as possible, as a new connection request is a considerable resource overhead for the database, the operating system kernel, and the WebLogic Server. Consequently, pool shrinking should be disabled on production systems in order to keep this overhead at a minimum. If pool shrinking is enabled, idle pool connections will be closed and reopened once connection requests to the pool cannot be satisfied.

As these activities can take some time, the related application requests may take an unexpectedly long time which can lead users to assume that the system hangs. Information on how to optimize JDBC connection pool configurations is provided in Investigating JDBC Problems Pattern.

Top of Page

Analysis of a hanging WebLogic Server instance
General information on how to analyze a hanging WebLogic Server instance is provided in Generic Server Hang Pattern.

Most times it will be helpful to start with taking thread dumps from the hanging system in order to find out what is going on, e.g., what the different threads are doing and why they hang. Generally, thread dumps can be taken on production systems, however caution is necessary for very old versions of the JVM (<1.3.1_09), as they may crash during thread dumps. Also if the WebLogic Server instance has a huge number of threads, it will mean that the thread dump will take awhile to complete, while the rest of the threads are blocked.

Please take more than one thread dump (5 to 10) with a delay of some seconds in between. This gives you the possibility to check the progress of the different threads. Also it will show if the system actually hangs (no progress at all) or if the throughput is extremely slow, which can seem to be a hanging system.

Information on how to take thread dumps is provided in “Generic Server Hang” support pattern or in our documentation: http://e-docs.bea.com/wls/docs81/cluster/trouble.html.

Also please check if the complete WebLogic Server instance hangs or if it is the application that hangs. “Generic Server Hang” support pattern also includes this information.

Analyzing the thread dumps can show if one of the reasons mentioned in the previous section Why does the problem occur? actually is responsible for your hanging instance. If for example all your threads are in a DriverManager method like getConnection() then you have identified the root cause and need to change your application to use a DataSource or Driver.connect() instead of DriverManager.getConnection().

A very useful tool, Samurai, can be used to analyze thread dumps and to monitor the progress of threads between different thread dumps. This can be downloaded from dev2dev at:  http://dev2dev.bea.com/resourcelibrary/utilitiestools/adminmgmt.jsp.

A whitepaper on analyzing thread dumps on dev2dev: http://dev2dev.bea.com/products/wlplatform81/articles/thread_dumps.jsp will also be helpful in going deeper into the thread dumps to find out more about the server hang.

Top of Page

Tips and Tricks to optimize your JDBC code and JDBC connection pool configuration
There are some best practices both in the development of JDBC code and also in the configuration practice of JDBC connection pools that can help to avoid common problems and optimize resource usage so that hanging server instances should not happen.

JDBC Programming
In order to optimize resource usage in WebLogic Server and conserve database resources, you should use JDBC connection pools for your application’s JDBC calls. Connections created and destroyed in your application code generate an unnecessary overhead which should be avoided. For generic documentation on JDBC programming, see: http://e-docs.bea.com/wls/docs81/jdbc/rmidriver.html#1028977. Also details on JDBC performance tuning are at: http://e-docs.bea.com/wls/docs81/jdbc/performance.html#1027791.

You can view comprehensive information on JDBC that will help to optimize your JDBC code and the utilization of your JDBC resources on dev2dev Java Database Connectivity page at: http://dev2dev.bea.com/technologies/jdbc/index.jsp.

JDBC Connection Pool Configuration
The Investigating JDBC Problems Pattern has recommendations on how to configure a connection pool for production environments. In order to avoid hangs or bad performance, these configuration tips should be considered.

• WLS 8.1 Release Notes
• WLS 7.0 Release Notes
• WLS 6.1 Release Notes

Please note that changes have been made in WLS 8.1 SP3 to resolve CR134921, where for certain JDBC connections, the call to roll back a transaction was not being handled immediately because the driver had to wait for any currently-executing statement to return. 

Searching will also return Release Notes, as well as other Support Solutions and CR-related information as noted at Need Further Help?.  Contract customers who are logged in at http://support.bea.com/ will also see a Browse portlet for both Solutions and Bug Central where latest available CRs can be browsed by Product version.

1. Query AskBEA at http://support.bea.com/ using “jdbc server hang”, as an example, to discover other published solutions.  Contract Support Customers: Ensure you are logged to access available CR-related information.
2. Ask a more detailed question on one of BEA’s newsgroups at http://forums.bea.com
   

If this does not resolve your issue and you have a valid Support Contract, you can open a Support Case by logging in at: http://support.bea.com/ .

FEEDBACK

Please provide us input on whether or not this Support Diagnostic Pattern “JDBC Causes Server Hang” helped, any clarifications you needed, and any requests for new topics to Support Diagnostic Patterns.

DISCLAIMER NOTICE:

BEA Systems, Inc. provides the technical tips and patches on this Website for your use under the terms of BEA’s maintenance and support agreement with you. While you may use this information and code in connection with software you have licensed from BEA, BEA makes no warranty of any kind, express or implied, regarding the technical tips and patches.

Any trademarks referenced in this document are the property of their respective owners. Consult your product manuals for complete trademark information.