Monday, June 13, 2011

AppServer does (not) reconnect to a bounced database

Few weeks ago, I was facing some databases issues. Due to an unexpected overload on the server, the services were not responding, and a couple of scripts were configured to stop automatically the databases. Few minutes later, the scripts started again those databases, and everything went fine again (I won’t go in details of the reasons of overload and why scripts are bouncing the database, that’s not the subject here).
But one of these databases is a Peoplesoft database, having application server connections to it. The Peopletools version was 8.51.07, and unfortunately, was not able to reconnect automatically to the database once restarted. The front-end users were not able to connect anymore to the application until the application server is restarted, rather annoying.

I reproduce the case on the delivered PSOVM FSCM 9.1 FP1 Peopletools 8.51.07.
1. Connect to the front-end application, it is working fine
2. Stop the database, you are logged out from the front-end application
Note : click on link prompt the login page, logging out of the application send a strange error for the down status of the database (“Your User ID and/or Password are invalid”)
3. Start the database
4. Open a new browser, no connection possible, raising error “bea.jolt.ApplicationException: TPESVCFAIL – application level service failure
5. Stop and start the application server
6. Open a new browser, new connections work fine again

There are two known bugs about it, one wrongly mentioned a fix within 8.51.08, the other one (#12358153) mentioned the patch 8.51.10.
Let’s have a try :
1. Install the patch 8.51.10 on top of the delivered PSOVM FSCM 9.1 FP1 Peopletools 8.51.07
2. Redeploy the PIA
3. Test the same scenario => now it is working, you can reconnect to the front-end application without bouncing the application server after a database stop and start. Good one.

So, that patch level 10 is probably the bare minimal to go in production with… even if a stop/start database is not expected.