Application server load balancing and failover

In a previous blog’s entry, I explained about Multiple application messaging servers on Peopletools 8.51 and facilities for load balance and failover on messaging domain server.
In a very similar way, the application server can also be used in load balance and failover configuration.
First of all, you need at least two applications servers, please note that if you are on the same server, then domain name and JSL (jolt) port should be different from each other. Of course, there are not many advantages to have this configuration if working on one and only one server, but rather good to have different server, if one crash, the other one takes place.
Here we go.
First application server :
 
And the second one :

Please, note that whether I name one “first” and the other one “second”, there is no precedence of one compared to the other by default (except if you define a weight in configuration.properties, but let’s leave it for later explanation).
Then starting each application server
 
Note that the number of listened ports regarding Jolt is defined is the psappsrv.cfg of the domain as following :
[JOLT Listener]
;=========================================================================
; Settings for JOLT Listener
;=========================================================================
Address=%PS_MACH%
Port=9000
Encryption=0
Min Handlers=5
Max Handlers=7
In the current case, the ports between 9000 and 9005 will be immediately in use after started :
[psadm2@psovmfscmfp1 APPDOM]$ netstat -an|grep 900
tcp        0      0 192.168.1.150:9000          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9001          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9002          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9003          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9004          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9005          0.0.0.0:*                   LISTEN
[psadm2@psovmfscmfp1 APPDOM]$
It means the second domain needs to be configured with a port above 9005.
 

Here’s the output from the database side, since my two apps are on the same machine, there’s nothing really useful so far, cannot distinguished which domain a database session is coming from :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
SQL> /

USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSQRYSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSQRYSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,

21 rows selected.

Now, webserver configuration to be done in configuration.properties file ($PS_HOME/webserv/peoplesoft//applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps).
This is a easy configuration, put the application servers separated by simple comma as following :
[psadm2@psovmfscmfp1 peoplesoft]$ more ./applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps/configuration.properties
<<...snipped...>>
# psserver must point to your application server machine name (recommended) or IP address
# and JOLT port.  The PeopleTools release of the application server must match the PeopleTools
# release of the PeopleSoft files on the web server.
#
# To enable jolt failover and load balancing, provide a list of application server
# domains in the format of;  psserver=AppSrvr:JSLport,...
# For example:  psserver=SERVER1:9000,SERVER2:9010,SERVER3:9020
## sd="Application Server Connect String", ld="This string embodies the failover and load balancing string for PIA to connect to Application Server domains.  Consult the PeopleB
ooks recommendations on load balancing and failover for more information.  The format is psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name.   Multiple d
omains should be comma separated", tip="Format: psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name, e.g  psserver=SERVER1.oracle.com:9000,SERVER2.oracle.
com:9010"
psserver=psovmfscmfp1:9000,psovmfscmfp1:9010
## sd="Jolt Pooling", dt="c", ld="Enable Jolt Pooling"
joltPooling=true
<<...snipped...>>
Configured as above, all the application servers have the same weight (the same priority) in the round-robin algorithm. This algorithm affect ingoing transaction randomly to one or the other configured application server, there’s no priority, no load checkup, no network checkup, it’s just random (find out more in the My Oracle Support note E-WS: Load Balancing Options for Web Server and Application Server [ID 618367.1]).
But we could also introduced weight (from 1 to 10) on application server to make one precedence on the other as following :
psserver=psovmfscmfp1:9000#10,psovmfscmfp1:9010#1
In that case, the first server can takes extra work for an heavy load compared to the second one. 

Not only the load balance will be set, but the failover as well.

After the first connection to the front end application, we can check the client status of both application servers :
 
We can see a first pool as been instantiated.
Since round-robin algorithm is randomly given connection to the application server, it’s not always easy to show when alone, furthermore, the pool connection are making it even less simple.
But you probably can check it from the back end :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
  2* order by 2
SQL> /

USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    NGA,,192.168.1.202,APPDOM,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,,192.168.1.202,APPDOM2,PSAPPSRV,
…
Here, from the same client machine, different users are in connected through the 2 applications servers (APPDOM and APPDOM2).

Now, let’s simulate a application server crash :

The online users won’t be disconnected, and the opened connections are automatically redirected to the second application server (failover) :

A second pool is now created, but that one on APPDOM2. Everything will be transparent to the front end users.

Find out more in the OTN hosted Peoplebooks : Working with Jolt Configuration Options

Nicolas.

PS: whilst I was writing this blog entry, I was reading an interesting note as well recommending the use of IP address instead of the server name for security reason in the configuration.properties (worth reading it Health Check Alert: Consider setting PSSERVER for web server JOLT connections to refer to an IP-address:port combination [ID 957484.1]).