On The Peoplesoft Road: September 2011

Friday, September 30, 2011

Application server load balancing and failover

In a previous blog’s entry, I explained about Multiple application messaging servers on Peopletools 8.51 and facilities for load balance and failover on messaging domain server.
In a very similar way, the application server can also be used in load balance and failover configuration.
First of all, you need at least two applications servers, please note that if you are on the same server, then domain name and JSL (jolt) port should be different from each other. Of course, there are not many advantages to have this configuration if working on one and only one server, but rather good to have different server, if one crash, the other one takes place.
Here we go.
First application server :

And the second one :

Please, note that whether I name one “first” and the other one “second”, there is no precedence of one compared to the other by default (except if you define a weight in configuration.properties, but let’s leave it for later explanation).
Then starting each application server

Note that the number of listened ports regarding Jolt is defined is the psappsrv.cfg of the domain as following :
[JOLT Listener]
;=========================================================================
; Settings for JOLT Listener
;=========================================================================
Address=%PS_MACH%
Port=9000
Encryption=0
Min Handlers=5
Max Handlers=7
In the current case, the ports between 9000 and 9005 will be immediately in use after started :
[psadm2@psovmfscmfp1 APPDOM]$ netstat -an|grep 900
tcp        0      0 192.168.1.150:9000          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9001          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9002          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9003          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9004          0.0.0.0:*                   LISTEN
tcp        0      0 192.168.1.150:9005          0.0.0.0:*                   LISTEN
[psadm2@psovmfscmfp1 APPDOM]$
It means the second domain needs to be configured with a port above 9005.

Here’s the output from the database side, since my two apps are on the same machine, there’s nothing really useful so far, cannot distinguished which domain a database session is coming from :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
SQL> /

USERNAME                       PROGRAM                                          CLIENT_INFO
------------------------------ ------------------------------------------------ ----------------------------------------------------------------
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSQRYSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSQRYSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSSUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBDSP,
SYSADM                         PSSUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSUBHND,
SYSADM                         PSPUBDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBDSP,
SYSADM                         PSPUBHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSPUBHND,
SYSADM                         PSBRKDSP@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKDSP,
SYSADM                         PSMONITORSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3 VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSMONITORSRV,
SYSADM                         PSBRKHND@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSBRKHND,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,
SYSADM                         PSSAMSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSSAMSRV,
SYSADM                         PSAPPSRV@psovmfscmfp1.phoenix.nga (TNS V1-V3)    VP1,psadm2,psovmfscmfp1.phoenix.nga,,PSAPPSRV,

21 rows selected.

Now, webserver configuration to be done in configuration.properties file ($PS_HOME/webserv/peoplesoft//applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps).
This is a easy configuration, put the application servers separated by simple comma as following :
[psadm2@psovmfscmfp1 peoplesoft]$ more ./applications/peoplesoft/PORTAL.war/WEB-INF/psftdocs/ps/configuration.properties
<<...snipped...>>
# psserver must point to your application server machine name (recommended) or IP address
# and JOLT port. The PeopleTools release of the application server must match the PeopleTools
# release of the PeopleSoft files on the web server.
#
# To enable jolt failover and load balancing, provide a list of application server
# domains in the format of; psserver=AppSrvr:JSLport,...
# For example: psserver=SERVER1:9000,SERVER2:9010,SERVER3:9020
## sd="Application Server Connect String", ld="This string embodies the failover and load balancing string for PIA to connect to Application Server domains. Consult the PeopleB
ooks recommendations on load balancing and failover for more information. The format is psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name. Multiple d
omains should be comma separated", tip="Format: psserver=AppSrvr:JSLport where AppSrvr is your fully qualified domain name, e.g psserver=SERVER1.oracle.com:9000,SERVER2.oracle.
com:9010"
psserver=psovmfscmfp1:9000,psovmfscmfp1:9010
## sd="Jolt Pooling", dt="c", ld="Enable Jolt Pooling"
joltPooling=true
<<...snipped...>>
Configured as above, all the application servers have the same weight (the same priority) in the round-robin algorithm. This algorithm affect ingoing transaction randomly to one or the other configured application server, there’s no priority, no load checkup, no network checkup, it’s just random (find out more in the My Oracle Support note E-WS: Load Balancing Options for Web Server and Application Server [ID 618367.1]).
But we could also introduced weight (from 1 to 10) on application server to make one precedence on the other as following :
psserver=psovmfscmfp1:9000#10,psovmfscmfp1:9010#1
In that case, the first server can takes extra work for an heavy load compared to the second one.

Not only the load balance will be set, but the failover as well.

After the first connection to the front end application, we can check the client status of both application servers :

We can see a first pool as been instantiated.
Since round-robin algorithm is randomly given connection to the application server, it’s not always easy to show when alone, furthermore, the pool connection are making it even less simple.
But you probably can check it from the back end :
SQL> select username,program, client_info from v$session where username = 'SYSADM'
2* order by 2
SQL> /

Monday, September 19, 2011

3 months update later, finally

I did not maintained this blog for quite a while during this last quarter, I should apology to the readers.
What happened during that time ? “Finally” is probably the word to keep in mind for that period.

CentOS 6.0
In July, CentOS finally released the version 6.0. It has been a very long waiting and many people moved away for an other (equivalent) distribution, I named Scientific Linux (see below). I do not know if the CentOS dev. team made the rights choice to go for the version 6.0 before the update 5.7, but truly enough, no update at all for several months is quite unusual and rather disappointed even though that’s a free distribution.
I did not tested this version yet, but according to my reading across a couple of article on Internet, it could have been done much better especially after that long (long) waiting.
Now, we may have to wait for the update 6.1 which has been promised very soon, still have to define “soon”.

CentOS 5.7
Few days ago, earlier in Sept., we also get that update I was looking for several months too. Much appreciated, the upgrade from 5.6 went smoothly, as usual. The only thing for that one was the delay since the RHEL 5.7 date release. Again, a lot of debates on that…

Oracle Linux 6.1
Of course, this is also free (the use, not the support), but developed by a company, a big company, so the time to get the new version is quite different compared to CentOS.
Oracle Linux is available since the 1st of June, 5.7 since August.

Scientific Linux 6.1 and 5.7
An other free distribution of a RHEL recut, Scientific Linux interest has considerably grown since CentOS took too much time for the newest release delivery.
Whether the update 5.7 has just been released, their update 6.1 is already out for a month. Some can argue the developers behind are also paid… still, this distribution has a good chance to see new customers.

Oracle VM 3.0 (August, the 23rd)
After several years of promises, Oracle has finally released a new version of their virtualization software, Oracle VM 3.0. A lot of new things has to be discovered, according to Oracle, it is much better than ever. But have they not too late compared the the concurrent like VMWare vSphere Hypervisor ?

Open World
On Oracle database and Peoplesoft sides, no major thingy, nothing unusual or non expected, the summer was rather quiet. So, no worries.
But the Oracle Open World is just in the corner by now. Probably we have to count on new Peopletools version 8.52 around that event, maybe new Peoplesoft OVM templates as well (?). Saying about Oracle Open World, this year I won’t attend. I hope attendees will report back on their blog about the exciting sessions.

Other
Above all, and most importantly my son is born also the 23rd of August, which explain the delay and absence of blog update a bit before and since then… but I’ll be back.
The only “problem” I have now, the room which was used to be my server room is occupied. I have to find new space for my noisy server and NAS. But that’s the part of the fun to have children !

Nicolas.

Multiple application messaging servers

In a standard mode, when dealing within messaging, one domain is configured as followed, and most of the time it is enough :

However, if you have to deal with a high number of messages, to process a maximum number of messages, you may want to create multiple domains to handle those messages.
Let’s create a new one as below :

Then, coming back on the domain status’ page, we see that new domain :

We have to activate it to make this new domain processing messages. Change the status, and click “update” button :

Here we go. An error is raised, we cannot have more than one master domain. And being master is set by default for a given domain.

What is that ?
A master domain can be used to balance ingoing/outgoing messages between master and slave(s) domains. Particularly useful when dealing a very high number of messages in a short time.

We can handle it in two ways.

1. Static slave
In the configuration file appsrv.cfg, we can change a parameter value – Domain Slave Mode – from 0 to 1 to make the domain a slave domain.
Change from default value

To

Reload the application server domain, and coming back to the domain status’ page to make it active :

After updating the status, we can see the domain APPDOM2 activated and also that’s a slave domain. It is fixed, there’s no way to change but modify the configuration file manually and reload again the domain.
Without any failover set, it is probably the easiest way to achieve the goal : handle a high number of messages, you could make multiple slaves domains all managed by one and only master domain.
Using multiple slaves domains can also be useful, you could set a load balance in between all of them (click on “Master/Slave LoadBalance”) to configure all the priorities, such as following :

2. Dynamic slave
An other way for handling a high number of messages and multiple domains would be to make slave domain dynamic. It is maybe a little bit more tricky, but also probably more reliable and scalable.
A dynamic slave domain can become a master domain in case of failure of the current master domain. Really nice features if the messages have to be processed asap or with a minimum of down-time.
So, we return back to the original domain status’ page, the second domain APPDOM2 is inactive (it is not set as static slave in the configuration file):

Click on “SetUp Failover”, then check “enable Failover”, define a failover time (when the slave domain has to become master after original master failure, here 5 minutes) and most importantly check “Dynamic Slave Option”. Finally set a failover priority (1, 2, 3…) and save :

It makes that second domain defined as dynamic slave as shown above in the failover setting as well as below in the domain status page :

From now, we have two domains which proceed with the messages, given the fact that if the first one fails, the second will become the master.
Just to show it, here I stopped the first domain APPDOM, once the failover time is off (as shown earlier, 5 minutes), we can see the second domain APPDOM2 becoming master (it is not “dynamic slave” anymore), and the first obviously becoming inactive :

When the first domain APPDOM is restarted, it becomes the slave domain whereas the second domain APPDOM2 remains the master.

Making choice between static and dynamic slave domain may not be an easy task, it depends of your all architecture and requirements you have in messages processing. The first might be useful on not so big environment whereas the second is more than recommended on big environments (you don’t want to make dynamic domain on non-messages dedicated domain).

Enjoy,

Nicolas.

Find out more on OTN hosted Peoplebooks : http://download.oracle.com/docs/cd/E24150_01/pt851h2/eng/psbooks/tiba/book.htm?File=tiba/htm/tiba13.htm