Application messaging troubleshooting : endless ping node

It’s really common to use a small configuration for environments such as DEMO, SYS or even for development databases to avoid the server to be overloaded within useless processes. But doing so, you should be sure it is not set too small otherwise you could have some issues when setting the messaging up.
Here’s an example of a common issue a Peoplesoft DBA might be face to.
I’m working here on the PSOVM CRM 9.1 based on Peopletools 8.50.09. Of course, it is assuming the Pub/Sub service has been set to “Yes” as shown below:


Checking the local node for proper configuration (user and password must be set) :
 
So far, so good.
Now, let’s configure the gateway. For initial configuration, load the connector first :
 

And try to ping the gateway to confirm it works :


Now, let’s configure the node in the gateway :

Default user id is “administrator”, default password is “password”.


Now try to ping that new added node :


It will process endlessly until reaching the timeout.

Note that there’s no error messages in any log files, nothing to help.
Having a look into the configuration file, psappsrv.cfg, PSAPPSRV has only 1 min. instance :

Change it to a minimum of 2 :
 
Reconfigure and restart the application server :
 

Note there’s now 2 processes PSAPPSRV.

Coming back in the front end application, trying again to ping the local node within the gateway configuration as previously failed :

Few seconds later, it is successful :


So, when you want to work within the messaging, always set at least 2 as a minimum instances for PSAPPSRV process in the configuration file.

As a side note, if you fill a wrong Peopletools release as shown below, it will fail but that time with a clear explanation (Integration Gateway: PeopleSoftTargetConnector - invalid PeopleTools release). So, be sure about the correctness  of the release until the patch level :



Nicolas.

Peoplesoft PIA silent deployment

Whether the silent Weblogic installation is documented in the Peopletools Installation Guide (Task 2-1-5: Installing Oracle WebLogic on Linux or UNIX in Silent Mode), the silent PIA deployment is not. Is that because it does not exists ? Is that because there’s no use of that ? Well, it exists and in few cases it can help. Especially if you are applying Peopletools patch on regular basis. As you may know, it is most often required to redeploy the domain.

Last month I replied to a resurrected thread on Peoplesoft OTN Forum regarding a silent deployment of the PIA. I was initially – last year - thinking that was not possible. But working a lot with the PSOVM since then, I saw the PIA deployment on “fly” during the OVM template installation.

Going through the PSOVM App/Batch/PIA server once deployed, there’s a script which is called to deploy the PIA : /opt/oracle/psft/vm/installpia.sh
The script is relatively easy to understand :
PT_HOME=/opt/oracle/psft/pt
TOOLS_HOME=$PT_HOME/tools
BEA_HOME=$PT_HOME/bea
SILENT_INSTALL_FILE=install.properties
 
echo "Doing PIA silent install"
if [ ! -d $TOOLS_HOME/tmp ]; then
    mkdir $TOOLS_HOME/tmp
fi
 
cd $TOOLS_HOME/tmp
 
 
# If there is a pre-existing silent response file we
# remove it
if [ -f $SILENT_INSTALL_FILE ]; then
    rm -f $SILENT_INSTALL_FILE
fi
 
echo PS_HOME=$TOOLS_HOME >> $SILENT_INSTALL_FILE
echo DOMAIN_NAME=peoplesoft >> $SILENT_INSTALL_FILE
echo SERVER_TYPE=weblogic >> $SILENT_INSTALL_FILE
echo BEA_HOME=$BEA_HOME >> $SILENT_INSTALL_FILE
echo USER_ID=system >> $SILENT_INSTALL_FILE
echo USER_PWD=password >> $SILENT_INSTALL_FILE
echo INSTALL_ACTION=CREATE_NEW_DOMAIN >> $SILENT_INSTALL_FILE
echo DOMAIN_TYPE=NEW_DOMAIN >> $SILENT_INSTALL_FILE
echo INSTALL_TYPE=SINGLE_SERVER_INSTALLATION >> $SILENT_INSTALL_FILE
echo WEBSITE_NAME=$PIA_SITENAME >> $SILENT_INSTALL_FILE
echo APPSERVER_NAME=localhost >> $SILENT_INSTALL_FILE
echo JSL_PORT=$JOLT_PORT >> $SILENT_INSTALL_FILE
echo HTTP_PORT=$PIA_HTTP_PORT >> $SILENT_INSTALL_FILE
echo HTTPS_PORT=$PIA_HTTPS_PORT >> $SILENT_INSTALL_FILE
echo AUTH_DOMAIN= >> $SILENT_INSTALL_FILE
echo WEB_PROF_NAME=DEV >> $SILENT_INSTALL_FILE
echo WEB_PROF_USERID=PTWEBSERVER >> $SILENT_INSTALL_FILE
echo WEB_PROF_PWD=PTWEBSERVER >> $SILENT_INSTALL_FILE
echo REPORTS_DIR=/var/tmp/psreports >> $SILENT_INSTALL_FILE
 
cd $TOOLS_HOME/setup/PsMpPIAInstall
sh setup.sh -javahome $TOOLS_HOME/jre -i silent \
-DRES_FILE_PATH=$TOOLS_HOME/tmp/$SILENT_INSTALL_FILE -tempdir $TOOLS_HOME/tmp

So, first build your own response file with all the required fields according to your environment and action you are doing (deployment, re-deployment), and call it within DRES_FILE_PATH. That parameter was unknown for me. It is rather a nice learn. Of course, it might require a lot of testing, but it could be helpful if you have a high number of environments to re-build time to time, especially useful after a Peopletools (patch) upgrade which can be done on regular basis.

Save the response file in a secure place, and eventually you’ll see the advantages of silent PIA deployment. Run the installer with the silent option avoids some painful time to remember what ports are used, what webprofile, what reports directory…
I was initially not convinced about the usefulness of that, but try it. Use it.

Nicolas.

Manage Peoplesoft with OEM Grid Control (3/3)

Last part of this series about Peoplesoft managed in OEM Grid Control, once the OEM Grid Control is installed with the Peoplesoft plug-in, once the Agent is installed on the targets with the Peoplesoft plug-in, it’s time for configuring the all thing in OEM GUI.

First, add the Peoplesoft target :
The user is the Peoplesoft domain owner (not necessarily the owner of the PS_HOME), for the PSOVM servers, that’s psadm2.

The discovery path is the PS_CFG_HOME. If it is missing from the environment variable, it is going to the user’s home directory. For the PSOVM servers, it is going to the psadm2’s home :

Choose the type you want to manage :
 

We can link the Oracle database when Peoplesoft is discovered (here the db server was not started) :

And finally, everything selected earlier is now recognized :

From now, we can start/stop (all in one shot if required) and configure the all Peoplesoft domain.

Let’s see the Process Scheduler possibilities :
Here, we can do whatever we are doing within the psadmin tool from the target host:
 
 

Let’s see the Application Server possibilities :    



And the WebServer :   
I won’t go through every single sub-menus over there, there are too many. But as you can see above, everything we – as Peoplesoft DBA - usually do from the backend, directly connected onto a server, can be done through OEM Grid Control. Rather nice, especially when managing a lot of environments. That’s maybe a bit of work for configuring everything properly, according to all the specificities of the environments, but that makes the life so easier once it is done that I think it is worth to do and to have on board.

Enjoy it.

Nicolas.

Manage Peoplesoft with OEM Grid Control (2/3)

After installing the OEM Grid Control 11gR1 – the OVM template, including the Peoplesoft plug-in for OMS as shown here, it’s time for the target system to be prepared. All the client servers have to have the OM Agent. By client, it means client for the OMS, say so the clients are the databases servers and Apps/Batch/PIA servers which I created based on PSOVM as shown in previous posts. But the Peoplesoft plug-in is crawling a PS_HOME, it has to be installed only on Apps/Batch/PIA servers.

1. The OEM Agent
Whether there’s space enough in $ORACLE_HOME (/u01/app/oracle/product) on the PSOVM database servers (user oracle), there’s no space left next to the Oracle client (/opt/oracle/psft/pt) on the PSOVM App/Batch/PIA servers (user psadm1).
Before installing the Agent on App/Batch/PIA, let’s make a dedicated folder where’s space allows us to install it, as following :
[root@psovmhcm /]# cd /
[root@psovmhcm /]# df -k .
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2             3876120    724696   3111424  19% /
[root@psovmhcm /]# mkdir oracle_agent
[root@psovmhcm /]# chown -R psadm1:oracle oracle_agent
[root@psovmhcm /]# cd /opt/oracle/psft/pt
[root@psovmhcm pt]# ln -s /oracle_agent .
[root@psovmhcm pt]# ls -lrt
total 56
drwx------  2 root   root   16384 Dec 20 19:44 lost+found
drwxr-xr-x  9 root   root    4096 Dec 20 20:08 bea
drwxr-xr-x  7 root   root    4096 Dec 20 20:12 oracle-client
drwxr-xr-x  5 root   root    4096 Jan 23 09:37 cobol
drwxr-xr-x 28 psadm1 oracle  4096 Jan 23 09:43 tools
lrwxrwxrwx  1 root   root      13 Mar 30 11:16 oracle_agent -> /oracle_agent
[root@psovmhcm pt]#

Remember, the PSOVMs are 64bit word sized (furthermore Peopletools 8.5x is only supported on 64bit OS), the Agent to be downloaded is from Linux x86-64, 11.1.0.1.0 :
http://www.oracle.com/technetwork/oem/grid-control/downloads/agentsoft-090381.html
Once downloaded, upload the file to the PSOVM OEM Grid Control previously installed, into the folder /u01/OracleHomes/Middleware/oms11g/sysman/agent_download/11.1.0.1.0 and unzip it.

Then, everything can be done through the OEM GUI. Much easier than ever, especially when deploying Agent through multiple target. However, screenshots below are coming from a deployment on one target only :

Choose the platform Linux-x86-64, that’s the one we’ve just upload to the OMS server :

Here you mention the owner of the Agent home (OMA_HOME),
=> for the PSOVM database servers it will be oracle,
=> for the PSOVM App/Batch/PIA servers it will be psadm1
Then, here you also mention the parent folder hosting the Agent home (OMA_HOME),
=> for the PSOVM database servers it will be /u01/app/oracle/product,
=> for the PSOVM App/Batch/PIA servers it will be /opt/oracle/psft/pt/oracle_agent 
Uncheck the run of root.sh, it has to be done manually later on (somehow it failed in my case when it was checked).
Give the sysman password :


Be patient during these phases, depending of the number of targets you are deploying to and of the network bandwidth, it can take a quite while :


Now, the new host has been added in the host’s list :

Do not forget to run root.sh (in $OMA_HOME). We can eventually check the agent status from the target host :

And the database as well in the database’s list :

Just to show it, I also deployed the Agent through multiple targets at once :

In the end, we got all the hosts targets

         
Warning :
1. Do not forget to run root.sh from the target Agent home, otherwise you may encountered the following error (RemoteOperationException: ERROR: NMO not setuid-root (Unix-only)) later on when managing the environment :

2. Do not forget to maintain the /etc/hosts according to the all environment. The /etc/hosts from OMS server should contains its own IP address, hostname and domain name, but also all the target servers’ references (IP, hostname and domain). Here’s the one I’m using on the OMS host server :
[oracle@ovmoem111 ~]$ more /etc/hosts
127.0.0.1       localhost.localdomain   localhost
192.168.1.160 ovmoem111.phoenix.nga ovmoem111
192.168.1.133   psovmdb.phoenix.nga psovmdb
192.168.1.143   psovmcrm.phoenix.nga psovmcrm
192.168.1.142   psovmcrmdb.phoenix.nga psovmcrmdb
192.168.1.145   psovmfscm.phoenix.nga psovmfscm
192.168.1.144   psovmfscmdb.phoenix.nga psovmfscmdb
192.168.1.147   psovmhcm.phoenix.nga psovmhcm
192.168.1.146   psovmhcmdb.phoenix.nga psovmhcmdb
192.168.1.148   psovmhcmdb2.phoenix.nga psovmhcmdb2
[oracle@ovmoem111 ~]$

The /etc/hosts file from target hosts must contains their own IP address, hostname and domain name as well as the ones of the OMS server. Here’s an example on one of the target host :
[root@psovmcrm ~]# more /etc/hosts
127.0.0.1       localhost.localdomain   localhost
::1     localhost6.localdomain6 localhost6
192.168.1.143   psovmcrm.phoenix.nga psovmcrm
192.168.1.160 ovmoem111.phoenix.nga ovmoem111
[root@psovmcrm ~]#

If not doing so, you can have the following error (not possible to configure target database for instance) :

2. The Peoplesoft plug-in for the Agent 
To be done only on App/Batch/PIA PSOVM servers.
To be able to manage the Peoplesoft environment from OEM Grid Control, the Agent must have the Peoplesoft plug-in. This is the same package used for the plug-in previously installed on OMS.
To be downloaded from https://edelivery.oracle.com (free registration), Product “Peoplesoft Enterprise”, Platform “Linux x86-64” (but the results do not differ, it does not matter for Peoplesoft packages), then choose the pack “Oracle Application Management Pack 1.0.0.2.0 for PeopleSoft Enterprise Media Pack v2” part number B60821-02.
Download the part number V22794-01 named “PeopleSoft Enterprise Environment Management Plug-in for Oracle Enterprise Manager (8.51.0.1.0)”. Despite the name, it is compatible on Peopletools 8.48 and above, moreover it is also compatible with OEM Grid Control 11gR1.

The Peoplesoft plug-in installation on Agent will be the easiest part, using the silent installation option as following.
Warning : the Peoplesoft plug-in for OEM Agent must be done whilst the psconfig.sh has not been previously invoked in the session (the PS environment must not be set), make it not happening.

Here I rename the psconfig.sh to avoid all confusion.
login as: root
root@192.168.1.143's password:
Last login: Wed Mar 30 11:19:04 2011 from 192.168.1.202
[root@psovmcrm ~]# su - psadm1
[psadm1@psovmcrm ~]$ cd $PS_HOME
[psadm1@psovmcrm tools]$ pwd
/opt/oracle/psft/pt/tools
[psadm1@psovmcrm tools]$ mv psconfig.sh _psconfig.sh
[psadm1@psovmcrm tools]$ exit
logout
[root@psovmcrm ~]# su - psadm1
-bash: /opt/oracle/psft/pt/tools/psconfig.sh: No such file or directory

From the Peoplesoft plug-in, unzip the file 5016553.zip (present under the downloaded Peoplesoft plug-in package) and copy it on your local server for installation (the target OMA host).
[psadm1@psovmcrm ~]$ cd /tmp
[psadm1@psovmcrm tmp]$ scp -r root@ovmoem111:/nfs/software/OracleCD/OEM/PeoplesoftPlugin/8.51/Disk1/5016553 .
root@ovmoem111's password:
<…snipped…>
[psadm1@psovmcrm tmp]$

You must set the ORACLE_HOME variable to the OMA_HOME and stop the Agent.
[psadm1@psovmcrm tmp]$ export ORACLE_HOME=/opt/oracle/psft/pt/oracle_agent/agent11g
[psadm1@psovmcrm tmp]$ cd $ORACLE_HOME/oui/bin
[psadm1@psovmcrm bin]$ cd ../../bin
[psadm1@psovmcrm bin]$ ./emctl stop agent

Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Agent is Not Running
[psadm1@psovmcrm bin]$

Then, from the OUI Installer of Agent, run the silent installation.
[psadm1@psovmcrm bin]$ cd -
/opt/oracle/psft/pt/oracle_agent/agent11g/oui/bin
[psadm1@psovmcrm bin]$ ./runInstaller.sh -silent -ignoreSysPrereqs -waitForCompletion -responseFile /tmp/5016553/Disk1/response/psemagent.rsp -sourceLoc /tmp/5016553/Disk1/stage/products.xml -invPtrLoc $ORACLE_HOME/oraInst.loc ORACLE_HOME=$ORACLE_HOME
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 2055 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2011-04-01_07-56-41AM. Please wait ...Oracle Universal Installer, Version 11.1.0.8.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.

You can find the log of this install session at:
/home/psadm1/oraInventory/logs/installActions2011-04-01_07-56-41AM.log
.................................................................................................... 100% Done.

Loading Product Information
............................................................... 100% Done.

Analyzing dependencies
............................................................... 100% Done.

-----------------------------------------------------------------------------
Summary
Global Settings
    Source: /tmp/5016553/Disk1/stage/products.xml
    Oracle Home: /opt/oracle/psft/pt/oracle_agent/agent11g (agent11g1)
    Installation Type: Typical
Product Languages
   English
Space Requirements
   / Required 23MB (includes 7MB temporary) : Available 1.82GB
New Installations (1 products)
   PeopleSoft Environment Management Plug-in (Agent) 8.51.0.1.0
-----------------------------------------------------------------------------

Installation in progress (Friday, April 1, 2011 7:58:52 AM EDT)
...............................................................................................                                 95% Done.
Install successful

Linking in progress (Friday, April 1, 2011 7:59:03 AM EDT)
Link successful

Setup in progress (Friday, April 1, 2011 7:59:03 AM EDT)
Setup successful

End of install phases.(Friday, April 1, 2011 7:59:11 AM EDT)
Starting to execute configuration assistants
Configuration assistant "PeopleSoft Environment Management Agent Plug-in" succeeded
The installation of PeopleSoft Environment Management Plug-in (Agent) was successful.
Please check '/home/psadm1/oraInventory/logs/silentInstall2011-04-01_07-56-41AM.log' for more details.
[psadm1@psovmcrm bin]$

The Agent is restarted automatically, we can check it.
[psadm1@psovmcrm bin]$ cd -
/opt/oracle/psft/pt/oracle_agent/agent11g/bin
[psadm1@psovmcrm bin]$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 11.1.0.1.0
OMS Version       : 11.1.0.1.0
Protocol Version  : 11.1.0.0.0
Agent Home        : /opt/oracle/psft/pt/oracle_agent/agent11g
Agent binaries    : /opt/oracle/psft/pt/oracle_agent/agent11g
Agent Process ID  : 1731
Parent Process ID : 1708
Agent URL         : https://psovmcrm.phoenix.nga:3872/emd/main/
Repository URL    : https://ovmoem111.phoenix.nga:4900/em/upload
Started at        : 2011-04-01 07:59:47
Started by user   : psadm1
Last Reload       : 2011-04-01 07:59:47
Last successful upload                       : 2011-04-01 08:01:25
Total Megabytes of XML files uploaded so far :     0.43
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    52.60%
Last successful heartbeat to OMS             : 2011-04-01 08:05:27
---------------------------------------------------------------
Agent is Running and Ready
[psadm1@psovmcrm bin]$

Rename back to the original name the psconfig.sh for further connection.
[psadm1@psovmcrm bin]$ cd /opt/oracle/psft/pt/tools
[psadm1@psovmcrm tools]$ mv _psconfig.sh psconfig.sh
[psadm1@psovmcrm tools]$

From now, we should be able to add this Peoplesoft environment in the OEM Grid Control.
So far we have this :

Nest step is then to add Peoplesoft target. To be continued here.