Sunday, March 13, 2011

PSOVM and Active Data Guard : Batch Server

After setting a Physical Standby Database and enabling Active Data Guard on Peoplesoft OVM, it is probably the nicest part for the Peoplesoft guys. Going further and look for Batch processes running onto that secondary database. Much useful for the big and heavy report, for instance.
As said earlier, the latest PSOVM is coming with the Peopletools 8.51, we can get all the latest technologies offered.
The new property regarding the standby database in the process scheduler configuration file allow us such thing, here in red from psprcs.cfg
[Startup]
;=========================================================================
; Database Signon settings
;=========================================================================
DBName=
DBType=
UserId=
UserPswd=
ConnectId=
ConnectPswd=
ServerName=
StandbyDBName=
StandbyDBType=
StandbyUserId=
StandbyUserPswd=
It is working only for an enabled Oracle Active Data Guard database.
At the end of this stage, the global architecture will be as follow (in red, the change compared to the previous state) :
PS_ACTIVEDG_037
1. Required scripts
Warning : 4 scripts are required to set up the configuration to run processes on read-only database through the Batch Server. Unfortunately, PSOVM does not offer any default scripts ($PS_HOME/scripts), so an additional Peoplesoft home installation is required before going further.
Those scripts are psadmin.sql ($PS_HOME/scripts/unix), createlocalsynonyms.sql, createdblinktoprimarydb.sql and createremotesynonyms.sql (from $PS_HOME/scripts). They are delivered on default Peoplesoft installation, but so far, not in PSOVM.
Eventually, here the copy onto the primary site (hosting primary database ) from an other Linux Peoplesoft installation I made on my own :
[root@psovmhcmdb ~]# su - oracle
[oracle@psovmhcmdb ~]$ scp root@192.168.1.141:/apps/psoft/pt851/scripts/unix/psadmin.sql .
root@192.168.1.141's password:
psadmin.sql                                   100% 2714     2.7KB/s   00:00
[oracle@psovmhcmdb ~]$ scp root@192.168.1.141:/apps/psoft/pt851/scripts/createlocalsynonyms.sql .
root@192.168.1.141's password:
createlocalsynonyms.sql                       100% 1986     1.9KB/s   00:01
[oracle@psovmhcmdb ~]$ scp root@192.168.1.141:/apps/psoft/pt851/scripts/createdblinktoprimarydb.sql .
root@192.168.1.141's password:
createdblinktoprimarydb.sql                   100% 1739     1.7KB/s   00:00
[oracle@psovmhcmdb ~]$ scp root@192.168.1.141:/apps/psoft/pt851/scripts/createremotesynonyms.sql .
root@192.168.1.141's password:
createremotesynonyms.sql                      100% 4485     4.4KB/s   00:00
2. Create a secondary ACCESSID/SYMBOLICID on primary db – psadmin.sql
2.1
PSADMIN.SQL on primary database
Run the script psadmin.sql will create an other Peoplesoft “owner” (ACCESSID) beside the default one SYSADM :
[oracle@psovmhcmdb ~]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb ~]$ sqlplus / as sysdba
@psadmin
=> user : SYSBKP
=> pwd : SYSBKP
=> default permanent tablespace : psdefault

2.2 Insert the corresponding line in PS.PSDBOWNER for the new ACCESSID linked to the standby database
SQL> conn sysbkp/SYSBKP
Connected.
SQL> insert into ps.psdbowner values ('H91STBY','SYSBKP');
1 row created.
SQL> commit;

2.3 Add a new SYMBOLICID
Open Application Designer, and create a new SYMBOLICID for the newly created ACCESSID (step 2.1)
PS_ACTIVEDG_004
PS_ACTIVEDG_005
PS_ACTIVEDG_006 PS_ACTIVEDG_007

3. Create a dedicated application user attached to the secondary SYMBOLICID
Within the front-end application, copy a user (e.g. PS) which is able to manage process scheduler and application server.
PS_ACTIVEDG_009 PS_ACTIVEDG_010
PS_ACTIVEDG_011
Make sure you set the secondary SYMBOLICID created earlier on step 2.3 (here SYSADM2) :
PS_ACTIVEDG_012 

We can check the PSACCESSPRFL table to see the new line :
SQL> select * from sysadm.psaccessprfl;
SYMBOLICID                          VERSION
-------------------------------- ----------
ACCESSID
----------------------------------------------------------------
ACCESSPSWD                                                        ENCRYPTED
---------------------------------------------------------------- ----------
SYSADM2                                 157
sBzLcoZSrag=
sBzLcoZSrag=                                                              1

SYSADM1                                   1
sBzLcYlPrag=
sBzLcYlPrag=                                                              1

4. Create database link on primary db to the primary db – createdblinktoprimarydb.sql
Before going further, a small explanation. Running a process through the process scheduler is never purely read-only even if the process itself is only selecting data. The process scheduler has to update a couple of tables, for instance to state the batch status. So, here it will be used a very nice little tricky thing. We’ll create a database link to the primary database on the primary (remember, whether the secondary database is open, it is open in read-only mode), it will become a remote database link onto the secondary database once replicated (it is assume the tnsnames.ora file on primary and secondary are same).
On the primary database, connected as the secondary ACCESSID SYSADM2 run :
SQL> CREATE DATABASE LINK PRIMARY CONNECT TO sysadm IDENTIFIED BY SYSADM USING 'H91TMPLT';
Database link created.
5. Create local synonyms – createlocalsynonyms.sql
The local synonyms are synonyms for the read-only objects (tables and views). Bear in mind to run it whenever you create customized records.
On the primary database, connected as the secondary ACCESSID SYSADM2 run the following script, it will create two subsequent scripts to be run :
[oracle@psovmhcmdb ~]$ more createlocalsynonyms.sql
...
SET ECHO ON
REM Create local synonyms for tables
SET HEADING OFF
Spool CREATELOCALTABLESYNONYMS.SQL
SELECT 'CREATE SYNONYM ' || TABLE_NAME || ' FOR SYSADM.' || TABLE_NAME || ';'
FROM ALL_TABLES
WHERE OWNER = 'SYSADM'
ORDER BY TABLE_NAME
;
Spool off

REM Create local synonyms for views
Spool CREATELOCALVIEWSYNONYMS.SQL
SET HEADING OFF
SELECT 'CREATE SYNONYM ' || VIEW_NAME || ' FOR SYSADM.' || VIEW_NAME || ';'
FROM ALL_VIEWS
WHERE OWNER = 'SYSADM'
ORDER BY VIEW_NAME
;
Spool off

[oracle@psovmhcmdb ~]$ sqlplus sysbkp/SYSBKP
SQL*Plus: Release 11.1.0.7.0 - Production on Fri Mar 11 10:11:06 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> @createlocalsynonyms.sql
...
<snipped>
...
SQL> !ls
createdblinktoprimarydb.sql  CREATELOCALTABLESYNONYMS.SQL  createremotesynonyms.sql  psadmin.sql  scripts  templates
createlocalsynonyms.sql      CREATELOCALVIEWSYNONYMS.SQL   psadmin.log               scp.sh       setup

SQL> @CREATELOCALTABLESYNONYMS.SQL
...
<snipped>
...
SQL> @CREATELOCALVIEWSYNONYMS.SQL
...
<snipped>
...
SQL>

6. Create remote synonyms – createremotesynonyms.sql
The remote synonyms are synonyms for the read-writable objects (the tables updated by the process scheduler itself). Bear in mind to run it whenever you create customized records.
On the primary database, connected as the secondary ACCESSID SYSADM2 run :
SQL> @createremotesynonyms.sql
...
<snipped>
...
SQL> Spool CREATEREMOTESYNONYMS.LOG
SQL>
SQL> CREATE or REPLACE SYNONYM PS_AE_QE_AETEST_TAO1 FOR PS_AE_QE_AETEST_TAO1@PRIMARY;
Synonym created.
...
<snipped>
...
SQL> CREATE or REPLACE SYNONYM PS_BAT_TIMINGS_FN FOR PS_BAT_TIMINGS_FN@PRIMARY;
Synonym created.
SQL>
SQL> Spool off

7. On the secondary site, confirm everything has been replicated
SQL> conn sysbkp/SYSBKP
Connected.
SQL> select * from ps.psdbowner;

DBNAME                           OWNERID
-------------------------------- --------------------------------
H91STBY                          SYSBKP
H91TMPLT                         SYSADM

SQL> select count(*), object_type from user_objects group by object_type;
  COUNT(*) OBJECT_TYPE
---------- ----------------------------------------------------------------------------
         1 DATABASE LINK
     47701 SYNONYM

8. Configure the Batch server
8.1 Configure the tnsnames.ora file, it must contain both databases (primary and secondary)
[root@psovmhcm ~]# cat /etc/tnsnames.ora
H91TMPLT=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.146)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=H91TMPLT)))
H91STBY=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.148)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SID=H91TMPLT)))
[root@psovmhcm ~]#

8.2 Modify the psprcs.cfg file
[psadm2@psovmhcm ~]$ pwd
/home/psadm2
[psadm2@psovmhcm ~]$ ls
java0.log  oradiag_psadm2  psft
[psadm2@psovmhcm ~]$ cd psft/pt/8.51/appserv/prcs/PRCSDOM/
[psadm2@psovmhcm PRCSDOM]$
[psadm2@psovmhcm PRCSDOM]$ more psprcs.cfg
[Startup]
;=========================================================================
; Database Signon settings
;=========================================================================
DBName=H91TMPLT
DBType=ORACLE
UserId=PS
UserPswd=PS
ConnectId=people
ConnectPswd=peop1e
ServerName=192.168.1.146
StandbyDBName=H91STBY
StandbyDBType=ORACLE
StandbyUserId=PSBKP
StandbyUserPswd=PSBKP


8.3 Reconfigure the modified PRCSDOM batch server
[psadm2@psovmhcm PRCSDOM]$ cd $PS_HOME/appserv
[psadm2@psovmhcm appserv]$ ./psadmin

PSADMIN -- Tools Release: 8.51.02
Copyright (c) 1996, 2010, Oracle. All rights reserved.

--------------------------------
PeopleSoft Server Administration
--------------------------------

  1) Application Server
  2) Process Scheduler
  3) Search Server
  4) Replicate Config Home
  q) Quit

Command to execute (1-4, q): 2
--------------------------------------------
PeopleSoft Process Scheduler Administration
--------------------------------------------

  1) Administer a domain
  2) Create a domain
  3) Delete a domain
  4) Import domain configuration
  q) Quit

Command to execute (1-4, q) : 1
Domain list:
   1) PRCSDOM
Select domain: 1
--------------------------------------------
PeopleSoft Process Scheduler Administration
--------------------------------------------
     Domain Name: PRCSDOM

  1) Boot this domain
  2) Domain shutdown menu
  3) Domain status menu
  4) Configure this domain
  5) TUXEDO command line (tmadmin)
  6) Edit configuration/log files menu
  7) Clean IPC resources of this Domain
  q) Quit

Command to execute (1-7, q) : 4
This option will shutdown the domain.
Do you want to continue? (y/n) [n] :y
<…snipped…>
----------------------------------------------
Quick-configure menu -- domain: PRCSDOM
----------------------------------------------
      Features                      Settings
     ==========                    ==========
  1) App Engine        : Yes    5) DBNAME     :[H91TMPLT]
  2) Master Scheduler  : Yes    6) DBTYPE     :[ORACLE]
                                7) PrcsServer :[OVMDEMO]
                                8) UserId     :[PS]
                                9) UserPswd   :[PS]
                               10) ConnectID  :[people]
                               11) ConnectPswd:[peop1e]
                               12) ServerName :[192.168.1.146]
                               13) Log/Output Dir:[%PS_SERVDIR%/log_output]
                               14) SQRBIN     :[%PS_HOME%/bin/sqr/%PS_DB%/bin]
                               15) AddToPATH  :[.]

      Actions
     =========
  3) Load config as shown
  4) Custom configuration
  h) Help for this menu
  q) Return to previous menu

HINT: Enter 5 to edit DBNAME, then 3 to load
Enter selection (1-15, h, or q): 3
Performing load prechecks ...
Loading validation table...
  setting DBName=H91TMPLT
  setting DBType=ORACLE
  setting UserId=PS
  setting UserPswd=PS
  setting ConnectId=people
  setting ConnectPswd=peop1e
  setting ServerName=192.168.1.146
  setting PrcsServerName=OVMDEMO
  setting Log/Output Directory=%PS_SERVDIR%/log_output
  setting Add to PATH=.
  setting SQRBIN=%PS_HOME%/bin/sqr/%PS_DB%/bin
New CFG file written with modified Startup parameters

Log Directory entry not found in configuration file.
Setting Log Directory to the default... [PS_SERVDIR/LOGS]
Spawning enabled for server PSDSTSRV.
Configuration file successfully created.
CFG setting changes completed
Loading configuration...
Domain configuration complete.


8.4
Start the modified PRCSDOM batch server
--------------------------------------------
PeopleSoft Process Scheduler Administration
--------------------------------------------
     Domain Name: PRCSDOM

  1) Boot this domain
  2) Domain shutdown menu
  3) Domain status menu
  4) Configure this domain
  5) TUXEDO command line (tmadmin)
  6) Edit configuration/log files menu
  7) Clean IPC resources of this Domain
  q) Quit

Command to execute (1-7, q) : 1
Archived a copy of the domain configuration to /home/psadm2/psft/pt/8.51/appserv/prcs/PRCSDOM/Archive/psprcs.cfg
Attempting to boot bulletin board...
tmadmin - Copyright (c) 2007-2008 Oracle.
Portions * Copyright 1986-1997 RSA Data Security, Inc.
All Rights Reserved.
Distributed under license by Oracle.
Tuxedo is a registered trademark.
No bulletin board exists. Entering boot mode.

> INFO: Oracle Tuxedo, Version 10.3.0.0, 64-bit, Patch Level 031
Booting admin processes ...
exec BBL -A :
        process id=1910 ... Started.
1 process started.
Attaching to active bulletin board.

> Attempting to boot ...
INFO: Oracle Tuxedo, Version 10.3.0.0, 64-bit, Patch Level 031

Booting server processes ...
exec PSMSTPRC -o ./LOGS/stdout -e ./LOGS/stderr -A -- -CD H91TMPLT -PS OVMDEMO -A start -S PSMSTPRC :
        process id=1914 ... Started.
exec PSAESRV -o ./LOGS/stdout -e ./LOGS/stderr -- -CD H91TMPLT -S PSAESRV :
        process id=1919 ... Started.
exec PSAESRV -o ./LOGS/stdout -e ./LOGS/stderr -- -CD H91TMPLT -S PSAESRV :
        process id=1923 ... Started.
exec PSAESRV -o ./LOGS/stdout -e ./LOGS/stderr -- -CD H91TMPLT -S PSAESRV :
        process id=1927 ... Started.
exec PSDSTSRV -o ./LOGS/stdout -e ./LOGS/stderr -p 1,600:1,1 -sPostReport -- -CD H91TMPLT -PS OVMDEMO -A start -S PSDSTSRV :
        process id=1931 ... Started.
exec PSPRCSRV -o ./LOGS/stdout -e ./LOGS/stderr -sInitiateRequest -- -CD H91TMPLT -PS OVMDEMO -A start -S PSPRCSRV :
        process id=1936 ... Started.
exec PSMONITORSRV -o ./LOGS/stdout -e ./LOGS/stderr -A -- -ID 51639 -PS OVMDEMO -S PSMONITORSRV :
        process id=1944 ... Started.
7 processes started.


8.5 On secondary database, check for the connections
SQL> conn / as sysdba
Connected.
SQL> select instance_name,host_name from v$instance;

INSTANCE_NAME    HOST_NAME
---------------- ----------------------------------------------------------------
H91TMPLT         psovmhcmdb2

SQL> select username,machine,program from v$session where username is not null;
USERNAME   MACHINE                        PROGRAM
---------- ------------------------------ ------------------------------------------------
SYSBKP     psovmhcm.phoenix.nga           PSMONITORSRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSPRCSRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSDSTSRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSAESRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSAESRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSAESRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSBKP     psovmhcm.phoenix.nga           PSMSTPRC@psovmhcm.phoenix.nga (TNS V1-V3)
PUBLIC     psovmhcmdb.phoenix.nga         oracle@psovmhcmdb.phoenix.nga (TNS V1-V3)
PUBLIC     psovmhcmdb.phoenix.nga         oracle@psovmhcmdb.phoenix.nga (TNS V1-V3)
PUBLIC     psovmhcmdb.phoenix.nga         oracle@psovmhcmdb.phoenix.nga (TNS V1-V3)
PUBLIC     psovmhcmdb.phoenix.nga         oracle@psovmhcmdb.phoenix.nga (TNS V1-V3)
SYS        psovmhcmdb2                    sqlplus@psovmhcmdb2 (TNS V1-V3)

12 rows selected.
We can see in the above output the process scheduler processes coming in with the secondary ACCESSID connection user.

9. Test, DDDAUDIT setting to read-only
9.1
We’ll first try to run a very well known process onto that secondary database. We first have to make the process as “read-only” on the process definition page.
PS_ACTIVEDG_020 PS_ACTIVEDG_021
9.2 Run the process DDDAUDIT
PS_ACTIVEDG_025
9.3 First result
Well, according to the databases connections, nothing is coming onto the standby database, on the contrary, a sqr connection is made from the client batch server onto the primary database.
The command line used to launch the SQR is not changed compared to a default one, still pointing to the primary database…
SQR Command = /opt/oracle/psft/pt/tools/bin/sqr/ORA/bin/sqr /opt/oracle/psft/pt/tools/sqr/dddaudit.sqr */*@H91TMPLT   "-o/home/psadm2/psft/pt/8.51/appserv/prcs/PRCSDOM/log_output/SQR_DDDAUDIT_851/dddaudit_851.out" -xb -xi  "-i/home/psadm2/psft/pt/8.51/appserv/prcs/PRCSDOM/log_output/SQR_DDDAUDIT_851/,/opt/oracle/psft/pt/tools/sqr/,/,/,/"  -zif/opt/oracle/psft/pt/tools/sqr/pssqr.unx  "-f/home/psadm2/psft/pt/8.51/appserv/prcs/PRCSDOM/log_output/SQR_DDDAUDIT_851/dddaudit_851.PDF" -PRINTER:PD    H91TMPLT 851 PS psovmhcm
A bit disappointing, but there’s an clear explanation, SQR and COBOL processes are not designed yet to run on a standby database, despite the read-only property on the process definition page. It was well explained during one of the sessions of the last Open World as I reported on OOW2010 – Day 4. It is planned for a future version, you can read it as well on the Peopletools 8.51 roadmap.
So, no SQR on standby database, let’s try the simplest program ever, AEMINITEST.

10. Test, AEMINITEST setting to read-only
10.1
Make the AEMINITEST read-only
PS_ACTIVEDG_029
10.2 Run the AEMINITEST
Once it is running, we can see an incoming connection to the primary from the secondary database (as SYSADM – through the database link) to update the process scheduler’s tables :
Here an output from the primary database :
USERNAME    MACHINE                  PROGRAM
----------- ------------------------ ------------------------------------------------
SYSADM      psovmhcm.phoenix.nga     PSPRCSRV@psovmhcm.phoenix.nga (TNS V1-V3)
SYSADM      psovmhcmdb2              oracle@psovmhcmdb2 (TNS V1-V3)               <--
SYSADM      psovmhcm.phoenix.nga     PSMONITORSRV@psovmhcm.phoenix.nga (TNS V1-V3)
I was setting the SQLTrace (=7) on Process scheduler to see how it is behave when running a read-only process. It is rather interesting to see the trace file :
=> it connect onto the primary database to retrieve the object type and few other information.
H91TMPLT RC=0 Dur=0.000993 COM Stmt=SELECT VERSION FROM PSVERSION WHERE OBJECTTYPENAME = 'SYS'
H91TMPLT RC=0 Dur=0.000069 COM Stmt=SELECT OBJECTTYPENAME, VERSION FROM PSVERSION
H91TMPLT RC=0 Dur=0.000128 COM Stmt=SELECT MESSAGE_SET_NBR FROM PSMSGSETDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000004 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000047 COM Stmt=SELECT SETCNTRLVALUE FROM PS_SET_CNTRL_TBL WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000004 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000079 COM Stmt=SELECT PRCSTYPE, PRCSNAME FROM PS_PRCSDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000005 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000060 COM Stmt=SELECT SERVERNAME FROM PS_SERVERDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000005 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000048 COM Stmt=SELECT PRCSTYPE, OPSYS, DBTYPE FROM PS_PRCSTYPEDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000004 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000064 COM Stmt=SELECT PRCSJOBNAME FROM PS_PRCSJOBDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000036 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000085 COM Stmt=SELECT URL_ID FROM PSURLDEFN WHERE VERSION > :1 UNION SELECT URL_ID FROM PSURLDEL
H91TMPLT RC=0 Dur=0.000004 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000005 Bind-2 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000000 COM Stmt=SELECT SCOPRID, SCLANG  FROM PSSCDICTDEFN WHERE VERSION > :1
H91TMPLT RC=0 Dur=0.000000 Bind-1 type=8 length=4 value=3
H91TMPLT RC=0 Dur=0.000005 Commit
H91TMPLT RC=0 Dur=0.000049 Disconnect

=> it connect onto the secondary database to run the AE, even some update appear to run onto the standby (which work fine since, as explained earlier, it is updating remote synonyms tables) :
H91STBY RC=0 Dur=0.018112 COM Stmt=SELECT 'PS_DOES_TABLE_EXIST' FROM PSOPTIONS
H91STBY RC=0 Dur=0.001922 COM Stmt=SELECT  DBLENGTHTYPE,LANGUAGE_CD,MULTICURRENCY,MULTIJOB,MULTICOMPANY,HELP_CONTEXT_LAST,GRANT_A
H91STBY RC=0 Dur=0.000084 Disconnect
notSamTran RC=0 Dur=0.000021 Open Cursor Handle=0000000009954E60
H91STBY RC=0 Dur=0.000041 Disconnect
notSamTran RC=0 Dur=0.000012 Open Cursor Handle=0000000009954E60
H91STBY RC=0 Dur=0.000057 COM Stmt=SELECT JOBINSTANCE, MAINJOBINSTANCE FROM PSPRCSRQST WHERE PRCSINSTANCE = :1
H91STBY RC=0 Dur=0.000005 Bind-1 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000052 Disconnect
notSamTran RC=0 Dur=0.000073 Open Cursor Handle=0000000009954E60
H91STBY RC=0 Dur=0.000119 COM Stmt=UPDATE PSPRCSRQST SET RUNSTATUS = :1,PRCSRTNCD = :2,CONTINUEJOB = :3,BEGINDTTM = CAST(SYSTIMES
H91STBY RC=0 Dur=0.000004 Bind-1 type=2 length=1 value=7
H91STBY RC=0 Dur=0.000004 Bind-2 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000004 Bind-3 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000004 Bind-4 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000030 COM Stmt=UPDATE PSPRCSQUE SET RUNSTATUS = :1,PRCSRTNCD = :2,CONTINUEJOB = :3 WHERE PRCSINSTANCE = :4
H91STBY RC=0 Dur=0.000004 Bind-1 type=2 length=1 value=7
H91STBY RC=0 Dur=0.000005 Bind-2 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000005 Bind-3 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000005 Bind-4 type=8 length=4 value=852
H91STBY RC=0 Dur=0.004184 Commit
H91STBY RC=0 Dur=0.000019 Disconnect
H91STBY RC=0 Dur=0.000518 COM Stmt=SELECT PROCESS_INSTANCE, AE_RTNCD, AE_CRITICAL_PHASE, AE_STEP_DONE, MARKET, TO_CHAR(ASOF_DT,'Y
H91STBY RC=0 Dur=0.000005 Bind-1 type=2 length=2 value=PS
H91STBY RC=0 Dur=0.000005 Bind-2 type=2 length=8 value=psovmhcm
H91STBY RC=0 Dur=0.000004 Bind-3 type=2 length=10 value=AEMINITEST
H91STBY RC=0 Dur=0.000090 COM Stmt=SELECT OPRID, RUN_CNTL_ID, AE_APPLID FROM PS_AERUNCONTROL WHERE PROCESS_INSTANCE = :1
H91STBY RC=0 Dur=0.000005 Bind-1 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000077 COM Stmt=SELECT OPRID, RUN_CNTL_ID, AE_APPLID, MIN(AE_DISABLE_RESTART) FROM PS_AETEMPTBLMGR WHERE PROCE
H91STBY RC=0 Dur=0.000005 Bind-1 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000095 COM Stmt=INSERT INTO PS_AERUNCONTROL (PROCESS_INSTANCE, OPRID, RUN_CNTL_ID, AE_APPLID, RUN_DTTM, AE_RTN
H91STBY RC=0 Dur=0.000004 Bind-1 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000004 Bind-2 type=2 length=2 value=PS
H91STBY RC=0 Dur=0.000004 Bind-3 type=2 length=8 value=psovmhcm
H91STBY RC=0 Dur=0.000004 Bind-4 type=2 length=10 value=AEMINITEST
notSamTran RC=0 Dur=0.000052 Open Cursor Handle=0000000009954F98

=> in the end, the process scheduler reconnect one more time onto the primary database :
H91STBY RC=0 Dur=0.000217 COM Stmt=UPDATE PS_AETEMPTBLMGR SET AE_DEDICATED = 1 WHERE 0 = 1
H91STBY RC=0 Dur=0.002722 Commit
notSamTran RC=0 Dur=0.000026 Open Cursor Handle=0000000009954E60
H91STBY RC=0 Dur=0.000065 COM Stmt=UPDATE PSPRCSRQST SET RUNSTATUS = :1,PRCSRTNCD = :2,CONTINUEJOB = :3,ENDDTTM = CAST(SYSTIME
H91STBY RC=0 Dur=0.000004 Bind-1 type=2 length=1 value=9
H91STBY RC=0 Dur=0.000005 Bind-2 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000005 Bind-3 type=6 length=4 value=1
H91STBY RC=0 Dur=0.000005 Bind-4 type=8 length=4 value=852
H91STBY RC=0 Dur=0.000042 COM Stmt=UPDATE PSPRCSQUE SET RUNSTATUS = :1,PRCSRTNCD = :2,CONTINUEJOB = :3 WHERE PRCSINSTANCE = :4
H91STBY RC=0 Dur=0.000013 Bind-1 type=2 length=1 value=9
H91STBY RC=0 Dur=0.000004 Bind-2 type=6 length=4 value=0
H91STBY RC=0 Dur=0.000004 Bind-3 type=6 length=4 value=1
H91STBY RC=0 Dur=0.000004 Bind-4 type=8 length=4 value=852
H91STBY RC=0 Dur=0.003032 Commit
H91STBY RC=0 Dur=0.000023 Disconnect
H91TMPLT RC=0 Dur=0.000143 Disconnect
H91TMPLT RC=0 Dur=0.000109 Disconnect
H91TMPLT RC=0 Dur=0.000114 Disconnect
H91TMPLT RC=0 Dur=0.000100 Disconnect
H91TMPLT RC=0 Dur=0.000140 Disconnect
H91TMPLT RC=0 Dur=0.000140 Disconnect
H91TMPLT RC=0 Dur=0.000004 Commit
H91TMPLT RC=0 Dur=0.000124 Disconnect
H91STBY RC=0 Dur=0.000005 Commit
H91STBY RC=0 Dur=0.000102 Disconnect
notSamTran RC=0 Dur=0.000018 Open Cursor Handle=0000000009954F98
H91TMPLT RC=0 Dur=0.000096 COM Stmt=SELECT UPPER(MACHINENAME),PROCESSID,TO_CHAR(CAST((PTATREGDTTM) AS TIMESTAMP),'YYYY-MM-DD-H
H91TMPLT RC=0 Dur=0.000005 Commit
H91TMPLT RC=0 Dur=0.000054 Disconnect

By curiosity, I revert the read-only property to the default unchecked (read-write) for the AEMINITEST, and re-run it. No surprise, the tracefile shows only primary database connections.

Right now, except failover, we got a very nice and complete PSOVM environment ! Enjoy it !

Read my next article :

PSOVM and Active Data Guard : enabling ADG

To go further, it is assumed you have set a Physical Standby Database on Peoplesoft OVM as shown in my previous blog entry : PSOVM and Activate Data Guard : Physical Standby Database
As explained in previous post, we created a Physical Standby Database, it should be relatively straightforward to enable the Active Data Guard. That will allow end users to query (in read only mode) the standby database whilst the redo are applying onto that standby database, feature known to be real-time query.
At the end of this stage, the architecture will be as follow (in blue the change compared to the previous post) :
PS_ACTIVEDG_035
1. dg_broker_start init parameter
Set the dg_broker_start to true on primary database and on secondary database
SQL> alter system set dg_broker_start=true;
System altered.
2. Create the dataguard configuration
On the primary site, start the Data Guard Manager client command line for the configuration settings :
[oracle@psovmhcmdb dbs]$ dgmgrl
DGMGRL for Linux: Version 11.1.0.7.0 - 64bit Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/oracle@H91TMPLT (connection to the primary database)
Connected.
DGMGRL> create configuration psovmdgmgrl
> as primary database is H91TMPLT
> connect identifier is H91TMPLT;
Configuration "psovmdgmgrl" created with primary database "h91tmplt"
DGMGRL> add database H91STBY
> as connect identifier is H91STBY
> maintained as physical;
Database "h91stby" added
DGMGRL>
DGMGRL> enable configuration
Enabled.
DGMGRL> show configuration

Configuration
  Name:                psovmdgmgrl
  Enabled:             YES
  Protection Mode:     MaxPerformance
  Databases:
    h91tmplt - Primary database
    h91stby  - Physical standby database

Fast-Start Failover: DISABLED
Current status for "psovmdgmgrl":
SUCCESS

DGMGRL>
3. Turn off the redo apply (dgmgrl)
In order to have an active data guard, we should stop the replication first (run dgmgrl on primary site) :
DGMGRL> edit database h91stby set state=apply-off;
Succeeded.
DGMGRL>
4. Read-only mode
Now, we can open the secondary database in read-only mode (on the further startups, that mode will be the default one):
SQL> alter database open read only;
Database altered.
5. Turn on the redo apply (dgmgrl)
Now, we can turn the redo apply on again (run dgmgrl on primary site)
DGMGRL> edit database h91stby set state=apply-on;
Succeeded.
DGMGRL>
6. Test
We will see if the replication is working whilst the standby database is open in read-only mode and “query-able” as SYSADM user.
6.1 On the secondary database
SQL> select instance_name,host_name from v$instance
SQL> /
INSTANCE_NAME    HOST_NAME
---------------- ----------------------------------------------------------------
H91TMPLT         psovmhcmdb2

SQL> conn sysadm/SYSADM
Connected.
SQL> select count(*) from psoprdefn;

  COUNT(*)
----------
       155
SQL> select oprid from psoprdefn where trunc(LASTPSWDCHANGE)=trunc(sysdate);

no rows selected
6.2 Within the front-end application, we’ll duplicate PS user as following
PS_ACTIVEDG_001
PS_ACTIVEDG_002 PS_ACTIVEDG_003

6.3 And now, on the secondary database, re-run the previous query, it should return one more row
SQL> select oprid from psoprdefn where trunc(LASTPSWDCHANGE)=trunc(sysdate);
OPRID
---------------------------------------------------
NICOLAS

SQL> select count(*) from psoprdefn;
  COUNT(*)
----------
       156

SQL>
=> the new created user is coming to the secondary database, and we can check it directly from PSOPRDEFN (that would not be possible without Active Data Guard).
7. For the record, show the dgmgrl status (configuration and databases)
[oracle@psovmhcmdb ~]$ dgmgrl
DGMGRL for Linux: Version 11.1.0.7.0 - 64bit Production
Copyright (c) 2000, 2005, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/oracle@H91TMPLT
Connected.
DGMGRL> show configuration

Configuration
  Name:                psovmdgmgrl
  Enabled:             YES
  Protection Mode:     MaxPerformance
  Databases:
    h91tmplt - Primary database
    h91stby  - Physical standby database

Fast-Start Failover: DISABLED
Current status for "psovmdgmgrl":
SUCCESS

DGMGRL> show database h91stby
Database
  Name:            h91stby
  Role:            PHYSICAL STANDBY
  Enabled:         YES
  Intended State:  APPLY-ON
  Instance(s):
    H91TMPLT

Current status for "h91stby":
SUCCESS

DGMGRL> show database h91tmplt
Database
  Name:            h91tmplt
  Role:            PRIMARY
  Enabled:         YES
  Intended State:  TRANSPORT-ON
  Instance(s):
    H91TMPLT

Current status for "h91tmplt":
SUCCESS

DGMGRL>
Now we can talk, query a database without loading the primary database while the redo continue to be applied. That's really helpful, especially for users who run a lot of reports through a tier query tool. Once more, PSOVM allow us to test it easily before implementing it in real live.
But it can be not enough. An other great feature coming with the latest PSOVM and Peopletools 8.51 is the StandbyDBName configuration possiblity on Batch and AppServer. That's obviously the next step. To be continued.

Read my next articles :

Saturday, March 12, 2011

PSOVM and Active Data Guard : Physical Standby Database

Peoplesoft Oracle VM (PSOVM) templates are really nice.  That allows customers to test Peoplesoft Application in an easy way. Easy and quick deployment.
Moreover, assuming you have space enough to host the VMs and a spare server to host the Oracle VM Server, the cost is nearly zero. 
=> That's the perfect tool for the business team, the latest PSOVM HCM9.1 is coming within the Features Pack 1 (Dec-2010). All the new functionalities are included and there's also the optional installation of the 30-days trial Cobol license. 
=> That's the perfect tool for the development team, the latest PSOVM HCM9.1 is coming within the latest Peopletools version (8.51.02 - the patch 07 is already released, but except bugs fixes, it should not make big differences), and it’s also coming within the binaries for AppDesigner and DataMover. Developpers can now test everything they want without much configuration and installation efforts. 
=> That's (almost) the perfect tool for the Peoplesoft DBA, in addition of the latest Peopletools version, the latest PSOVM HCM9.1 is coming within one of the latest Oracle Database version. Unfortunately, the Oracle database is not the last version, it is 11gR1 (11.1.0.7). Some new features of 11gR2 would be great to have, to name only two : segment creation deferrable (do not occupied space when table is empty) and few on Active Data Guard (ASH, STANDBY_MAX_DATA_DELAY).
But 11gR1 is good enough to test the Active Data Guard that Peoplesoft customers are looking for a while. A really nice feature to get a reporting database whilst the redolog continue to be applied (it’s known as real-time query). And that will be the subject here.
So, the exercise will be to have a database that end users will be able to query without over-loading the primary database, but having the latest data. First of all, create a Physical Standby Database.
Since primary and secondary databases will come from the same templates, we are sure about the same OS (and version) (OEL5.2) and the same database version (11.1.0.7).
1. The configuration
1.1
Install a working PSOVM environment as shown here
=> 1 database server, the primary site (psovmhcmdb), hosting the primary database (db name H91TMPLT, unique name H91TMPLT).
=> 1 App/Batch/PIA server
1.2 Install an additional database server, the secondary site (psovmhcmdb2) hosting the secondary database (db name H91TMPLT, unique name H91STBY). This one is deployed as a normal db server (see step 1.1), to make it simpler, the database name is named as the primary one, H91TMPLT. The PSOVM deployment scripts will create all the necessary folders and files.
1.3 Here below a schema of the target configuration we are currently looking for at this stage. 

  PS_ACTIVEDG_034Note 1 : the DB_NAME and DB_UNIQUE_NAME  of the database on primary site is H91TMPLT
Note 2 : the DB_NAME of the database on secondary site is H91TMPLT, DB_UNIQUE_NAME is H91STBY
2. First settings on primary db (primary site – psovmhcmdb)
It is assumed the primary database server has been successfully deployed and database name is H91TMPLT.
2.1 Archive log mode
The delivered database in PSOVM Database Server is not in archivelog mode, it has to be changed :
[oracle@psovmhcmdb .ssh]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb .ssh]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 04:49:35 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup mount
ORACLE instance started.
Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size              96472648 bytes
Database Buffers          184549376 bytes
Redo Buffers                4923392 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.
SQL> alter database open;
2.2 spfile
The delivered database in PSOVM Database Server does not use a spfile but a pfile, let’s make it
SQL> show parameter spfile
NAME         TYPE         VALUE
------------ ------------ ------------------------------
spfile       string      
SQL> create spfile from pfile;

File created.
SQL> startup force
ORACLE instance started.

Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size              96472648 bytes
Database Buffers          184549376 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.
SQL> show parameter spfile

NAME         TYPE         VALUE
------------ ------------ ------------------------------
spfile       string       /u01/app/oracle/product/11.1.0/db_1/dbs/spfileH91TMPLT.ora

2.3 Create the directory of archive log files
We will create a dedicated folder for the archive log files onto the primary database server, first, have a look where it should go.
[oracle@psovmhcmdb oradata]$ df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda2             3177204   1037312   2139892  33% /
/dev/xvda1               31077      8277     21196  29% /boot
tmpfs                  2097152         0   2097152   0% /dev/shm
/dev/xvdb1             9851308   4480692   4870196  48% /u01
/dev/xvdc1            41286796  20908248  18281264  54% /opt/oracle/psft/ptdb <—18Gb free seems good for us
[root@psovmhcmdb ~]# cd /opt/oracle/psft/ptdb
[root@psovmhcmdb ptdb]# mkdir -p arch/H91TMPLT
[root@psovmhcmdb ptdb]# chown -R oracle:dba arch
[root@psovmhcmdb ptdb]# su - oracle
[oracle@psovmhcmdb ~]$ cd /u01/app/oracle/oradata
[oracle@psovmhcmdb oradata]$ ln -s /opt/oracle/psft/ptdb/arch/H91TMPLT H91TMPLT_arch
[oracle@psovmhcmdb oradata]$ ls -lrt
total 0
lrwxrwxrwx 1 oracle oinstall 38 Jan 23 06:18 H91TMPLT -> /opt/oracle/psft/ptdb/oradata/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 05:00 H91TMPLT_arch -> /opt/oracle/psft/ptdb/arch/H91TMPLT
......

Let’s make the database using it
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size              96472648 bytes
Database Buffers          184549376 bytes
Redo Buffers                4923392 bytes
Database mounted.
SQL> alter system set log_archive_dest='/u01/app/oracle/oradata/H91TMPLT_arch';

System altered.
SQL> alter database open;
Database altered.
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /u01/app/oracle/oradata/H91TMPLT_arch
Oldest online log sequence     8
Next log sequence to archive   10
Current log sequence           10
SQL>

2.4 Force logging
The primary database must be turned on to force logging :
SQL> alter database force logging;
Database altered.
SQL>

3. Remove db file on secondary site - psovmhcmdb2
It is assumed the secondary database server has been successfully deployed and database name is H91TMPLT. When the secondary database server has been deployed, it created the database, we don’t need that one, all the datafiles must be removed.
3.1 Check for the datafiles
[oracle@psovmhcmdb2 ~]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb2 ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 05:10:50 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select distinct substr(name,1,instr(name,'/',-1)) from v$datafile
  2  union
  3  select distinct substr(name,1,instr(name,'/',-1)) from v$tempfile
  4  union
  5  select distinct substr(member,1,instr(member,'/',-1)) from v$logfile;

SUBSTR(NAME,1,INSTR(NAME,'/',-1))
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/H91TMPLT/

3.2 Shutdown the secondary database
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@psovmhcmdb2 ~]$
3.3 Remove the db file
[oracle@psovmhcmdb2 ~]$ rm -f /u01/app/oracle/oradata/H91TMPLT/*
4. Copy the db file of primary to secondary site - psovmhcmdb2
4.1
Check the dbf of primary database and create the script to copy all the datafiles
SQL> set lines 200 pages 0 head off feed off trims on
SQL> spool scp.sh
SQL> select 'gzip -c '||name||' | ssh psovmhcmdb2 "gunzip -c > '||name||'"' from v$datafile;
...
gzip -c /u01/app/oracle/oradata/H91TMPLT/waapp.dbf | ssh psovmhcmdb2 "gunzip -c > /u01/app/oracle/oradata/H91TMPLT/waapp.dbf"
SQL> spool off
 4.2 Shutdown the primary db
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> quit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[oracle@psovmhcmdb ~]$
4.3 Copy the dbf from primary to secondary site (execute the script create earlier at step 4.1). Good enough, the name of the two databases on both sites are the same, consequently the path directories are the same, not change required. (note: the script will work if a ssh-keygen has been configured between the servers)
[oracle@psovmhcmdb ~]$ chmod +x scp.sh
[oracle@psovmhcmdb ~]$ ./scp.sh
5. Open the primary db on the primary site - psovmhcmdb
5.1
create pfile from spfile
[oracle@psovmhcmdb ~]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 06:43:33 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> create pfile from spfile;
File created.
SQL> quit
Disconnected
[oracle@psovmhcmdb ~]$

5.2 Set the init parameters required for the physical standby settings (in bold the changes made)
[oracle@psovmhcmdb ~]$ cd $ORACLE_HOME/dbs
[oracle@psovmhcmdb dbs]$ vi initH91TMPLT.ora
H91TMPLT.__db_cache_size=184549376
H91TMPLT.__java_pool_size=4194304
H91TMPLT.__large_pool_size=4194304
H91TMPLT.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
H91TMPLT.__pga_aggregate_target=96468992
H91TMPLT.__sga_target=289406976
H91TMPLT.__shared_io_pool_size=0
H91TMPLT.__shared_pool_size=88080384
H91TMPLT.__streams_pool_size=0
*._unnest_subquery=false
*.compatible='11.1.0.7.0'
*.control_files='/u01/app/oracle/oradata/H91TMPLT/control01.ctl'
*.db_block_size=8192
*.db_domain='us.oracle.com'
*.db_file_multiblock_read_count=16
*.DB_FILES=256
*.DB_NAME='H91TMPLT'
#*.dispatchers='(PROTOCOL=TCP) (SERVICE=XDB)'
*.job_queue_processes=10
#*.log_archive_dest='/u01/app/oracle/oradata/H91TMPLT_arch'
*.NLS_LENGTH_SEMANTICS='CHAR'
*.OPEN_CURSORS=1000
*.pga_aggregate_target=95420416
*.processes=150
*.remote_login_passwordfile='NONE'
*.sga_target=287309824
*.undo_management='AUTO'
*.undo_tablespace='PSUNDOTS'
#####
log_archive_dest_1='LOCATION=/u01/app/oracle/oradata/H91TMPLT_arch VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=H91TMPLT'
log_archive_dest_2='SERVICE=H91STBY VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=H91STBY'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
DB_UNIQUE_NAME=H91TMPLT
LOG_ARCHIVE_CONFIG='DG_CONFIG=(H91TMPLT,H91STBY)'
standby_file_management=auto
5.3 recreate spfile from pfile
[oracle@psovmhcmdb dbs]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 06:51:06 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> create spfile from pfile;
File created.
5.4 Open the primary database
SQL> startup mount
ORACLE instance started.
Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size              96472648 bytes
Database Buffers          184549376 bytes
Redo Buffers                4923392 bytes
Database mounted.
SQL>

5.5 Create standby controlfile and pfile for secondary db
SQL> alter database create standby controlfile as '/tmp/H91STBY.ctl';
Database altered.
SQL> alter database open;
Database altered.
SQL> create pfile='/tmp/initH91STBY.ora' from spfile;
File created.

5.6 Set more parameters on primary database
SQL> alter system set standby_file_management=auto;
System altered.
SQL> alter system set FAL_SERVER=H91STBY;
System altered.
SQL> alter system set FAL_CLIENT=H91TMPLT;
System altered.
SQL>
6. tnsnames.ora
Set the primary and secondary databases in tnsnames.ora on both servers, primary and secondary sites, as following (assuming the servers name have been properly added into /etc/hosts) :
[oracle@psovmhcmdb dbs]$ more $ORACLE_HOME/network/admin/tnsnames.ora
# tnsnames.ora Network Configuration File:
H91TMPLT =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = psovmhcmdb.phoenix.nga)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = H91TMPLT.us.oracle.com )
      (SID = H91TMPLT )
    )
  )

H91STBY =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = psovmhcmdb2.phoenix.nga)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = H91TMPLT.us.oracle.com )
      (SID = H91TMPLT )
    )
  )

7. Mount the secondary db on secondary site - psovmhcmdb2
7.1
Copy the standby controlfile and pfile created at step 5.5 onto the secondary site
[oracle@psovmhcmdb2 H91TMPLT]$ pwd
/u01/app/oracle/oradata/H91TMPLT
[oracle@psovmhcmdb2 H91TMPLT]$ scp root@psovmhcmdb:/tmp/H91STBY.ctl control01.ctl
[oracle@psovmhcmdb2 H91TMPLT]$ cd $ORACLE_HOME/dbs
[oracle@psovmhcmdb2 dbs]$ scp root@psovmhcmdb:/tmp/initH91STBY.ora initH91TMPLT.ora
[oracle@psovmhcmdb2 dbs]$
7.2 Set the init parameters of the secondary database
[oracle@psovmhcmdb2 dbs]$ vi initH91TMPLT.ora
H91TMPLT.__db_cache_size=184549376
H91TMPLT.__java_pool_size=4194304
H91TMPLT.__large_pool_size=4194304
H91TMPLT.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
H91TMPLT.__pga_aggregate_target=96468992
H91TMPLT.__sga_target=289406976
H91TMPLT.__shared_io_pool_size=0
H91TMPLT.__shared_pool_size=88080384
H91TMPLT.__streams_pool_size=0
*._unnest_subquery=false
*.compatible='11.1.0.7.0'
*.control_files='/u01/app/oracle/oradata/H91TMPLT/control01.ctl'
*.db_block_size=8192
*.db_domain='us.oracle.com'
*.db_file_multiblock_read_count=16
*.DB_FILES=256
*.DB_NAME='H91TMPLT'
*.DB_UNIQUE_NAME='H91STBY'
*.job_queue_processes=10
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(H91TMPLT,H91STBY)'
*.log_archive_dest_1='LOCATION=/u01/app/oracle/oradata/H91TMPLT_arch VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=H91STBY'
*.log_archive_dest_2='SERVICE=H91STBY VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=H91TMPLT'
*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'
*.LOG_ARCHIVE_DEST_STATE_2='ENABLE'
*.NLS_LENGTH_SEMANTICS='CHAR'
*.OPEN_CURSORS=1000
*.pga_aggregate_target=95420416
*.processes=150
*.remote_login_passwordfile='NONE'
*.sga_target=287309824
*.undo_management='AUTO'
*.undo_tablespace='PSUNDOTS'
standby_file_management=auto
fal_server=H91TMPLT
fal_client=H91STBY
7.3 Create the directory of archive log files on the secondary site
[root@psovmhcmdb2 ~]# mkdir -p /opt/oracle/psft/ptdb/arch/H91TMPLT
[root@psovmhcmdb2 ~]# chown -R oracle:dba /opt/oracle/psft/ptdb/arch
[root@psovmhcmdb2 ~]# su - oracle
[oracle@psovmhcmdb2 ~]$ cd /u01/app/oracle/oradata
[oracle@psovmhcmdb2 oradata]$ ln -s /opt/oracle/psft/ptdb/arch/H91TMPLT H91TMPLT_arch
[oracle@psovmhcmdb2 oradata]$ ls -lrt
total 0
lrwxrwxrwx 1 oracle oinstall 38 Mar 10 05:02 H91TMPLT -> /opt/oracle/psft/ptdb/oradata/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 07:11 H91TMPLT_arch -> /opt/oracle/psft/ptdb/arch/H91TMPLT
[oracle@psovmhcmdb2 oradata]$
7.4 Mount the secondary database
[oracle@psovmhcmdb2 dbs]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb2 dbs]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 07:12:22 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> create spfile from pfile;
File created.
SQL> startup mount
ORACLE instance started.

Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size              96472648 bytes
Database Buffers          184549376 bytes
Redo Buffers                4923392 bytes
Database mounted.
SQL>

8. Standby redologs
8.1
Create the standby redolog directoy on primary site - psovmhcmdb
[root@psovmhcmdb ptdb]# cd /opt/oracle/psft/ptdb/arch/H91TMPLT/../..
[root@psovmhcmdb ptdb]# mkdir -p redo/H91TMPLT
[root@psovmhcmdb ptdb]# chown -R oracle:dba redo
[root@psovmhcmdb ptdb]# su - oracle
[oracle@psovmhcmdb ~]$ cd /u01/app/oracle/oradata/
[oracle@psovmhcmdb oradata]$ ln -s /opt/oracle/psft/ptdb/redo/H91TMPLT H91TMPLT_redo
[oracle@psovmhcmdb oradata]$ ls -lrt
total 0
lrwxrwxrwx 1 oracle oinstall 38 Jan 23 06:18 H91TMPLT -> /opt/oracle/psft/ptdb/oradata/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 05:00 H91TMPLT_arch -> /opt/oracle/psft/ptdb/arch/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 07:15 H91TMPLT_redo -> /opt/oracle/psft/ptdb/redo/H91TMPLT
8.2 Create the standby redolog directoy on secondary site - psovmhcmdb2
[root@psovmhcmdb2 ~]# cd /opt/oracle/psft/ptdb/arch/H91TMPLT/../..
[root@psovmhcmdb2 ptdb]# mkdir -p redo/H91TMPLT
[root@psovmhcmdb2 ptdb]# chown -R oracle:dba redo
[root@psovmhcmdb2 ptdb]# su - oracle
[oracle@psovmhcmdb2 ~]$ cd /u01/app/oracle/oradata/
[oracle@psovmhcmdb2 oradata]$ ln -s /opt/oracle/psft/ptdb/redo/H91TMPLT H91TMPLT_redo
[oracle@psovmhcmdb2 oradata]$ ls -lrt
total 0
lrwxrwxrwx 1 oracle oinstall 38 Mar 10 05:02 H91TMPLT -> /opt/oracle/psft/ptdb/oradata/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 07:11 H91TMPLT_arch -> /opt/oracle/psft/ptdb/arch/H91TMPLT
lrwxrwxrwx 1 oracle oinstall 35 Mar 10 07:25 H91TMPLT_redo -> /opt/oracle/psft/ptdb/redo/H91TMPLT
8.3 Create the standby redolog on primary db
[oracle@psovmhcmdb oradata]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb oradata]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 07:17:07 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> SELECT GROUP#, BYTES FROM V$LOG;
    GROUP#      BYTES
---------- ----------
         1  104857600
         2  104857600
         3  104857600

SQL> SELECT GROUP#, BYTES FROM V$STANDBY_LOG;
no rows selected
SQL> ALTER DATABASE ADD STANDBY LOGFILE
  2  ('/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo01.log') size 100m,
  3  ('/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo02.log') size 100m,
  3  ('/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo03.log') size 100m
SQL> /

Database altered.
SQL> SELECT GROUP#, BYTES FROM V$STANDBY_LOG;
    GROUP#      BYTES
---------- ----------
         4  104857600
         5  104857600
         6  104857600

SQL>
8.4 Create the standby redolog on secondary db
[oracle@psovmhcmdb2 oradata]$ export ORACLE_SID=H91TMPLT
[oracle@psovmhcmdb2 oradata]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 07:25:47 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> ALTER DATABASE ADD STANDBY LOGFILE
  2  '/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo01.log' size 100m,
  3  '/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo02.log' size 100m,
  4* '/u01/app/oracle/oradata/H91TMPLT_redo/stb_redo03.log' size 100m
SQL> /

Database altered.
9. Activate the standby mechanism on secondary db
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;
Database altered.
SQL>
10. First test
10.1
Force a switch of logfile on primary db
SQL> ALTER SYSTEM SWITCH LOGFILE;
10.2 Check if archivelog files have been applied on secondary db
SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
no rows selected
10.3 Error issued
The archivelog files have not been applied onto the secondary database, looking into the alert.log file of the primary database, few errors are issued :
2011-03-10 07:44:34.979000 -08:00
Errors in file /u01/app/oracle/diag/rdbms/h91tmplt/H91TMPLT/trace/H91TMPLT_arc2_30195.trc:
ORA-01017: invalid username/password; logon denied
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
      returning error ORA-16191
------------------------------------------------------------
Errors in file /u01/app/oracle/diag/rdbms/h91tmplt/H91TMPLT/trace/H91TMPLT_arc2_30195.trc:
ORA-16191: Primary log shipping client not logged on standby
PING[ARC2]: Heartbeat failed to connect to standby 'H91STBY'. Error is 16191.
=> a password file is required, and by default, the delivered database in PSOVM does not have one.
11. Password files
11.1
Change remote_login_passwordfile to exclusive on secondary db
SQL> alter system set remote_login_passwordfile=exclusive scope=spfile;
System altered.
11.2 Create password file for secondary db, take care, the password file must be named within the DB_UNIQUE_NAME if the database, not within the DB_NAME. So, here on the secondary database DB_UNIQUE_NAME=H91STBY
[oracle@psovmhcmdb2 oradata]$ cd $ORACLE_HOME/dbs
[oracle@psovmhcmdb2 dbs]$ orapwd file=orapwH91STBY password=oracle entries=5
11.3 Bounce the secondary db and activate the standby mechanism
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.

Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size             109055560 bytes
Database Buffers          171966464 bytes
Redo Buffers                4923392 bytes
Database mounted.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

Database altered.

11.4 Change remote_login_passwordfile to exclusive on primary db
SQL> alter system set remote_login_passwordfile=exclusive scope=spfile;
System altered.
SQL> quit
11.5 Create password file for primary db
[oracle@psovmhcmdb oracle]$ cd $ORACLE_HOME/dbs
[oracle@psovmhcmdb dbs]$ ls
ab_+ASM.dat  hc_H91TMPLT.dat  initdw.ora        init.ora      lk+ASM      lkORCL     orapworcl       spfileH91TMPLT.ora
hc_+ASM.dat  hc_orcl.dat      initH91TMPLT.ora  initorcl.ora  lkH91TMPLT  orapw+ASM  spfile+ASM.ora
[oracle@psovmhcmdb dbs]$ orapwd file=orapwH91TMPLT password=oracle entries=5

11.6 Bounce the primary db
[oracle@psovmhcmdb dbs]$ sqlplus / as sysdba
SQL*Plus: Release 11.1.0.7.0 - Production on Thu Mar 10 07:52:36 2011
Copyright (c) 1982, 2008, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  288104448 bytes
Fixed Size                  2159032 bytes
Variable Size             100666952 bytes
Database Buffers          180355072 bytes
Redo Buffers                4923392 bytes
Database mounted.
Database opened.

12. Second test
12.1
Force a switch of logfile on primary db
SQL> ALTER SYSTEM SWITCH LOGFILE;
System altered.
12.2 Check if archivelog files have been applied on secondary db
SQL> SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;
SEQUENCE# FIRST_TIME         NEXT_TIME
---------- ------------------ ------------------
        11 10-MAR-11          10-MAR-11
        12 10-MAR-11          10-MAR-11
        13 10-MAR-11          10-MAR-11
        14 10-MAR-11          10-MAR-11
        15 10-MAR-11          10-MAR-11
        16 10-MAR-11          10-MAR-11
        17 10-MAR-11          10-MAR-11
        18 10-MAR-11          10-MAR-11
        19 10-MAR-11          10-MAR-11
        20 10-MAR-11          10-MAR-11
        21 10-MAR-11          10-MAR-11
        22 10-MAR-11          10-MAR-11
        23 10-MAR-11          10-MAR-11
        24 10-MAR-11          10-MAR-11
        25 10-MAR-11          10-MAR-11
        26 10-MAR-11          10-MAR-11
        27 10-MAR-11          10-MAR-11
        28 10-MAR-11          10-MAR-11
        29 10-MAR-11          10-MAR-11
        30 10-MAR-11          10-MAR-11
        31 10-MAR-11          10-MAR-11
        32 10-MAR-11          10-MAR-11

22 rows selected.
SQL>
=> success, all the created archivelog files have been shipped to the secondary database.
We have now a fully working physical standby, that's good but the target is to be able to query against that database, that's not possible in that current state, so next step is to enable the Active Data Guard. To be continued.
Read my next articles :

Nicolas.