Snapshot standby database

Oracle 11g has a feature in this area called : Snapshot standby database. A Snapshot Standby Database is a fully update-able standby database that is created by converting a physical standby database into a snapshot standby database for read and write operation for a short period of time thus making it possible to process transactions independently of the primary database

These steps applies to Oracle Server - Enterprise Edition - Version 11.1.0.6 to 11.2.0.2 [Release 11.1 to 11.2]

As per the MOS ID 443720.1  Snapshot database has following characteristics

1. Snapshot standby database continues to receive and archive redo data from primary but does not archive it

2. Redo data received from the primary database is applied automatically once it is converted back into a physical standby database.

3. Data from the primary database is always protected as the archives are being received and stored in place.

4. All local updates will be discarded when snapshot database is converted back to physical standby database.

5. If the primary database moves to new database branch (for example, because of a Flashback Database or an OPEN RESETLOGS), the snapshot standby database will continue accepting redo from new database branch.

6. Snapshot standby database cannot be the target of a switchover or failover. A snapshot standby database must first be converted back into a physical standby database before performing a role transition to it.

7. After a switchover or failover between the primary database and one of the physical or logical standby databases in a configuration, the snapshot standby database can receive redo data from the new primary database after the role transition.

8. Snapshot standby database cannot be the only standby database in a Maximum Protection Data Guard configuration.

Steps to convert the Physical Standby to Snapshot Standby Database

1) If not already configured , configure flash recovery area as given below

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=5G;

SQL>  ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='/u01/ora/flashback'


2) Bring the physical standby database to mount stage. 

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

3) Stop managed recovery if it is active. 

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; 

4) Convert physical standby database to snapshot standby database.

ALTER DATABASE CONVERT TO SNAPSHOT STANDBY; 

The database is dismounted during conversion and must be restarted.

SQL> STARTUP

Once the database is restarted  any transaction can be executed .

SQL> select open_mode,database_role from v$database;

OPEN_MODE  DATABASE_ROLE
---------- ----------------
READ WRITE SNAPSHOT STANDBY


Steps to convert Snapshot Standby Database back to Physical Standby



SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;



SQL>ALTER DATABASE CONVERT TO PHYSICAL STANDBY;




SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;




SQL> select open_mode,database_role from v$database;

OPEN_MODE  DATABASE_ROLE
---------- ----------------
MOUNTED    PHYSICAL STANDBY 







Start the media recovery process.




SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; 




Or for Active dataguard

SQL> alter database recover managed standby database disconnect from session using current logfile;

Once a snapshot standby database has been converted back into a physical standby database and restarted, Redo Apply can be started and all redo received by the snapshot standby database will be applied to the physical standby database.

Open Physical Standby For Read Write Testing and Flashback

Here I am discussing the steps to open the Standby database in read write mode testing purposes and then move it back to standby database using the flashback technology. 

In Standby database 

A ) Set up a flash recovery area.

B )  Make sure to give enough space for to FRA

SQL> ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=5G;

SQL>  ALTER SYSTEM SET DB_RECOVERY_FILE_DEST='/u01/ora/flashback'

B ) Cancel Redo Apply and create a guaranteed restore point.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; 

SQL> CREATE RESTORE POINT restore_point_standby GUARANTEE FLASHBACK DATABASE; 

To Confirim the details of restore point, query the below

SQL> select NAME,SCN,TIME from v$restore_point; 

In Primary Database 

Switch logs so the SCN of the restore point is archived on the physical standby database

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT; 

Defer log archive destinations pointing to the standby 

SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=DEFER; 

In Standby database 

A ) Activate the physical standby database:

SQL> ALTER DATABASE ACTIVATE STANDBY DATABASE; 

SQL> select CONTROLFILE_TYPE from v$database;

CONTROL
-------
CURRENT

B) Then open the database.

SQL> ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE;

SQL> ALTER DATABASE OPEN; 

Revert the active standby database back to Physical standby database 

Once done with your testing

    A1. Mount the database.
    A2. Flashback the database to restore point.

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> FLASHBACK DATABASE TO RESTORE POINT restore_point_standby ; 

SQL> select controlfile_type from v$database;

CONTROL
--------------
BACKUP 

B ) Convert to Standby database

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY; 

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;


SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; 

SQL> select controlfile_type from v$database;
CONTROL
--------------
STANDBY 

A ) Put the standby database in managed recovery mode. Let archive gap resolution fetch all missing archived redo log files and allow Redo Apply to apply the gap

In Primary database 

A ) Re-enable archiving for the physical standby database:

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT; 
SQL> ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;  

In Standby database 

A ) Open the database in Read only mode

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL; 

SQL> ALTER DATABASE OPEN READ ONLY;

B ) Drop the restore point


SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;


SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT; 

SQL> DROP RESTORE POINT restore_point_standby ; 

reference: Oracle documentation, MOS ID 805438.1

Oracle 11g R2 Clusterware installation

Oracle 11g R2 is a two phase installation process as below

1. Oracle Grid installation
2. ORACLE RDBMS (software) installation

before going into installation, below points are important to remember and understand as per the MOS note

• 11gR2 Clusterware needs to be up and running before 11gR2 Real Application Clusters database is installed
• The GRID home consists of the Oracle Clusterware and ASM.  ASM should not be in a separate home.
• The 11gR2 Clusterware can be installed in "Standalone" mode for ASM and/or "Oracle Restart" single node support. This clusterware is a subset of the full clusterware described in this document.
• The 11gR2 Clusterware can be run by itself or on top of vendor clusterware.  See the certification matrix for certified combinations. Ref: Note: 184875.1 "How To Check The Certification Matrix for Real Application Clusters"
• The GRID Home and the RAC/DB Home must be installed in different locations.
• The 11gR2 Clusterware requires a shared OCR files and voting files.  These can be stored on ASM or a cluster filesystem.
• The OCR is backed up automatically every 4 hours to <GRID_HOME>/cdata/<scan name>/ and can be restored via ocrconfig. 
• The voting file is backed up into the OCR at every configuration change and can be restored via crsctl. 
• The 11gR2 Clusterware requires at least one private network for inter-node communication and at least one public network for external communication.  Several virtual IPs need to be registered with DNS.  This includes the node VIPs (one per node), SCAN VIPs (three).  This can be done manually via your network administrator or optionally you could configure the "GNS" (Grid Naming Service) in the Oracle clusterware to handle this for you (note that GNS requires its own VIP).  
• A SCAN (Single Client Access Name) is provided to clients to connect to.  For more informantion on SCAN see Note: 887522.1
• The root.sh script at the end of the clusterware installation starts the clusterware stack.  For information on troubleshooting root.sh issues see Note: 1053970.1
• Only one set of clusterware daemons can be running per node. 
• On Unix, the clusterware stack is started via the init.ohasd script referenced in /etc/inittab with "respawn".
• A node can be evicted (rebooted) if a node is deemed to be unhealthy.  This is done so that the health of the entire cluster can be maintained.  For more information on this see: Note: 1050693.1 "Troubleshooting 11.2 Clusterware Node Evictions (Reboots)"
• Either have vendor time synchronization software (like NTP) fully configured and running or have it not configured at all and let CTSS handle time synchonization.  See Note: 1054006.1 for more information.
• If installing DB homes for a lower version, you will need to pin the nodes in the clusterware or you will see ORA-29702 errors.  See Note 946332.1 and Note:948456.1 for more information.
• The clusterware stack can be started by either booting the machine, running "crsctl start crs" to start the clusterware stack, or by running "crsctl start cluster" to start the clusterware on all nodes.  Note that crsctl is in the <GRID_HOME>/bin directory.  Note that "crsctl start cluster" will only work if ohasd is running.
• The clusterware stack can be stopped by either shutting down the machine, running "crsctl stop crs" to stop the clusterware stack, or by running "crsctl stop cluster" to stop the clusterware on all nodes.  Note that crsctl is in the <GRID_HOME>/bin directory.
• Killing clusterware daemons is not supported.
• Instance is now part of .db resources in "crsctl stat res -t" output, there is no separate .inst resource for 11gR2 instance.

so here is the installation process.

1. run the installation command from either software home or CD

./runInstaller

Select the "Install and grid option for a cluster" and click Next

Grid infrastructure can be installed by choosing Typical or Advance option.

Typical is the most simplest installation type. Advance option can let us configure ASM, storage etc.. In our case, we are picking 'Typical' installation type. Hit Next to proceed further.

 Now this below screen is an important one! we'll need to perform following things here

a. Add node details
b. Set up Public and Private network interfaces
c. SCAN IP
d. SSH Connectivity test


As part of the installation, local node details are displayed e.g. public and VIP names. You can add more node by hitting the ADD button and entering the details of each node.

 THE SSH CONNECTIVITY:

The SSH Connectivity configuration between the node members is a important part of Cluster installation. Therefore, if you haven't done that already then now is your chance :)

You'd  need to enter the Grid Infrastructure software owner  details and click the Test button to test and build the connectivity. To assign the right interface type for public and private interfaces, click on the Identify Network Interfaces button. All the network interfaces in the previous release were wrongly assigned to the Private type. However, with 11g R2, you can see that public and private interfaces have been assigned to the correct interface types. You could also choose the right type for the individual interface name using the drop-down list on the Identify Network Interfaces dialogue box. Click on OK and close the Identify Network Interfaces window and click on Next to continue further, as shown in the following screenshot:

The next (Install Locations) screen requires you to make a decision about 

where you are going to configure the OCR and voting disks, whether it will be on a shared disk or on ASM. Unlike the previous versions, with 11g R2, you can configure the critical components of a cluster, that is, OCR and voting disk files on ASM as well. If the ASM option is being selected over the shared storage, you need to enter and confirm the SYSASM Password subsequently, as shown in the following screenshot:

Also, on the Install Locations screen, enter the correct values for the Oracle
base and software locations

Choose one of the options from the drop-down list against Cluster RegistryType: Filesystem or AutomaticStorage Management. If the ASM option is selected, then provide a password for ASM super user, that is, ASMSYS use and assign the relevant OS group. Bypass the password warning message 

here, in case the password doesn't fit into the Oracle recommendation. Click on Next to continue. After choosing ASM as the storage option for the OCR 

and the voting disk, the subsequent screen requires your interaction for creating an ASM disk group in order to place the OCR and voting disks.

7.  Click on the Change Directory Path button to define the storage path in order to discover the storage (disks) for ASM, as shown in the following screenshot:

For example, enter a string value such as /dev/sd*1 in the Change Disk
Discovery Path dialog box and click on OK.


8.  Upon discovering the ASM disks, select the required disks from the list and enter the disk group name and also choose External type for the Redundancy option. Click on Next to continue further. Accept the default inventory location, and click on Next to proceed further on the Create Inventory
screen.

9.  OUI then initiates the prerequisite verifications on the Prerequisites Checks screen. You will then need to progress towards the Summary screen, provided that no problems were found in the prerequisite checks. However, if any concerns are raised during the prerequisites checks, resolve them
before you continue with the installation process.

10.  Verify the summary details given on the Summary screen and click on Next to begin the installation process.

11.  The actual Grid Infrastructure installation process will then kick off on the
Setup screen.

12.  Towards approaching the 100% progress, a pop-up window Execute Configuration scripts will be displayed with instructions to run orainstRoot.sh and root.sh scripts sequentially on all cluster node
members, as shown in the following screenshot:

Therefore, open a new terminal, log in as root user and run the orainstRoot. sh script on the local node. After the script on the local node successfully completes, move on to another node and execute the script again. You need to execute the same script on the rest of the nodes of a cluster.

13.  After executing the orainstRoot.sh script successfully on all nodes, turn back to the first node to execute the root.sh script as root user. After completion, move to the second node and execute the same script. After executing the script successfully on each node of a cluster, click on the OK
button to close the dialog box.

14.  If, for any reason, the root.sh script is not run successfully, then refer to the  deconfigure/reconfigure section at the end of the chapter to know more about how to resume from a failed installation. After the scripts are successfully run on each of the nodes of the cluster, continue further configuration of other options such as, ASM creation, Listener creation, Private interconnect, and the cluster verification utility on the Setup screen. Once everything is successfully completed, click on Finish to end the Grid Infrastructure installation process.



BASH-DBA: Oracle RAC components
BASH-DBA: Finding details of Oracle 11g RAC interconnect
BASH-DBA: Oracle 11g Clusterware Installation Requirements
BASH-DBA: Oracle 11g R1 Clusterware installation
BASH-DBA: Oracle 11g R1 Clusterware post-installation checks
BASH-DBA: How to Start/Stop Oracle Clusterware


source: MOS, 11g R1 R2 RAC essentials

Oracle RMAN Incrmental Backups and Block Change Tracking

Incremental backups capture on a block-by-block basis changes in your database since a previous incremental backup.The starting point for an incremental backup strategy is a level 0 incremental backup that backs up all blocks in the database. The Level 1 incremental backups which are taken at regular intervals contain only changed blocks since a previous incremental backup

Types of incremental Backups:

You can make two types of incremental backups

1. Differential Incremental Backups
2. Cumulative Incremental Backups

Differential Incremental Backups:

Its the backup of al data blocks that changes after level  or a level 1 backup. RMAN first looks for a level 1 backup, if not available to goes on finding level 0 backup and backups every change since that level 0 backup.

Here how to make a level 0 backup

RMAN> backup incremental level 0 database;

Note that level 0 backup can be an image copy or in backup set form

Here is how you make level 1 backup

RMAN> backup incremental level 1 database;

Cumulative Incremental Backups

Backups all the data block changes since the last level 0 backup.

Here is how to make cumulative incremental backup

RMAN> backup incremental level 1 cumulative database;

Recovery of Incremental Backups:

Recovery from an incremental backup is faster than recovery using redo logs alone. During a restore from incremental backup, the level 0 backup is used as the starting point, then changed blocks are updated based on level 1 backups where possible to avoid re-applying changes from redo one at a time. Recovering with incremental backups requires no additional effort on your part. If incremental backups are available, RMAN will use them during recovery.

Change tracking file:

Enterprise Edition includes RMAN's change tracking feature for incremental backups which improves incremental backup performance by recording changed blocks in each datafile in a change tracking file. If change tracking is enabled, RMAN uses the change tracking file to identify changed blocks for incremental backup, thus avoiding the need to scan every block in the datafile.

Find out if Block change tracking is enabled:

SQL> desc v$block_change_tracking

Name                     Null?                   Type
-------------------      --------------          ------------------
STATUS                                         VARCHAR2(10)
FILENAME                                    VARCHAR2(513)
BYTES                                            NUMBER

The status tells if the file is enabled or disabled. The filename tells the name of the file along with its size in the BYTES column.

If the file is in use, than how effective it is ?

The V$BACKUP_DATAFILE view contains a column called USED_CHANGE_TRACKING.
A value of YES for this column for an incremenat backup level > 0 means that RMAN used
the tracking file to speed up the incremental backup. This can help determine how effective
the the tracking file in minimizing the I/O activity during an incremental backup. The following
query can be used:

SQL> select file#,
                   avg(datafile_blocks),
                    avg(blocks_read),
                    avg(blocks_read/datafile_blocks) * 100 as “% read for backup”
          from   v$backup_datafile
          where  incremental_level > 0
             and  used_change_tracking = ‘YES’
          group  by file#
          order  by file#;


How to enable block change tracking:

SQL> alter database enable block change tracking   using file ‘/u01/oradata/change_tracking.f’;

REUSE clause can be specified if the file already exists.

SQL> alter database enable block change tracking using file ‘/u01/oradata/change_tracking.f’ reuse;

Alternatively, if you have the DB_CREATE_FILE_DEST parameter set (for Oracle Managed Files),
then you can simply issue:

SQL> alter database enable block change tracking;

In this case, Oracle crates an Oracle Managed File (OMF) in the directory specified by DB_CREATE_FILE_DEST to keep track of block changes.

If DB_CREATE_FILE_DEST is not set and you do not specify a file name, following error will result:

ORA-19773: must specify change tracking file name

Disabling block change tracking:

SQL> alter database disable block change tracking.

This command also removes the change tracking file.

Note :  Fast Incremental Backups were not possible on standby databases till 10g but from 11g Fast Incremental Backups are available on standby databases also.




Source: RMAN Fast Incremental Backups [ID 262853.1]

Oracle :View Cluster Interconnects through SQLPLUS

To verify the interconnect settings, query the (G)V$CLUSTER_INTERCONNECTS and
the (G)V$CONFIGURED_INTERCONNECTS

SQL> SELECT * FROM V$CLUSTER_INTERCONNECTS;

NAME            IP_ADDRESS       IS_ SOURCE
--------------- ---------------- --- -------------------------------
en1             10.62.144.137    NO  Oracle Cluster Repository
en2             10.62.115.218    NO  Oracle Cluster Repository

SQL> SELECT * FROM V$CONFIGURED_INTERCONNECTS;

NAME            IP_ADDRESS       IS_ SOURCE
--------------- ---------------- --- -------------------------------
en1             10.62.144.137    NO  Oracle Cluster Repository
en2             10.62.115.218    NO  Oracle Cluster Repository
en0             162.50.100.162   YES Oracle Cluster Repository

SQL> DESC V$CONFIGURED_INTERCONNECTS

Name
 --------------------------------------------------------------------
 NAME
 IP_ADDRESS
 IS_PUBLIC
 SOURCE

Oracle RMAN: Industry Standards & best practices for Stable/Reliable backups

Following are some of the best practices that we can adopt in order to have must stable and reliable oracle backups

1. Turn on block checking.

Block checking is enabled as below
                                     
SQL> show parameter db_block_checking

NAME                                 TYPE VALUE
------------------------------------ ---- ---------
db_block_checking                 string FALSE
                                     
SQL> alter system set db_block_checking = true scope=both;

When set to 'TRUE' this allows oracle to detect early presence of corrupt blocks in the database.This has a slight performance overhead but can detect corruption caused by underlying disk, storage system, or I/O system problems.

2.  Block Change Tracking tracking (incremental backups 10g & higher)

Change Tracking File maintains  information that allows the RMAN incremental backup process to avoid reading data that has not yet been modified since the last backup. When Block Change Tracking is not used, all blocks must be read to determine if they have been modified since the last backup.

SQL> alter database enable block change tracking using file '/u01/oradata/chg_trcing/chg_tracking.f';

Once set the file can be queried from SQLPLUS as below


SQL> SELECT filename, status, bytes  FROM v$block_change_tracking;

for further information on incremental backups and block change tracking follow the below link

http://docs.oracle.com/cd/B19306_01/backup.102/b14192/bkup004.htm

3.  Archive log destination.

Very important to have more than one archive log destinations. The reason is if an archivelog is corrupted or lost, by having multiple copies in multiple locations, the other logs will still be available and could be used.

This is how an another archive log location can be added to the database

SQL> alter system set log_archive_dest_2='location=/new/location/archive2' scope=both;

4. Duplex redo log groups and members

If an online log is deleted or becomes corrupt, you will have another member that can be used to recover if required.

SQL> alter database add logfile member '/new/location/redo21.log' to group 1;

Below SQL can be used to find out the number of members in the group

SQL> SELECT a.group#, count(a.member) FROM v$logfile a, v$log b WHERE a.group# = b.group# group by a.group#  order by 1;

    GROUP# COUNT(A.MEMBER)

---------- ---------------

         1               2

         2               2

         3               2

         4               2

         5               2

         6               2

         7               2

5.  RMAN  CHECK LOGICAL option.

While taking backups with RMAN, using 'check logical ' option checks the  logical corruption within a block, in addition to the normal checksum verification. This is the best way to ensure that you will get a good backup.

RMAN> backup check logical database plus archivelog delete input;

for further information, see the below link

http://docs.oracle.com/cd/B28359_01/backup.111/b28270/rcmvalid.htm

6. Test the backups.

Use 'Validate' command to test your backups. This will do everything except actually restore the database. This is the best method to determine if your backup is good and usable before being in a situation where it is critical and issues exist.RMAN> restore validate database;

See below for more information

http://docs.oracle.com/cd/B28359_01/backup.111/b28270/rcmvalid.htm

7. When using RMAN have each datafile in a single backup piece

When doing a partial restore RMAN must read through the entire piece to get the datafile/archivelog requested. The smaller the backup piece the quicker the restore can complete. This is especially relevent with tape backups of large databases or where the restore is only on individual / few files.

However, very small values for filesperset will also cause larger numbers of backup pieces to be created, which can reduce backup performance and increase processing time for maintenance operations. So those factors must be weighed against the desired restore performance.

RMAN> backup database filesperset 1 plus archivelog delete input;

8. Maintain your RMAN catalog/controlfile

Choose your retention policy carefully. Make sure that it complements your tape subsystem retention policy, requirements for backup recovery strategy. If not using a catalog, ensure that your CONTROL_FILE_RECORD_KEEP_TIME parameter matches your retention policy.

SQL> alter system set control_file_record_keep_time=31 scope=both;

This will keep 31 days of backup records in the control file.

For more details :
BASH-DBA: Relation between RMAN retention period and ...
Note 461125.1 .

9. Control file backup

Ensure autobackup parameter in RMAN is always set to 'ON'.This will ensure that you always have an up to date controlfile available that has been taken at the end of the current backup, rather then during the backup itself.

RMAN> configure controlfile autobackup on;

Also, keep your backup logs. These logs contain parameters for your tape access, locations on controlfile backups that can be utilized if complete loss occurs.

10. Test your recovery

During a recovery situation this will let you know how the recovery will go without
actually doing it, and can avoid having to restore source datafiles again.
SQL> recover database test;

11. In RMAN backups do not specify 'delete all input' when backing up archivelogs

REASON: Delete all input' will backup from one destination then delete both copies of the
archivelog where as 'delete input' will backup from one location and then delete what has
been backed up. The next backup will back up those from location 2 as well as new logs
from location 1, then delete all that are backed up. This means that you will have the
archivelogs since the last backup available on disk in location 2 (as well as backed up
once) and two copies backup up prior to the previous backup.

See Note 443814.1 Managing multiple archive log destinations with RMAN for details.

References: [ID 388422.1], Oracle documentation

How to Start/Stop Oracle 10g/11g Clusterware

Typically the Oracle cluster-ware starts up automatically during startup. However, this can be controlled using the following commands.

10gR1

init.crs stop

The above command is used to stop the cluster ready services. eg. used while applying patches or any other plan outages.

init.crs start

used in Oracle 10.1.0.4 onward. In prior versions (10.1.0.3 and 10.1.0.2), node needs to be reboot if CRS needs to be started with the exception of patch set application process that already have clusterware starting script.

init.crs disable

This command disables Clusterware from being started in a subsequent reboot.

init.crs enable

Enables Clusterware to be started in a subsequent reboot.

10gR2 and High Version (11gR1, 11gR2)

below are the self explanatory commands to start/stop cluster services in 10g R2 or higher

crsctl start crs

crsctl stop crs

The individual components of the clusterware like init.cssd init.evmd init.crsd should not be used manually.

For version 10g R1 to lower that 10gR2, the location of the init scripts is dependent on the OS.

For Solaris, the scripts are in /etc/init.d/
For HP, the scripts are in /sbin/init.d
For AIX, the scripts are in /etc
For Linux, the scripts are in /etc/init.d

further reading!
BASH-DBA: Verifying Oracle Cluster prerequisites (CLUVFY utility)
BASH-DBA: Oracle 11g R1 Clusterware installation
BASH-DBA: Oracle 11g R1 Clusterware post-installation checks
BASH-DBA: Finding details of Oracle 11g RAC interconnect