Patching and Upgrading (Oracle Clusterware 11gR2)

As with other database software Oracle Clusterware 11g Release 2 will require occasional patches and upgrades. Patches are interim releases between major versions of the Oracle Clusterware product. For example, moving from 11.2.0.1 to 11.2.0.2 would be considered a patch. Upgrades occur when you are moving between major versions of the Oracle Clusterware product. An example of an upgrade would be moving from Oracle Clusterware version 11.1 to 11.2. In this section we will first look at patching of Oracle Clusterware 11g Release 2 and then we will look at upgrading.

Patching Oracle Clusterware 11g Release 2

If you have patches to apply to Oracle Clusterware and Oracle RAC, you should always apply the Clusterware patches first, followed by the patches to the Oracle Database. You can choose to patch Oracle Clusterware and not the Oracle Database but the reverse is not true. If you wish to patch the Oracle Database you must patch the Oracle Clusterware first. Thus the Oracle Clusterware version number must always be higher or the same as the patch set level as the Oracle Database.

So it more then normal to have 11gR2 Clusterware for Oracle 10g database!

There are three basic kinds of Oracle patches. These include:

• CRS Merge Label Request (MLR) patches, also known as One-Off patches
• CRS Bundle Patches (BP) and PSU Patches
• Critical Patch Updates (CPU)
• Patch Set Updates (PSU)

Let’s look at each of these in more detail next. We will then look at how to search for existing patch sets on the Oracle Support Website.

CRS Merge Label Request (MLR) Patches

When you experience bugs in your database that require immediate correction, you should contact Oracle support and open a support request (SR). After opening the SR the Oracle support representative may determine that your problem is an existing bug, and may identify an existing MLR patch for that bug. Sometimes the MLR patch will not be available for your operating system platform, in which case you can ask that Oracle support request Oracle development port the patch to the operating system that you are using.

Other times you may have found a bug that is new. As a result Oracle will open a support request and will request that development review the bug. As a result of this review by development they may produce a patch for you to apply and this would also be considered and MLR patch. It may also be that Oracle will not be able to create an MLR patch and that they will indicate that you will need to wait for the next patch set (see CRS bundle patches and CRS patch sets later in this section).

It is important to note that MLR patches are not regression tested. Therefore it’s important that you carefully test your database and make sure that it operates normally after the application of a patch. MLR patches are generally installed using the Oracle opatch utility. You should always read the associated patch set install instructions before you attempt to install the patch set.

One off patches are installed in-place. This means that the patch is applied to the existing GRID_HOME or ORACLE_HOME. Also in many cases one-off patches can be applied in a rolling fashion. This means that the patch is applied to each node, one at a time. This results in limited service interruptions to your users. Check the patch install instructions to determine if it can be installed using a rolling upgrade method.

CRS Bundle Patches (BP) and Patch Set Updates (PSU’s)

A CRS Bundle Patch Set is a fully regression tested patch set. It typically contains a number of MLR patches and other patches that Oracle things are important to include in the BP. BP’s are installed with the opatch utility.

BP patches can be installed in-place or out of place. As with One-off patches in place installs means that the patch is applied to the existing GRID_HOME or ORACLE_HOME. An out-of-place install means that the patch is applied to a new GRID_HOME or ORACLE_HOME rather than the existing home, and that you swing over the cluster to using that new home location to start using the newly applied patches.

Many BP patches can be applied in a rolling fashion. This means that the patch is applied to each node, one at a time. This results in limited service interruptions to your users. Check the patch install instructions to determine if it can be installed using a rolling upgrade method.

CRS (clusterware) BP’s are now released on a regularly scheduled basis in January, April, July and October and are called Patch Set Updates (PSU). PSU patch sets contain recommended bug fixes from Oracle plus the security patches contained in the CPU patches (see more about CPU patches later in this section). PSU’s are numbered using the fifth minor number of the database version. For example, if you have Oracle version 11.2.0.1.0 installed, the next PSU you would install would be 11.2.0.1.1.

PSU’s are intended to be low risk patch sets. They contain fixes that fix critical technical issues, that impact a large number of Oracle customers and have already been proven effective in the field. Since they also include CPU updates, PSU’s also address current security issues. PSU’s do not contain any patches that would require re-certification or that would require configuration changes.

There are a number of different PSU’s including Grid Infrastructure PSU’s and Oracle Database PSU’s. The GI PSU’s contain all the Database PSU’s and therefore you need only run the GI PSU’s if you are running Oracle RAC. If you are not running a clustered environment then you would only need to install the database PSU’s. As always consult MOS and the readme documents of the patch sets to ensure that you are properly installing the patch.

Critical Patch Updates (CPU)

CPU patch sets are patch sets that are targeted towards security fixes associated with Oracle Database products. CPU’s are released four times a year in the middle part of January, April, July and October and they are cumulative in nature. CPU patch sets are designed to only address security issues and no other bug fixes are intended to be included in CPU patch sets with the exception to any fixes that might conflict with the related security fixes. You should see no functionality changes with a CPU patch set.
!!!Searching For Existing Patch Sets
Oracle provides an easy way to search for existing patch sets. There are three methods that you can use. These methods are:

1. Finding the patch on Oracle MOS, download and install.
2. Using Oracle Enterprise Manager.
3. If you are applying a CPU, use the www.oracle.com website.

Manually Searching for Patches

If you have access to the My Oracle Support (MOS) support page you can search for recommended patches for various Oracle products. Note that the MOS pages sometimes change their look and feel, so these instructions may not be 100% accurate.

After logging into the MOS homepage, you will see a tab at the top of the page titled “Patches and Updates”. Click on the “Patches and Updates” tab. On the next page you will see an option to select specific products. Click on the link that indicates that the link is for Oracle patches (at this time it says “Oracle, Siebel and Hyperion products”.

Next the patches and Updates screen will appear. From this page you can search for specific patch sets. The page also offers a quick link to the latest patch sets for the Oracle Database and a link to the Oracle recommended patches. After you have selected the link you are interested, complete the search information requested. Typically you will be asked for the database release number you currently are on, the platform it is running on and other related information. Enter the information requested (some type of information are optional) and select the go button. MOS will then present a list of patches and you will have an opportunity to review the readme file associated with the patch and then download the patch.

Using OEM to Search For Patches

You can configure Oracle Enterprise Manager to manage database patching for you. OEM will interface with Metalink Oracle Support (MOS) via your Metalink support ID. OEM will notify you if patches are available for both Clusterware and the Oracle database. You can opt to download and install those patches using OEM.

Personally, as member of old school I do avoid this kind of patching....suggesting to you to decide for your own. Do remember that in many situations EM may not be available!

Downloading CPU Patches from www.oracle.com

You can find information on CPU patch sets at http://www.oracle.com/technology/deploy/security/alerts.htm . You will need a valid Oracle support contract to download the CPU patch sets. To download patch sets for Oracle log into Metalink Oracle Support (MOS). There is a tab on the MOS homepage that is called Patches and Updates (as of this writing). Click on that tab. There you will have several options to choose from including an option to quick links to the latest available patch sets.

Using OPatch

OPatch is an Oracle supplied Java utility that supports patch application and rollback. OPatch keeps track of the patches that have been installed. Also with OPatch you can perform rolling upgrades of an Oracle Cluster. In this section we will provide a general overview of the patch install process. We will then look at the OPatch utility in more detail and then finally we will look at the application of a patch with the OPatch command.

Overview of Oracle Clusterware/RAC Patching with OPatch

Applying a patch with OPatch varies based on the instructions associated with the patch to be applied. Each patch has a readme file associated with it that contains the instructions you should follow to install the patch. Instructions will differ for RAC and non-RAC databases.

OPatch supports three modes of patching. These modes are:

• All-Node patching (all at once)
• Minimum downtime patching
• Rolling patching

All-Node patching

OPatch will apply the patch on the local node first, then propagate the patch to the other nodes. All instances are shutdown during the patching process. The summary of operations for all-node patching includes:

• Shutdown all Oracle instances on all nodes.
• Apply the patch to all nodes.
• Bring all the nodes up

Minimum downtime patching

OPatch will apply the patch to the local node. It then will start patching a sub-set of other nodes, that it will apply the patch too. Once the patch is applied to those nodes, it will propagate the patch to the remaining nodes. This method leads to limited downtime during patching. The only downtime would occur between the point-in-time that you shutdown the second sub-set of nodes and start-up the first set of nodes that was patched. The summary of operations for minimum downtime patching includes:

• Assume 3 nodes are being patched.
• Shutdown all instances for the Oracle Home to be patched on Node 1.
• Apply the patch to the Oracle Home on node 1.
• Shutdown all instances for the Oracle Home to be patched on Node 2.
• Apply the patch to the Oracle Home on node 2.
• Shutdown all instances for the Oracle Home to be patched on Node 3.
• Bring up the instances on Nodes 1 and 2.
• Apply the patch to the Oracle Home on node 3.
• Startup the instance on Node 3.

Rolling patching

 Each node is patched and brough up while the other nodes stay up and running. There is no downtime when using this patch method. The summary of operations for minimum downtime patching includes:

• Assume 3 nodes are being patched.
• Shutdown all instances for the Oracle Home to be patched on Node 1.
• Apply the patch to the Oracle Home on node 1.
• Start the instance on Node 1.
• Shutdown all instances for the Oracle Home to be patched on Node 2.
• Apply the patch to the Oracle Home on node 2.
• Start the instance on Node 2.
• Shutdown all instances for the Oracle Home to be patched on Node 3.
• Apply the patch to the Oracle Home on node 3.
• Start the instance on Node 3.

The patch readme file will indicate if rolling patching is available for the patch. Carefully review the patch set readme and any other files that come with the patch set to determine the proper way to install the patch.

The OPatch Utility

The OPatch utility is used to patch Oracle Clusterware software and Oracle RAC and Non-RAC Database software. The OPatch utility requires that the variable ORACLE_HOME be set. When running OPatch you should set the value of ORACLE_HOME to the location of the product to be patched. Thus, if you are patching Oracle Clusterware you will set ORACLE_HOME to the location of GRID_HOME. Because of this, and other occasions when it is helpful to be able to set the Oracle environment to point to GIRD_HOME, we recommend that you actually add an entry in the /etc/oratab file called grid which points to the GRID_HOME. Then you can use the oraenv program (or coraenv) to configure your environment easily. You can also include the ORACLE_HOME location when running commands from OPatch. You can also use the –oh parameter when calling OPatch and include the ORACLE_HOME location there as seen in this example:

Opatch lsinventory –detail –oh /u01/app/11.2.0/grid

Once you have configured $ORACLE_HOME correctly, you can begin the OPatch patch application process. Following the install instructions, you will typically run the pre-patch scripts, OPatch and then the post-patch scripts.

You can use the OPatch –help switch to get help for the OPatch utility. You can run OPatch –help and OPatch will present a general help screen. If you want help for a specific function you can type in the name of that function followed by the –help switch as seen in this example:

Opatch apply –help
Opatch lsinventory -help

Patch Set Conflicts

Before applying a patch set, OPatch will check for conflicts with other patch sets that you may have already installed. If it detects a conflict you will need to contact Oracle Support and as them to open an SR and request that a patch set merge request be submitted to support. You will not be able to install the patch set until this merge patch set is created. Because of the possibility of conflicts between patch sets, it’s a good idea to keep track of the patch sets that you have installed and submit this list to Oracle when requesting the creation of a patch set. This can reduce the time it take to create a patch set or the time it will take to wait for a merge patch set to be created.

OPatch can detect the following types of conflicts:

• Bug Superset
• Bug Conflict
• File Conflict
• Combination Conflict

Bug Superset

If all the bugs fixed by a patch in the system are also fixed by the patch to be applied, then this patch (the patch to be applied) is considered to be a superset of the patch already applied. If a bug superset condition is detected, it is not considered an error situation. All the subset patches are removed from the system and the new patch is applied.
For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown:

If you apply a patch E that fixes bugs 5,6,7,8,9, and 10 then patch E will be the superset of patch C and D.

If you want OPatch to error out if the current patch bugs-to-fix is a superset or the same as an installed patch bugs-fixed in the Oracle home directory, you can use the -no_bug_superset flag.
$ OPatch/opatch apply -no_bug_superset

Bug Conflict

If a set of bugs to be fixed by the current interim patch includes some but not all bugs already fixed by one or more previously installed interim patches it is called a bug conflict. You must remove the bug conflict before you proceed with the patching by using the apply command with -force flag, that rolls back the conflicting patches before applying the new one.

For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown in previous picture. If you apply a patch E that fixes bugs 1,3,5,7,9, and 10, you will find that this patch has fixed bugs 1,3,5,7,9, and 10, but has opened bugs 2,4,6, and, 8. This is a conflict situation.

File Conflict

If a set of files to be patched by the current interim patch include files already patched by one or more previously installed interim patches and it is not a bug superset, it is called a file conflict. You must remove the file conflict before you proceed with the patching by using the apply command with -force flag, that rolls back the conflicting patches before applying the new one.

Combination Conflict

If a set of patches has a combination of bug superset, and bug or file conflict, it is called a Combination Conflict. It is an error situation. In this case, OPatch removes all conflicting patches as well as the subset patches and then re-applies the new patch.

For example, consider a scenario where there are four patches A,B,C, and D applied in a system, each of which fixes 2 bugs as shown in previous picture. Patch C is the subset of patch D. Patch A and patch B are conflicting patches of patch D. If you apply this patch D that fixes bugs 1,3,5,6,7, and 8 with -force flag, you will find that OPatch would have rolled back patches A, B, and C and would have applied patch D.

The End

Keep in mind that in patching is one of the most risking operation. So having a proper database backup as well as general "rollback plan" is something that will save hours of bad mouth taste!

Cheers!