How to identified active files/executables when using opatch to deinstall software

During a opatch rollback operation, we saw the following error message in the opatch logfile:

[Feb 19, 2023 2:29:51 PM] [INFO]    Prerequisite check "CheckActiveFilesAndExecutables" failed.
                                    The details are:


                                    Following active files/executables/libs are used by ORACLE_HOME :/sw/oracle/product/19.18
                                    /sw/oracle/product/19.18/lib/libclntsh.so.19.1

Reason:
Some processes are still using the /sw/oracle/product/19.18/lib/libclntsh.so.19.1.
Use the fuser utility with verbose output to find the process:

fuser -v /sw/oracle/product/19.18/lib/libclntsh.so.19.1
                     USER        PID ACCESS COMMAND
/sw/oracle/product/19.18/lib/libclntsh.so.19.1:
                     oracle    48439 ....m prometheus_orac
                     oracle    595787 ....m ggsci

There were two open processes using the file libclntsh.so.19.1: 1. the Golden Gate Manager 2. a utility called prometheus_oracle_exporter

Solution:

log in as the Golden Gate software owner
ggsci --> info all --> list all processes
stop mgr !

One of two processes quit its handler on the file:

 fuser -v /sw/oracle/product/19.18/lib/libclntsh.so.19.1
                     USER        PID ACCESS COMMAND
/sw/oracle/product/19.18/lib/libclntsh.so.19.1:
                     oracle    48439 ....m prometheus_orac

For the prometheus agent, we simply kill the agent, and the output from fuser now reveals that no file handlers are open:

kill 48439
fuser -v /sw/oracle/product/19.18/lib/libclntsh.so.19.1

What is the catcon.pl script used in a Multitenant Database installation?

What is the catcton.pl script?

From Doc ID 1932340.1:

Oracle has provided script catcon.pl to execute scripts at Container and Pluggable database at once. In a CDB, the catcon.pl script is the best way to run SQL scripts and SQL statements. It can run them in the root and in specified PDBs in the correct order, and it generates log files.

It can be found in the $ORACLE_HOME/rdbms/admin folder.

Example of usage

To execute preupgrd.sql at CDB and all PDBs, copy preupgrd.sql and utlppkf.sql from the software version you want to upgrade to, temporary location (say /u01/oracle) PDB should be in open state before executing script. Its status can be checked using

SYS@cdb> connect / as sysdba
SYS@cdb>SQL>show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 SALES                          READ WRITE NO

At OS prompt, execute:

cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catcon.pl -d /u01/oracle -l /home/oracle/preupgrd_logs -b preupgrade_log_base_name preupgrd.sql

The arguments used are:

-d = directory containing the file to be run (Location of preupgrade script)
-l = directory to use for spool log files
-b = base name for log and spool file names

Not that if neither the -c nor the -C parameter is specified, then catcon.pl runs the script in all containers by default:

-c - container(s) in which to run sqlplus scripts for example, -c 'PDB1 PDB2'
-C - container(s) in which NOT to run sqlplus scripts, i.e. skip all

Sources:
"How to execute sql scripts in Multitenant environment (catcon.pl) (Doc ID 1932340.1)"
"How to recompile invalid objects in all PDBs at the same time (Doc ID 2880643.1)"

Both from Oracle Support.

How to fix the pending_dst_session error when running the preupgrade script

If you receive an error during the precheck-phase of an upgrade from 18c to 19c, which points to "pending_dst_sessions", like this:

you can follow Doc ID 1509653.1 "Updating the RDBMS DST version in 12c Release 1 (12.1.0.1 ) and above using DBMS_DST"

In short, execute the following:

SELECT version FROM v$timezone_file;

In my case the result was

   VERSION
----------
        32

Then, check the following:

col property_name format a40
col value format a40
set lines 200
SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;

Result:

PROPERTY_NAME                            VALUE
---------------------------------------- --------------------
DST_PRIMARY_TT_VERSION                   32
DST_SECONDARY_TT_VERSION                 31
DST_UPGRADE_STATE                        UPGRADE

Then, create a file called "dst.sql" and add the following:

col property_name format a40
col value format a40
set lines 200
alter session set "_with_subquery"=materialize;
alter session set "_simple_view_merging"=TRUE;
set serveroutput on
VAR numfail number
BEGIN
DBMS_DST.UPGRADE_DATABASE(:numfail,
parallel => TRUE,
log_errors => TRUE,
log_errors_table => 'SYS.DST$ERROR_TABLE',
log_triggers_table => 'SYS.DST$TRIGGER_TABLE',
error_on_overlap_time => FALSE,
error_on_nonexisting_time => FALSE);
DBMS_OUTPUT.PUT_LINE('Failures:'|| :numfail);
END;
/
VAR fail number
BEGIN
DBMS_DST.END_UPGRADE(:fail);
DBMS_OUTPUT.PUT_LINE('Failures:'|| :fail);
END;
/

SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;
exit

Execute the script:

oracle@myserver.mydomain.com:[testdb01]# sqlplus / as sysdba @dst.sql

Output:

Connected to:
Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production
Version 18.14.0.0.0


Session altered.


Session altered.

Table list: "GSMADMIN_INTERNAL"."AQ$_CHANGE_LOG_QUEUE_TABLE_L"
Number of failures: 0
Table list: "GSMADMIN_INTERNAL"."AQ$_CHANGE_LOG_QUEUE_TABLE_S"
Number of failures: 0
Table list: "DVSYS"."SIMULATION_LOG$"
Number of failures: 0
Table list: "DVSYS"."AUDIT_TRAIL$"
Number of failures: 0
Failures:0

PL/SQL procedure successfully completed.

An upgrade window has been successfully ended.
Failures:0

PL/SQL procedure successfully completed.


PROPERTY_NAME                            VALUE
---------------------------------------- --------------------
DST_PRIMARY_TT_VERSION                   32
DST_SECONDARY_TT_VERSION                 0
DST_UPGRADE_STATE                        NONE

SQL> exit

The error should now be fixed and you can try to run the prechecks for the upgrade once more.

utlu122s.sql is replaced from version 19c

During upgrades, you will notice that any old script that you've assembled to upgrade 12c and 18c databases, will break when they try to run the script utlu122s.sql. Note that it has been rename from version 19c and onwards, and is now called utlusts.sql.

Thanks to Rajasekhar Amudala for pointing this out on his blog.

What do do when an upgrade from 12cR1 to 18c is seemingly frozen in Phase #:50

During an upgrade from 12.1 to 18.0 my upgrade process seemed to be completely frozen at stage 50:

Restart  Phase #:47   [proddb01] Files:1    Time: 0s
*************   Final Catproc scripts   ************
Serial   Phase #:48   [proddb01] Files:1    Time: 4s
Restart  Phase #:49   [proddb01] Files:1    Time: 0s
**************   Final RDBMS scripts   *************
Serial   Phase #:50   [proddb01] Files:1

The catupgrd0.log file located in the $ORACLE_HOME/cfgtoollogs/proddb01/upgrade20191103120250 showed that the last entry was

12:23:40 SQL> update tabpart$
12:23:40   2    set hiboundval='4294967296', bhiboundval = '06C52B5F614961'
12:23:40   3    where obj# in (select t1.obj# as part_obj_no
12:23:40   4      from sys.tabpart$ t1
12:23:40   5        left join sys.tabpart$ tmax
12:23:40   6          on (t1.bo#=tmax.bo# and t1.part# < tmax.part#)
12:23:40   7        join sys.tab$ t3 on (t1.bo#=t3.obj#)
12:23:40   8      where tmax.part# is null and t1.hiboundval is not null
12:23:40   9        and t1.hiboundlen = 10 and bitand(t3.property/power(2, 75), 1) = 1);

This may happen when your database have tables with lots and lots of partitions. A check against the database proves this:

select table_owner,count(*)
from dba_tab_partitions
where table_owner <> 'SYS'
group by table_owner
order by 2 desc;

TABLE_OWNER	COUNT(*)
USER1	3497525
USER2	569471
USER3	288664
USER4	59857
USER5	28032
USER6	7239
USER7	1481

In our case, the grant total upgrade time was 10 hours and 11 minutes.
The "update tabpart$" statement above counted for 9,5 hours of this time.

The team responsible for application design later revealed that they have been meaning to redesign the tables to cut down on the number of partitions, but not yet come around to it.

Lession learned....

;-)

What is the admsi.pl utility that is used during EBS patching?

Thanks to the blog post by Atul Kumar for explain this.
It's a tool for generating instance-specific patch installation instructions.

This tool was refered to in the beginning of a readme-file for a particular patch. The readme-file stated:

Instructions For Applying This Patch
Execute the following command to generate your instance specific installation instructions Source the Applications environment file Run the Patch Application Assistant by entering "admsi.pl".

A bit further down in the same readme-file:

Apply The Patch

For 12.0.X / 12.1.X / pre-upgrade patches (using adpatch), you must shut down all Application tier services before performing the tasks in this section. For 12.2.X patches (using adop), you can perform the tasks in this section without shutting down the Application tier services. Apply patch [required] This patch contains the following unified driver file to be applied with AutoPatch: u28389569.drv (This is the unified driver)

$ perl $AD_TOP/bin/admsi.pl -patch_top=/software/EBS/28389569
Gathering Information..
Enter the APPS user password:
Logfile for this session is located at admsi.log
Generating installation instructions for patch 28389569..
Updating database....
install_28389569.html generated successfully
install_28389569.txt generated successfully

The generated files will contain the exact patching instructions.

Parameters that can be removed after upgrade to 12.2

From Information For Parameters _upgrade_optim and _upgrade_capture_noops (Doc ID 2182783.1):


Below parameters gets added after upgrade to 12.2:

_upgrade_optim = FALSE
_upgrade_capture_noops = FALSE

These parameters are part of upgrade optimizations. They do not have any effect on upgrade currently.

Since these parameters does not have any impact, it can be removed safely after upgrade to 12.2.


Can be removed, like this:

echo "alter system reset \"_upgrade_optim\" scope=spfile SID='*';" | sqlplus / as sysdba
echo "alter system reset \"_upgrade_capture_noops\" scope=spfile SID='*';" | sqlplus / as sysdba

What is the "datapatch" utility released in Oracle 12c?

From Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1):

Datapatch is the new tool that enables automation of post-patch SQL actions for RDBMS patches.

and

Datapatch can be executed to complete post-patch SQL actions after the database is restarted following patch application.
For patches that do not have post-patch SQL actions to be performed, calling datapatch is a no-op.
For patches that do have post-patch SQL instructions to be invoked on the database instance, datapatch will automatically detect ALL pending actions (from one installed patch or multiple installed patches) and complete the actions as appropriate.

How does Oracle detect ALL pending actions?

Datapatch determines the requisite apply/rollback actions by matching an internal repository with the patch inventory.


When should datatach be invoked?

Datapatch should be invoked when the database is restarted after a patching session.

From 12.1.0.2 and onwards, Oracle Corporation points out that they have made some change to the datapatch utility. The most important in my opinion is that datapatch now assumes "the role of catbundle when applying bundles/PSU."

This means that

catbundle.sql has been deprecated and patch registry is maintained only in registry$sqlpatch.
Application of the PSU does not update registry$history table anymore.
The status for patch application is now entirely maintained in registry$sqlpatch


Here's an abbreviated output from using datapatch as part of applying Oracles latest PSU, Database Patch Set Update 12.1.0.2.170814 (Includes CPUJul2017):

1. Download and unpack the patch

[oracle@lx01oric software]$ unzip p26609783_121020_Linux-x86-64.zip

2. Shutdown your oracle instance and the listener
3. Check for conflicts:

[oracle@lx01oric software]$ cd 26609783
oracle@lx01oric 26609783]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.1.0.1.3
Copyright (c) 2017, Oracle Corporation.  All rights reserved.

PREREQ session

Oracle Home       : /u01/oracle/product/db/12.1/12.1.0.2.0
Central Inventory : /u01/oraInventory
   from           : /u01/oracle/product/db/12.1/12.1.0.2.0/oraInst.loc
OPatch version    : 12.1.0.1.3
OUI version       : 12.1.0.2.0
Log file location : /u01/oracle/product/db/12.1/12.1.0.2.0/cfgtoollogs/opatch/opatch2017-09-06_13-41-14PM_1.log

Invoking prereq "checkconflictagainstohwithdetail"

Prereq "checkConflictAgainstOHWithDetail" passed.

OPatch succeeded.

4. Apply the patch:

[oracle@lx01oric 26609783]$ opatch apply
Oracle Interim Patch Installer version 12.2.0.1.9
Copyright (c) 2017, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/product/db/12.1/12.1.0.2.0
Central Inventory : /u01/oraInventory
   from           : /u01/oracle/product/db/12.1/12.1.0.2.0/oraInst.loc
OPatch version    : 12.2.0.1.9
OUI version       : 12.1.0.2.0
Log file location : /u01/oracle/product/db/12.1/12.1.0.2.0/cfgtoollogs/opatch/opatch2017-09-06_14-16-35PM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   19769480  20299023  20831110  21359755  21948354  22291127  23054246  24006101  24732082  25171037  25755742  26609783  

Do you want to proceed? [y|n]

... output abbreviated.

Composite patch 26609783 successfully applied.
Log file location: /u01/oracle/product/db/12.1/12.1.0.2.0/cfgtoollogs/opatch/opatch2017-09-06_14-16-35PM_1.log

OPatch succeeded.

5. Start your database. When opened, execute datapatch:

[oracle@lx01oric] cd $ORACLE_HOME/OPatch
[oracle@lx01oric OPatch]$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 Production on Wed Sep  6 14:54:49 2017
Copyright (c) 2012, 2016, Oracle.  All rights reserved.

Log file for this invocation: /u01/oracle/cfgtoollogs/sqlpatch/sqlpatch_16001_2017_09_06_14_54_50/sqlpatch_invocation.log

Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...
Bootstrap timed out after 240 seconds
Retrying bootstrap after retryable errors....done
Determining current state...done

Current state of SQL patches:
Bundle series PSU:
  ID 170814 in the binary registry and not installed in any PDB

Adding patches to installation queue and performing prereq checks...
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED
    Nothing to roll back
    The following patches will be applied:
      26609783 (DATABASE PATCH SET UPDATE 12.1.0.2.170814)

Installing patches...
Patch installation complete.  Total patches installed: 2

Validating logfiles...
Patch 26609783 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/oracle/cfgtoollogs/sqlpatch/26609783/21481899/26609783_apply_CDBVEG_CDBROOT_2017Sep06_15_01_26.log (no errors)
Patch 26609783 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/oracle/cfgtoollogs/sqlpatch/26609783/21481899/26609783_apply_CDBVEG_PDBSEED_2017Sep06_15_09_34.log (no errors)
SQL Patching tool complete on Wed Sep  6 15:18:53 2017
[oracle@lx01oric OPatch]$ vi /u01/oracle/cfgtoollogs/sqlpatch/26609783/21481899/26609783_apply_CDBVEG_PDBSEED_2017Sep06_15_09_34.log

6. If applicable, activate the OJVM Mitigation patch.

According to Oracle, this patch is applicable in "situations where the latest OJVM PSU cannot be installed immediately... The "Mitigation Patch" is an interim solution to protect against all currently known (Jul 2015) Oracle JavaVM security vulnerabilities in the database until such time as the OJVM PSU can be installed."

Since this is my situation, I install the mitigation patch:

[oracle@lx01oric 26609783]$ cd $ORACLE_HOME/rdbms/admin
[oracle@lx01oric admin]$ sqlplus / as sysdba @dbmsjdev.sql
[oracle@lx01oric admin]$ sqlplus / as sysdba
SQL> exec dbms_java_dev.disable;
PL/SQL procedure successfully completed.
SQL> 

How to unzip multiple files on a Unix-based operating systems

When extracting multiple zip files, remember to use single quotes to mask the zip-files:

unzip '*.zip'

For example, when unpacking the files needed to install Oracle 11.2.0.4:

[oracle@lx01 Downloads]$ ls -latr
total 5586768
-rwxr-x---.  1 root   root     1395582860 Sep  2 10:14 p13390677_112040_Linux-x86-64_1of7.zip
-rwxr-x---.  1 root   root     1151304589 Sep  2 10:14 p13390677_112040_Linux-x86-64_2of7.zip
-rwxr-x---.  1 root   root     1205251894 Sep  2 10:15 p13390677_112040_Linux-x86-64_3of7.zip
-rwxr-x---.  1 root   root      656026876 Sep  2 10:16 p13390677_112040_Linux-x86-64_4of7.zip
-rwxr-x---.  1 root   root      599170344 Sep  2 10:16 p13390677_112040_Linux-x86-64_5of7.zip
-rwxr-x---.  1 root   root      488372844 Sep  2 10:16 p13390677_112040_Linux-x86-64_6of7.zip
-rwxr-x---.  1 root   root      119521122 Sep  2 10:16 p13390677_112040_Linux-x86-64_7of7.zip
drwx------. 17 oracle oinstall       4096 Sep  2 10:27 ..
drwxr-xr-x.  2 oracle oinstall       4096 Sep  2 10:34 .
[oracle@lx01 Downloads]$ unzip 'p13390677_112040_Linux-x86-64_*.zip'

Without the quotes, unzip will look for "zip files within zip files".

Thanks to Chris Jean for the tip!

Oracle 11g out-of-place vs. in-place database upgrades

Starting with Oracle 11gR2, Oracle changed their patching strategy.

Previously, a patch set was supposed to be downloaded and installed into an existing ORACLE_HOME.
After 11.2, a "patch set" is really a full release, bundled in a number of zip files.

The default patching method is the "out-out-place", which means that every time customers need to upgrade to a new patch set (say, 11.2.0.4), a new ORACLE_HOME should be created, and the new binaries should be installed here.

So you will potentially end up with a number of ORACLE_HOME on your server, and you can choose to run your database out of either of these, upgrading whenever you please.

See Mike Dietrich's post about the same topic for more details.

How to fix error thrown by OPatch: java.lang.NullPointerException

Problem:

Opatch throws the following error when you are querying the inventory:

[oracle@lx01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.11
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/oraInventory
   from           : /u01/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.11
OUI version       : 11.2.0.1.0
Log file location : /u01/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-09-02_08-52-28AM_1.log

Lsinventory Output file location : /u01/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-09-02_08-52-28AM.txt

--------------------------------------------------------------------------------
java.lang.NullPointerException
 at oracle.sysman.oii.oiin.OiinNetOps.addNICInfo(OiinNetOps.java:144)
 at oracle.sysman.oii.oiin.OiinNetOps.computeNICList(OiinNetOps.java:109)
 at oracle.sysman.oii.oiin.OiinNetOps.(OiinNetOps.java:76)
 at oracle.sysman.oii.oiin.OiinNetOps.getNetOps(OiinNetOps.java:90)
 at oracle.sysman.oii.oiix.OiixNetOps.getFullHostName(OiixNetOps.java:49)
 at oracle.opatch.LsInventoryPrinter.printLocalMachineInfo(LsInventoryPrinter.java:2336)
 at oracle.opatch.LsInventorySession.loadAndPrintInventory(LsInventorySession.java:480)
 at oracle.opatch.LsInventorySession.process(LsInventorySession.java:317)
 at oracle.opatch.OPatchSession.main(OPatchSession.java:2120)
 at oracle.opatch.OPatch.main(OPatch.java:626)
Local Machine Information::
Hostname: lx01.oric.no
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

Cause:
In my case, the file /etc/hosts was incorrectly configured.

Solution:
Configure the file /etc/hosts correctly.

[root@lx01 ~]# ifconfig 
enp0s3: flags=4163  mtu 1500
        inet 192.168.0.55  netmask 255.255.255.0  broadcast 192.168.0.255

Check the /etc/hosts:

[root@lx01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.55 lx01.oric.no lx01
[oracle@lx01 ~]$ vi /etc/hosts

Update the incorrect ip address from 192.168.1.55 to 192.168.0.55, and save the file.

Opatch will now display expected output:

[oracle@lx01 ~]$ opatch lsinventory
Oracle Interim Patch Installer version 11.2.0.3.11
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/product/11.2.0/dbhome_1
Central Inventory : /u01/oraInventory
   from           : /u01/oracle/product/11.2.0/dbhome_1/oraInst.loc
OPatch version    : 11.2.0.3.11
OUI version       : 11.2.0.1.0
Log file location : /u01/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/opatch2015-09-02_09-07-43AM_1.log

Lsinventory Output file location : /u01/oracle/product/11.2.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2015-09-02_09-07-43AM.txt

--------------------------------------------------------------------------------
Local Machine Information::
Hostname: lx01.oric.no
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1): 

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

How to disable an event parameter in the database

During an Upgrade from 11.2.0.3 to 11.2.0.4 I had to remove an event-parameter in the database.

The syntax for this is:

alter system set event="10852 trace name context off" scope=spfile;

How to confirm that the latest PSU has been applied to your ORACLE_HOME

Use opatch lsinventory to extract the information. In my case:

hostname:PRODDB01>opatch lsinventory | egrep -i 'PSU|DATABASE PATCH SET UPDATE'

Patch description:  "Database Patch Set Update : 11.2.0.4.2 (18031668)" 
Sub-patch  17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Patch description:  "Database Patch Set Update : 11.2.0.3.7 (16619892)"
Sub-patch  16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)"
Sub-patch  14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)"
Sub-patch  14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)"
Sub-patch  13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)"
Sub-patch  13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)"
Sub-patch  13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)"