Monday, June 30, 2014

log_archive_format default value

Having trouble finding out why the value of the parameter LOG_ARCHIVE_FORMAT does not seem to apply to your database?

From Oracle 10g and onwards, the default value for LOG_ARCHIVE_FORMAT is %t_%S_%r.dbf

However, this parameter will only make a difference if you are NOT using a "Fast Recovery Area" (in Version 10 Oracle called the same thing a "Flash Recovery Area")

If you have defined a FRA by using the parameters db_recovery_file_dest and db_recovery_file_dest_size, the archivelogs will be automatically named according to OMF (Oracle Managed Files) standard and the value of LOG_ARCHIVE_FORMAT will be ignored.

An example of an OMF managed file in the FRA:
/fra/PRODDB01/archivelog/2014_06_30/o1mv_1_1_9vxzch5_.arc

From the documentation:
"The fast recovery area is an Oracle Database managed space that can be used to hold RMAN disk backups, control file autobackups and archived redo log files. The files placed in this location are maintained by Oracle Database and the generated file names are maintained in Oracle Managed Files (OMF) format."

At first glance, letting the LOG_ARCHIVE_FORMAT default to such a poor naming standard may seem like a bad idea, but I guess it illustrates the point of defining an FRA vs. using conventional archive log destinations.

Wednesday, June 25, 2014

How to gather dictionary statistics and fixed-objects statistics

Use the procedures

dbms_stats.gather_dictionary_stats
and
dbms_stats.gather_fixed_objects_stats

Examples:
BEGIN
   DBMS_STATS.GATHER_DICTIONARY_STATS (
     estimate_percent  => DBMS_STATS.AUTO_SAMPLE_SIZE
     ,method_opt        => 'FOR ALL COLUMNS SIZE AUTO'
     ,degree            => NULL
     ,cascade           => DBMS_STATS.AUTO_CASCADE
     ,granularity       => 'AUTO'
     ,no_Invalidate     => DBMS_STATS.AUTO_INVALIDATE);
 END;
 /

 BEGIN
   DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;
 END;
 /

Used in a simple shell script:
#!/usr/bin/ksh
# Gather dictionary and fixed objects stats as recommended by oracle

cat << EoF > ${SCRIPT_BASE}/sql/gather_dict_stats.sql
alter session set nls_date_format='DD-MM-YYYY HH24:MI:SS';
spool ${SCRIPT_BASE}/log/gather_dict_stats.log append
set timing on
select 'Start time: ' || sysdate as "startime" from dual;
execute dbms_stats.gather_dictionary_stats(estimate_percent=>DBMS_STATS.AUTO_SAMPLE_SIZE,method_opt=>'FOR ALL COLUMNS SIZE AUTO',degree=>DBMS_STATS.DEFAULT_DEGREE,granularity=>'ALL',cascade=>DBMS_STATS.AUTO_CASCADE,options=>'GATHER AUTO',no_invalidate=>DBMS_STATS.AUTO_INVALIDATE);
execute dbms_stats.gather_fixed_objects_stats(NULL);
select 'End time: ' || sysdate as "endtime" from dual;
exit
EoF
sqlplus / as sysdba @${SCRIPT_BASE}/sql/gather_dict_stats.sql
rm -r ${SCRIPT_BASE}/sql/gather_dict_stats.sql
exit


Good sources for further reading: Maria Colgan's blog

Tuesday, June 24, 2014

Where to find the database characterset in the dictionary

SQL> select value$ from sys.props$ where name = 'NLS_CHARACTERSET';

 VALUE$
 --------------------------------------------------------------------------------
 WE8MSWIN1252

 SQL> select * from nls_database_parameters WHERE PARAMETER='NLS_CHARACTERSET';

 PARAMETER                      VALUE
 ------------------------------ ----------------------------------------
 NLS_CHARACTERSET               WE8MSWIN1252

How to work around dbca error "The Oracle system identifier ... already exists, specify another SID"

If you have previously created a database through dbca and saved it as a template for future use, you may see the following error when trying to specify the same SID again:
The Oracle system identifier ... alerady exists, specify another SID
Solution to the problem is to remove the entry for the SID in /etc/oratab

Then click next in dbca, and the configuration wizard will proceed.

Thanks to Elena for pointing this out in her blog.

Friday, June 20, 2014

Potential cause of ORA-12547: TNS:lost contact

I was running a script against the database directly on the host, when the script timed out with:

ERROR:
 ORA-12547: TNS:lost contact

After a search on Oracle's support site I found doc 555565.1: Troubleshooting ORA-12547 TNS: Lost Contact

One potential cause was incorrect permissions on the oracle binary in the $ORACLE_HOME/bin directory. I checked my old OH and compared it with the new OH:

server1>cd $ORACLE_HOME
server1>pwd
 /u01/oracle/product/11204
server1>ls -latr bin/oracle
 -rwxr-x--x    1 ora11g   dba       309704130 Jun 11 11:00 bin/oracle
server1>ls -latr /u01/oracle/product/11.2.0.2/bin/oracle
 -rwsr-s--x    1 ora11g   dba       279291858 Mar 12 13:42 /u01/oracle/product/11.2.0.2/bin/oracle
As can be seen from the above statement, the oracle server executable was lacking the setuid bit

This bit is needed, since we need to make sure that any user who runs the executable file will inherit the user ID of the owner (or group) of the executable file. Note that LOCAL connections would work just fine, even if the permissions on the oracle executable is set incorrectly; only connections that are passed from the listener to the Oracle server are affected. These sessions must be able to spawn a new server process.

Oracle points out that this error is commonly seen "in environments where the listener is running in the GRID home and servicing connections to an instance in a different $ORACLE_HOME."

Connections via the listener are failing with ORA-12547. It is likely in this scenario that LOCAL or BEQ connections to the instance are successful.

Solution to my particular problem this time was to give $ORACLE_HOME/bin/oracle the proper permissions:
server1> sqlplus / as sysdba
SQL> shutdown immediate

server1>chmod 6751 oracle
server1>ls -latr oracle
 -rwsr-s--x    1 ora11g   dba       309704130 Jun 11 11:00 oracle

server1> sqlplus / as sysdba
SQL> startup

After this change the script ran through with no ORA-12547 errors.

Sources: Oracle System Administration Guide

Thursday, June 19, 2014

How to disable or drop an SQL Profile

Find the profiles currently saved in the database:
SELECT NAME, CREATED,LAST_MODIFIED,TYPE,STATUS,FORCE_MATCHING
FROM DBA_SQL_PROFILES
ORDER BY CREATED DESC;


NAME CREATED LAST_MODIFIED TYPE STATUS FORCE_MATCHING
SYS_SQLPROF_0146b2c081f9011a 19.06.2014 08:11:39,000000 19.06.2014 08:11:39,000000 MANUAL ENABLED NO
SYS_SQLPROF_0146b2bf0d430118 19.06.2014 08:10:03,000000 19.06.2014 08:10:03,000000 MANUAL ENABLED NO
SYS_SQLPROF_0146af25274a0117 18.06.2014 15:23:06,000000 18.06.2014 15:23:06,000000 MANUAL ENABLED NO
SYS_SQLPROF_0146af1a61f40116 18.06.2014 15:11:20,000000 18.06.2014 15:11:20,000000 MANUAL ENABLED NO
SYS_SQLPROF_0146af18d0960115 18.06.2014 15:09:37,000000 18.06.2014 15:09:37,000000 MANUAL ENABLED NO
SYS_SQLPROF_0146af150fc30114 18.06.2014 15:05:31,000000 18.06.2014 15:05:31,000000 MANUAL ENABLED NO
SYS_SQLPROF_01462313623f0113 22.05.2014 10:36:51,000000 22.05.2014 10:36:51,000000 MANUAL ENABLED NO
SYS_SQLPROF_01461857e9680112 20.05.2014 08:35:53,000000 20.05.2014 08:35:53,000000 MANUAL ENABLED NO
SYS_SQLPROF_014614c584ab0111 19.05.2014 15:57:07,000000 19.05.2014 15:57:07,000000 MANUAL ENABLED NO

To disable the profile, which means they will not any longer be considered by the CBO until you once again accept them:
set serveroutput on
 begin
   dbms_sqltune.alter_sql_profile(name=>'SYS_SQLPROF_0146af25274a0117', attribute_name=>'STATUS',value=>'DISABLED');
 end;
/
To completely drop the profile:
set serveroutput on
 begin
   dbms_sqltune.drop_sql_profile(name=>'SYS_SQLPROF_0146af25274a0117');
end;
/

What does the TERRITORY component of the NLS_LANG parameter control?

From OTN:

"The territory component of the NLS_LANG parameter controls the operation of a subset of globalization support features.
It specifies conventions such as the default date, monetary, and numeric formats. Each supported territory has a unique name, for example, AMERICA, FRANCE, or CANADA.
If the territory is not specified, then the value is derived from the language value."


For examples please see the Globalization Support Guide

Wednesday, June 18, 2014

How to work around ORA-12705 when connecting to your database

You get:

ERROR:
 ORA-12705: Cannot access NLS data files or invalid environment specified
Solution:
Incorrect or misspelled NLS_LANG.

For example
NLS_LANG=AMERICAN_NORWAY.UTF-8
is incorrect.

However
export NLS_LANG=AMERICAN_NORWAY.UTF8
is correct.

How to install the instant client from Oracle on a Linux application server

This is how I quite quickly installed sqlplus on a Linux appserver on request from my customer:

1. Download the two files
instantclient-basic-linux.x64-11.2.0.4.0.zip
instantclient-sqlplus-linux.x64-11.2.0.4.0.zip
from the otn network.


2. Create Directory for the installation files
mkdir  /u01/oracle/product/admin/install

3. Ftp the files over to the server

4. Put the installation files in the install Directory:
mv /tmp/instantclient*.zip /u01/oracle/product/admin/install

5. Unzip the files:
unzip instantclient-basic-linux.x64-11.2.0.4.0.zip -d /u01/oracle/product
unzip instantclient-sqlplus-linux.x64-11.2.0.4.0.zip -d /u01/oracle/product
The files will be unzipped to the Directory /u01/oracle/Product/instantclient_11_2

6. Create a Directory for networking:
mkdir $ORACLE_HOME/network/admin

7. set up the .bash_profile:
export ORACLE_BASE=/u01/oracle
export ORACLE_HOME=/u01/oracle/product/instantclient_11_2
export PATH=$PATH:$HOME/bin:$ORACLE_HOME
export LD_LIBRARY_PATH=/u01/oracle/product/instantclient_11_2
export NLS_LANG=NORWEGIAN_NORWAY.UTF8
export TNS_ADMIN=$ORACLE_HOME/network/admin

8. Create file tnsnames.ora, add an entry for each of the databases you'd like to Access via sqlplus.

All done!


Note that you do need *both* the basic and the sqlplus instantclient zip files to be unpacked into the same directory. If not, you will be missing Libraries in your installation and you'll receive the following error when launching sqlplus:

sqlplus: error while loading shared libraries: libclntsh.so.11.1: cannot open shared object file: No such file or Directory

Friday, June 13, 2014

Workaround for RMAN-11003 and ORA-32010 during cloning



At the end of the duplication procedure, my script threw the following error:

RMAN-06162: sql statement: alter system reset  db_unique_name scope=spfile
 RMAN-08031: released channel: c1
 RMAN-08031: released channel: c2
 RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of Duplicate Db command at 06/12/2014 17:40:14
 RMAN-05501: aborting duplication of target database
 RMAN-03015: error occurred in stored script Memory Script
 RMAN-03009: failure of sql command on clone_default channel at 06/12/2014 17:40:14
 RMAN-11003: failure during parse/execution of SQL statement: alter system reset  db_unique_name scope=spfile
 ORA-32010: ORA-32010: cannot find entry to delete in SPFILE 

 Recovery Manager complete.

After a brief search I found that my init.ora was the culprit: I had accidently left out single quotes in the db_file_name_convert parameter:

*.db_file_name_convert=('/u02/oradata/proddb01/datafile','/u02/oradata/testdb01','/u02/oradata/proddb01/tempfile',/u02/oradata/testdb01')

Oracle Support Dok ID 1384262.1 "ORA-32010: cannot find entry to delete in SPFILE" points this out.

The spfile was not in $OH/dbs physically, although Oracle believes it does:
SQL> show parameter spfile

NAME        TYPE        VALUE
 ---------- ----------- ---------------------------------------------
 spfile     string      /u01/oracle/product/11204/dbs/spfiletestdb01.ora

To prove it, I attempted to create an spfile:
SQL> create spfile from pfile;
 create spfile from pfile
 *
 ERROR at line 1:
 ORA-32002: cannot create SPFILE already being used by the instance 
When attempting to mount:
SQL> alter database mount;
 alter database mount
 *
 ERROR at line 1:
 ORA-01103: database name PRODDB01 in control file is not TESTDB01
So the control files are incorrect, written to during the duplication, and RMAN was unfortunately interrupted before the duplication procedure completed.

Solution:
Recreate the controlfile from the Source database, and open the auxiliary database with this new controlfile instead of the one that was generated during duplication.


1. Log onto the source server on which the PRODDB01 database is hosted, and execute
SQL> alter database backup controlfile to trace as '/tmp/proddb01_cntrfile.sql';
2. Transfer the file /tmp/proddb01_cntrfile.sql to the testserver
3. Shutdown the auxiliary database
4. Edit the script:
* Go to the section that says "Set #2. RESETLOGS case". You can strip away anything above this section, as you won't be needing it.
* Use the physical paths valid on the auxiliary server
* Exchange "REUSE DATABASE" with "SET DATABASE"
From Source database:
STARTUP NOMOUNT
 CREATE CONTROLFILE REUSE DATABASE "PRODDB01" RESETLOGS  ARCHIVELOG
     MAXLOGFILES 16
     MAXLOGMEMBERS 3
     MAXDATAFILES 200
     MAXINSTANCES 8
     MAXLOGHISTORY 1168
 LOGFILE
   GROUP 1 '/u02/oradata/proddb01/redo1.log'  SIZE 1000M BLOCKSIZE 512,
   GROUP 2 '/u02/oradata/proddb01/redo2.log'  SIZE 1000M BLOCKSIZE 512,
   GROUP 3 '/u02/oradata/proddb01/redo3.log'  SIZE 1000M BLOCKSIZE 512,
   GROUP 4 '/u02/oradata/proddb01/redo4.log'  SIZE 1000M BLOCKSIZE 512
 -- STANDBY LOGFILE
 DATAFILE
   '/u02/oradata/proddb01/datafile/system.428.688578763',
   '/u02/oradata/proddb01/datafile/undotbs1.489.688574787',
   '/u02/oradata/proddb01/datafile/sysaux.436.688579377'
.
.
.
CHARACTER SET AL32UTF8
;
The new edited version:
STARTUP NOMOUNT
 CREATE CONTROLFILE REUSE
 SET DATABASE "TESTDB01" RESETLOGS  ARCHIVELOG
     MAXLOGFILES 16
     MAXLOGMEMBERS 3
     MAXDATAFILES 200
     MAXINSTANCES 8
     MAXLOGHISTORY 1168
 LOGFILE
   GROUP 1 '/u02/oradata/testdb01/redo1.log'  SIZE 1024M BLOCKSIZE 512,
   GROUP 2 '/u02/oradata/testdb01/redo2.log'  SIZE 1024M BLOCKSIZE 512,
   GROUP 3 '/u02/oradata/testdb01/redo3.log'  SIZE 1024M BLOCKSIZE 512,
   GROUP 4 '/u02/oradata/testdb01/redo4.log'  SIZE 1024M BLOCKSIZE 512
 -- STANDBY LOGFILE
 DATAFILE
  '/u02/oradata/testdb01/system.428.688578763',
   '/u02/oradata/testdb01/undotbs1.489.688574787',
   '/u02/oradata/testdb01/sysaux.436.688579377',
   '/u02/oradata/testdb01/users.476.688577593',
   '/u02/oradata/testdb01/undotbs1.510.688564845',
   '/u02/oradata/testdb01/sl_part_data_2006.473.688577939',
.
.
.
CHARACTER SET AL32UTF8
;
5. Run the script on the newly cloned database:
SQL> sqlplus / as sysdba @proddb01_cntrfile.sql
Oracle instance started

 Total System Global Area    5344731136 bytes

 Fixed Size                     2255784 bytes
 Variable Size               1459618904 bytes
 Database Buffers            3875536896 bytes
 Redo Buffers                   7319552 bytes

Control file created.

6. Mount the database
SQL> alter database mount;

Database mounted.
7. Open the database
SQL> alter database open resetlogs;

Database opened.


Finally I had to handle TEMP tablespaces, as the alert log would tell you:
Dictionary check beginning
 Tablespace 'TEMP' #3 found in data dictionary,
 but not in the controlfile. Adding to controlfile.
 *********************************************************************
 WARNING: The following temporary tablespaces contain no files.
          This condition can occur when a backup controlfile has
          been restored.  It may be necessary to add files to these
          tablespaces.  That can be done using the SQL statement:

          ALTER TABLESPACE  ADD TEMPFILE

          Alternatively, if these temporary tablespaces are no longer
          needed, then they can be dropped.
            Empty temporary tablespace: TEMP

Add a file to the temporary tablespace using the REUSE clause:
SQL> alter tablespace temp add tempfile '/u02/oradata/testdb01/temp01.dbf' size 1024M reuse autoextend on next 32M maxsize unlimited;

Thursday, June 12, 2014

How to solve RMAN-06217 during cloning from active database


RMAN-03002: failure of Duplicate Db command at 06/12/2014 12:28:00
 RMAN-05501: aborting duplication of target database
 RMAN-06217: not connected to auxiliary database with a net service name

Workaround is to connect to the auxiliary database using username and password, followed by the TNS alias.
So instead of
 connect target sys/<password>@PRODDB01
 connect rmanuser/<password>@RMANCAT
 connect auxiliary /
use
 connect target sys/<password>@PRODDB01
 connect rmanuser/<password>@RMANCAT
 connect auxiliary sys/<password>@AUX01

Examples of RMAN duplicate database scripts

I prefer to set the parameters db_file_name_convert and log_file_name_convert whenever possible, to simplify the redirection of restored files.

Here is an example when parameter log_file_name_convert is not set. We are also specifying the auxiliary channels with vendor specific tape settings:

run { 
allocate auxiliary channel t1 type sbt PARMS="BLKSIZE=1048576" maxpiecesize 32G maxopenfiles 64; 
allocate auxiliary channel t2 type sbt PARMS="BLKSIZE=1048576" maxpiecesize 32G maxopenfiles 64;
send 'NB_ORA_POLICY=mypolicy,NB_ORA_CLIENT=myclient-bkp.mydomain,NB_ORA_SCHED=myschedule'; 
set until scn 5886541624244; 
duplicate target database to AUXDB1 
logfile group 1 ('/data/oracle/u01/AUXDB1/redo01.log') size 128M, 
        group 2 ('/data/oracle/u02/AUXDB1/redo02.log') size 128M, 
        group 3 ('/data/oracle/u03/AUXDB1/redo03.log') size 128M;
} 

Another example, when both db_file_name_convert and log_file_name_convert are set.
In smaller environments, you may not need to bother specifying different backup polices etc, you simply specify 'SBT_TAPE' and the


run {
allocate auxiliary channel c1 type 'SBT_TAPE';
allocate auxiliary channel c2 type 'SBT_TAPE';
set until time "to_date('01.05.2014 18:00:00','DD.MM.YYYY HH24:MI:SS')";
duplicate target database to AUXDB1; 
}

How to set the db_file_name_convert and log_file_name_convert parameters

In this example, I want to redirect the files to a slightly different path during a clone from prod to test.

In prod, files are to be found under:

* /u02/oradata/proddb01/datafile

In test, I want to be placed under:

* /u02/oradata/testdb01

Furthermore, some tempfiles are placed differently than regular datafiles in prod. In test I do not need or want multiple destinations; all files should be placed under /u02/oradata/testdb01. Therefore, my db_file_name_convert parameter must have multiple pairs of source and target locations.

For the log files, they could all be placed under similar locations, so the redirection string can simply contain the only thing that will differ: the ORACLE_SID.

When using a pfile:

db_file_name_convert=('/u02/oradata/proddb01/datafile','/u02/oradata/testdb01',
                      '/u02/oradata/proddb01/tempfile',/u02/oradata/testdb01')

log_file_name_convert=('proddb01','testdb01')
When using an spfile:
alter system set db_file_name_convert='/u02/oradata/proddb01/datafile','/u02/oradata/testdb01',
                                      '/u02/oradata/proddb01/tempfile',/u02/oradata/testdb01' scope=spfile;

alter system set log_file_name_convert='proddb01','testdb01' scope=spfile;

It is also supported to use the log_file_name_convert multiple times in the parameter file, like this:
log_file_name_convert=('/u01/app/oracle/oradata/proddb01/onlinelog/','/u04/oradata/testdb01/onlinelog/')
log_file_name_convert=('/u01/app/oracle/flash_recovery_area/proddb01/onlinelog/','/u04/oradata/testdb01/onlinelog/')
Oracle Docs:
db_file_name_convert
log_file_name_convert

How to pin SGA in memory on an AIX server

On AIX, if you want to pin the SGA in memory, set the parameter LOCK_SGA to TRUE.

A prerequisite for locking the SGA is that you set the proper capabilities for the user owning and spawning the Oracle rdbms processes.

Check the set capabilities as follows:
[root@myserver] lsuser -a capabilities oracle
 oracle
Set capabilities:
[root@myserver] chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle
 [root@myserver] lsuser -a capabilities oracle
 oracle capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE
 [root@myserver]


Wednesday, June 11, 2014

How to work around SEVERE: OUI-10197:Unable to create a new Oracle Home when using clone.pl

When cloning a database home onto a new server, the clone.pl script threw the following error:

perl clone.pl ORACLE_HOME=/u01/oracle/product/11204 ORACLE_HOME_NAME=11204 

SEVERE: OUI-10197:Unable to create a new Oracle Home at /u01/oracle/product/11204. Oracle Home already exists at this location. Select another location.
The server was freshly installed, new volume Groups, logical Volume Groups and file systems newly created.

I then realized that since the software was tared and untared into a new directory, there would potentially be an existing Oracle Inventory present, and I would needed to "detach" the ORACLE_HOME before cloning it. The tarball did indeed contain an Inventory:

 myserver>cd $ORACLE_HOME
 myserver>cat oraInst.loc.org
 inventory_loc=/u01/oracle/oraInventory
 inst_group=dba
Since there was no such directory present, I created it:
myserver>mkdir /u01/oracle/oraInventory/ContentsXML/
myserver>cd /u01/oracle/oraInventory/ContentsXML/
myserver>vi inventory.xml
In the file inventory.xml, I added the following minimal information:
<!-- Copyright (c) 1999, 2013, Oracle and/or its affiliates.
All rights reserved. -->
 <!-- Do not modify the contents of this file by hand. -->
 <inventory>
 <version_info>
    <saved_with>11.2.0.4.0</saved_with>
    <minimum_ver>2.1.0.6.0</minimum_ver>
 </version_info>
 <home_list>
 <home idx="1" loc="/u01/oracle/product/11204" name="11204" type="O">
 </home></home_list>
 <compositehome_list>
 </compositehome_list>
 </inventory>

I then added
 -invPtrLoc /u01/oracle/product/11204/oraInst.loc
to file $ORACLE_HOME/clone/config/cs.properties

Next, I detached the ORACLE_HOME:
myserver>cd $ORACLE_HOME/oui/bin
 myserver>./runInstaller -detachHome ORACLE_HOME=/u01/oracle/product/11204 -invPtrLoc /u01/oracle/product/11204/oraInst.loc
 Starting Oracle Universal Installer...

 Checking swap space: must be greater than 500 MB.   Actual 8192 MB    Passed
 The inventory pointer is located at /u01/oracle/product/11204/oraInst.loc
 The inventory is located at /u01/oracle/oraInventory
 'DetachHome' was successful.

Finally I was able to clone the ORACLE_HOME:

perl clone.pl ORACLE_HOME=/u01/oracle/product/11204 ORACLE_HOME_NAME=11204 ORACLE_BASE=/u01/oracle OSDBA_GROUP=dba

 ********************************************************************************

 Your platform requires the root user to perform certain pre-clone
 OS preparation.  The root user should run the shell script 'rootpre.sh' before
 you proceed with cloning.  rootpre.sh can be found at
 /u01/oracle/product/11204/clone directory.
 Answer 'y' if the root user has run 'rootpre.sh' script.

 ********************************************************************************

 Has 'rootpre.sh' been run by the root user? [y/n] (n)
 y
 ./runInstaller -clone -waitForCompletion  "ORACLE_HOME=/u01/oracle/product/11204" "ORACLE_HOME_NAME=11204" "ORACLE_BASE=/u01/oracle" "oracle_install_OSDBA=dba" -silent -noConfig -nowait -invPtrLoc /u01/oracle/product/11204/oraInst.loc
 Starting Oracle Universal Installer...

 Checking swap space: must be greater than 500 MB.   Actual 8192 MB    Passed
 Preparing to launch Oracle Universal Installer from /tmp/OraInstall2014-06-11_03-12-31PM. Please wait ...Oracle Universal Installer, Version 11.2.0.4.0 Production
 Copyright (C) 1999, 2013, Oracle. All rights reserved.

 You can find the log of this install session at:
  /u01/oracle/oraInventory/logs/cloneActions2014-06-11_03-12-31PM.log
 .................................................................................................... 100% Done.



 Installation in progress (Wednesday, June 11, 2014 3:12:45 PM CEST)
 ................................................................................                                                80% Done.
 Install successful

 Linking in progress (Wednesday, June 11, 2014 3:12:54 PM CEST)
 Link successful

 Setup in progress (Wednesday, June 11, 2014 3:15:32 PM CEST)
 Setup successful

 End of install phases.(Wednesday, June 11, 2014 3:15:59 PM CEST)
 WARNING:
 The following configuration scripts need to be executed as the "root" user.
 /u01/oracle/product/11204/root.sh
 To execute the configuration scripts:
     1. Open a terminal window
     2. Log in as "root"
     3. Run the scripts

 The cloning of 11204 was successful.
 Please check '/u01/oracle/oraInventory/logs/cloneActions2014-06-11_03-12-31PM.log' for more details.

How to deal with ORA-00020: maximum number of processes (%s) exceeded


I recently had a situation where access to the database was completely blocked because of the infamous error message
ORA-00020: maximum number of processes (%s) exceeded
Facts:
  • The database had processes set to 1000.
  • The Cloud Control agent was spawning hundreds of processes (obviously an error to troubleshoot as a separate action)
  • Connecting through sqlplus with os authentication (sqlplus / as sysdba) didn't work due to the same reason

    At that time, the database had to become available to the users again ASAP.

    When I have encountered these situations in the past, I have had to kill all the operating system processes and restart the instance. A brut-force method that is not particularly pretty, but sometimes necessary:

    for a in $(ps -ef |grep $ORACLE_SID | grep -v grep | awk '{ print $2}'); do 
    >kill -9 $a; 
    >done
    

    It normally does the job when you really have no other option.
    This time however, after having killed all the processes, Oracle still rejected connections to the database using sqlplus:

    sqlplus /nolog
    
     SQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 6 09:07:23 2014
    
     Copyright (c) 1982, 2013, Oracle.  All rights reserved.
     @ SQL> connect /
     ERROR:
     ORA-00020: maximum number of processes (1000) exceeded
    
    I then found the page by tech.e2sn.com that showed how to use sqlplus with the "preliminary connection".

    Simply by using
    
    sqlplus -prelim "/as sysdba"
    
    I was able to connect and shutdown the database with the abort option.
    sqlplus -prelim "/ as sysdba"
    
     SQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 6 09:09:15 2014
     Copyright (c) 1982, 2013, Oracle.  All rights reserved.
    
     SQL> shutdown abort
     ORACLE instance shut down.
     SQL> exit
     Disconnected from ORACLE
    
    After this point the database could once again be restarted:
    sqlplus / as sysdba
    
     SQL*Plus: Release 11.2.0.4.0 Production on Fri Jun 6 09:10:38 2014
     Copyright (c) 1982, 2013, Oracle.  All rights reserved.
     Connected to an idle instance.
    SQL> startup
     ORACLE instance started.
    
     Total System Global Area 2137886720 bytes
     Fixed Size                  2248080 bytes
     Variable Size            1258291824 bytes
     Database Buffers          855638016 bytes
     Redo Buffers               21708800 bytes
     Database mounted.
     Databasen opened.
    

    The article referred to above is worth reading, but in short, the -prelim option will not try to create private session structures in the SGA. This allows you to connect to perform debugging or shutdown operations.
  • Great feature in adrci for searching in the alert log


    A potentially very timesaving feature in Oracle's adrci is the ability to search in the alert log for specific text, as I had to do to find when a specific parameter was set:

    adrci
    
     ADRCI: Release 11.2.0.4.0 - Production on On Jun 11 12:03:34 2014
    
     Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
    
     ADR base = "/u01/oracle/admin/proddb01/diagnostic"
     adrci> show homes
     ADR Homes:
     diag/rdbms/proddb01/proddb01
     adrci> show alert -p "message_text like '%event%'"
    
     ADR Home = /u01/oracle/admin/proddb01/diagnostic/diag/rdbms/proddb01/proddb01:
     *************************************************************************
     Output the results to file: /tmp/alert_3343326_1_proddb01_1.ado
     "/tmp/alert_3343326_1_proddb01_1.ado" 44 lines, 2408 characters
     2013-08-25 14:35:46.344000 +02:00
     One of the following events caused this:
     2014-01-02 10:56:34.311000 +01:00
     OS Pid: 8455160 executed alter system set events '10852 trace name context forever, level 16384'
     2014-01-02 10:56:47.555000 +01:00
     ALTER SYSTEM SET event='10852 trace name context forever, level 16384' SCOPE=SPFILE;
     2014-01-10 00:43:02.945000 +01:00
       event                    = "10852 trace name context forever, level 16384"
     2014-01-31 19:32:59.471000 +01:00
       event                    = "10852 trace name context forever, level 16384"
     2014-02-01 09:12:59.653000 +01:00
       event                    = "10852 trace name context forever, level 16384"
     CLOSE: Active sessions prevent database close operation
     2014-02-01 18:10:54.100000 +01:00
       event                    = "10852 trace name context forever, level 16384"
     2014-06-10 19:38:42.536000 +02:00
     ALTER SYSTEM SET event='10852 trace name context forever, level 16384 off' SCOPE=SPFILE;
     2014-06-10 19:43:12.770000 +02:00
       event                    = "10852 trace name context off"
    

    Without much effort I was able to find that the parameter was set 02.01.2014, and switched off 10.06.2014.

    How to disable an event parameter in the database

    During an Upgrade from 11.2.0.3 to 11.2.0.4 I had to remove an event-parameter in the database.

    The syntax for this is:
    alter system set event="10852 trace name context off" scope=spfile;
    

    Friday, June 6, 2014

    Why does the TIMESTAMP# column in the AUD$ table contain NULL values?

    According to Oracle Support Note 427296.1:

    "In database version 10gR1 and above, the TIMESTAMP# column is obsoleted in favor of the new NTIMESTAMP# column."

    So when exchanging the TIMESTAMP# with the NTIMESTAMP# column, my script works as intended, while it had previously showed NULL values:

    SELECT DBID "CURRENT DBID" FROM V$DATABASE;
    
     SET TIMING ON
     SET LINES 200
     COL "Earliest" format a30
     col "Latest" format a30
    
     PROMPT Counting the DBIDs and the number of audit entries each
     PROMPT Could take a while...
     COL TIMESTAMP# FORMAT A3
     SELECT DBID,COUNT(*),MIN(NTIMESTAMP#) "Earliest", MAX(NTIMESTAMP#) "Latest"
     FROM AUD$
     GROUP BY DBID;
    

    Output:
    Counting the DBIDs and the number of audit entries each
     Could take a while...
    
           DBID   COUNT(*) Earliest                       Latest
     ---------- ---------- ------------------------------ ------------------------------
    2367413790       1867 05.06.2014 14.01.30,193254     06.06.2014 06.17.08,485629
    

    The views built upon AUD$, for example DBA_AUDIT_TRAIL and DBA_FGA_AUDIT_TRAIL, will of course reflect the correct columns from AUD$ (NTIMESTAMP#) in their own TIMESTAMP column.

    Thursday, June 5, 2014

    Multiple DBIDs in AUD$

    I wanted to test the DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL package, with use_last_arch_timestamp to TRUE, to only purge records one month older than the minimum value found.

    I found this value by using the function ADD_MONTHS to add 1 month on top of the minimum value found in AUD$
    SET SERVEROUTPUT ON
     DECLARE
     currdate DATE;
     last_archtime DATE;
    
     BEGIN
    
     currdate := SYSTIMESTAMP;
    
     ---------------------------------------------------------
     -- Get the oldest timestamp from AUD$, then add one month.
     -- Use this timestamp as the last archive timestamp in
     -- procedure SET_LAST_ARCHIVE_TIMESTAMP
     ---------------------------------------------------------
     SELECT ADD_MONTHS(
       (
        SELECT MIN(TIMESTAMP) 
        FROM DBA_AUDIT_TRAIL
        ), 1)
     INTO last_archtime
     FROM DUAL;
     DBMS_OUTPUT.PUT_LINE('last_archtime: ' || last_archtime);
    
      DBMS_AUDIT_MGMT.SET_LAST_ARCHIVE_TIMESTAMP (
        AUDIT_TRAIL_TYPE => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
        LAST_ARCHIVE_TIME => last_archtime);
    
     END;
     /
    
    Put all that in a file called set_last_timestamp_std.sql.
    First, check the DBA_AUDIT_MGMT_LAST_ARCH_TS for the last archive timestamp:
    SQL> SELECT * FROM DBA_AUDIT_MGMT_LAST_ARCH_TS;
    
    No rows selected.
    
    
    Execute the script created above:
    sqlplus / as sysdba @set_last_timestamp_std.sql
    
     last_archtime: 07.02.2009
    
     PL/SQL-procedure executed.
    
    Check the DBA_AUDIT_MGMT_LAST_ARCH_TS again:
    AUDIT_TRAIL          RAC_INSTANCE LAST_ARCHIVE_TS
     -------------------- ------------ ----------------------------------------
     STANDARD AUDIT TRAIL            0 07.02.2009 15.01.50,000000 +00:00
    

    I was now ready to execute the manual cleanup. Before I did so, I wanted to get an idea about how many rows that should be purged:
    SELECT COUNT(*)
    FROM DBA_AUDIT_TRAIL
    WHERE TIMESTAMP < (
    SELECT ADD_MONTHS(
       (SELECT TIMESTAMP
        FROM (SELECT TIMESTAMP FROM DBA_AUDIT_TRAIL ORDER BY TIMESTAMP ASC)
        WHERE ROWNUM <=1), 1)
     FROM DUAL)
    ;
    
      COUNT(*)
    ----------
       126405
    
    
    Compare with the total number of rows:
    SQL> SELECT COUNT(*)FROM DBA_AUDIT_TRAIL;
    
      COUNT(*)
    ----------
      33 664,540
    
    Sweet. 126405 records would be purged. I then executed:
    BEGIN
    DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL(
    audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    use_last_arch_timestamp => TRUE);
    END;
    /
    
    The purge succeeded. But when I checked the number of rows again, it still returned 126405 rows. What I found was that what Oracle was executing the following statement internally when using the DBMS_AUDIT_MGMT package:
    DELETE FROM SYS.AUD$ WHERE DBID = 2367413790 AND NTIMESTAMP# < to_timestamp('2009-02-07 15:01:50', 'YYYY-MM-DD HH24:MI:SS.FF') AND ROWNUM <= 10000;
    
    So I tested to select the rows using the same predicate that was used during the purge:
    SQL> SELECT COUNT(*) FROM SYS.AUD$ WHERE DBID = 2367413790 AND NTIMESTAMP# < to_timestamp('2009-02-07 15:01:50', 'YYYY-MM-DD HH24:MI:SS.FF');
    
       COUNT(*)
     ----------
              0
    
    checked again against the dba_audit_trail
    SQL> SELECT COUNT(*) FROM DBA_AUDIT_TRAIL WHERE TIMESTAMP  < to_timestamp('2009-02-07 15:01:50', 'YYYY-MM-DD HH24:MI:SS.FF');
    
       COUNT(*)
     ----------
         126405
    
    So there are indeed records that are older than '2009-02-07 15:01:50'. Why is it not caught when querying the AUD$ table, only the DBA_AUDIT_TRAIL? Of course! The AUD$ table also has a reference to the DBID. And since the database was recently cloned, it has cycled through another incarnation:
    SQL> select DBID,MIN(NTIMESTAMP#)
    2  FROM AUD$
    3  GROUP BY DBID;
    
    DBID MIN(NTIMESTAMP#)
    ---------- ----------------------------
    2367413790 19.05.2014 07.07.13,675010
    848951741 07.01.2009 13.01.50,802413
    
    So the fact that minimum timestamp for DBID 2367413790 is 19.05.2014 is correct after all:
    SQL> SELECT MIN(TIMESTAMP) FROM DBA_AUDIT_TRAIL WHERE DBID=2367413790;
    
    MIN(TIMEST
    ----------
    19.05.2014
    
    In fact, the majority of the audit trail records are from a previous incarnation:
    SQL> select count(*) from aud$ where dbid = 848951741;
    
    COUNT(*)
    ----------
    33 612,411
    
    SQL> select DBID,MAX(NTIMESTAMP#)
    2  FROM AUD$
    3  GROUP BY DBID;
    
    DBID MAX(NTIMESTAMP#)
    ---------- --------------------------------
    2367413790 05.06.2014 08.42.59,749967
    848951741 15.05.2014 21.41.52,247344
    
    So the size of the AUD$ is 7481 MB:
    SQL> SELECT SUM(BYTES)/1024/1024 FROM DBA_SEGMENTS WHERE SEGMENT_NAME = 'AUD$';
    
    SUM(BYTES)/1024/1024
    --------------------
    7481
    
    Now the question is: since the procedure DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL with the parameter use_last_arch_timestamp set to TRUE only attempts to purge the rows from AUD$ that has the same DBID as the current database incarnation, will a "purge all" directive, indicated by use_last_arch_timestamp set to FALSE be equally selective? Since this is a test system, I tried it out by putting the following statements into a script:
    SET LINES 200
    SET SERVEROUTPUT ON
    
    SELECT DBID,COUNT(*)
    FROM AUD$
    GROUP BY DBID;
    
    BEGIN
    DBMS_AUDIT_MGMT.CLEAN_AUDIT_TRAIL(
    audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,
    use_last_arch_timestamp => FALSE);
    END;
    /
    
    SELECT DBID,COUNT(*)
    FROM AUD$
    GROUP BY DBID;
    
    Execute it:
    sqlplus / as sysdba @clean_all_audit_trail.sql
    
    Result:
    DBID   COUNT(*)
    ---------- ----------
    2367413790      52560
    848951741   33612411
    
    PL/SQL-procedure executed.
    
    No rows selected.
    
    
    SQL> SELECT SUM(BYTES)/1024/1024 FROM DBA_SEGMENTS WHERE SEGMENT_NAME = 'AUD$';
    
    SUM(BYTES)/1024/1024
    --------------------
    ,0625
    

    So a "purge all" directive will certainly wipe out all of your audit trail, regardless of the presence of multiple DBIDs.

    Purging "up until a the last archive timestamp" will only select the audit entries for your current database incarnation's DBID.

    Audit parameters. What do they mean - short description



    When managing your audit trail, you will need to be familiar with these settings. Here is a short description of what they mean.
    SQL> SELECT NAME, VALUE FROM V$PARAMETER WHERE NAME LIKE '%audit%';
    

    NAME VALUE
    audit_sys_operations TRUE
    audit_file_dest /u01/oracle/admin/slyt/adump
    audit_syslog_level  
    audit_trail DB_EXTENDED

  • AUDIT_TRAIL
    When set to "DB", Oracle directs audit records to the database audit trail (the SYS.AUD$ table), except for mandatory and SYS audit records, which are always written to the operating system audit trail.

    When set to "DB,EXTENDED", oracle behaves as it would when AUDIT_TRAIL=DB, but also enables you to capture the SQL statement used in the action that was audited.

    When set to "NONE", standard auditing is disabled

    When set to "OS", Oracle directs all audit records to an operating system file.

    When set to "XML", Oracle writes to the operating system audit record file in XML format

    When set to "XML, EXTENDED", oracle behaves as it would with "AS AUDIT_TRAIL=XML", but also includes SQL text and SQL bind information

  • AUDIT_SYS_OPERATIONS
    Enables or disables the auditing of top-level operations, which are SQL statements directly issued by users when connecting with SYSDBA or SYSOPER privileges.

  • AUDIT_FILE_DEST
    specifies the operating system directory into which the audit trail is written when the AUDIT_TRAIL initialization parameter is set to os, xml, or xml,extended. It is also the location to which mandatory auditing information is written and, if so specified by the AUDIT_SYS_OPERATIONS initialization parameter, audit records for user SYS.

  • AUDIT_SYSLOG_LEVEL
    Allows SYS and standard OS audit records to be written to the system audit log using the SYSLOG utility


    Note: when you set the audit_trail parameter in the spfile, DO NOT use qotation marks around the values: Incorrect:
    alter system set audit_trail='DB,EXTENDED' scope=spfile;
    
    ORA-00096: invalid value DB,EXTENDED for parameter audit_trail, must be from among extended, xml, none, os, db
    

    Correct:
    alter system set audit_trail=db,extended scope=spfile;
    
    System altered.
    
  • Tuesday, June 3, 2014

    How to use the slibclean utility on AIX to clean up unused modules in memory

    On AIX, oracle will leave links to library usage in memory.

    This can prevent you from for example applying a patch using opatch.
    To clean up, use the slibclean utility, as demonstrated below:

    First, check for unused links by using the genld and genkld utilities:
    root@myserver> genld -l | grep /u01/oracle/product/11204
    root@myserver> genkld | grep /u01/oracle/product/11204
    
    Output will be similar to the following:
    900000002fb6000    141a3 /u01/oracle/product/11204/lib/libdbcfg11.so
    900000002f9e000    17eb6 /u01/oracle/product/11204/lib/libclsra11.so
    9000000036f6000   20ef63 /u01/oracle/product/11204/lib/libocrb11.so
    900000002efd000    a0f25 /u01/oracle/product/11204/lib/libocr11.so
    900000003055000   6a0dc1 /u01/oracle/product/11204/lib/libhasgen11.so
    900000002dcf000     cb95 /u01/oracle/product/11204/lib/libocrutl11.so
    900000002dcd000     1d7d /u01/oracle/product/11204/lib/libskgxn2.so
    900000010ba7000  2ddd1de /u01/oracle/product/11204/lib/libttsh11.so
    900000002a64000    1701f /u01/oracle/product/11204/lib/libons.so
    900000002a7c000   3508c9 /u01/oracle/product/11204/lib/libnnz11.so
    90000000cb99000  400d9b5 /u01/oracle/product/11204/lib/libolapapi11.so
    900000002173000     c05e /u01/oracle/product/11204/lib/libcorejava.so
    90000000211d000    5597d /u01/oracle/product/11204/lib/libxdb.so
    900000002007000     125f /u01/oracle/product/11204/lib/libodm11.so
    900000001f37000    cf4de /u01/oracle/product/11204/lib/libskgxp11.so
    
    Then clean up:
    root@myserver> slibclean
    
    Links are now gone:
    root@myserver> genkld | grep /u01/oracle/product/11204
    root@myserver>
    

    How to check progress of a long running statistics gathering job

    If you have a long-running statistics job running, you can check it from v$session_longops:

    For example, you execute:
    SQL> EXECUTE dbms_stats.gather_dictionary_stats;
    PL/SQL procedure successfully completed.
    

    Check progress with:
    SQL> select sofar, totalwork,units,start_time,time_remaining,message  
         from v$session_longops
         where opname = 'Gather Dictionary Schema Statistics';
    
    SOFAR  TOTALWORK UNITS                START_TIM TIME_REMAINING MESSAGE
    ---------- ---------- -------------------- --------- -------------- ------------------------------------------------
           423        423 Objects              03-JUN-14              0 Gather Dictionary Schema Statistics: Dictionary Schema : 423 out of 423 Objects done
    

    Monday, June 2, 2014

    How to solve "unknown nfs status return value: -1" when trying to mount an NFS drive on AIX on Linux

    My error:
    [root@mylinuxserver/]# mount -o nfsvers=2 -o nolock mynfsserver:/u01/software /myshare
    mount: mynfsserver:/u01/software failed, reason given by server: unknown nfs status return value: -1
    
    Solution: Add the client's ip-address and the name to the NFS servers /etc/hosts file, then retry the mount command.

    Share is now mounted and usable:
    [root@mylinuxserver/]# df -h -F nfs
    Filesystem            Size  Used Avail Use% Mounted on
    mynfsserver:/u01/software     705G  654G   52G  93% /myshare
    
    Apparently the error can also be overcome by making sure the client is set up with "reversed lookup" in your DNS server.
    In other words, nslookup should be able to resolve both the hostname of the client and the ip address of the client.

    How to display Linux kernel and version information

    List full distribution name:
    [root@myserver/]# cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 5.2 (Tikanga)
    
    List kernel version:
    [root@myserver/]# cat /proc/version
    Linux version 2.6.18-92.el5xen (brewbuilder@ls20-bc2-13.build.redhat.com) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:31:30 EDT 2008
    
    List processor type:
    [root@myserver/]# uname -p
    x86_64
    
    List all:
    root@myserver/]# uname -a
    Linux myserver.mydomain.com 2.6.18-92.el5xen #1 SMP Tue Apr 29 13:31:30 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
    

    From RHEL 7, you can use "hostnamectl":
    [root@myhost ~]# hostnamectl
       Static hostname: myhost.mydomain.com
             Icon name: computer-vm
               Chassis: vm
            Machine ID: ******
               Boot ID: ******
        Virtualization: vmware
      Operating System: Red Hat Enterprise Linux Server 7.8 (Maipo)
           CPE OS Name: cpe:/o:redhat:enterprise_linux:7.8:GA:server
                Kernel: Linux 3.10.0-1127.19.1.el7.x86_64
          Architecture: x86-64
    
    The contributor slm gives a good explaination on the background for hostnamectl in this article on stackexchange.com