Showing posts with label automated maintenance tasks. Show all posts
Showing posts with label automated maintenance tasks. Show all posts

Wednesday, August 7, 2024

How to find a failed job in the data dictionary tables

My adrci output showed a failed job (abbreviated):
ADR Home = /u01/oracle/diag/rdbms/mydb01/mydb01:
*************************************************************************
INCIDENT_ID          PROBLEM_KEY                                                 CREATE_TIME
-------------------- ----------------------------------------------------------- -------------------------
9720716              ORA 600 [kdxolrs2_comp: samekey]                            2024-08-06 17:37:32.674000 +02:00
9720796              ORA 600 [kdxolrs2_comp: samekey]                            2024-08-06 17:37:24.616000 
2 rows fetched (*** more available ***)

adrci> show incident -mode detail -p "incident_id=9720716"

   INCIDENT_ID                   9720716
   STATUS                        ready
   CREATE_TIME                   2024-08-06 17:37:32.674000 +02:00
   PROBLEM_ID                    21
   PROBLEM_KEY                   ORA 600 [kdxolrs2_comp: samekey]
   FIRSTINC_TIME                 2024-08-06 17:37:24.616000 +02:00
   LASTINC_TIME                  2024-08-06 17:37:32.674000 +02:00
   KEY_NAME                      PQ
   KEY_NAME                      Service
   KEY_VALUE                     SYS$USERS
   KEY_NAME                      Module
   KEY_VALUE                     DBMS_SCHEDULER
   KEY_NAME                      Action
   KEY_VALUE                     ORA$AT_OS_OPT_SY_125693
   OWNER_ID                      1
   INCIDENT_FILE                 /u01/oracle/diag/rdbms/mydb01/mydb01/incident/incdir_9720716/mydb01_p007_11096_i9720716.trc
   OWNER_ID                      1
   INCIDENT_FILE                 /u01/oracle/diag/rdbms/mydb01/mydb01/trace/mydb01_p007_11096.trc
   OWNER_ID                      1
   INCIDENT_FILE                 /u01/oracle/diag/rdbms/mydb01/mydb01/incident/incdir_9720716/mydb01_mz00_1311372_i9720716_a.trc
1 row fetched

The file mydb01_p007_11096_i9720716.trc shows:
*** MODULE NAME:(DBMS_SCHEDULER) 2024-08-06T17:37:32.699564+02:00
*** ACTION NAME:(ORA$AT_OS_OPT_SY_125693) 2024-08-06T17:37:32.699567+02:00

[TOC00002]
========= Dump for incident 9720716 (ORA 600 [kdxolrs2_comp: samekey]) ========

*** 2024-08-06T17:37:32.709272+02:00
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
[TOC00003]
----- Current SQL Statement for this session (sql_id=82xfc2m10quym) -----
select /*+ opt_param('_optimizer_use_auto_indexes' 'on')  
parallel_index(t, "mypk",8)  
dbms_stats cursor_sharing_exact use_weak_name_resl dynamic_sampling(0) 
no_monitoring xmlindex_sel_idx_tbl 
opt_param('optimizer_inmemory_aware' 'false') 
no_substrb_pad  no_expand index_ffs(t,"mypk") */ 
count(*) as nrw,approx_count_distinct(sys_op_lbid(632968,'L',t.rowid)) as nlb,null as ndk,sys_op_countchg(substrb(t.rowid,1,15),1) 
as clf from "myuser"."mytable" t where "ID" is not null
[TOC00003-END]
To find the record in the data dictionary holding the history for this run, use the following query:
set lines 300
col client_name format a30
col job_name format a30
col job_status format a20
col job_start_time format a30
col job_duration format a20
col job_info format a80

SELECT client_name,job_name,job_status,to_char(job_start_time,'dd.mm.yyyy hh24:mi:ss') "job_start_time" ,job_duration,job_info
FROM  dba_autotask_job_history 
WHERE JOB_NAME = 'ORA$AT_OS_OPT_SY_125693';
CLIENT_NAME JOB_NAME JOB_STATUS job_start_time JOB_DURATION JOB_INFO
auto optimizer stats collection ORA$AT_OS_OPT_SY_125693 FAILED 06.08.2024 17:00:02 +000 00:37:34 ORA-12801: error signaled in parallel query server P007 ORA-00600: internal error code, arguments: [kdxolrs2_comp: samekey], [0], [0], [0], [0], [], [], [], [], [], [], []
Further details can be seen in the view DBA_OPTSTAT_OPERATION_TASKS, if necessary.

Tuesday, May 18, 2021

How to find the opening and closing time for the maintenance windows in an Oracle database

To see the last 7 days of opening/closing time for the maintenance windows, use this query:
SELECT window_name,to_char(start_time,'dd.mm.yyyy hh24:mi:ss') "start time", duration 
FROM dba_autotask_schedule 
ORDER BY start_time desc fetch first 7 rows only;
Example output:
WINDOW_NAME start time DURATION
SATURDAY_WINDOW 19.06.2021 06:00:00 +00 20:00:00.000000
FRIDAY_WINDOW 18.06.2021 22:00:00 +00 04:00:00.000000
THURSDAY_WINDOW 17.06.2021 22:00:00 +00 04:00:00.000000
WEDNESDAY_WINDOW 16.06.2021 22:00:00 +00 04:00:00.000000
TUESDAY_WINDOW 15.06.2021 22:00:00 +00 04:00:00.000000
MONDAY_WINDOW 14.06.2021 22:00:00 +00 04:00:00.000000
SUNDAY_WINDOW 13.06.2021 06:00:00 +00 20:00:00.000000

Thursday, April 19, 2018

How do I find which objects that were analyzed during an autostats job?


Step 1

Find the start and stop times of the maintenance window:
select window_name, cast(last_start_date as date) "Start Time", DURATION, cast(last_start_date+duration as date) "End Time"
FROM DBA_SCHEDULER_WINDOWS
WHERE ENABLED='TRUE'
AND WINDOW_NAME LIKE UPPER('%&&day%');

Which results in:
WINDOW_NAME Start Time DURATION End Time
WEDNESDAY_WINDOW 18.04.2018 23:59:00 +00 04:00:00.000000 19.04.2018 03:59:00

So my Wednesday window starts at 23:59 and ends 4 hours later, at 03:59.

Step 2
Add the query above as a subquery in a query against DBA_OPTSTAT_OPERATIONS, to get all the IDs for the sub-steps that was executed during this window:
select ID
from DBA_OPTSTAT_OPERATIONS 
where operation='gather_database_stats (auto)' 
and CAST(start_time AS DATE) BETWEEN 
                                ( SELECT CAST (LAST_START_DATE AS DATE)
                                  FROM  DBA_SCHEDULER_WINDOWS
                                  WHERE ENABLED='TRUE'
                                  AND   WINDOW_NAME LIKE UPPER('%&&day%')
                                 )
                              AND 
                                 (
                                  SELECT CAST (LAST_START_DATE+DURATION AS DATE)
                                  FROM  DBA_SCHEDULER_WINDOWS
                                  WHERE ENABLED='TRUE'
                                  AND   WINDOW_NAME LIKE UPPER('%&&day%')
                                 )
;

Step 3
Put everything together by querying DBA_OPTSTAT_OPERATION_TASKS, to see which tasks and which objects that were analyzed during the time window:

-- get the details for every task executed by "gather_database_stats (auto)" 
SELECT opid,
       target,
       target_objn,
       target_type,
       target_size,
       cast(start_time as date) "Start Time",
       cast(end_time as date) "End Time", 
       status
FROM DBA_OPTSTAT_OPERATION_TASKS 
WHERE OPID IN ( -- get the unique ID for the "gather_database_stats (auto)" jobs
                SELECT ID FROM DBA_OPTSTAT_OPERATIONS 
                WHERE OPERATION='gather_database_stats (auto)' 
                AND CAST (START_TIME AS DATE) BETWEEN 
                                                -- get the start and stop times for the last window
                                                -- for a particular weekday
                                                (SELECT CAST(LAST_START_DATE AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
                                                AND (
                                                      SELECT CAST(LAST_START_DATE+DURATION AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
)
order by CAST(start_time as DATE) asc;

Sample output (displaying the first and last 5 rows of the record set):
OPID TARGET TARGET_OBJN TARGET_TYPE TARGET_SIZE Start Time End Time STATUS
48848
SYS.ET$16E30DF80001
6177051
TABLE
0
18.04.2018 23:59:30 18.04.2018 23:59:30 FAILED
48848
SYS.I_SEQ1
79
INDEX
1
18.04.2018 23:59:30 18.04.2018 23:59:31 COMPLETED
48848
SYS.SEQ$
74
TABLE
7
18.04.2018 23:59:30 18.04.2018 23:59:31 COMPLETED
48848
SYS.OPTSTAT_HIST_CONTROL$
506
TABLE
1
18.04.2018 23:59:31 18.04.2018 23:59:31 COMPLETED
48848
SYS.REG$
5775
TABLE
1
18.04.2018 23:59:31 18.04.2018 23:59:31 COMPLETED
...
48885
SYS.WRH$_SERVICE_STAT.WRH$_SERVICE_STAT_2490506650_826
11986128
TABLE PARTITION
5
19.04.2018 02:00:48 19.04.2018 02:00:48 COMPLETED
48885
SYS.WRH$_SEG_STAT_PK.WRH$_SEG_STAT_2490506650_826
11986126
INDEX PARTITION
1
19.04.2018 02:00:48 19.04.2018 02:00:48 COMPLETED
48885
SYS.WRH$_SGASTAT_U.WRH$_SGASTAT_2490506650_826
11986114
INDEX PARTITION
1
19.04.2018 02:00:48 19.04.2018 02:00:48 COMPLETED
48885
SYS.WRH$_SEG_STAT.WRH$_SEG_STAT_2490506650_826
11986124
TABLE PARTITION
5
19.04.2018 02:00:48 19.04.2018 02:00:48 COMPLETED
48885
SYS.WRH$_SEG_STAT.WRH$_SEG_STAT_2490506650_826
11986124
TABLE PARTITION
5
19.04.2018 02:00:48 19.04.2018 02:00:48 COMPLETED

Check the overall status of your run:
SELECT target_type,status,count(*)
FROM DBA_OPTSTAT_OPERATION_TASKS 
WHERE OPID IN ( -- get the unique ID for the "gather_database_stats (auto)" jobs
                SELECT ID FROM DBA_OPTSTAT_OPERATIONS 
                WHERE OPERATION='gather_database_stats (auto)' 
                AND CAST(START_TIME AS DATE) BETWEEN 
                                                -- get the start and stop times for the last window
                                                -- for a particular weekday
                                                (SELECT CAST(LAST_START_DATE AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
                                                AND (
                                                      SELECT CAST(LAST_START_DATE+DURATION AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
)
group by target_type,status
order by status;

Result:
TARGET_TYPE STATUS COUNT(*)
INDEX COMPLETED
501
INDEX PARTITION COMPLETED
177
TABLE COMPLETED
394
TABLE PARTITION COMPLETED
362
TABLE FAILED
54


To further filter out those objecs that failed, use:
SELECT distinct target,target_objn,target_type
FROM DBA_OPTSTAT_OPERATION_TASKS 
WHERE OPID IN ( -- get the unique ID for the "gather_database_stats (auto)" jobs
                SELECT ID FROM DBA_OPTSTAT_OPERATIONS 
                WHERE OPERATION='gather_database_stats (auto)' 
                AND CAST(START_TIME AS DATE) BETWEEN 
                                                -- get the start and stop times for the last window
                                                -- for a particular weekday
                                                (SELECT CAST(LAST_START_DATE AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
                                                AND (
                                                      SELECT CAST(LAST_START_DATE+DURATION AS DATE)
                                                      FROM DBA_SCHEDULER_WINDOWS
                                                      WHERE ENABLED='TRUE'
                                                      AND WINDOW_NAME LIKE UPPER('%&&day%')
                                                     )
)
AND status = 'FAILED'
order by TARGET;

Which in my case revealed that the following objects need attention:

TARGET TARGET_OBJN TARGET_TYPE
SYS.ET$16E30DF80001
6177051
TABLE
SYS.X$LOGMNR_CONTENTS
4294951535
TABLE
SYS.X$LOGMNR_REGION
4294951631
TABLE

Wednesday, February 8, 2017

How to check if the "gather_database_stats" automatic maintenance jobs has succseeded

select id,operation,to_char(start_time,'dd.mm.yyyy hh24:mi:ss') "starttime",
                        to_char(end_Time,'dd.mm.yyyy hh24:mi:ss') "endtime",
                        to_dsinterval(to_char(end_time-start_Time,'dd.mm.yyyy hh24:mi:ss')) "elapsed",
                        status,job_name
from DBA_OPTSTAT_OPERATIONS
where operation = 'gather_database_stats (auto)'
order by start_time desc
fetch first 5 rows only;

Gave the following result:
ID OPERATION starttime endtime elapsed STATUS JOB_NAME
58778
gather_database_stats (auto) 07.02.2017 22:00:02 08.02.2017 02:00:15 +00 04:00:13.178197 TIMED OUT ORA$AT_OS_OPT_SY_1965
58585
gather_database_stats (auto) 06.02.2017 22:00:06 07.02.2017 02:00:24 +00 04:00:18.089479 TIMED OUT ORA$AT_OS_OPT_SY_1963
58273
gather_database_stats (auto) 05.02.2017 06:00:04 06.02.2017 02:00:26 +00 20:00:21.968028 TIMED OUT ORA$AT_OS_OPT_SY_1950
58088
gather_database_stats (auto) 04.02.2017 06:00:02 05.02.2017 02:00:32 +00 20:00:29.696085 TIMED OUT ORA$AT_OS_OPT_SY_1945
58038
gather_database_stats (auto) 03.02.2017 22:00:02 04.02.2017 02:00:17 +00 04:00:15.314408 TIMED OUT ORA$AT_OS_OPT_SY_1943

The output indicates that the Automatic maintenance job never get to finish before the maintenance Windows Closes.

You can examine another view, dba_opstat_operation_tasks, to see the details on what the job didn't have time to do before the window closed:

select  opid, target,target_type, 
        to_char(start_Time,'dd.mm.yyyy hh24:mi:ss') "starttime",
        to_char(end_time,'dd.mm.yyyy hh24:mi:ss') "endtime"
        ,status
from dba_optstat_operation_tasks 
where opid=58778
order by start_Time desc
;

Sample output:
OPID TARGET TARGET_TYPE starttime endtime STATUS
58778
SCOTT.TABLE1.SYS_SUBP25275 TABLE 08.02.2017 02:00:11 08.02.2017 02:00:11 TIMED OUT
58778
SCOTT.TABLE1.SYS_SUBP25267 TABLE 08.02.2017 02:00:11 08.02.2017 02:00:11 TIMED OUT

Notice how the starttime and endtime is identical; this is because the window Closes, and the job aborts. Oracle then records this timestamp as the start time and end time for the Objects it never had time to analyze.


The Predefined Maintenance Windows in Oracle 12c

Oracle documentation states:

"By default there are seven predefined maintenance windows, each one representing a day of the week. The weekend maintenance windows, SATURDAY_WINDOW and SUNDAY_WINDOW, are longer in duration than the weekday maintenance windows. The window group MAINTENANCE_WINDOW_GROUP consists of these seven windows."

Verify by executing:
SELECT WINDOW_NAME,REPEAT_INTERVAL,DURATION,to_char(LAST_START_DATE,'dd.mm.yyyy hh24:mi') "laststart",
    to_char(NEXT_START_DATE,'dd.mm.yyyy hh24:mi') "nextstart",ENABLED "window enabled?",ACTIVE "window open?"
FROM DBA_SCHEDULER_WINDOWS 
WHERE RESOURCE_PLAN='DEFAULT_MAINTENANCE_PLAN';

WINDOW_NAME REPEAT_INTERVAL DURATION laststart nextstart window enabled? window open?
SUNDAY_WINDOW freq=daily;byday=SUN;byhour=6;byminute=0; bysecond=0 +00 20:00:00.000000 05.02.2017 06:00 12.02.2017 06:00 TRUE FALSE
SATURDAY_WINDOW freq=daily;byday=SAT;byhour=6;byminute=0; bysecond=0 +00 20:00:00.000000 04.02.2017 06:00 11.02.2017 06:00 TRUE FALSE
FRIDAY_WINDOW freq=daily;byday=FRI;byhour=22;byminute=0; bysecond=0 +00 04:00:00.000000 03.02.2017 22:00 10.02.2017 22:00 TRUE FALSE
THURSDAY_WINDOW freq=daily;byday=THU;byhour=22;byminute=0; bysecond=0 +00 04:00:00.000000 02.02.2017 22:00 09.02.2017 22:00 TRUE FALSE
WEDNESDAY_WINDOW freq=daily;byday=WED;byhour=22;byminute=0; bysecond=0 +00 04:00:00.000000 01.02.2017 22:00 08.02.2017 22:00 TRUE FALSE
TUESDAY_WINDOW freq=daily;byday=TUE;byhour=22;byminute=0; bysecond=0 +00 04:00:00.000000 07.02.2017 22:00 14.02.2017 22:00 TRUE FALSE
MONDAY_WINDOW freq=daily;byday=MON;byhour=22;byminute=0; bysecond=0 +00 04:00:00.000000 06.02.2017 22:00 13.02.2017 22:00 TRUE FALSE

The weekend Windows starts at 6 in the morning and run for 20 hours.
The weekday Windows starts at 22:00 and runs until 02:00 in the morning.

An overview of some important data Dictionary views can be found here

Monday, November 14, 2016

A short history of the Automated Maintenance Tasks in Oracle

Predefined Automatic System Tasks were introduced in Oracle 10g, and consisted of two jobs:

1. Automated statistics collection
2. The Automated segment space advisor

Two scheduler jobs were set up by default to run these automated tasks: The GATHER_STATS_JOB and the AUTO_SPACE_ADVISOR_JOB.

You could enable or disable the jobs using the procedures enable or disable in the dbms_scheduler package:
EXECUTE DBMS_SCHEDULER.ENABLE('GATHER_STATS_JOB');
EXECUTE DBMS_SCHEDULER.DISABLE('GATHER_STATS_JOB');

In Oracle 11g, Oracle redesigned the the Automatic System Tasks and introduced the concept of Automated Maintenance Tasks.
At the same time, a third automated task was introduced: The Automated SQL Tuning advisor.

Oracle removed the GATHER_STATS_JOB and AUTO_SPACE_ADVISOR_JOB altogether and instead introduced a new package called DBMS_AUTO_TASK_ADMIN.

To enable or disable an Automatic tasks using DBMS_AUTO_TASK_ADMIN, see this short post.

To check the status of your automated maintenance tasks, see this post.

How to check the status of the Automated Maintenance Tasks in Oracle 11g

To see the currently enabled jobs in your database, use the dictionary view DBA_AUTOTASK_OPERATION
which displays all automated maintenance task operations for each client:
select client_name,operation_name,status  
from DBA_AUTOTASK_OPERATION;
Example from one of my 11g databases:
CLIENT_NAME OPERATION_NAME STATUS
auto optimizer stats collection auto optimizer stats job DISABLED
auto space advisor auto space advisor job ENABLED
sql tuning advisor automatic sql tuning task DISABLED

Thursday, February 25, 2016

How to enable and disable Automatic SQL tuning

Enable:
BEGIN
dbms_auto_task_admin.enable(client_name => 'sql tuning advisor', operation => NULL, window_name => NULL);
END;

Disable:
BEGIN
dbms_auto_task_admin.disable(client_name => 'sql tuning advisor', operation => NULL, window_name => NULL);
END;

If desirable, Oracle also lets you enable or disable all automated maintenance tasks for all windows. Do this by calling the ENABLE or DISABLE procedure with no arguments:
EXECUTE DBMS_AUTO_TASK_ADMIN.DISABLE;



Sources:
dbms_auto_task_admin
Configuring Automated Maintenance Tasks