How to see the number of huge pages configured on a Linux server

On a RHEL server, to see if hugespages have been configured, you can use

cat /proc/meminfo|grep -iE 'tables|huge'

PageTables:        33368 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    5121
HugePages_Free:        9
HugePages_Rsvd:        9
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:        10487808 kB

From the above, we can see that the total amount of huge pages is 5121.
Each page is 2M in size, as can be seen from Hugepagesize.
The total size of hugepages amounts to 10242M, or 10G.

Frank Pachot has written a really neat pice of code to format the output from sysctl to display huge pages usage on a Linux server.

awk '/Hugepagesize:/{p=$2} / 0 /{next} / kB$/{v[sprintf("%9d GB %-s",int($2/1024/1024),$0)]=$2;next} {h[$0]=$2} /HugePages_Total/{hpt=$2} /HugePages_Free/{hpf=$2} {h["HugePages Used (Total-Free)"]=hpt-hpf} END{for(k in v) print sprintf("%-60s %10d",k,v[k]/p); for (k in h) print sprintf("%9d GB %-s",p*h[k]/1024/1024,k)}' /proc/meminfo|sort -nr|grep --color=auto -iE "^|( HugePage)[^:]*"

The output is much easier to draw conclusions from, and it gives quite a lof of other useful information about your system, too:

32767 GB VmallocTotal:   34359738367 kB                    16777215
24 GB CommitLimit:    26025928 kB                          12707
19 GB SwapTotal:      20971516 kB                          10239
19 GB SwapFree:       20969176 kB                          10238
19 GB MemTotal:       20596632 kB                          10056
12 GB DirectMap1G:    12582912 kB                           6144
10 GB Hugetlb:        10487808 kB                           5121
10 GB HugePages_Total:    5121
9 GB HugePages Used (Total-Free)
9 GB DirectMap2M:    10082304 kB                           4923
7 GB MemAvailable:    8173656 kB                           3991
6 GB Cached:          6329396 kB                           3090
5 GB Inactive:        5862704 kB                           2862
4 GB Inactive(file):  4679556 kB                           2284
2 GB Committed_AS:    2912096 kB                           1421
1 GB MemFree:         1835428 kB                            896
1 GB Inactive(anon):  1183148 kB                            577
1 GB AnonPages:       1056944 kB                            516
1 GB Active(file):    1502496 kB                            733
1 GB Active:          1514284 kB                            739
0 GB VmallocUsed:       30708 kB                             14
0 GB Unevictable:       12376 kB                              6
0 GB SwapCached:          432 kB                              0
0 GB SUnreclaim:       101008 kB                             49
0 GB SReclaimable:     375752 kB                            183
0 GB Slab:             476760 kB                            232
0 GB Shmem:            139612 kB                             68
0 GB Percpu:             4960 kB                              2
0 GB PageTables:        33448 kB                             16
0 GB Mlocked:           12376 kB                              6
0 GB Mapped:           322284 kB                            157
0 GB KReclaimable:     375752 kB                            183
0 GB KernelStack:        6080 kB                              2
0 GB HugePages_Surp:        0
0 GB HugePages_Rsvd:        9
0 GB Hugepagesize:       2048 kB                              1
0 GB HugePages_Free:        9
0 GB Dirty:              1272 kB                              0
0 GB DirectMap4k:      403328 kB                            196
0 GB Buffers:            2852 kB                              1
0 GB Active(anon):      11788 kB                              5

Solution for [WARNING] ORA-01081: cannot start already-running ORACLE - shut it down first when running dbca

I received the following error multiple times, when executing dbca like below:

dbca -createDatabase -responsefile ./cdb.rsp -silent
 2023-09-14 15:05:45.435 CEST ] Prepare for db operation
DBCA_PROGRESS : 10%
[ 2023-09-14 15:05:45.505 CEST ] Copying database files
DBCA_PROGRESS : 12%
[ 2023-09-14 15:05:47.250 CEST ] [WARNING] ORA-01081: cannot start already-running ORACLE - shut it down first

DBCA_PROGRESS : 40%
DBCA_PROGRESS : 100%
[ 2023-09-14 15:05:47.616 CEST ] [FATAL] Recovery Manager failed to restore datafiles. Refer logs for details.
DBCA_PROGRESS : 10%
DBCA_PROGRESS : 0%

At first I thought it was because I had forgotten to terminate some processes owned by oracle. And I also made sure that /etc/oratab was empty before I started.

Reason:

Turns out, there were old memory segments left behind that dbca considers a part of a running instance. Even if

ps -fu oracle

gave no processes in the list.

Solution:

su - oracle
sqlplus / as sysdba
shutdown abort

Given that you've set ORACLE_SID to the same value you used in your response file, this will properly shut down your instance, and clean up the memory segments left behind from the previous run.

How to find the number of huge pages to set on a PostgreSQL server

Thanks to Ibrar Ahmed for posting the article "Tune Linux Kernel Parameters For PostgreSQL Optimization"

Log on to the server as the os user that owns the PostgreSQL server. On my server, this user is called "postgres":

su - postgres

Create a file called find_hp.sh. Insert the following:

#!/bin/bash
pid=`head -1 $PGDATA/postmaster.pid`
echo "Pid:            $pid"
peak=`grep ^VmPeak /proc/$pid/status | awk '{ print $2 }'`
echo "VmPeak:            $peak kB"
hps=`grep ^Hugepagesize /proc/meminfo | awk '{ print $2 }'`
echo "Hugepagesize:   $hps kB"
hp=$((peak/hps))
echo Set Huge Pages:     $hp

Make sure the environment variable $PGDATA is set. Give the script execution rights:

chmod 755 find_hp.sh

Execute it, and it will tell you how many huge pages you need:

 ./find_hp.sh
Pid:             128678
VmPeak:          68986484 kB
Hugepagesize:    2048 kB
Set Huge Pages:  33684

I can now proceed to allow for 33684 huge pages on my system with

sysctl -w vm.nr_hugepages=33684

How to solve ORA-02017: integer value required when trying to adjust parameters

I was trying to rectify an issue where the alert log of the database kept throwing errors like this:

ORA-01013: user requested cancel of current operation
ORA-00600: internal error code, arguments: [KGL-heap-size-exceeded], [0x114890D28], [6], [1263979496], [], [], [], [], [], [], [], []

Reading Doc ID 330239.1 "Memory Notification: Library Cache Object loaded into SGA / ORA-600 [KGL-heap-size-exceeded]", I realized that these errors are not causing any failure, as they are simply warnings that Oracle implemented to alert DBAs about potential large occupants of database memory.

There are two hidden parameters which control these messages:

set lines 200
col name format a40
col description format a80
col KSPPSTVL format a20
 select
 nam.ksppinm NAME,
 nam.ksppdesc DESCRIPTION,
 val.KSPPSTVL
 from
 x$ksppi nam,
 x$ksppsv val
 where nam.indx = val.indx and nam.ksppinm like '%kgl_large_heap_%_threshold%';

NAME                                 DESCRIPTION                                                        KSPPSTVL
-----------------------------------  ------------------------------------------------------------------ --------------
_kgl_large_heap_warning_threshold    maximum heap size before KGL writes warnings to the alert log      524288000
_kgl_large_heap_assert_threshold     maximum heap size before KGL raises an internal error              524288000

These are the default values, 50M. Not very much. Instead, I want a warning to be written to the alert log at 1500M, and an ORA-00600 error to be thrown at 2G.

The first instruction works just fine:

alter system set "_kgl_large_heap_warning_threshold"=1572864000 scope=spfile;

System altered.

The secoond fails with:

alter system set "_kgl_large_heap_assert_threshold"=2147483648 scope=spfile;

ERROR at line 1:
ORA-02017: integer value required

Why?

The answer was found in the document Doc ID 2049098.1 Setting Parameter SORT_AREA_SIZE Throws error ORA-2017: "Integer Value Required", which states

"The parameter SORT_AREA_SIZE has a 2G limit"

and points out that instead of setting the value to 2147483648 bytes, it should be lowered to a value below 2G.

So the solution was to reduce the value by one single byte, from

alter system set "_kgl_large_heap_assert_threshold"=2147483648 scope=spfile;

to

alter system set "_kgl_large_heap_assert_threshold"=2147483647 scope=spfile;

and Oracle happily accepted the new value.

How to solve ORA-27102: out of memory on startup

You want to increase your SGA to a larger value, and you have taken care to increase the value of shmmax in the /etc/sysctl.conf file.

When starting up the database with increased values for sga_max_size and sga_target, you hit

SQL> startup
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
Additional information: 1543503872
Additional information: 1

Solution:
Increase the value of shmall as well. Shmall indicates the total amount of shared memory that the system can use at one time (measured in pages).

To find the page size:

getconf PAGE_SIZE
4096

Make sure that the shmall is larger, not identical, than the setting of your SGA.
For example, if you want a 12G SGA, do not set shmall to 3145728 which would be the number of pages for 12G

Set it to a value larger instead, for example 3670016, which is the number of pages equivalent to 14G:

3670016*4096=15032385536
15032385536/1024/1024/1024 = 14G

As root, open /etc/sysctl.conf in an editor.
Search for the string kernel.shmall
Change the value
Save and exit.

Then, still as root, execute

sysctl -p

Check that the new settings are in place:

/sbin/sysctl -a | grep shm

The database should now be possible to open.

How to check memory consumption and swapping on a Linux server

The easiest way to check for memory consumption on Linux is in my opinion the "free" utility:

oracle@prodserver1:[proddb01]# free -m
             total       used       free     shared    buffers     cached
Mem:         16056       9027       7029          0        233       1093
-/+ buffers/cache:       7700       8356
Swap:         8191       2674       5517

Add totals with the -t flag:

 free -t -m
             total       used       free     shared    buffers     cached
Mem:         16056       9029       7027          0        233       1093
-/+ buffers/cache:       7701       8354
Swap:         8191       2674       5517
Total:       24248      11703      12545

So in this case I am using 7701 MB of memory, while 8354 MB is free for use.

To get a quick glance, you can have a look at the file /proc/meminfo, grep for the highlights, like this:

 egrep  'Mem|Cache|Swap' /proc/meminfo
MemTotal:       16442312 kB
MemFree:         7255224 kB
Cached:          1120380 kB
SwapCached:        45632 kB
SwapTotal:       8388600 kB
SwapFree:        5650100 kB

A good option for finding out if a server is swapping, is by using vmstat:

oracle@myserver# vmstat -w 10 4
procs -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
 r  b         swpd         free         buff        cache   si   so    bi    bo   in   cs  us  sy  id  wa  st
 3  0      2772764       659396        20340     10764836    4    4    19     9    3    4  13   6  81   0   0
 6  0      2771740       735076        20364     10766444   82    0    82     7 68360 52089  57  30  13   0   0
10  0      2771484       767816        20376     10766628    8    0    12     4 67349 52155  55  31  15   0   0
 7  0      2771228       747480        22832     10768888   34    0   495    12 67799 52119  57  30  13   0   0

where * The flag -w indicates wide output * Sample every 10 second * 4 iterations Columns: * si: Amount of memory swapped in from disk (/s) * so: Amount of memory swapped to disk (/s) When a lot of negative swapping occurs the value of "so" (swap out) is increasing.