Ivan Kartik - Oracle and Linux Blog

Installation of Oracle Database 12c Release 2 on Redhat 7, Oracle Linux 7

Here is short installation HOWTO for Oracle Database 12c Release 2 for Redhat 7, Oracle Linux 7 (and possibly CentOS 7 which is not officially certified and supported distribution). This post covers Oracle Database software installation (only). In case you want to use ASM or other Grid features and you don't have Grid Infrastructure 12c R2 installed refer to my previous post: http://ivan.kartik.sk/index.php?controller=post&action=view&id_post=76 If you don't need ASM or GI features you can continue with this part directly.
This post also contains some steps which were used during Oracle Grid Infrastructure 12c R2 installation, so you can skip those if you followed my GI installation HOWTO.

1. OS preparation

# - stands for "root" action
$ - stands for "oracle" (owner of Oracle software)

Note: You can skip this step if you have installed the Grid Infrastructure Software.

# groupadd dba
# groupadd oinstall
# useradd -g oinstall -G dba oracle

Create ORACLE_HOME directory. (Note: In my example I use /opt/oracle/12201 directory, change it to your preferred location):

# mkdir -p /opt/oracle/12201
# chown -R oracle:oinstall /opt/oracle

2. Install required packages

Note: You can skip this step completely once you have installed the Grid Infrastructure Software (as described in my previous post related to Grid Infrastructure installation).

First install required packages (use this command if your computer has access to Internet or you have registered and enabled your media as yum repository).

Add/set kernel parameters to/in /etc/sysctl.d/99-sysctl.conf file:

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Execute sysctl -p command to apply settings:

# sysctl -p

Add/set limits to/in /etc/security/limits.d/99-oracle-rdbms.conf file:

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728

4. Software download and install

Download installation archive (linuxx64_12201_database.zip) from http://www.oracle.com/technetwork/database/enterprise-edition/downloads/oracle12c-linux-12201-3608234.html

Unpack the installation archive and start the installer:

$ unzip linuxx64_12201_database.zip
$ cd database
$ ./runInstaller

Next steps after installation

In case you want startup your Oracle Database(s) automatically and you don't have Oracle Grid Infrastructure installed, you can follow the steps described in the post Configuring automatic startup of Oracle Database under systemd on RHEL 7/OEL 7/CentOS 7 at http://ivan.kartik.sk/index.php?controller=post&action=view&id_post=65

 

 

 

Bad bad systemd...

This post will be short...
I'm not a big fan of systemd and honestly I can't understand how this disease (which became an epidemy) could have happened to Linux. Sometimes I wish an eternal insomnia to person who is responsible for this (I am joking of course). Anyway systemd is (unfortunately) part of my OS and (despite numerous benefits) it often offers me surprising moments.

[root@ol7-01 ~]# reboot
Failed to start reboot.target: Connection timed out
See system logs and 'systemctl status reboot.target' for details.
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.

Of course the same result was produced by systemctl reboot, init 6, reboot -f -r now or even systemctl -f reboot

Last resort has been in using the deadly force in order to reboot the system:

[root@ol7-01 ~]# systemctl -ff reboot
Rebooting.

I used the "deadly force" term for "-ff" (you can use --force --force, or -f -f) because it's appropriate for logic behind this directive. Take a look what man page for systemd says:

"When used with halt, poweroff, reboot or kexec, execute the selected operation without shutting down all units. However, all processes will be killed forcibly and all file systems are unmounted or remounted read-only. This is hence a drastic but relatively safe option to request an immediate reboot. If --force is specified twice for these operations, they will be executed immediately without terminating any processes or unmounting any file systems. Warning: specifying --force twice with any of these operations might result in data loss."

So use this option as a last resort.

Installation of Oracle Grid Infrastructure 12c Release 2 on Redhat Linux 7 and Oracle Linux 7

I prepared short installation HOWTO Oracle Grid Infrastructure 12c Release 2 for Redhat 7, Oracle Linux 7 (and Centos 7). Installation on first two (supported) Linux distros was almost fine, just some workaround were needed. But you can expect some problems on (unsupported) Centos distro where some workarounds were needed in order to have fully working product. These specific workarounds you can at the end of this post. This post also tries to answer some potential questions you may have.

Oracle 12cR2 brought interesting new feature called Oracle ASM Filter Driver (ASMFD) which is in fact replacement for ASMLIB. As official documentation states ASMFD simplifies configuration and management of disk devices for Oracle ASM (just like AMSLIB does) moreover ASMFD is using a "Filter Driver" that rejects any invalid I/O request in order to protect ASM disks by filtering out non-Oracle I/O operations which could cause accidental overwrites thus corruption. Note that ASMFD cannot be used simultaneously with ASMLIB.

So (logically) this article covers installation of Oracle GI 12.2.0.1 aka 12cR2 for x86-64 Linux using ASMFD feature. I've used "Minimal" version of Linux installation.

1. OS preparation

# - stands for "root" user action
$ - stands for "oracle" user (owner of Oracle software) action

# groupadd dba
# groupadd oinstall
# useradd -g oinstall -G dba oracle

Create $GI_HOME directory. (Note: In my example I use /opt/oraclecrs/12201 directory, change it to your preferred location):

# mkdir -p /opt/oraclecrs/12201
# chown -R oracle:oinstall /opt/oraclecrs

2. Install required packages

Install required packages (use this command if your computer has access to Internet or you have registered and enabled your media as yum repository). Note: Some packages are required by Database installation which I expect it will be the next step once Oracle GI has been installed. If you are using Oracle Linux or in case of Redhat if have configured OL public YUM repository then you can use "yum install oracle-database-server-12cR2-preinstall" and it will install all required packages and also perform Linux parameters configuration described in next step (So, you can skip this step and step#3 as well). Note, in case of Redhat this package will install also UEK kernel.

# yum install unzip xorg-x11-utils xorg-x11-apps xorg-x11-utils compat-libcap1 libstc++-devel \ 
gcc-c++ glibc-devel nfs-utils-1.2.3-15 ksh libaio-devel sysstat smartmontools

3. Set OS kernel parameters and user limits

Add/set kernel parameters to/in /etc/sysctl.d/99-sysctl.conf

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

Execute sysctl -p command:

# sysctl -p

Add/set limits to/in /etc/security/limits.d/99-oracle-rdbms.conf:

oracle   soft   nofile    1024
oracle   hard   nofile    65536
oracle   soft   nproc    16384
oracle   hard   nproc    16384
oracle   soft   stack    10240
oracle   hard   stack    32768
oracle   hard   memlock    134217728
oracle   soft   memlock    134217728

4. Software download and deploy

Create install directory in /home/oracle for downloaded software:

$ mkdir /home/oracle/install

Download linuxx64_12201_grid_home.zip file from http://www.oracle.com/technetwork/database/enterprise-edition/downloads/oracle12c-linux-12201-3608234.html to the /home/oracle/install directory.

Deploy GRID software:

$ cd /opt/oraclecrs/12201
$ unzip -q /home/oracle/install/linuxx64_12201_grid_home.zip

Note: Since Oracle 12cR2 you can unzip archive within $GI_HOME.

5. Install package cvuqdisk RPM package

Install $GI_HOME/cv/rpm/cvuqdisk-1.0.10-1.rpm package from $GI_HOME. Note in case of Oracle Linux with internet connection or Redhat with configured OL public YUM repository you can use "yum install cvuqdisk" command.

# yum localinstall /opt/oraclecrs/12201/cv/rpm/cvuqdisk-1.0.10-1.rpm

6. ASM disk configuration (ASMFD)

Now we need to prepare (stamp) the disks that will be used for Oracle ASM. Note in my case /dev/sdb and /dev/sdc are dedicated for ASM.

# /opt/oraclecrs/12201/bin/asmcmd afd_label DATA_0000 /dev/sdb --init
# /opt/oraclecrs/12201/bin/asmcmd afd_label DATA_0001 /dev/sdc --init

Just simple check if disks are prepared:

# /opt/oraclecrs/12201/bin/asmcmd afd_lslbl /dev/sdb
--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DATA_0000                             /dev/sdb

# /opt/oraclecrs/12201/bin/asmcmd afd_lslbl /dev/sdc
--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DATA_0001                             /dev/sdc

Possible error:

# ./bin/asmcmd afd_label DATA_0000 /dev/sdb
/opt/oraclecrs/12201/bin/kfod: line 22: /scratch/app/user/product/12.2.0/grid/bin/kfod.bin: No such file or directory

Solution:

Modify $GI_HOME/bin/kfod (in my case /opt/oraclecrs/12201/bin/kfod ) and replace path in OHOME variable to correct one.
Change this entry: OHOME=/scratch/app/user/product/12.2.0/grid to this: OHOME=/opt/oraclecrs/12201

 

7. Run gridSetup in order to configure Grid infrastructure

$ cd /opt/oraclecrs/12201
$ ./gridSetup.sh

 8. Use "Configure Oracle ASM Filter Driver" option during ASM disk group creation

 undefined

When your installation was finished successfully you can log on to ASM instance in order to check v$asm_disk.

 

SQL> select path, name, library, header_status from v$asm_disk;

PATH            NAME            LIBRARY                                                          HEADER_STATU
--------------- --------------- ---------------------------------------------------------------- ------------
AFD:DATA_0000   DATA_0000       AFD Library - Generic , version 3 (KABI_V3)                      MEMBER
AFD:DATA_0001   DATA_0001       AFD Library - Generic , version 3 (KABI_V3)                      MEMBER

As you can see AFD Library (thus ASMFD) is being used. Output is very similar to output from ASM Library (ASMLIB).
Just for comparison there is output from ASMLIB configuration:

SQL> select path, name, library, header_status from v$asm_disk;

PATH            NAME            LIBRARY                                                          HEADER_STATU
--------------- --------------- ---------------------------------------------------------------- ------------
ORCL:DATA_0000  DATA_0000       ASM Library - Generic Linux, version 2.0.12 (KABI_V2)            MEMBER
ORCL:DATA_0001  DATA_0001       ASM Library - Generic Linux, version 2.0.12 (KABI_V2)            MEMBER

 

Possible Errors and Solutions:

Problem:
[INS-41223] ASM Filter Driver is not supported on this platform

AFD-620: AFD is not supported on this operating system version: ...
AFD-9201: Not Supported

Solution: Well, message is clear, you are using unsupported Linux distribution (e.g.: Centos). If you have a good reason to play with Centos see "Centos specific important steps" bellow, otherwise I would suggest to use certified Linux distribution (e.g.: Redhat Enteprise Linux, Oracle Linux).


Problem:
[INS-41223] ASM Filter Driver is not supported on this platform
Action - To proceed, do not specify or select the Oracle ASM Filter Driver option.
Additional Information:
AFD-9202: AFD can not be installed/loaded because ASMLib is installed.
AFD-9201: Not Supported

Solution: Again, message is very clear. You can not use ASMFD while ASMLIB is in use. So you have to choose one. If there is reason for using ASMLIB then don't check the "Configure Oracle ASM Filter Driver" option during ASM disk group creation.

Problem:
bin/kfod: line 22: /scratch/app/user/product/12.2.0/grid/bin/kfod.bin: No such file or directory

Solution:
Edit $GI_HOME/bin/kfod script and replace path in OHOME=/scratch/app/user/product/12.2.0/grid to correct one.

 

Centos specific important steps:

So you are stubborn, well then... :-) Couple of tweaks have to be perfomed prior "gridSetup" execution (before step #7).

1. Edit $GI_HOME/lib/osds_acfslib.pm file and add following line:

($release =~ /^centos-release/) ||    # Centos Linux

- after line 359 (as next line after "enterprise-release" condition)
- after line 911 (as next line "enterprise-release" condition, originally at 910 line)
- after line 1146 (as next line "enterprise-release" condition, originally at 1144 line)

2. Edit /etc/redhat-release file

# cp /etc/redhat-release /etc/redhat-release.old
# sed -e 's/Centos/Redhat/' /etc/redhat-release.old > /etc/redhat-release

 Now you can execute gridSetup.

 

Next step after installation:

Installation of Oracle Database 12c Release 2 Database Software - http://ivan.kartik.sk/index.php?controller=post&action=view&id_post=78

 

Oracle Database 12c Release 2 available for download

Pretty long awaited second release of 12c (On-Premise) version is (apparently) officially available for download at: http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

Currently Oracle Solaris SPARC, Oracle Solaris x86-64 and (of course) Linux x86-64 versions are available.
As always I suggest to read a New features Guide in order to explore new things brought by current release: http://docs.oracle.com/database/122/NEWFT/toc.htm

Enjoy learning and exploring...

 

Oracle Linux 7.3 release available

Just one week after RHEL 7.3 release Oracle has announced the general availability of Oracle Linux 7.3 as well.

There is lot of improvements and features but the most notable fact about OL 7.3 is that comes with Unbreakable Enterprise Kernel 4 (UEK4) and older UEK3 is not shipped in distribution while it still supported. Starting with 7.3 version OL supports UEFI Secure Boot. The bad news is that kpatch package has been removed from the distribution.

For more information about new version see the release notes at: https://docs.oracle.com/cd/E52668_01/E79987/html/index.html

You can download the ISOs either from MOS site or Oracle Software Delivery Cloud (aka edelivery.oracle.com).

 

 

Oracle Database 12cR2 online documentation available

Following the Cloud-first strategy Oracle has announced the general availability of Oracle Database 12c Release 2 via the new Oracle Exadata Express Cloud Service. Software for on-premises installation has not been released yet but if you want to learn what's new or what is coming in the 12cR2 you can browse documentation which is available at: http://docs.oracle.com/en/database/

ACFS issue with latest UEK3 kernels on OEL 7/RHEL 7

One of my colleagues has been facing to weird behaviour of ACFS during installation of two nodes RAC on RHEL 7 and OEL7 (started on RHEL, then tried OEL). ACFS volume creation (during mkfs execution) one of the nodes has either hung or thrown error like this:

mkfs.acfs: version                   = 12.1.0.2.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/goldengate-32
mkfs.acfs: CLSU-00100: operating system function: ioctl failed with error data: 1
mkfs.acfs: CLSU-00101: operating system error message: Operation not permitted
mkfs.acfs: CLSU-00103: error location: OI_0
mkfs.acfs: ACFS-00546: failed to change on-disk signature
mkfs.acfs: ACFS-01004: /dev/asm/goldengate-32 was not formatted.

After couple of days of his struggling I decided to help him to find out the problem because my colleague is very experienced thus I knew that he has done everything correctly (well, definitely it wasn't his first rodeo). Moreover I have done similar installation just few days before, without any problem. And that was interesting. Firstly I checked logs and of course configuration (udev, multipath, etc.), it had been correct and traced (using one of my closest friends since 2000) that process with this result:

2216  stat("/dev/asm/goldengate-34", {st_mode=S_IFBLK|0770, st_rdev=makedev(251, 17409), ...}) = 0
2216  open("/dev/ofsctl", O_RDWR)       = 9
2216  ioctl(9, 0xffffffffc1387015, 0x7ffdbee52880) = -1 EPERM (Operation not permitted)

 And this was really weird as permissions for these devices are defined by Udev configuration which is created during installation.

Gathered all important information about environment I've created an action plan. Installation of the same environment on my virtual machine. Well I'd rather say almost the same as I decided to not install the clustered environment in first step and of course used hardware was different and hypervisor has been as well. Version of all software was pretty much same as in original (my colleague's) environment, except one so important thing - Linux kernel. I decided to use original (non UEK) kernel first.
Installation went smoothly, ACFS had smooth configuration and later functionality too, so two options (or unanswered question) remained - whether it's kernel related or RAC related problem where the first option was my preference according to previous tracing.
So I've installed exact version (3.8.13-118.6.2.el7uek) of Unbreakable Kernel provided by Oracle just as my colleague did. And here is the result (Note that it was an intention to perform all steps manually):

$ uname -a
Linux el1 3.8.13-118.6.2.el7uek.x86_64 #2 SMP Thu May 19 13:15:51 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ acfsdriverstate supported
ACFS-9200: Supported

# acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.

# lsmod | grep ora
oracleacfs           3498177  0
oracleadvm            594197  0
oracleoks             503994  2 oracleacfs,oracleadvm

$ asmcmd volinfo --all
no volumes found
$ asmcmd volcreate -G GG GOLDENGATE -s 4G
$ asmcmd volinfo --all
Diskgroup Name: GG

         Volume Name: GOLDENGATE
         Volume Device: /dev/asm/goldengate-34
         State: ENABLED
         Size (MB): 4096
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

$ mkfs -t acfs /dev/asm/goldengate-34
mkfs.acfs: version                   = 12.1.0.2.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/goldengate-34
mkfs.acfs: CLSU-00100: operating system function: ioctl failed with error data: 1
mkfs.acfs: CLSU-00101: operating system error message: Operation not permitted
mkfs.acfs: CLSU-00103: error location: OI_0
mkfs.acfs: ACFS-00546: failed to change on-disk signature
mkfs.acfs: ACFS-01004: /dev/asm/goldengate-34 was not formatted.

Bingo! I've got the same problem on my test machine, so I was able to reproduce the problem. Now it's clear that is not related to a clustered configuration. Ok, let's see the trace output:

2061  stat("/dev/asm/goldengate-34", {st_mode=S_IFBLK|0770, st_rdev=makedev(251, 17409), ...}) = 0
2061  open("/dev/ofsctl", O_RDWR)       = 9
2061  ioctl(9, 0xffffffffc1387015, 0x7ffd26305310) = -1 EPERM (Operation not permitted)

and permissions:

$ ll /dev/asm/.asm_ctl_spec
brwxrwx--- 1 root dba 251, 0 Jun  1 23:13 /dev/asm/.asm_ctl_spec
$ ll /dev/asm/goldengate-34
brwxrwx--- 1 root dba 251, 17409 Jun  1 23:24 /dev/asm/goldengate-34
$ ll /dev/ofsctl
brw-rw-r-- 1 root dba 250, 0 Jun  1 23:24 /dev/ofsctl

Output from tracing was the same and permissions are the same as defined in Udev rules and also the are correct. So according to this result I decided to remove ACFS related configuration and check the configuration and functionality of ACFS again using several older UEK3 kernels. Here is the result from the test of one of them (3.8.13-118.4.2.el7uek.x86_64):

$ uname -a
Linux el1 3.8.13-118.4.2.el7uek.x86_64 #2 SMP Tue Mar 22 20:46:48 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ acfsdriverstate supported
ACFS-9200: Supported

# acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.
# lsmod | grep ora
oracleacfs           3498177  0
oracleadvm            594197  0
oracleoks             503994  2 oracleacfs,oracleadvm

$ asmcmd volinfo --all
no volumes found
$ asmcmd volcreate -G GG GOLDENGATE -s 4G
$ asmcmd volinfo --all
Diskgroup Name: GG

         Volume Name: GOLDENGATE
         Volume Device: /dev/asm/goldengate-34
         State: ENABLED
         Size (MB): 4096
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage:
         Mountpath:

$ mkfs -t acfs /dev/asm/goldengate-34
mkfs.acfs: version                   = 12.1.0.2.0
mkfs.acfs: on-disk version           = 39.0
mkfs.acfs: volume                    = /dev/asm/goldengate-34
mkfs.acfs: volume size               = 4294967296  (   4.00 GB )
mkfs.acfs: Format complete.

# mount -t acfs /dev/asm/goldengate-34 /mnt/
# mount | grep mnt
/dev/asm/goldengate-34 on /mnt type acfs (rw,relatime,device,rootsuid,ordered)
# touch /mnt/testfile
# ll /mnt/testfile
-rw-r--r-- 1 root root 0 Jun  2 00:52 /mnt/testfile

 It works, we were able to format, mount and use the ACFS volume thus now we know that downgrade the UEK3 kernel is a workaround which solves the issue with ACFS until time when bug will be fixed by newer release of UEK3 kernel.

 A bonus case: What if the Sys Admin will upgrade kernel thus we use the "buggy" version of UEK3 kernel on already configured ACFS volumes? I did several repeating tests on already configured and previously working ACFS volume. Let see what will happen:

1st run:

$ uname -a
Linux el1 3.8.13-118.6.2.el7uek.x86_64 #2 SMP Thu May 19 13:15:51 PDT 2016 x86_64 x86_64 x86_64 GNU/Linux

# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed

$ asmcmd volenable -G GG GOLDENGATE
$ asmcmd volinfo --all
Diskgroup Name: GG

         Volume Name: GOLDENGATE
         Volume Device: /dev/asm/goldengate-34
         State: ENABLED
         Size (MB): 4096
         Resize Unit (MB): 64
         Redundancy: UNPROT
         Stripe Columns: 8
         Stripe Width (K): 1024
         Usage: ACFS
         Mountpath: /mnt

# mount -t acfs /dev/asm/goldengate-34 /mnt/
# mount | grep mnt
/dev/asm/goldengate-34 on /mnt type acfs (rw,relatime,device,rootsuid,ordered)
# touch /mnt/testfile2
# ll /mnt/testfile*
-rw-r--r-- 1 root root 0 Jun  2 00:52 /mnt/testfile
-rw-r--r-- 1 root root 0 Jun  2 01:18 /mnt/testfile2

Result of 1st run - working.

2nd run:

# Kernel panic - not syncing: Holding spin lock
rid: 1925, comm: mount Tainted: PF 0 3.8.13-118.6.2.e17uek.x86_64 #2
all Trace:
[<ffffffff81574fb0>] panic+Oxc8/0x1d7
[<ffffffffa0484a7a>] __KsPanic+0x9a/Oxa0 [oracleoks]
[<ffffffffa072275d>] ? OfsDoMountRecovery+Oxldd/Ox2b0 [oracleacfs]
[<ffffffffa0722a5d>] OfsDoPhaselRecovery+0x22d/Ox4e0 [oracleacfs]
[<ffffffffa0722e28>] OfsWaitForRecoveryToComplete+0x118/0x3c0 [oracleacfs]
[<ffffffffa0667446>] OfsSetupVolume+Oxc6/0x2ff0 [oracleacfs]
[<ffffffffa048507c>] ? KsSleepEvent+Ox8c/Oxce [oracleoks]
[<ffffffff81081e90>] ? wake_up_bit+Ox30/0x30
[<ffffffffa04851bc>] ? KsCreateSystemThread+Ox10c/Ox1b0 [oracleoks]
[<ffffffffa066b6dd>] OfsMountVolume+0x136d/Ox27e0 [oracleacfs]
[<ffffffffa0766e8f>] ofs_fill_sb+0x6f/Ox7e0 [oracleacfs]
[<ffffffff8118af58>] mount_bdev+0x1b8/0x200
[<ffffffffa0766e20>] ? ofs_parse_flags+0x140/0x140 [oracleacfs]
[<ffffffffa0763ec7>] ofs_mount+Ox107/0x2f0 [oracleacfs]
[<ffffffff8118b8e9>] mount_fs+0x39/0x1b0
[<ffffffff811a5dc7>] ? alloc_vfsmnt+Oxd7/0x1b0
[<ffffffff811a5f3f>] vfs_kern_mount+Ox5f/Oxf0
[<ffffffff811a8250>] do_mount+0x220/0xaf0
[<ffffffff8114343b>] ? strndup_user+Ox4b/Oxf0
[<ffffffff811a8ba3>] sys_mount+0x83/0xc0
[<ffffffff81587179>] system_call_fastpath+0x16/0x1b

Result of the 2nd run was kernel panic, machine hung of course.

3rd run:

# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
acfsutil plogconfig: CLSU-00100: operating system function: ioctl failed with error data: 22
acfsutil plogconfig: CLSU-00101: operating system error message: Invalid argument
acfsutil plogconfig: CLSU-00103: error location: OI_0
acfsutil plogconfig: ACFS-03500: Unable to access kernel persistent log entries.
ACFS-9225: Failed to start OKS persistent logging.
ACFS-9322: completed

Result of 3rd run: finished with CLSU-00100: operating system function: ioctl failed with error data: 22 , CLSU-00101: operating system error message: Invalid argument, CLSU-00103: error location: OI_0 error messages. After several tries of manual startup it has started successfully.

I did seven more runs and final result was 4x working, 5x  kernel panic, 1x error message during startup, so there is 60% chance that your system won't work.

Final conclusions

  • problem is not related to clustered environment
  • based on several tests (not shown in this article) 11gR2 (11.2.0.4 + APR 2016 PSU) and 12cR1 (12.1.0.2 + Apr 2016 PSU) are affected by this issue
  • problem is related to specific UEK3 kernel versions, to be precise: 3.8.13-118.6.1.el7uek and 3.8.13-118.6.2.el7uek
  • (currently) last properly working version of UEK3 is 3.8.13-118.4.2.el7uek
  • supported version of default kernels (non UEK) is one of possible solution/workaround
  • using older UEK3 kernel is a temporary workaround, not solution until the bug will be fixed. There are two reasons to have updated kernel (security, bugfix)

 

Update:

Updated kernel 3.8.13-118.8.1.el7uek.x86_64 has been released and this bug doesn't occur with this version (and probably later versions which will come in future).

 Hope that helps...

 

Home ← Older posts