One-off patch on Oracle 19.15 RAC nodes on Linux 7.9
One-off patch on Oracle 19.15 RAC nodes on Linux 7.9 to fix internal error
========================================================================
Apply one off patch on RAC in a rolling fashion
=====================================================
OS: Linux 7.9 Redhat
DB : Oracle 19.15.0.0.0
Number of Nodes in RAC : 2
Server1 FQDN : node1.xyz.com
Server2 FQDN : node2.xyz.com
Node 1: node1
Node2: node2
DB name: ORCLP
Instance names: ORCLP1, ORCLP2
one off patch number: 33421721
Reason for Patch
====================
To fix internal error
Software details:
======================
node1 ORCLP1 $
node1 ORCLP1 $ cd /u01/app/oracle/software/
node1 ORCLP1 $ cd 1915/
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $ ls -lt
total 2887384
-rw-r–r– 1 oracle oinstall 152541 Mar 9 13:05 p33421721_1915000DBRU_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 124109254 Aug 20 2022 p6880880_190000_Linux-x86-64.zip
-rwxr-xr-x 1 oracle oinstall 2832388023 Aug 20 2022 p33803476_190000_Linux-x86-64.zip
-rw-rw-r– 1 oracle oinstall 4436 Jul 26 2022 PatchSearch.xml
drwxr-xr-x 4 oracle oinstall 4096 Jul 26 2022 33421721
drwxr-x— 8 oracle oinstall 4096 Apr 7 2022 33803476
node1 ORCLP1 $
node1 ORCLP1 $ cd 33421721/
Applied Patches on Node 1 server
==================================
node1 ORCLP1 $
node1 ORCLP1 $ $ORACLE_HOME/OPatch/opatch lsinventory |grep applied
Patch 33815596 : applied on Sat Aug 20 11:06:58 CEST 2022
Patch 33806152 : applied on Sat Aug 20 10:56:01 CEST 2022
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
node1 ORCLP1 $ pwd
/u01/app/oracle/software/1915/33421721
node1 ORCLP1 $
node1 ORCLP1 $ cd
node1 ORCLP1 $ cd /u01/app/oracle
node1 ORCLP1 $ ls -lrt *sh
-rwxr-x— 1 oracle oinstall 3509 Aug 25 2021 hugepages_settings.sh
-rwxr-xr-x 1 oracle oinstall 679 Apr 9 2022 db_services.sh
node1 ORCLP1 $
To Take pre output of all services
=====================================
node1 ORCLP1 $ sh db_services.sh
stty: standard input: Inappropriate ioctl for device
node1 ORCLP1 $ cd
node1 ORCLP1 $
node1 ORCLP1 $ ls -lrt |tail
-rw-r–r– 1 oracle oinstall 57351 Oct 15 05:13 ORCLP1_services_15_Oct_2022_05_13.log
-rw-r–r– 1 oracle oinstall 62065 Mar 11 04:38 ORCLP1_services_11_Mar_2023_04_38.log
node1 ORCLP1 $
node1 ORCLP1 $ cd /u01/app/oracle/software/1915/33421721
node1 ORCLP1 $ echo $ORACLE_HOME
/u01/app/oracle/product/19.0.0/dbhome_1
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
node1 ORCLP1 $ export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
node1 ORCLP1 $ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
node1 ORCLP1 $
To check Prerequisites
=======================
node1 ORCLP1 $ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.35
Copyright (c) 2023, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.35
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_04-39-44AM_1.log
Invoking prereq “checkconflictagainstohwithdetail”
Prereq “checkConflictAgainstOHWithDetail” passed.
OPatch succeeded.
Take backup of OraInventory
==================================
node1 ORCLP1 $
Connect to root user
=================
node1 ORCLP1 $ sudo -s
bash-4.2#
bash-4.2#
bash-4.2# cd /u01/app/
bash-4.2#
bash-4.2# cp -RP oraInventory oraInventory.bkp.20230310
bash-4.2#
bash-4.2# ls -ld oraI*
drwxrwx— 6 grid oinstall 4096 Mar 11 04:39 oraInventory
drwxr-x— 7 root root 4096 Jul 3 2022 oraInventory.bkp.20220703
drwxr-x— 6 root root 4096 Mar 11 04:40 oraInventory.bkp.20230310
bash-4.2#
bash-4.2# exit
exit
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
Check status of database on node and node 2
==============================================
node1 ORCLP1 $
node1 ORCLP1 $ srvctl status database -d ORCLP_LDC -f -v
Instance ORCLP1 is running on node node1 with online services ORCLP. Instance status: Open.
Instance ORCLP2 is running on node node2 with online services ORCLP. Instance status: Open.
node1 ORCLP1 $
node1 ORCLP1 $ pwd
/u01/app
node1 ORCLP1 $
node1 ORCLP1 $ cd
node1 ORCLP1 $ pwd
/export/home/oracle
Take output of “opatch lsinventory -detail” in a file
=============================================================
node1 ORCLP1 $ $ORACLE_HOME/OPatch/opatch lsinventory -detail > opatch_20230311.txt
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
node1 ORCLP1 $ vi opatch_20230311.txt
node1 ORCLP1 $
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
node1 ORCLP1 $
node1 ORCLP1 $ pwd
/export/home/oracle
node1 ORCLP1 $
node1 ORCLP1 $
node1 ORCLP1 $ sqlplus “/as sysdba”
SQL*Plus: Release 19.0.0.0.0 – Production on Sat Mar 11 04:44:16 2023
Version 19.15.0.0.0
Copyright (c) 1982, 2022, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.15.0.0.0
SQL> exit
Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.15.0.0.0
node1 ORCLP1 $ sudo -s
Set environemnt to ASM1 — on first node
===============================================
bash-4.2#
bash-4.2# . oraenv
ORACLE_SID = [ORCLP1] ? +ASM1
The Oracle base has been changed from /u01/app/oracle to /u01/app/grid
bash-4.2# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
bash-4.2# crsctl stat res -t -init
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.asm
1 ONLINE ONLINE node1 STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE node1 STABLE
ora.crf
1 ONLINE ONLINE node1 STABLE
ora.crsd
1 ONLINE ONLINE node1 STABLE
ora.cssd
1 ONLINE ONLINE node1 STABLE
ora.cssdmonitor
1 ONLINE ONLINE node1 STABLE
ora.ctssd
1 ONLINE ONLINE node1 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE node1 STABLE
ora.evmd
1 ONLINE ONLINE node1 STABLE
ora.gipcd
1 ONLINE ONLINE node1 STABLE
ora.gpnpd
1 ONLINE ONLINE node1 STABLE
ora.mdnsd
1 ONLINE ONLINE node1 STABLE
ora.storage
1 ONLINE ONLINE node1 STABLE
——————————————————————————–
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# crsctl stat res -t
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Local Resources
——————————————————————————–
ora.LISTENER.lsnr
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.chad
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.helper
OFFLINE OFFLINE node1 IDLE,STABLE
OFFLINE OFFLINE node2 IDLE,STABLE
ora.net1.network
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.ons
ONLINE ONLINE node1 STABLE
ONLINE ONLINE node2 STABLE
ora.proxy_advm
OFFLINE OFFLINE node1 STABLE
OFFLINE OFFLINE node2 STABLE
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.DATA.dg(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.DATA_2.dg(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.DATA_3.dg(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node2 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node1 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node1 STABLE
ora.MGMTLSNR
1 ONLINE ONLINE node1 169.254.21.141 192.1
68.2.11,STABLE
ora.RECO.dg(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.RECO_NEW.dg(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.asm(ora.asmgroup)
1 ONLINE ONLINE node1 Started,STABLE
2 ONLINE ONLINE node2 Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
1 ONLINE ONLINE node1 STABLE
2 ONLINE ONLINE node2 STABLE
ora.node1.vip
1 ONLINE ONLINE node1 STABLE
ora.node2.vip
1 ONLINE ONLINE node2 STABLE
ora.ORCLP_ldc.ORCLP.svc
1 ONLINE ONLINE node2 STABLE
2 ONLINE ONLINE node1 STABLE
ora.ORCLP_ldc.db
1 ONLINE ONLINE node1 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
2 ONLINE ONLINE node2 Open,HOME=/u01/app/o
racle/product/19.0.0
/dbhome_1,STABLE
ora.ORCLP_ldc.testbefswitch.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
ora.cvu
1 ONLINE ONLINE node1 STABLE
ora.mgmtdb
1 ONLINE ONLINE node1 Open,STABLE
ora.qosmserver
1 ONLINE ONLINE node1 STABLE
ora.rhpserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE node2 STABLE
ora.scan2.vip
1 ONLINE ONLINE node1 STABLE
ora.scan3.vip
1 ONLINE ONLINE node1 STABLE
——————————————————————————–
bash-4.2#
bash-4.2# pwd
/export/home/oracle
bash-4.2#
bash-4.2#
bash-4.2# crsctl stat res -t > db_resource.txt
You have new mail in /var/spool/mail/oracle
bash-4.2#
bash-4.2#
bash-4.2# ls -l db_resource.txt
-rw-r–r– 1 root root 4875 Mar 11 04:45 db_resource.txt
bash-4.2#
Stop the CRS on node1
==========================
bash-4.2#
bash-4.2# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘node1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘node1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server ‘node1’
CRS-2673: Attempting to stop ‘ora.ORCLP_ldc.ORCLP.svc’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.chad’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.qosmserver’ on ‘node1’
CRS-2677: Stop of ‘ora.ORCLP_ldc.ORCLP.svc’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.ORCLP_ldc.db’ on ‘node1’
CRS-2677: Stop of ‘ora.qosmserver’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.ORCLP_ldc.db’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.chad’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.chad’ on ‘node2’
CRS-2677: Stop of ‘ora.chad’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.mgmtdb’ on ‘node1’
CRS-2677: Stop of ‘ora.mgmtdb’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.MGMTLSNR’ on ‘node1’
CRS-2677: Stop of ‘ora.MGMTLSNR’ on ‘node1’ succeeded
CRS-33673: Attempting to stop resource group ‘ora.asmgroup’ on server ‘node1’
CRS-2673: Attempting to stop ‘ora.DATA_2.dg’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.DATA_3.dg’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.RECO.dg’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.RECO_NEW.dg’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘node1’
CRS-2677: Stop of ‘ora.DATA_2.dg’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.RECO.dg’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.DATA_3.dg’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.RECO_NEW.dg’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘node1’
CRS-2677: Stop of ‘ora.cvu’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.node1.vip’ on ‘node1’
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘node1’
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘node1’
CRS-2677: Stop of ‘ora.node1.vip’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘node1’
CRS-2677: Stop of ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.asmnet1.asmnetwork’ on ‘node1’
CRS-2677: Stop of ‘ora.asmnet1.asmnetwork’ on ‘node1’ succeeded
CRS-33677: Stop of resource group ‘ora.asmgroup’ on server ‘node1’ succeeded.
CRS-2672: Attempting to start ‘ora.MGMTLSNR’ on ‘node2’
CRS-2672: Attempting to start ‘ora.qosmserver’ on ‘node2’
CRS-2672: Attempting to start ‘ora.scan2.vip’ on ‘node2’
CRS-2672: Attempting to start ‘ora.scan3.vip’ on ‘node2’
CRS-2672: Attempting to start ‘ora.node1.vip’ on ‘node2’
CRS-2672: Attempting to start ‘ora.cvu’ on ‘node2’
CRS-2676: Start of ‘ora.cvu’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.node1.vip’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.MGMTLSNR’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.scan2.vip’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN2.lsnr’ on ‘node2’
CRS-2676: Start of ‘ora.scan3.vip’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN3.lsnr’ on ‘node2’
CRS-2676: Start of ‘ora.qosmserver’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN2.lsnr’ on ‘node2’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN3.lsnr’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.mgmtdb’ on ‘node2’
CRS-2676: Start of ‘ora.mgmtdb’ on ‘node2’ succeeded
CRS-2672: Attempting to start ‘ora.chad’ on ‘node2’
CRS-2676: Start of ‘ora.chad’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘node1’
CRS-2677: Stop of ‘ora.ons’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘node1’
CRS-2677: Stop of ‘ora.net1.network’ on ‘node1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘node1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.storage’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘node1’
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.storage’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘node1’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘node1’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘node1’
CRS-2677: Stop of ‘ora.ctssd’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘node1’
CRS-2677: Stop of ‘ora.cssd’ on ‘node1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘node1’
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘node1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘node1’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘node1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘node1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
You have new mail in /var/spool/mail/oracle
bash-4.2# exit
exit
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
node1 ORCLP1 $
node1 ORCLP1 $ cd /u01/app/oracle/software/1915/
node1 ORCLP1 $ ls -lrt
total 2887384
drwxr-x— 8 oracle oinstall 4096 Apr 7 2022 33803476
drwxr-xr-x 4 oracle oinstall 4096 Jul 26 2022 33421721
-rw-rw-r– 1 oracle oinstall 4436 Jul 26 2022 PatchSearch.xml
-rwxr-xr-x 1 oracle oinstall 2832388023 Aug 20 2022 p33803476_190000_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 124109254 Aug 20 2022 p6880880_190000_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 152541 Mar 9 13:05 p33421721_1915000DBRU_Linux-x86-64.zip
node1 ORCLP1 $ cd 33421721/
node1 ORCLP1 $ export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
node1 ORCLP1 $ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
node1 ORCLP1 $
node1 ORCLP1 $
node1 ORCLP1 $
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
Apply patch 33421721 on node1
==========================
node1 ORCLP1 $
node1 ORCLP1 $ opatch apply -local
Oracle Interim Patch Installer version 12.2.0.1.35
Copyright (c) 2023, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.35
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_04-50-14AM_1.log
Verifying environment and performing prerequisite checks…
OPatch continues with these patches: 33421721
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u01/app/oracle/product/19.0.0/dbhome_1’)
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…
Applying interim patch ‘33421721’ to OH ‘/u01/app/oracle/product/19.0.0/dbhome_1’
Patching component oracle.rdbms, 19.0.0.0.0…
Patch 33421721 successfully applied.
Log file location: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_04-50-14AM_1.log
OPatch succeeded.
You have new mail in /var/spool/mail/oracle
node1 ORCLP1 $
Verify the applied patches
============================
node1 ORCLP1 $ $ORACLE_HOME/OPatch/opatch lsinventory |grep applied
Patch 33421721 : applied on Sat Mar 11 04:50:46 CET 2023
Patch 33815596 : applied on Sat Aug 20 11:06:58 CEST 2022
Patch 33806152 : applied on Sat Aug 20 10:56:01 CEST 2022
node1 ORCLP1 $
Apply Patch on node2
=====================
Node2 FQDN: node2.xyz.com
Node 2: node2
node2 ORCLP2 $ sudo -s
bash-4.2#
Take OraInventory backup
===================================
bash-4.2# cd /u01/app
bash-4.2# cp -RP oraInventory oraInventory.bkp.20230310
bash-4.2#
bash-4.2# ls -ld oraI*
drwxrwx— 5 grid oinstall 4096 Mar 11 04:48 oraInventory
drwxr-x— 5 root root 4096 Mar 11 04:54 oraInventory.bkp.20230310
bash-4.2#
bash-4.2#
bash-4.2# exit
exit
node2 ORCLP2 $ pwd
/u01/app/oracle
node2 ORCLP2 $
node2 ORCLP2 $ ls -lrt *sh
-rwxrwxrwx 1 oracle oinstall 3509 Aug 25 2021 hugepages_settings.sh
-rwxrwxrwx 1 oracle oinstall 680 Sep 10 2021 db_services.sh
-rw-r—– 1 oracle oinstall 1179 Nov 1 2021 load_average.sh
node2 ORCLP2 $
node2 ORCLP2 $ sh db_services.sh
stty: standard input: Inappropriate ioctl for device
node2 ORCLP2 $
node2 ORCLP2 $ cd
node2 ORCLP2 $
node2 ORCLP2 $ pwd
/export/home/oracle
node2 ORCLP2 $
node2 ORCLP2 $ ls -lrt |tail
-rw-r–r– 1 oracle oinstall 34 Mar 2 2022 oracle.ahf.loc
-rw-r—– 1 oracle oinstall 45313 Apr 9 2022 ORCLP2_services_09_Apr_2022_11_07.log
-rw-r–r– 1 oracle oinstall 61278 Mar 11 04:54 ORCLP2_services_11_Mar_2023_04_54.log
node2 ORCLP2 $
node2 ORCLP2 $ date
Sat Mar 11 04:54:49 CET 2023
node2 ORCLP2 $
node2 ORCLP2 $ ORCLP2_services_11_Mar_2023_04_54.log
-bash: ORCLP2_services_11_Mar_2023_04_54.log: command not found
node2 ORCLP2 $
node2 ORCLP2 $ vi ORCLP2_services_11_Mar_2023_04_54.log
node2 ORCLP2 $ cd /u01/app/oracle/software/
node2 ORCLP2 $ cd 1915/
node2 ORCLP2 $
node2 ORCLP2 $ $ORACLE_HOME/OPatch/opatch lsinventory -detail > opatch_20230311.txt
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $ ls -lrt
total 2890712
drwxr-x— 8 oracle oinstall 4096 Apr 7 2022 33803476
drwxr-xr-x 4 oracle oinstall 4096 Jul 26 2022 33421721
-rw-rw-r– 1 oracle oinstall 4436 Jul 26 2022 PatchSearch.xml
-rwxr-xr-x 1 oracle oinstall 2832388023 Aug 20 2022 p33803476_190000_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 124109254 Aug 20 2022 p6880880_190000_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 152541 Mar 10 15:57 p33421721_1915000DBRU_Linux-x86-64.zip
-rw-r–r– 1 oracle oinstall 3409024 Mar 11 04:57 opatch_20230311.txt
node2 ORCLP2 $
node2 ORCLP2 $ export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
node2 ORCLP2 $ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $ $ORACLE_HOME/OPatch/opatch lsinventory |grep applied
Patch 33815596 : applied on Sat Aug 20 12:30:12 CEST 2022
Patch 33806152 : applied on Sat Aug 20 12:25:32 CEST 2022
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $ cd 33421721/
node2 ORCLP2 $
Verify prerequisites on node2
===================================
node2 ORCLP2 $ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.2.0.1.35
Copyright (c) 2023, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.35
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_04-58-22AM_1.log
Invoking prereq “checkconflictagainstohwithdetail”
Prereq “checkConflictAgainstOHWithDetail” passed.
OPatch succeeded.
Verify all services on node2
============================
connect to root user
node2 ORCLP2 $ sudo -s
bash-4.2#
bash-4.2#
bash-4.2# . oraenv
ORACLE_SID = [ORCLP2] ? +ASM2
The Oracle base has been changed from /u01/app/oracle to /u01/app/grid
bash-4.2#
bash-4.2#
bash-4.2# crsctl stat res -t -init
——————————————————————————–
Name Target State Server State details
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.asm
1 ONLINE ONLINE node2 Started,STABLE
ora.cluster_interconnect.haip
1 ONLINE ONLINE node2 STABLE
ora.crf
1 ONLINE ONLINE node2 STABLE
ora.crsd
1 ONLINE ONLINE node2 STABLE
ora.cssd
1 ONLINE ONLINE node2 STABLE
ora.cssdmonitor
1 ONLINE ONLINE node2 STABLE
ora.ctssd
1 ONLINE ONLINE node2 OBSERVER,STABLE
ora.diskmon
1 OFFLINE OFFLINE STABLE
ora.drivers.acfs
1 ONLINE ONLINE node2 STABLE
ora.evmd
1 ONLINE ONLINE node2 STABLE
ora.gipcd
1 ONLINE ONLINE node2 STABLE
ora.gpnpd
1 ONLINE ONLINE node2 STABLE
ora.mdnsd
1 ONLINE ONLINE node2 STABLE
ora.storage
1 ONLINE ONLINE node2 STABLE
——————————————————————————–
bash-4.2#
bash-4.2# crsctl stat res -t > db_resource.txt
bash-4.2# vi db_resource.txt
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
bash-4.2#
Stop CRS on node2 to apply patch in a rolling fashion
=====================================================================
bash-4.2# crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘node2’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘node2’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on server ‘node2’
CRS-2673: Attempting to stop ‘ora.ORCLP_ldc.ORCLP.svc’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.qosmserver’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.chad’ on ‘node2’
CRS-2677: Stop of ‘ora.ORCLP_ldc.ORCLP.svc’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.ORCLP_ldc.db’ on ‘node2’
CRS-2677: Stop of ‘ora.qosmserver’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.ORCLP_ldc.db’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.chad’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.chad’ on ‘b0d0hxs2’
CRS-2677: Stop of ‘ora.chad’ on ‘b0d0hxs2’ succeeded
CRS-2673: Attempting to stop ‘ora.mgmtdb’ on ‘node2’
CRS-2677: Stop of ‘ora.mgmtdb’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.MGMTLSNR’ on ‘node2’
CRS-2677: Stop of ‘ora.MGMTLSNR’ on ‘node2’ succeeded
CRS-33673: Attempting to stop resource group ‘ora.asmgroup’ on server ‘node2’
CRS-2673: Attempting to stop ‘ora.DATA_2.dg’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.DATA_3.dg’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.RECO.dg’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.RECO_NEW.dg’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘node2’
CRS-2677: Stop of ‘ora.DATA_2.dg’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.RECO.dg’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.RECO_NEW.dg’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.DATA_3.dg’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘node2’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.node2.vip’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘node2’
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.node2.vip’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘node2’
CRS-2677: Stop of ‘ora.cvu’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.ASMNET1LSNR_ASM.lsnr’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.asmnet1.asmnetwork’ on ‘node2’
CRS-2677: Stop of ‘ora.asmnet1.asmnetwork’ on ‘node2’ succeeded
CRS-33677: Stop of resource group ‘ora.asmgroup’ on server ‘node2’ succeeded.
CRS-2672: Attempting to start ‘ora.MGMTLSNR’ on ‘b0d0hxs2’
CRS-2672: Attempting to start ‘ora.qosmserver’ on ‘b0d0hxs2’
CRS-2672: Attempting to start ‘ora.scan2.vip’ on ‘b0d0hxs2’
CRS-2672: Attempting to start ‘ora.scan3.vip’ on ‘b0d0hxs2’
CRS-2672: Attempting to start ‘ora.node2.vip’ on ‘b0d0hxs2’
CRS-2672: Attempting to start ‘ora.cvu’ on ‘b0d0hxs2’
CRS-2676: Start of ‘ora.node2.vip’ on ‘b0d0hxs2’ succeeded
CRS-2676: Start of ‘ora.scan2.vip’ on ‘b0d0hxs2’ succeeded
CRS-2676: Start of ‘ora.cvu’ on ‘b0d0hxs2’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN2.lsnr’ on ‘b0d0hxs2’
CRS-2676: Start of ‘ora.scan3.vip’ on ‘b0d0hxs2’ succeeded
CRS-2672: Attempting to start ‘ora.LISTENER_SCAN3.lsnr’ on ‘b0d0hxs2’
CRS-2676: Start of ‘ora.MGMTLSNR’ on ‘b0d0hxs2’ succeeded
CRS-2676: Start of ‘ora.qosmserver’ on ‘b0d0hxs2’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN2.lsnr’ on ‘b0d0hxs2’ succeeded
CRS-2676: Start of ‘ora.LISTENER_SCAN3.lsnr’ on ‘b0d0hxs2’ succeeded
CRS-2672: Attempting to start ‘ora.mgmtdb’ on ‘b0d0hxs2’
CRS-2676: Start of ‘ora.mgmtdb’ on ‘b0d0hxs2’ succeeded
CRS-2672: Attempting to start ‘ora.chad’ on ‘b0d0hxs2’
CRS-2676: Start of ‘ora.chad’ on ‘b0d0hxs2’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘node2’
CRS-2677: Stop of ‘ora.ons’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘node2’
CRS-2677: Stop of ‘ora.net1.network’ on ‘node2’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘node2’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.storage’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘node2’
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.storage’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘node2’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘node2’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘node2’
CRS-2677: Stop of ‘ora.ctssd’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘node2’
CRS-2677: Stop of ‘ora.cssd’ on ‘node2’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘node2’
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘node2’
CRS-2677: Stop of ‘ora.gipcd’ on ‘node2’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘node2’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘node2’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# exit
exit
node2 ORCLP2 $ pwd
/u01/app/oracle/software/1915/33421721
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $ export ORACLE_HOME=/u01/app/oracle/product/19.0.0/dbhome_1
node2 ORCLP2 $ export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
node2 ORCLP2 $
node2 ORCLP2 $ pd
-bash: pd: command not found
node2 ORCLP2 $
node2 ORCLP2 $
node2 ORCLP2 $ pwd
/u01/app/oracle/software/1915/33421721
node2 ORCLP2 $
Apply patch 33421721 on node2
=================================
node2 ORCLP2 $
node2 ORCLP2 $ opatch apply -local
Oracle Interim Patch Installer version 12.2.0.1.35
Copyright (c) 2023, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.35
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_05-03-32AM_1.log
Verifying environment and performing prerequisite checks…
OPatch continues with these patches: 33421721
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u01/app/oracle/product/19.0.0/dbhome_1’)
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…
Applying interim patch ‘33421721’ to OH ‘/u01/app/oracle/product/19.0.0/dbhome_1’
Patching component oracle.rdbms, 19.0.0.0.0…
Patch 33421721 successfully applied.
Log file location: /u01/app/oracle/product/19.0.0/dbhome_1/cfgtoollogs/opatch/opatch2023-03-11_05-03-32AM_1.log
OPatch succeeded.
node2 ORCLP2 $
node2 ORCLP2 $
Verify applied patches on Node2
======================================
node2 ORCLP2 $ $ORACLE_HOME/OPatch/opatch lsinventory |grep applied
Patch 33421721 : applied on Sat Mar 11 05:04:02 CET 2023
Patch 33815596 : applied on Sat Aug 20 12:30:12 CEST 2022
Patch 33806152 : applied on Sat Aug 20 12:25:32 CEST 2022
You have new mail in /var/spool/mail/oracle
node2 ORCLP2 $
node2 ORCLP2 $
Start CRS on Node2
===========================
node2 ORCLP2 $
node2 ORCLP2 $ sudo -s
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# . oraenv
ORACLE_SID = [ORCLP2] ? +ASM2
The Oracle base has been changed from /u01/app/oracle to /u01/app/grid
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
bash-4.2#
bash-4.2#
bash-4.2#
bash-4.2# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
bash-4.2#
bash-4.2#
bash-4.2# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
See also:
- OPatch apply failed with error code 73 in Oracle
- Step by Step procedure for dry-run upgrade on RAC
- Step by Step upgrade GI from 12.2 to 19.12 on AIX
- http://oracle.com