Errors while running rootupgrade.sh GI upgrade RAC
Errors while running rootupgrade.sh script on Node2 in a Grid upgrade of two node RAC
Overview of the Issue
On a 2 Node RAC , rootupgrade.sh script execution completed successfully on Node1, but received errors while running on Node2.
Env: Oracle Grid 12c to 19c on Linux
Errors: In the 1st Attempt of rootupgrade.sh script on Node2
CRS-2883: Resource ‘ora.cssd’ failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/sweep” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/lck” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata_pv” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/alert” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/cdump” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/log” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/stage” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/incident” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata_dgif” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/incpkg” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
CRS-4000: Command Start failed, or completed with errors.
2021/06/22 18:24:22 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 1575.
Solution: With root user
Killed processes that were associated with below programs.
1)/etc/init.d/init.ohasd run
2)/u00/grid/oracle/product/19.0.0.0/grid/bin/ohasd.bin
cd /u00/app/oracle/diag/crs/Node2/crs
Node2:/u00/app/oracle/diag/crs/Node2/crs#chown oracle:oinstall *
Errors : In the Second attempt of rootupgrade.sh on Node2
CRS-2883: Resource ‘ora.cssd’ failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
CRS-4000: Command Start failed, or completed with errors.
2021/06/22 18:30:31 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 1575.
Solution:
cd /u00/grid/oracle/product/19.0.0.0/grid/crs/install
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs/install#chown oracle:oinstall *
For the complete list of commands & output and the sequence of steps.Check below steps
Step 1) Verify the processes and connect as root and kill
Node2:~#ps -ef|grep pmon
oracle 29463 20179 0 18:16 pts/1 00:00:00 grep –color=auto pmon
Node2:~#ps -ef|grep css
root 5216 1 0 Feb27 ? 18:50:05 /opt/Symantec/sdcssagent/IDS/bin/sisidsdaemon
sisips 5263 1 0 Feb27 ? 02:35:31 /opt/Symantec/sdcssagent/IPS/bin/sisipsdaemon
sisips 5296 1 0 Feb27 ? 00:00:00 /opt/Symantec/sdcssagent/IPS/bin/sisipsutildaemon
oracle 29658 20179 0 18:17 pts/1 00:00:00 grep –color=auto css
Node2:~#
Node2:~#ps -ef|grep has
root 5269 1 0 Apr10 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 Type=simple
root 12619 1 0 Jun21 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
root 12783 12619 0 Jun21 ? 00:02:53 /u00/grid/oracle/product/19.0.0.0/grid/bin/ohasd.bin restart
oracle 29770 20179 0 18:17 pts/1 00:00:00 grep –color=auto has
root 30789 1 1 Jun21 ? 00:29:21 /u00/grid/oracle/product/19.0.0.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
Node2:~#
Node2:~#kill -9 5269 12619 12783
-bash: kill: (5269) – Operation not permitted
-bash: kill: (12619) – Operation not permitted
-bash: kill: (12783) – Operation not permitted
Node2:~#
Node2:~#
Node2:~#ps -ef|grep has
root 5269 1 0 Apr10 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 Type=simple
root 12619 1 0 Jun21 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
root 12783 12619 0 Jun21 ? 00:02:53 /u00/grid/oracle/product/19.0.0.0/grid/bin/ohasd.bin restart
oracle 30064 20179 0 18:17 pts/1 00:00:00 grep –color=auto has
root 30789 1 1 Jun21 ? 00:29:22 /u00/grid/oracle/product/19.0.0.0/grid/bin/ohasd.bin reboot _ORA_BLOCKING_STACK_LOCALE=AMERICAN_AMERICA.AL32UTF8
Node2:~#
Node2:~#sudo su – root
Last login: Tue Jun 22 09:34:31 GMT 2021
Node2:~#
Node2:~#
Node2:~#kill -9 5269 12619 12783 30789
Node2:~#ps -ef|grep has
root 30310 1 0 18:18 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
root 30311 1 0 18:18 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 Type=simple
root 30559 30139 0 18:18 pts/1 00:00:00 grep –color=auto has
Node2:~#
Node2:~#kill -9 30310 30311
Node2:~#
Node2:~#
Step 2 ) Verify “has” processes after killing old has processes
Node2:~#ps -ef|grep has
root 950 30139 0 18:18 pts/1 00:00:00 grep –color=auto has
root 30668 1 0 18:18 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
root 30669 1 0 18:18 ? 00:00:00 /bin/sh /etc/init.d/init.ohasd run >/dev/null 2>&1 Type=simple
Node2:~#
Node2:~#
Step 3) Set the Environment
Node2:~#. oraenv
ORACLE_SID = [root] ? +ASM2
ORACLE_HOME = [/home/oracle] ?
ORACLE_BASE environment variable is not being set since this
information is not available for the current user ID root.
You can set ORACLE_BASE manually if it is required.
Resetting ORACLE_BASE to its previous value or ORACLE_HOME
The Oracle base has been set to /home/oracle
Node2:~#
Node2:~#
Step 4) Check the shell scripts under GRID_HOME
Node2:~#cd /u00/grid/oracle/product/19.0.0.0/grid/
Node2:/u00/grid/oracle/product/19.0.0.0/grid#ls -l *sh
-rwxrwxr-x 1 oracle oinstall 3294 Jun 21 13:11 gridSetup.sh
-rwxr-xr-x 1 root oinstall 495 Jun 21 13:12 root.sh
-rwxr-xr-x 1 root oinstall 504 Jun 21 13:12 rootupgrade.sh
-rwxrwxr-x 1 oracle oinstall 628 Jun 21 13:12 runcluvfy.sh
Step 5) Run the rootupgrade.sh
Node2:/u00/grid/oracle/product/19.0.0.0/grid#
Node2:/u00/grid/oracle/product/19.0.0.0/grid#./rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u00/grid/oracle/product/19.0.0.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u00/app/oracle/crsdata/Node2/crsconfig/rootcrs_Node2_2021-06-22_06-19-45PM.log
2021/06/22 18:19:56 CLSRSC-595: Executing upgrade step 1 of 18: ‘UpgradeTFA’.
2021/06/22 18:19:56 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:19:56 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:21:36 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:21:36 CLSRSC-595: Executing upgrade step 2 of 18: ‘ValidateEnv’.
2021/06/22 18:21:37 CLSRSC-595: Executing upgrade step 3 of 18: ‘GetOldConfig’.
2021/06/22 18:21:37 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/06/22 18:21:54 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:22:30 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2021/06/22 18:22:30 CLSRSC-595: Executing upgrade step 4 of 18: ‘GenSiteGUIDs’.
2021/06/22 18:22:30 CLSRSC-595: Executing upgrade step 5 of 18: ‘UpgPrechecks’.
2021/06/22 18:22:31 CLSRSC-595: Executing upgrade step 6 of 18: ‘SetupOSD’.
2021/06/22 18:22:31 CLSRSC-595: Executing upgrade step 7 of 18: ‘PreUpgrade’.
2021/06/22 18:22:31 CLSRSC-595: Executing upgrade step 8 of 18: ‘CheckCRSConfig’.
2021/06/22 18:22:32 CLSRSC-595: Executing upgrade step 9 of 18: ‘UpgradeOLR’.
2021/06/22 18:22:32 CLSRSC-595: Executing upgrade step 10 of 18: ‘ConfigCHMOS’.
2021/06/22 18:23:04 CLSRSC-595: Executing upgrade step 11 of 18: ‘UpgradeAFD’.
2021/06/22 18:23:05 CLSRSC-595: Executing upgrade step 12 of 18: ‘createOHASD’.
2021/06/22 18:23:06 CLSRSC-595: Executing upgrade step 13 of 18: ‘ConfigOHASD’.
2021/06/22 18:23:06 CLSRSC-329: Replacing Clusterware entries in file ‘oracle-ohasd.service’
2021/06/22 18:23:20 CLSRSC-595: Executing upgrade step 14 of 18: ‘InstallACFS’.
2021/06/22 18:23:23 CLSRSC-595: Executing upgrade step 15 of 18: ‘InstallKA’.
2021/06/22 18:23:24 CLSRSC-595: Executing upgrade step 16 of 18: ‘UpgradeCluster’.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start ‘ora.evmd’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘Node2’
CRS-2676: Start of ‘ora.mdnsd’ on ‘Node2’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘Node2’
CRS-2676: Start of ‘ora.gpnpd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘Node2’
CRS-2676: Start of ‘ora.gipcd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.crf’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘Node2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘Node2’
CRS-2676: Start of ‘ora.diskmon’ on ‘Node2’ succeeded
CRS-2676: Start of ‘ora.crf’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘Node2’
CRS-2883: Resource ‘ora.cssd’ failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/sweep” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/lck” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata_pv” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/alert” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/cdump” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/log” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/stage” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/incident” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/metadata_dgif” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
PRVG-2031 : Owner of file “/u00/app/oracle/diag/crs/Node2/crs/incpkg” did not match the expected value on node “Node2”. [Expected = “oracle(54321)” ; Found = “root(0)”]
CRS-4000: Command Start failed, or completed with errors.
2021/06/22 18:24:22 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 1575.
Node2:/u00/grid/oracle/product/19.0.0.0/grid#cd /u00/app/oracle/diag/crs/Node2/crs/
Step 6) Error received in the above step. Verify the owner ship of files under crs directory
Node2:/u00/app/oracle/diag/crs/Node2/crs#ls -lrt
total 6392
drwxrwxr-x 2 root oinstall 4096 Aug 22 2015 metadata_dgif
drwxrwxr-x 2 root oinstall 4096 Aug 22 2015 incpkg
drwxrwxr-x 2 root oinstall 4096 Apr 10 20:07 metadata
drwxrwxr-x 2 root oinstall 4096 Apr 11 06:05 alert
drwxrwxr-x 5 root oinstall 4096 Jun 21 13:55 log
drwxrwxr-x 2 root oinstall 4096 Jun 21 17:21 metadata_pv
drwxrwxr-x 18 root oinstall 282624 Jun 22 18:24 incident
drwxrwxr-x 2 root oinstall 442368 Jun 22 18:24 sweep
drwxrwxr-x 2 root oinstall 425984 Jun 22 18:24 stage
drwxrwxr-x 2 root oinstall 4096 Jun 22 18:24 lck
drwxrwxr-x 3 root oinstall 172032 Jun 22 18:24 cdump
drwxrwxr-x 2 oracle oinstall 5165056 Jun 22 18:24 trace
Node2:/u00/app/oracle/diag/crs/Node2/crs#cd ..
Node2:/u00/app/oracle/diag/crs/Node2#ls -lrt
total 4
drwxrwxr-x 14 oracle oinstall 4096 Jun 21 14:10 crs
Step 7 ) Change the ownership of files and directories to oracle and group to oinstall under crs directory
Node2:/u00/app/oracle/diag/crs/Node2#cd crs/
Node2:/u00/app/oracle/diag/crs/Node2/crs#chown oracle:oinstall *
Node2:/u00/app/oracle/diag/crs/Node2/crs#ls -l
total 6392
drwxrwxr-x 2 oracle oinstall 4096 Apr 11 06:05 alert
drwxrwxr-x 3 oracle oinstall 172032 Jun 22 18:24 cdump
drwxrwxr-x 18 oracle oinstall 282624 Jun 22 18:24 incident
drwxrwxr-x 2 oracle oinstall 4096 Aug 22 2015 incpkg
drwxrwxr-x 2 oracle oinstall 4096 Jun 22 18:24 lck
drwxrwxr-x 5 oracle oinstall 4096 Jun 21 13:55 log
drwxrwxr-x 2 oracle oinstall 4096 Apr 10 20:07 metadata
drwxrwxr-x 2 oracle oinstall 4096 Aug 22 2015 metadata_dgif
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 17:21 metadata_pv
drwxrwxr-x 2 oracle oinstall 425984 Jun 22 18:24 stage
drwxrwxr-x 2 oracle oinstall 442368 Jun 22 18:24 sweep
drwxrwxr-x 2 oracle oinstall 5165056 Jun 22 18:24 trace
Node2:/u00/app/oracle/diag/crs/Node2/crs#
Node2:/u00/app/oracle/diag/crs/Node2/crs#
Step 8) Run again rootupgrade.sh script
Node2:/u00/app/oracle/diag/crs/Node2/crs#cd /u00/grid/oracle/product/19.0.0.0/grid/
Node2:/u00/grid/oracle/product/19.0.0.0/grid#./rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u00/grid/oracle/product/19.0.0.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u00/app/oracle/crsdata/Node2/crsconfig/rootcrs_Node2_2021-06-22_06-26-31PM.log
2021/06/22 18:26:41 CLSRSC-595: Executing upgrade step 1 of 18: ‘UpgradeTFA’.
2021/06/22 18:26:42 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:26:42 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:28:22 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:28:22 CLSRSC-595: Executing upgrade step 2 of 18: ‘ValidateEnv’.
2021/06/22 18:28:23 CLSRSC-595: Executing upgrade step 3 of 18: ‘GetOldConfig’.
2021/06/22 18:28:23 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/06/22 18:28:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:29:22 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2021/06/22 18:29:22 CLSRSC-595: Executing upgrade step 4 of 18: ‘GenSiteGUIDs’.
2021/06/22 18:29:22 CLSRSC-595: Executing upgrade step 5 of 18: ‘UpgPrechecks’.
2021/06/22 18:29:23 CLSRSC-595: Executing upgrade step 6 of 18: ‘SetupOSD’.
2021/06/22 18:29:23 CLSRSC-595: Executing upgrade step 7 of 18: ‘PreUpgrade’.
2021/06/22 18:29:23 CLSRSC-595: Executing upgrade step 8 of 18: ‘CheckCRSConfig’.
2021/06/22 18:29:24 CLSRSC-595: Executing upgrade step 9 of 18: ‘UpgradeOLR’.
2021/06/22 18:29:25 CLSRSC-595: Executing upgrade step 10 of 18: ‘ConfigCHMOS’.
2021/06/22 18:29:56 CLSRSC-595: Executing upgrade step 11 of 18: ‘UpgradeAFD’.
2021/06/22 18:29:57 CLSRSC-595: Executing upgrade step 12 of 18: ‘createOHASD’.
2021/06/22 18:29:59 CLSRSC-595: Executing upgrade step 13 of 18: ‘ConfigOHASD’.
2021/06/22 18:30:02 CLSRSC-595: Executing upgrade step 14 of 18: ‘InstallACFS’.
2021/06/22 18:30:04 CLSRSC-595: Executing upgrade step 15 of 18: ‘InstallKA’.
2021/06/22 18:30:06 CLSRSC-595: Executing upgrade step 16 of 18: ‘UpgradeCluster’.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start ‘ora.evmd’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘Node2’
CRS-2676: Start of ‘ora.evmd’ on ‘Node2’ succeeded
CRS-2676: Start of ‘ora.mdnsd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘Node2’
CRS-2676: Start of ‘ora.gpnpd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘Node2’
CRS-2676: Start of ‘ora.gipcd’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.crf’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘Node2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘Node2’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘Node2’
CRS-2676: Start of ‘ora.diskmon’ on ‘Node2’ succeeded
CRS-2676: Start of ‘ora.crf’ on ‘Node2’ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘Node2’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘Node2’ succeeded
CRS-2883: Resource ‘ora.cssd’ failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-41053: checking Oracle Grid Infrastructure for file permission issues
CRS-4000: Command Start failed, or completed with errors.
2021/06/22 18:30:31 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsupgrade.pm line 1575.
Step 9) Errors received again in the above step.
Node2:/u00/grid/oracle/product/19.0.0.0/grid#cd /u00/grid/oracle/product/19.0.0.0/grid/crs
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs#ls -l
total 48
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 conf
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 config
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 include
drwxr-xr-x 2 root oinstall 4096 Jun 21 13:54 init
drwxrwxr-x 3 oracle oinstall 4096 Jun 22 18:29 install
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 lib
drwxr-x–T 2 oracle oinstall 4096 Jun 21 13:54 log
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 mesg
drwxrwxr-x 2 oracle oinstall 4096 Jun 21 13:12 sbs
drwxr-xr-x 2 root oinstall 4096 Jun 21 13:12 template
drwxr-x–T 2 oracle oinstall 4096 Jun 21 13:54 trace
drwxr-xr-x 3 root oinstall 4096 Jun 21 13:54 utl
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs#cd install/
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs/install#ls -l
total 2200
-rwxrwxr-x 1 oracle oinstall 3391 Jun 21 13:12 CLSR.pm
-rwxrwxr-x 1 oracle oinstall 1294 Jun 21 13:54 cmdllroot.sh
-rwxrwxr-x 1 oracle oinstall 1520 Jun 21 13:12 crsconfig_addparams.sbs
-rwxr-xr-x 1 root oinstall 271 Jun 21 13:12 crsconfig_fileperms.excl
-rwxr-x— 1 root oinstall 7317 Jun 21 13:12 crsconfig_params
-rwxrwxr-x 1 oracle oinstall 9354 Jun 21 13:12 crsconfig_params.sbs
-rwxrwxr-x 1 oracle oinstall 22239 Jun 21 13:12 crsconvert.pm
-rwxrwxr-x 1 oracle oinstall 17217 Jun 21 13:12 crsconvtoext.pm
-rwxr-xr-x 1 root oinstall 10000 Jun 21 13:12 crscpcfg.pm
-rwxr-xr-x 1 root oinstall 51384 Jun 21 13:12 crsdeconfig.pm
-rwxr-xr-x 1 root oinstall 36090 Jun 21 13:12 crsdowngrade.pm
-rwxrwxr-x 1 oracle oinstall 71 Jun 21 13:36 crsgenconfig_params
-rwxr-xr-x 1 root oinstall 136065 Jun 21 13:12 crsgpnp.pm
-rwxr-xr-x 1 root oinstall 107528 Jun 21 13:12 crsinstall.pm
-rwxr-xr-x 1 root oinstall 18859 Jun 21 13:12 crska.pm
-rwxr-xr-x 1 root oinstall 89916 Jun 21 13:12 crspatch.pm
-rwxr-xr-x 1 root oinstall 8035 Jun 21 13:12 crstfa.pm
-rwxr-xr-x 1 root oinstall 200600 Jun 21 13:12 crsupgrade.pm
-rwxr-xr-x 1 root oinstall 503206 Jun 21 13:11 crsutils.pm
-rwxr-xr-x 1 root oinstall 7175 Jun 21 13:12 crsxag.pm
-rwxr-xr-x 1 oracle oinstall 4097 Jun 21 13:54 dropdb
-rwxr-xr-x 1 oracle oinstall 3517 Jun 21 13:12 dropdb.pl
-rwxrwxr-x 1 oracle oinstall 2925 Jun 21 13:12 HASLoad.pm
-rwxrwxr-x 1 oracle oinstall 68 Jun 21 13:12 inittab
-rwxrwxr-x 1 oracle oinstall 584 Jun 21 13:12 install.excl
-rwxrwxr-x 1 oracle oinstall 2670 Jun 21 13:12 install_gi.excl
-rwxrwxr-x 1 oracle oinstall 0 Jun 21 13:12 install.incl
-rwxrwxr-x 1 oracle oinstall 27 Jun 21 13:12 installRemove.excl
-rwxr-xr-x 1 root oinstall 111872 Jun 21 13:12 oraacfs.pm
-rwxr-xr-x 1 root oinstall 49544 Jun 21 13:12 oraafd.pm
-rwxr-xr-x 1 root oinstall 102299 Jun 21 13:12 oraasm.pm
-rwxr-xr-x 1 root oinstall 19036 Jun 21 13:12 oracdp.pm
-rwxr-xr-x 1 root oinstall 30096 Jun 21 13:12 orachm.pm
-rwxrwxr-x 1 oracle oinstall 220 Jun 21 13:12 oracle-ohasd.conf
-rwxrwxr-x 1 oracle oinstall 685 Jun 21 13:12 oracle-ohasd.service
-rwxr-xr-x 1 root oinstall 24616 Jun 21 13:12 oraClusterwareComp.pm
-rwxr-xr-x 1 root oinstall 19768 Jun 21 13:12 oracss.pm
-rwxr-xr-x 1 root oinstall 0 Jun 21 13:55 oragpnp.pm
-rwxr-xr-x 1 root oinstall 6856 Jun 21 13:12 oraios.pm
-rwxr-xr-x 1 root oinstall 86147 Jun 21 13:12 oraocr.pm
-rwxr-xr-x 1 root oinstall 42394 Jun 21 13:12 oraohasd.pm
-rwxr-xr-x 1 root oinstall 20062 Jun 21 13:12 oraolr.pm
-rwxr-xr-x 1 root oinstall 4931 Jun 21 13:12 oraons.pm
-rwxr-xr-x 1 root oinstall 17943 Jun 21 13:12 oraqosmserver.pm
-rwxr-xr-x 1 root oinstall 59869 Jun 21 13:12 orasrvm.pm
-rwxrwxr-x 1 oracle oinstall 414 Jun 21 13:12 paramfile.crs
-rw-r–r– 1 root oinstall 182 Jun 21 13:55 ParentDirPerm_Node2.txt
-rwxrwxr-x 1 oracle oinstall 4189 Jun 21 13:12 perlhasgen.pm
-rwxrwxr-x 1 oracle oinstall 9084 Jun 21 13:12 post_gimr_ugdg.pl
drwxrwx— 2 root oinstall 4096 Jun 21 13:54 rhpdata
-rwxr-xr-x 1 root oinstall 24668 Jun 21 13:12 rootcrs.pl
-rwxr-x— 1 root oinstall 8064 Jun 21 13:12 rootcrs.sh
-rwxr-xr-x 1 root oinstall 10305 Jun 21 13:12 roothas.pl
-rwxr-x— 1 root oinstall 3944 Jun 21 13:12 roothas.sh
-rwxr-xr-x 1 root oinstall 3593 Jun 21 13:12 s_crsconfig_defs
-rwxr-xr-x 1 root oinstall 994 Jun 22 18:26 s_crsconfig_Node2_env.txt
-rwxr-xr-x 1 root oinstall 118143 Jun 21 13:12 s_crsutils.pm
-rwxr-xr-x 1 root oinstall 3917 Jun 21 13:12 s_orachm.pm
-rwxr-xr-x 1 root oinstall 19742 Jun 21 13:12 s_oraocr.pm
-rwxr-xr-x 1 root oinstall 7074 Jun 21 13:12 s_oraolr.pm
-rwxr-x— 1 root oinstall 68324 Jun 21 13:12 tfa_setup
Step 10) Change ownership under /u00/grid/oracle/product/19.0.0.0/grid/crs/install directory.
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs/install#chown oracle:oinstall *
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs/install#ls -l
total 2200
-rwxrwxr-x 1 oracle oinstall 3391 Jun 21 13:12 CLSR.pm
-rwxrwxr-x 1 oracle oinstall 1294 Jun 21 13:54 cmdllroot.sh
-rwxrwxr-x 1 oracle oinstall 1520 Jun 21 13:12 crsconfig_addparams.sbs
-rwxr-xr-x 1 oracle oinstall 271 Jun 21 13:12 crsconfig_fileperms.excl
-rwxr-x— 1 oracle oinstall 7317 Jun 21 13:12 crsconfig_params
-rwxrwxr-x 1 oracle oinstall 9354 Jun 21 13:12 crsconfig_params.sbs
-rwxrwxr-x 1 oracle oinstall 22239 Jun 21 13:12 crsconvert.pm
-rwxrwxr-x 1 oracle oinstall 17217 Jun 21 13:12 crsconvtoext.pm
-rwxr-xr-x 1 oracle oinstall 10000 Jun 21 13:12 crscpcfg.pm
-rwxr-xr-x 1 oracle oinstall 51384 Jun 21 13:12 crsdeconfig.pm
-rwxr-xr-x 1 oracle oinstall 36090 Jun 21 13:12 crsdowngrade.pm
-rwxrwxr-x 1 oracle oinstall 71 Jun 21 13:36 crsgenconfig_params
-rwxr-xr-x 1 oracle oinstall 136065 Jun 21 13:12 crsgpnp.pm
-rwxr-xr-x 1 oracle oinstall 107528 Jun 21 13:12 crsinstall.pm
-rwxr-xr-x 1 oracle oinstall 18859 Jun 21 13:12 crska.pm
-rwxr-xr-x 1 oracle oinstall 89916 Jun 21 13:12 crspatch.pm
-rwxr-xr-x 1 oracle oinstall 8035 Jun 21 13:12 crstfa.pm
-rwxr-xr-x 1 oracle oinstall 200600 Jun 21 13:12 crsupgrade.pm
-rwxr-xr-x 1 oracle oinstall 503206 Jun 21 13:11 crsutils.pm
-rwxr-xr-x 1 oracle oinstall 7175 Jun 21 13:12 crsxag.pm
-rwxr-xr-x 1 oracle oinstall 4097 Jun 21 13:54 dropdb
-rwxr-xr-x 1 oracle oinstall 3517 Jun 21 13:12 dropdb.pl
-rwxrwxr-x 1 oracle oinstall 2925 Jun 21 13:12 HASLoad.pm
-rwxrwxr-x 1 oracle oinstall 68 Jun 21 13:12 inittab
-rwxrwxr-x 1 oracle oinstall 584 Jun 21 13:12 install.excl
-rwxrwxr-x 1 oracle oinstall 2670 Jun 21 13:12 install_gi.excl
-rwxrwxr-x 1 oracle oinstall 0 Jun 21 13:12 install.incl
-rwxrwxr-x 1 oracle oinstall 27 Jun 21 13:12 installRemove.excl
-rwxr-xr-x 1 oracle oinstall 111872 Jun 21 13:12 oraacfs.pm
-rwxr-xr-x 1 oracle oinstall 49544 Jun 21 13:12 oraafd.pm
-rwxr-xr-x 1 oracle oinstall 102299 Jun 21 13:12 oraasm.pm
-rwxr-xr-x 1 oracle oinstall 19036 Jun 21 13:12 oracdp.pm
-rwxr-xr-x 1 oracle oinstall 30096 Jun 21 13:12 orachm.pm
-rwxrwxr-x 1 oracle oinstall 220 Jun 21 13:12 oracle-ohasd.conf
-rwxrwxr-x 1 oracle oinstall 685 Jun 21 13:12 oracle-ohasd.service
-rwxr-xr-x 1 oracle oinstall 24616 Jun 21 13:12 oraClusterwareComp.pm
-rwxr-xr-x 1 oracle oinstall 19768 Jun 21 13:12 oracss.pm
-rwxr-xr-x 1 oracle oinstall 0 Jun 21 13:55 oragpnp.pm
-rwxr-xr-x 1 oracle oinstall 6856 Jun 21 13:12 oraios.pm
-rwxr-xr-x 1 oracle oinstall 86147 Jun 21 13:12 oraocr.pm
-rwxr-xr-x 1 oracle oinstall 42394 Jun 21 13:12 oraohasd.pm
-rwxr-xr-x 1 oracle oinstall 20062 Jun 21 13:12 oraolr.pm
-rwxr-xr-x 1 oracle oinstall 4931 Jun 21 13:12 oraons.pm
-rwxr-xr-x 1 oracle oinstall 17943 Jun 21 13:12 oraqosmserver.pm
-rwxr-xr-x 1 oracle oinstall 59869 Jun 21 13:12 orasrvm.pm
-rwxrwxr-x 1 oracle oinstall 414 Jun 21 13:12 paramfile.crs
-rw-r–r– 1 oracle oinstall 182 Jun 21 13:55 ParentDirPerm_Node2.txt
-rwxrwxr-x 1 oracle oinstall 4189 Jun 21 13:12 perlhasgen.pm
-rwxrwxr-x 1 oracle oinstall 9084 Jun 21 13:12 post_gimr_ugdg.pl
drwxrwx— 2 oracle oinstall 4096 Jun 21 13:54 rhpdata
-rwxr-xr-x 1 oracle oinstall 24668 Jun 21 13:12 rootcrs.pl
-rwxr-x— 1 oracle oinstall 8064 Jun 21 13:12 rootcrs.sh
-rwxr-xr-x 1 oracle oinstall 10305 Jun 21 13:12 roothas.pl
-rwxr-x— 1 oracle oinstall 3944 Jun 21 13:12 roothas.sh
-rwxr-xr-x 1 oracle oinstall 3593 Jun 21 13:12 s_crsconfig_defs
-rwxr-xr-x 1 oracle oinstall 994 Jun 22 18:26 s_crsconfig_Node2_env.txt
-rwxr-xr-x 1 oracle oinstall 118143 Jun 21 13:12 s_crsutils.pm
-rwxr-xr-x 1 oracle oinstall 3917 Jun 21 13:12 s_orachm.pm
-rwxr-xr-x 1 oracle oinstall 19742 Jun 21 13:12 s_oraocr.pm
-rwxr-xr-x 1 oracle oinstall 7074 Jun 21 13:12 s_oraolr.pm
-rwxr-x— 1 oracle oinstall 68324 Jun 21 13:12 tfa_setup
Step 11) Run rootupgrade.sh again. This time Successfuly completed.
Node2:/u00/grid/oracle/product/19.0.0.0/grid/crs/install#cd /u00/grid/oracle/product/19.0.0.0/grid/
Node2:/u00/grid/oracle/product/19.0.0.0/grid#./rootupgrade.sh
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u00/grid/oracle/product/19.0.0.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of “dbhome” have not changed. No need to overwrite.
The contents of “oraenv” have not changed. No need to overwrite.
The contents of “coraenv” have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u00/grid/oracle/product/19.0.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u00/app/oracle/crsdata/Node2/crsconfig/rootcrs_Node2_2021-06-22_06-35-31PM.log
2021/06/22 18:35:41 CLSRSC-595: Executing upgrade step 1 of 18: ‘UpgradeTFA’.
2021/06/22 18:35:42 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:35:42 CLSRSC-4012: Shutting down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:37:22 CLSRSC-4013: Successfully shut down Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:37:22 CLSRSC-595: Executing upgrade step 2 of 18: ‘ValidateEnv’.
2021/06/22 18:37:23 CLSRSC-595: Executing upgrade step 3 of 18: ‘GetOldConfig’.
2021/06/22 18:37:23 CLSRSC-464: Starting retrieval of the cluster configuration data
2021/06/22 18:37:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/06/22 18:38:23 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2021/06/22 18:38:23 CLSRSC-595: Executing upgrade step 4 of 18: ‘GenSiteGUIDs’.
2021/06/22 18:38:23 CLSRSC-595: Executing upgrade step 5 of 18: ‘UpgPrechecks’.
2021/06/22 18:38:23 CLSRSC-595: Executing upgrade step 6 of 18: ‘SetupOSD’.
2021/06/22 18:38:23 CLSRSC-595: Executing upgrade step 7 of 18: ‘PreUpgrade’.
2021/06/22 18:38:24 CLSRSC-595: Executing upgrade step 8 of 18: ‘CheckCRSConfig’.
2021/06/22 18:38:25 CLSRSC-595: Executing upgrade step 9 of 18: ‘UpgradeOLR’.
2021/06/22 18:38:25 CLSRSC-595: Executing upgrade step 10 of 18: ‘ConfigCHMOS’.
2021/06/22 18:38:57 CLSRSC-595: Executing upgrade step 11 of 18: ‘UpgradeAFD’.
2021/06/22 18:38:58 CLSRSC-595: Executing upgrade step 12 of 18: ‘createOHASD’.
2021/06/22 18:38:59 CLSRSC-595: Executing upgrade step 13 of 18: ‘ConfigOHASD’.
2021/06/22 18:39:03 CLSRSC-595: Executing upgrade step 14 of 18: ‘InstallACFS’.
2021/06/22 18:39:05 CLSRSC-595: Executing upgrade step 15 of 18: ‘InstallKA’.
2021/06/22 18:39:06 CLSRSC-595: Executing upgrade step 16 of 18: ‘UpgradeCluster’.
2021/06/22 18:41:05 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 19 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
2021/06/22 18:42:08 CLSRSC-595: Executing upgrade step 17 of 18: ‘UpgradeNode’.
Start upgrade invoked..
2021/06/22 18:42:13 CLSRSC-478: Setting Oracle Clusterware active version on the last node to be upgraded
2021/06/22 18:42:13 CLSRSC-482: Running command: ‘/u00/grid/oracle/product/19.0.0.0/grid/bin/crsctl set crs activeversion’
Started to upgrade the active version of Oracle Clusterware. This operation may take a few minutes.
Started to upgrade CSS.
CSS was successfully upgraded.
Started to upgrade Oracle ASM.
Started to upgrade CRS.
CRS was successfully upgraded.
Started to upgrade Oracle ACFS.
Oracle ACFS was successfully upgraded.
Successfully upgraded the active version of Oracle Clusterware.
Oracle Clusterware active version was successfully set to 19.0.0.0.0.
2021/06/22 18:43:22 CLSRSC-479: Successfully set Oracle Clusterware active version
2021/06/22 18:43:23 CLSRSC-476: Finishing upgrade of resource types
2021/06/22 18:43:33 CLSRSC-477: Successfully completed upgrade of resource types
Related posts