rootupgrade.sh (and root.sh for that matter) are restartable since 11.2.0.2 at least
A great deal of logging is available in $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_hostname.log
Checkpoint file : /u01/app/grid/Clusterware/ckptGridHA_grac31.xml
Log File : $GRID_HOME/cfgtoollogs/crsconfig/rootcrs_grac31.log
Note : This article simulates and fix a failed upgrade by having account qosadmim deactivated -
( see Note 1577072.1 for details )
Verify that we are ready to UPGRADE with orachk *
[oracle@grac31 ORACHECK225]$ ./orachk -u -o pre
Enter upgrade target version (valid versions are 11.2.0.3.0, 11.2.0.4.0, 12.1.0.1.0, 12.1.0.2.0):- 11.2.0.4.0
CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11203/grid?[y/n][y]
Checking ssh user equivalency settings on all nodes in cluster
Data collections completed. Checking best practices on grac31.
--------------------------------------------------------------------------------------
INFO => Prior to Upgrade Verify NO Materialized Views Being Refreshed (Automatically or Manually) for on11203
INFO => Special Considerations for SCAN and HAIP of 11.2.0.2 and above Grid Infrastructure
INFO => Minimizing Planned Downtime for Upgrades and Patching
INFO => Upgrade Related References
INFO => Lifetime Support Policy and Product Certification INFO
INFO => Capture performance baseline, backup important configuration files and batch, cron, DBMS_JOBS and DBMS_SCHEDULER jobs
INFO => Known Issues Integrating pre-11gR2 Databases with Grid Infrastructure
INFO => Be Aware of New Instant Client for Simplified Deployment
INFO => Oracle E-Business Suite interoperability and migration resources
INFO => Oracle Software Download Sites
WARNING => One or More Column Names in ALL_TAB_COLUMNS table are Reserved Words for on11203
WARNING => Some Users Needing Network ACLs for Oracle Utility Packages Found for on11203
WARNING => Package unixODBC-2.2.14-11.el6-i686 is recommended but NOT installed
WARNING => Package unixODBC-devel-2.2.14-11.el6-i686 is recommended but NOT installed
INFO => Steps to execute rootupgrade.sh
INFO => Save "Oracle Enterprise Manager Database Control" files and data with the emdwgrd Utility before upgrading database.
INFO => Information about ASM process parameter when its not set to default value
INFO => Information about Grid Infrastructure software installation directory
WARNING => Package unixODBC-devel-2.2.14-11.el6-x86_64 is recommended but NOT installed
---------------------------------------------------------------------------------
CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
Detailed report (html) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522/orachk_grac31_on11203_092214_152522.html
UPLOAD(if required) - /home/oracle/ORACHECK225/orachk_grac31_on11203_092214_152522.zip
Check that qosadmin account is disabled which aborts/stops the upgrade process ( see Doc ID 1577072.1 )
[root@grac31 Desktop]# qosctl qosadmin -listusers
AbstractLoginModule password: <-- oracle112
Sep 22, 2014 7:05:48 PM oracle.security.jazn.spi.xml.XMLRealmUser authenticate
INFO: User(jazn.com/qosadmin) is deactivated. AUTH FAILURE.
Sep 22, 2014 7:05:48 PM oracle.security.jazn.login.module.RealmLoginModule authenticate
SEVERE: [RealmLoginModule] authentication failed
Note : The above error will abort the rootupgrade.sh script later on
Prepare and Install new 11.2.0.4 GRID_HOME *
[grid@grac31 ~]$ env | grep -i ORAC
ORACLE_BASE=/u01/app/grid
ORACLE_HOME=/u01/app/11204_upg/grid
[grid@grac31 ~]$ ls -ld /u01/app/oracle
drwxrwxr-x. 7 oracle oinstall 4096 Sep 19 17:37 /u01/app/oracle
Create the GRID_HOME directory
[root@grac31 Desktop]# mkdir -p /u01/app/11204_upg/grid
[root@grac31 Desktop]# chown grid:oinstall /u01/app/11204_upg/grid
[root@grac31 Desktop]# chmod 775 /u01/app/11204_upg/grid
[root@grac31 Desktop]# ls -ld /u01/app/11204_upg/grid
drwxrwxr-x. 2 grid oinstall 4096 Sep 22 19:12 /u01/app/11204_upg/grid
Invoke Installer
[grid@grac31 grid]$ ./runInstaller
--> Upgrade GRID Infrastructure or Oracle ASM
[root@grac31 Desktop]# /u01/app/11204_upg/grid/rootupgrade.sh
..
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer
ASM upgrade has started on first node.
ASM upgrade has started on first node.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'grac31'
CRS-2673: Attempting to stop 'ora.crsd' on 'grac31'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'grac31'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'grac31'
..
CRS-2677: Stop of 'ora.drivers.acfs' on 'grac31' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'grac31' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'grac31'
CRS-2677: Stop of 'ora.gpnpd' on 'grac31' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'grac31' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0
ASM upgrade has finished on last node.
Failed to perform J2EE (OC4J) Container Resource upgrade at /u01/app/11204_upg/grid/crs/install/crsconfig_lib.pm line 9323.
/u01/app/11204_upg/grid/perl/bin/perl -I/u01/app/11204_upg/grid/perl/lib -I/u01/app/11204_upg/grid/crs/install
/u01/app/11204_upg/grid/crs/install/rootcrs.pl execution failed
Only ora.oc4j resource is OFFLINE
[root@grac31 Desktop]# crs
NAME TARGET STATE SERVER STATE_DETAILS
------------------------- ---------- ---------- ------------ ------------------
ora.oc4j OFFLINE OFFLINE
Fix the rootupgrade problem by following Note 1577072.1 *
For Case I - Upgrading Grid Infrastructure from 11.2.0.3 GI PSU 6 or higher to 11.2.0.4
Please perform the following workaround as grid user either before rootupgrade.sh is issued or after rootupgrade.sh fails with above error:
1. Activate the qosadmin user from 11.2.0.3 GRID_HOME as grid user:
grid@grac31 grid]$ env | grep GRID
GRID_HOME=/u01/app/11203/grid
[grid@grac31 grid]$ env | grep PATH
PATH=/u01/app/11203/grid/bin:.:/u01/app/11203/grid/bin:
Change qosadmin password to neworacle112
[grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin oracle112 neworacle112
AbstractLoginModule password: <<< enter oracle112
Please re-enter the password entered on the command line above
Enter Password: <<< enter oracle112
[grid@grac31 grid]$ qosctl qosadmin -listusers
AbstractLoginModule password: <<< enter neworacle112
oc4jadmin
JtaAdmin
qosadmin
The next steps are not needed but I want to have back ther original password in my test env
Change qosadmin password back to oracle112 ( here we use the default password again )
[grid@grac31 grid]$ qosctl qosadmin -setpasswd qosadmin neworacle112 oracle112
AbstractLoginModule password: <<< enter neworacle112
Please re-enter the password entered on the command line above
Enter Password: <<< enter neworacle112
[grid@grac31 grid]$ qosctl qosadmin -listusers
AbstractLoginModule password: <<< enter oracle112
oc4jadmin
JtaAdmin
qosadmi
Rerun rootupgrade.sh from updated 11.2.0.4 CRS Home
/u01/app/11204_upg/grid
[root@grac31 grid]# pwd
/u01/app/11204_upg/grid
[root@grac31 grid]# ./rootupgrade.sh
..
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-1115: Oracle Clusterware has already been upgraded.
ASM upgrade has finished on last node.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Related rootcrs_grac31.log entries:
> Oracle Clusterware version on node [grac31] is [11.2.0.4.0]
>End Command output
2014-09-23 08:38:29: Version String passed is: Oracle Clusterware version on node [grac31] is [11.2.0.4.0]
2014-09-23 08:38:29: Version Info returned is : 11.2.0.4.0
2014-09-23 08:38:29: Got CRS softwareversion for grac31: 11.2.0.4.0
2014-09-23 08:38:29: The software version on grac31 is 11.2.0.4.0
2014-09-23 08:38:29: Invoking "/u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first"
2014-09-23 08:38:29: trace file=/u01/app/11204_upg/grid/cfgtoollogs/crsconfig/srvmcfg0.log
2014-09-23 08:38:29: Executing /u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first
2014-09-23 08:38:29: Executing cmd: /u01/app/11204_upg/grid/bin/srvctl upgrade model -s 11.2.0.3.0 -d 11.2.0.4.0 -p first
2014-09-23 08:38:31: srvctl upgrade model -first ... passed
2014-09-23 08:38:31: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl stop oc4j
2014-09-23 08:38:31: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl stop oc4j '
2014-09-23 08:38:32: Removing file /tmp/filefdfk1C
2014-09-23 08:38:32: Successfully removed file: /tmp/filefdfk1C
2014-09-23 08:38:32: /bin/su exited with rc=2
2014-09-23 08:38:32: J2EE (OC4J) Container Resource Stop ... passed ...
2014-09-23 08:38:32: Running as user grid: /u01/app/11204_upg/grid/bin/srvctl disable oc4j
2014-09-23 08:38:32: s_run_as_user2: Running /bin/su grid -c ' /u01/app/11204_upg/grid/bin/srvctl disable oc4j '
2014-09-23 08:38:32: Removing file /tmp/file5FS4ZR
2014-09-23 08:38:32: Successfully removed file: /tmp/file5FS4ZR
2014-09-23 08:38:32: /bin/su exited with rc=1
2014-09-23 08:38:32: J2EE (OC4J) Container Resource Disable ... passed
2014-09-23 08:38:32: J2EE (OC4J) Container Resource stopped and disabled
2014-09-23 08:38:32: isLastNodeToUpgrade...
2014-09-23 08:38:32: isLastNodeToUpgrade: 1
2014-09-23 08:38:32: Executing /u01/app/11204_upg/grid/bin/crsctl set crs activeversion
2014-09-23 08:38:32: Executing cmd: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion
2014-09-23 08:38:32: Command output:
> CRS-1115: Oracle Clusterware has already been upgraded.
>End Command output
2014-09-23 08:38:32: /u01/app/11204_upg/grid/bin/crsctl set crs activeversion ... passed
2014-09-23 08:39:32: Rolling upgrade is set to 1
2014-09-23 08:39:32: End ASM rolling upgrade
2014-09-23 08:39:32: Executing as grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid
2014-09-23 08:39:32: Running as user grid: /u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid
2014-09-23 08:39:32: Invoking "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid" as user "grid"
2014-09-23 08:39:32: Executing /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid"
2014-09-23 08:39:32: Executing cmd: /bin/su grid -c "/u01/app/11204_upg/grid/bin/asmca -silent -upgradeLocalASM -lastNode /u01/app/11203/grid"
2014-09-23 08:39:45: Command output:
>
> ASM upgrade has finished on last node.
..
Rerun ./rootupgrade.sh a 3rd time ( now we don’t see any CRS-1115 ) *
[root@grac31 grid]# ./rootupgrade.sh
..
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11204_upg/grid/crs/install/crsconfig_params
Installing Trace File Analyzer
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Reference *
- GI Upgrade from 11.2.0.3.6+ to 11.2.0.4 or 12.1.0.1 Fails with User(qosadmin) is deactivated. AUTH FAILURE. (Doc ID 1577072.1)
- How to Downgrade 11.2 Grid Infrastructure Standalone (Oracle Restart) After Successful or Failed Upgrade (Doc ID 1364412.1)
Please, take a look on the documentations below:
