16 July 2014

Notes: Instructions on patching 11.2.0.x database to 11.2.0.4

Install 11.2.0.4 software

Run Xming
export DISPLAY=xxx:0.0
export TEMP=xxxx
cd your_stage_directorey
Check first what options are installed on your database:
select * from v$option where PARAMETER in
('Partitioning','OLAP','Oracle Label Security','Oracle Database Vault','Real Application Testing','Data Mining');
cd stage_dir/18031668
opatch prereq CheckConflictAgainstOHWithDetail -ph ./
               There should be no conflicts
opatch apply
This will apply 17478514  18031668. After that you need to apply two more patches.
ld: warning: option -Q appears more than once, first setting taken  -- You can safely ignore            Each patch takes few minutes
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> STARTUP
SQL> @catbundle.sql psu apply
select 
substr(id,1,10) id,
substr(action,1,10) action,
substr(version,1,8) version,
substr(BUNDLE_SERIES,1,6) bundle,
substr(comments,1,20) comments
from registry$history;

./runInstaller
.... takes 5-10 min

Patching

Set new line in oratab: (I like setting env this way, you don't have to)
vi /var/opt/oracle/oratab
new:/opt/oracle/product/db/11.2.0.4:Y
. oraenv with new

Install OPatch utility version 11.2.0.3.6 or later
            cd stage_dir/OPatch
            cp p6880880_112040_SOLARIS64.zip $ORACLE_HOME
            oh;unzip p6880880_112040_SOLARIS64.zip
export PATH=$ORACLE_HOME/OPatch:/usr/ccs/bin:$PATH
cd ../17551261; opatch apply
cd ../17767676; opatch apply

** clean up ~/temp and remove "new" from oraenv

Upgrade Database

Run this before upgrade: @$ORACLE_HOME/rdbms/admin/utlu112i.sql
PURGE DBA_RECYCLEBIN;
EXECUTE dbms_stats.gather_dictionary_stats;
show parameter recycle
alter system set recyclebin=off scope=spfile;
restart
export DISPLAY=xxx:0.0
export TEMP=xxx    

DR part

On the production:
alter system switch logfile;
alter system set db_recovery_file_dest_size=6G scope=both;
DGMGRL> DISABLE CONFIGURATION;
Both prod and DR: SQL> ALTER SYSTEM SET DG_BROKER_START=FALSE;
alter system set log_archive_dest_state_2='DEFER' scope=both;
show parameter dg_broker_config
Make a copy of the file
Shut down standby
CREATE RESTORE POINT UPG_11204 GUARANTEE FLASHBACK DATABASE;

cd $ORACLE_HOME//bin
./dbua
            (use all default option, except for:
            -- untick "Configure DB with OEM"
            Started at 7:40am - (50% at 8am) finished at 8:07am.
 Errors with ORA-6512 WMSYS.OWM_MIG_PKG can be ignored.

After upgrade

On both prod and DR:
export ORACLE_HOME=$ORACLE_HOME (old home)
oh; cd bin
stop listeners
. oraenv
xxx
oh
cp network/admin/*.ora file to a new home 
alter system set recyclebin=on scope=spfile;
restart
start listeners

DR part

Standby:
Make a copy of the existing password file
Copy the production password file from the old home
Copy the spfile files from the old home
Copy the broker files from the old home
cp network/admin/*.ora file to a new home 
vi /var/opt/oracle/oratab and change the oracle home
. oraenv
Mount the physical standby database(s) on the new Oracle home (upgraded version).
            Note: The standby database(s) should not be opened until the primary database upgrade is completed.

Production:
            alter system set log_archive_dest_state_2='ENABLE' scope=both;

Both : ALTER SYSTEM SET DG_BROKER_START=TRUE scope=both;
DGMGRL> ENABLE CONFIGURATION
alter system set db_recovery_file_dest_size=2G scope=both;  --put back the old size
If all ok, DROP RESTORE POINT UPG_11204;

Check DST is version 14:
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=1541322717962474&id=1579838.1&_afrWindowMode=0&_adf.ctrl-state=eps5f0qjm_490#aref_section32
SELECT version FROM v$timezone_file;

Verify the PSU Post Install steps were run in the DB
select substr(action_time,1,30) action_time, 

rman target xx@xx catalog
UPGRADE CATALOG
UPGRADE CATALOG (do it twice)

Change the housekeeping file
Change any other scripts/cron jobs
Change the Config in the Grid

Deinstall

Deinstall the home:
oh; cd ../11.2.0.3/network/admin
mv listener.ora listener.ora.old
set ORACLE_HOME=xxx (old home)
cd $ORACLE_HOME
$ORACLE_HOME/deinstall/deinstall




// posted by Tatjana (Tanja) Kane @ 7/16/2014 1 comments

Oracle SYS password change: Potential bug in scp

This was an interesting problem, because it happened after a very common and simple procedure.

Error

After changing SYS password on 11.2.0.2 database, we started receiving errors  ORA-1017 ORA-1031 ORA-16373 ORA-16778 ORA-16810

 We scp-ed the password file from RAC production nodes to DR nodes, and restarted the DG log apply, as per normal process. Data Guard started failing intermittently, with files not being applied on either of an instance thread for hours, hearbeat and HC were failing.
Database  use ASYNC log applym has sec_case_sensitive_logon=TRUE, no dedicated redo_transport_user user and only SYS is in pwfile.

 Note: 11.2.0.3 or 11.2.0.4 databases didn't have the same issue, which initially got us thinking that it was a 11.2.0.2 related bug, but it turned out to be a problem with scp.

Fix 

Simon checksum-ed oracle password file with "sum" command, and discovered that scp-ed files had a different check-sum. He took the valid orapw file and scp-ed to the other production node and two DR nodes, checksum-ed, restarted log apply and Data Guard started working.
2014-07-14 16:24:46.042 00000000 1856455312 DMON: Database DBDR_A returned ORA-16664 

2014-07-14 16:24:46.042 00000000 1856455312 for opcode = HEALTH_CHECK, phase = BEGIN, req_id = 1.1.1856455312 ....
....
2014-07-14 15:18:38.033 RSM detected log transport problem: log transport for database 'DBDR_A' has the following error. 
2014-07-14 15:18:38.033 ORA-01031: insufficient privileges 
2014-07-14 15:18:38.034 RSM0: HEALTH CHECK ERROR: ORA-16737: the redo transport service for standby database "DBDR_A" has an error 
2014-07-14 15:18:38.297 00000000 1856454759 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16778 
2014-07-14 15:18:38.300 00000000 1856454759 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16810 *** 
2014-07-14 15:20:51.155 4265 krsh.c Error 1031 received logging on to the standby *** 
2014-07-14 15:20:51.155 912 krsu.c Error 1031 connecting to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = xxxxx)(PORT = xxx1))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))' Error 1031 attaching to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))' *** 
2014-07-14 15:20:51.155 2864 krsi.c krsi_dst_fail: dest:2 err:1031 force:0 blast:1 
kcrrwkx: unknown error:1031 
 OCIServerAttach failed -1 .. Detailed OCI error val is 12514 and errmsg is 'ORA-12514: TNS:listener does not currently know of service requested in conn ect descriptor ' NSV1: Failed to connect to remote database DBDR_A. Error is ORA-12514 NSV1: Connect attempt to database DBDR_A failed, status = ORA-12514. 
Failed to send message to site DBDR_A. Error code is ORA-12514 ...
.....
*** 2014-07-14 15:20:51.155 4265 krsh.c
Error 1031 received logging on to the standby
*** 2014-07-14 15:20:51.155 912 krsu.c
Error 1031 connecting to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL =
 TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))'
Error 1031 attaching to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL =
TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))'
*** 2014-07-14 15:20:51.155 2864 krsi.c
krsi_dst_fail: dest:2 err:1031 force:0 blast:1
kcrrwkx: unknown error:1031

Labels:


// posted by Tatjana (Tanja) Kane @ 7/16/2014 0 comments

This page is powered by Blogger. Isn't yours?