16 July 2014

Oracle SYS password change: Potential bug in scp

This was an interesting problem, because it happened after a very common and simple procedure.

Error

After changing SYS password on 11.2.0.2 database, we started receiving errors  ORA-1017 ORA-1031 ORA-16373 ORA-16778 ORA-16810

 We scp-ed the password file from RAC production nodes to DR nodes, and restarted the DG log apply, as per normal process. Data Guard started failing intermittently, with files not being applied on either of an instance thread for hours, hearbeat and HC were failing.
Database  use ASYNC log applym has sec_case_sensitive_logon=TRUE, no dedicated redo_transport_user user and only SYS is in pwfile.

 Note: 11.2.0.3 or 11.2.0.4 databases didn't have the same issue, which initially got us thinking that it was a 11.2.0.2 related bug, but it turned out to be a problem with scp.

Fix 

Simon checksum-ed oracle password file with "sum" command, and discovered that scp-ed files had a different check-sum. He took the valid orapw file and scp-ed to the other production node and two DR nodes, checksum-ed, restarted log apply and Data Guard started working.
2014-07-14 16:24:46.042 00000000 1856455312 DMON: Database DBDR_A returned ORA-16664 

2014-07-14 16:24:46.042 00000000 1856455312 for opcode = HEALTH_CHECK, phase = BEGIN, req_id = 1.1.1856455312 ....
....
2014-07-14 15:18:38.033 RSM detected log transport problem: log transport for database 'DBDR_A' has the following error. 
2014-07-14 15:18:38.033 ORA-01031: insufficient privileges 
2014-07-14 15:18:38.034 RSM0: HEALTH CHECK ERROR: ORA-16737: the redo transport service for standby database "DBDR_A" has an error 
2014-07-14 15:18:38.297 00000000 1856454759 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16778 
2014-07-14 15:18:38.300 00000000 1856454759 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16810 *** 
2014-07-14 15:20:51.155 4265 krsh.c Error 1031 received logging on to the standby *** 
2014-07-14 15:20:51.155 912 krsu.c Error 1031 connecting to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = xxxxx)(PORT = xxx1))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))' Error 1031 attaching to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL = TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))' *** 
2014-07-14 15:20:51.155 2864 krsi.c krsi_dst_fail: dest:2 err:1031 force:0 blast:1 
kcrrwkx: unknown error:1031 
 OCIServerAttach failed -1 .. Detailed OCI error val is 12514 and errmsg is 'ORA-12514: TNS:listener does not currently know of service requested in conn ect descriptor ' NSV1: Failed to connect to remote database DBDR_A. Error is ORA-12514 NSV1: Connect attempt to database DBDR_A failed, status = ORA-12514. 
Failed to send message to site DBDR_A. Error code is ORA-12514 ...
.....
*** 2014-07-14 15:20:51.155 4265 krsh.c
Error 1031 received logging on to the standby
*** 2014-07-14 15:20:51.155 912 krsu.c
Error 1031 connecting to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL =
 TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))'
Error 1031 attaching to destination LOG_ARCHIVE_DEST_2 standby host '(DESCRIPTION = (LOAD_BALANCE = ON)(ADDRESS = (PROTOCOL =
TCP)(HOST = xxx)(PORT = xxx))(CONNECT_DATA = (SERVICE_NAME = DBDR_A)))'
*** 2014-07-14 15:20:51.155 2864 krsi.c
krsi_dst_fail: dest:2 err:1031 force:0 blast:1
kcrrwkx: unknown error:1031

Labels:


// posted by Tatjana (Tanja) Kane @ 7/16/2014
Comments: Post a Comment



<< Home

This page is powered by Blogger. Isn't yours?