Control-M: 23_EUQ - Event: JOB NOT OK App: GLOBAL TREASURY-IS-ECC-DEV-D30 Job: ORACLE-DB-ARCHIVES Folder: 9EUDV23_TRE-DATABASE-ECC
Incident Resolution Log 
1. Issue Summary 
The Oracle database instance was running normally, and the database role was Primary (HA DB). The scheduled archivelog backup job was triggered and initiated successfully. The backup utility (backint) completed the backup of archived redo logs and profile/log files. However, the overall job was marked as failed due to database connection errors encountered by BRARCHIVE during execution. 
 
2. Observations 
 
Oracle instance status: Running 
Database role: Primary, High Availability (HA) Database 
Archivelog backup: Started successfully 
backint utility: 
 
Successfully backed up 4 archived redo log files 
Successfully backed up related profile and log files 
 
 
Archived redo logs: 
 
Saved successfully 
Deleted after successful backup 
 
 
Backup utility return status: Success 
 
Despite the successful backup of files, BRARCHIVE reported repeated connection failures to the database. 
 
3. Error Details 
The following error was logged multiple times during the backup execution: 
Plain TextORA-01017: invalid username/password; logon denied``Show more lines 
This error occurred whenever BRARCHIVE attempted to connect to the database using OS authentication: 
Plain TextCONNECT /Show more lines 
Runtime environment details at the time of failure: 
Plain TextRunning user  : d30admOS user       : orad30Primary group : sapsysORACLE_SID    : D30Show more lines 
Due to this connection failure, BRARCHIVE could not consistently validate the database state and therefore completed with warnings, resulting in a return code = 1, which caused the wrapper script to flag the backup as failed. 
 
4. Root Cause 
A planned Kernel upgrade activity was in progress at the time the backup job was triggered. During the kernel upgrade: 
 
oraroot.sh file permissions were modified 
OS-level authentication was temporarily impacted 
As a result, the SAP Oracle owner (d30adm) was unable to authenticate via OS authentication (CONNECT /) 
This led to repeated ORA-01017 errors during the BRARCHIVE execution 
 
The initial archivelog backup attempts failed due to permission issues caused by the kernel upgrade. After the kernel upgrade activity was completed and permissions were restored, subsequent archivelog backups completed successfully. 
 
5. Actions Taken 
 
Reviewed BRARCHIVE and backup logs to validate backup activity 
Confirmed that archived redo logs were successfully backed up and deleted 
Correlated the backup failure timeline with the ongoing kernel upgrade activity 
Verified that once kernel upgrade activities were completed, subsequent backups succeeded without ORA-01017 errors 
 
7. Workaround 
 
Avoid triggering SAP Oracle backup jobs during kernel upgrade or OS maintenance windows 
Temporarily pause or reschedule backup jobs while system-level permission changes are in progress 
 
8. Solution / Preventive Measures 
 
Ensure oraroot.sh and related Oracle binaries retain correct ownership and permissions after kernel upgrades 
Validate OS authentication (sqlplus / as sysdba) post-maintenance 
Align backup schedules with maintenance windows to prevent overlap 
Perform a post-upgrade validation check for SAP and Oracle OS authentication before resuming scheduled jobs