ORA-16038 TNS-12535 ORA-03135 archiver hung

ORA-16038 TNS-12535 ORA-03135 archiver hung at timeline number.

The archiver hung at time/line number due to Possible N/W disconnect with Primary DB.
The RFS process on the standby gets the network disconnection error TNS-12535 and
the TT00 process on the primary throws the ORA-3135..

Below errors found in the alertlog:

ORA-16038: log 109 sequence# 107104 cannot be archived (Primary)
RFS[27]: Possible network disconnect with primary database(RFS process on Secondary)
ORA-03135 ON TTNN AT PRIMARY
RFS[29]: Possible network disconnect with primary database
RFS[27]: Possible network disconnect with primary database

Root cause:

The Error has occurred in Prod database due to “ network glitch “ between primay and DR.

Actions to be taken:

Coordinate with network team

Complete Sample alert look like below:

Subject: EM Event: Critical:ORADW1_ORADW11 – The archiver hung at time/line number:

Host=compnode05
Target type=Database Instance
Target name=ORADW1_ORADW11
Categories=Fault
Message=The archiver hung at time/line number:
Severity=Critical
Platform=x86_64
Operating System=Linux
Associated Incident Id=170935
Associated Incident Status=New
Associated Incident Owner=
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=None
Associated Incident Escalation Level=0
Event Type=Metric Alert
Event name=alertLog:archiveHungErrStack
Metric Group=Alert Log
Metric=Archiver Hung Alert Log Error
Metric value=ORA-16038: log 109 sequence# 107104 cannot be archived~LNS: Failed to archive log 109 thread 1 sequence 107104 (3135)
Key Column 1=Time/Line Number
Rule Name=Incident management rule set for all targets,Create incident for critical metric alerts
Rule Owner=System Generated
Update Details:
The archiver hung at time/line number:
Incident created by rule (Name = Incident management rule set for all targets,
Create incident for critical metric alerts [System generated rule]).

 

See Also

How to verify DR lag ?

cksum values for primary & on DR is not matching

http://oracle.com