Thursday, July 19, 2012

ORA-27037 followed by "Error: 2: No such file or directory"

Recently I had a situation on one of production database, occasion errors in alaert log which looks like:
Mon Jul  2 19:53:53 2012
Errors in file /u01/DB/mydb/10.2.0/admin/DB/udump/db2_ora_25488.trc:
ORA-07445: exception encountered: core dump [kpofgi()+2144] [SIGSEGV] [unknown code] [0x000000000] [] []
Mon Jul  2 19:54:12 2012
Trace dumping is performing id=[cdmp_20120702195412]
Mon Jul  2 19:55:02 2012
Errors in file /u01/DB/mydb/10.2.0/admin/DB/bdump/db2_diag_6145.trc:
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=ORA
Linux-ia64 Error: 2: No such file or directory
Additional information: 9
ORA-27037: Message 27037 not found; No message file for product=RDBMS, facility=
Mon Jul  2 19:57:03 2012
Completed checkpoint up to RBA [0x179c1.2.10], SCN: 21475188014
Mon Jul  2 19:59:54 2012
Incremental checkpoint up to RBA [0x179c1.e6.0], current log tail at RBA [0x179c1.2771.0]
Interesting was that many ORA-07445 on same database were placed "standalone" (with very same parameters as shown) and some are like shown example, following by other errors. After searching Internet haven't found a cause of such a behavior.

Because there were no other implications on database I left that case for better times.

And that time came very soon when one day, I was adding some commands in Linux crontab. Here is the interesting part (separate in two rows):
00,05,10,15,20,25,30,35,40,45,50,55 * * * * 
rm -f /u01/DB/mydb/10.2.0/admin/DB/udump/*.trc & rm -rf /u01/DB/mydb/10.2.0/admin/DB/bdump/cdmp* 
So every 5 minutes, crontab was deleting all trace files as well as cdump directories in shown directories. And that was it!

When happened that Oracle has a ORA which was creating trace file as well as dump directory entries and in the same moment crontab delete all of them, Oracle is left without files/directories in which it was wring data just moment ago.

Hope this helps someone.

Cheers!

1 comment:

  1. Make 15 minutes delay between deleting ....

    ReplyDelete