导读：某客户单位数据库出现异常，大致现象是：数据库状态是open的，但是其中一个数据文件无法访问，本文分享排查原因与解决问题的整个过程。

通过ls 查看文件都报错，如下所示：

1. 
   
2. [oracle@oracledata01 oracle]$ ls
   
3. ls: 无法访问zf4.dbf: 输入/输出错误<font face="Tahoma">
   
4. </font>

复制代码

同时dd命令也无法读取该文件：

1. 
   
2. [oracle@oracledata01 fd]$ dd if=zf4.dbf bs=8192 count=1 skip=1|od -x|header -1
   
3. bash: header: command not found
   
4. dd: 正在打开"zf4.dbf": 没有那个文件或目录<font face="Tahoma">
   
5. </font>

复制代码

此时在从数据库层面看到的信息类似如下：

1. 
   
2. Tue Apr 28 16:49:51 2020
   
3. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_9146.trc:
   
4. ORA-01157: 无法标识/锁定数据文件 11 - 请参阅 DBWR 跟踪文件
   
5. ORA-01110: 数据文件 11: '/oracle_data/oracle/zf4.dbf'
   
6. ORA-1157 signalled during: ALTER DATABASE OPEN...
   
7. Tue Apr 28 16:49:51 2020
   
8. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m000_9152.trc:
   
9. ORA-27037: 无法获得文件状态
   
10. Linux-x86_64 Error: 5: Input/output error
    
11. Additional information: 3
    
12. ORA-01122: 数据库文件 11 验证失败
    
13. ORA-01110: 数据文件 11: '/oracle_data/oracle/zf4.dbf'
    
14. ORA-01565: 标识文件 '/oracle_data/oracle/zf4.dbf' 时出错
    
15. ORA-27037: 无法获得文件状态
    
16. Linux-x86_64 Error: 5: Input/output error
    
17. Additional information: 3
    
18. Tue Apr 28 16:50:19 2020
    
19. USER (ospid: 9309): terminating the instance<font face="Tahoma">
    
20. </font>

复制代码

从上述信息来看，数据库第11号文件无法读取。

在我们进行恢复之前，客户进行了多次尝试，还遭遇了一些数据库坏块，也遇到了不少ora-00600 错误：

1. 
   
2. Wed Apr 29 03:10:12 2020
   
3. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j009_65816.trc:
   
4. ORA-12012: 自动执行作业 "ZX_XT"."JS_XT_XZGLXX_XZXK_FR" 出错
   
5. ORA-06502: PL/SQL: 数字或值错误 :  字符串缓冲区太小
   
6. ORA-06512: 在 "ZX_XT.PKG_双公示接收_邢_2019", line 1071
   
7. ORA-00942: 表或视图不存在
   
8. ORA-02063: 紧接着 line (起自 BM_SJJH)
   
9. alter database datafile 11 resize 20g
   
10. Completed: alter database datafile 11 resize 20g
    
11. Wed Apr 29 03:10:15 2020
    
12. Corrupt Block Found
    
13. CONT = 0, TSN = 1, TSNAME = SYSAUX
    
14. RFN = 3, BLK = 3, RDBA = 12582915
    
15. OBJN = 1, OBJD = -1, OBJECT = _NEXT_OBJECT, SUBOBJECT =
    
16. SEGMENT OWNER = SYS, SEGMENT TYPE = Invalid Type
    
17. Wed Apr 29 03:10:15 2020
    
18. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m006_66106.trc  (incident=246250):
    
19. ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], []
    
20. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_246250/orcl_m006_66106_i246250.trc
    
21. Wed Apr 29 03:10:15 2020
    
22. ......
    
23. Wed Apr 29 03:11:00 2020
    
24. Doing block recovery for file 3 block 3
    
25. Resuming block recovery (PMON) for file 3 block 3
    
26. Block recovery from logseq 50078, block 15827 to scn 14514058257813
    
27. Wed Apr 29 03:11:00 2020
    
28. Recovery of Online Redo Log: Thread 1 Group 3 Seq 50078 Reading mem 0
    
29. Mem# 0: /oracle_data/oradata/orcl/redo03.log
    
30. Block recovery completed at rba 50078.23957.16, scn 3379.1363764630
    
31. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m004_66271.trc  (incident=245043):
    
32. ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], []
    
33. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245043/orcl_m004_66271_i245043.trc
    
34. Use ADRCI or Support Workbench to package the incident.
    
35. See Note 411.1 at My Oracle Support for error and packaging details.
    
36. Wed Apr 29 03:11:01 2020
    
37. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m004_66271.trc:
    
38. ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], []
    
39. Wed Apr 29 03:11:02 2020
    
40. ......
    
41. Wed Apr 29 03:15:49 2020
    
42. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j000_65779.trc  (incident=245026):
    
43. ORA-00600: internal error code, arguments: [13011], [7907], [12713924], [44], [12687314], [0], [], [], [], [], [], []
    
44. ORA-01403: no data found
    
45. ORA-06512: at "SYS.DBMS_AQ_INV", line 1248
    
46. ORA-06512: at line 1
    
47. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245026/orcl_j000_65779_i245026.trc
    
48. Wed Apr 29 03:15:52 2020
    
49. Dumping diagnostic data in directory=[cdmp_20200429031552], requested by (instance=1, osid=65779 (J000)), summary=[incident=245026].
    
50. Use ADRCI or Support Workbench to package the incident.
    
51. See Note 411.1 at My Oracle Support for error and packaging details.
    
52. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_j000_65779.trc  (incident=245027):
    
53. ORA-00600: internal error code, arguments: [13011], [7907], [12713924], [44], [12687314], [0], [], [], [], [], [], []
    
54. ORA-01403: no data found
    
55. ORA-06512: at "SYS.DBMS_AQ_INV", line 1248
    
56. ORA-06512: at line 1
    
57. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_245027/orcl_j000_65779_i245027.trc
    
58. Use ADRCI or Support Workbench to package the incident.
    
59. ......
    
60. Wed Apr 29 04:05:07 2020
    
61. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_m003_76294.trc:
    
62. ORA-00600: 内部错误代码, 参数: [kdBlkCheckError], [3], [3], [18018], [], [], [], [], [], [], [], []
    
63. Wed Apr 29 04:05:08 2020
    
64. Dumping diagnostic data in directory=[cdmp_20200429040508], requested by (instance=1, osid=76263), summary=[incident=282250].
    
65. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_76263.trc  (incident=282251):
    
66. ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], []
    
67. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282251/orcl_ora_76263_i282251.trc
    
68. Use ADRCI or Support Workbench to package the incident.
    
69. See Note 411.1 at My Oracle Support for error and packaging details.
    
70. Wed Apr 29 04:05:10 2020
    
71. Sweep [inc][282251]: completed
    
72. Sweep [inc2][282265]: completed
    
73. Sweep [inc2][282251]: completed
    
74. Wed Apr 29 04:05:10 2020
    
75. Dumping diagnostic data in directory=[cdmp_20200429040510], requested by (instance=1, osid=76263), summary=[incident=282251].
    
76. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_76263.trc  (incident=282252):
    
77. ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], []
    
78. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282252/orcl_ora_76263_i282252.trc
    
79. Use ADRCI or Support Workbench to package the incident.
    
80. See Note 411.1 at My Oracle Support for error and packaging details.
    
81. Wed Apr 29 04:07:26 2020
    
82. ......
    
83. Wed Apr 29 04:24:01 2020
    
84. Sweep [inc][279857]: completed
    
85. Sweep [inc2][279857]: completed
    
86. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78503.trc  (incident=279858):
    
87. ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], []
    
88. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_279858/orcl_ora_78503_i279858.trc
    
89. Use ADRCI or Support Workbench to package the incident.
    
90. See Note 411.1 at My Oracle Support for error and packaging details.
    
91. Wed Apr 29 04:24:03 2020
    
92. Sweep [inc][279858]: completed
    
93. Sweep [inc2][279858]: completed
    
94. Wed Apr 29 04:24:04 2020
    
95. Dumping diagnostic data in directory=[cdmp_20200429042404], requested by (instance=1, osid=78503), summary=[incident=279858].
    
96. Wed Apr 29 04:25:57 2020
    
97. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78738.trc  (incident=282222):
    
98. ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], []
    
99. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282222/orcl_ora_78738_i282222.trc
    
100. Use ADRCI or Support Workbench to package the incident.
     
101. See Note 411.1 at My Oracle Support for error and packaging details.
     
102. Wed Apr 29 04:25:58 2020
     
103. Sweep [inc][282222]: completed
     
104. Sweep [inc2][282222]: completed
     
105. Wed Apr 29 04:25:58 2020
     
106. Dumping diagnostic data in directory=[cdmp_20200429042558], requested by (instance=1, osid=78738), summary=[incident=282222].
     
107. Errors in file /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/trace/orcl_ora_78738.trc  (incident=282223):
     
108. ORA-00600: 内部错误代码, 参数: [qosdFindObjRead: objcnt mismatch], [], [], [], [], [], [], [], [], [], [], []
     
109. Incident details in: /oracle_app/Oracle12c/diag/rdbms/orcl/orcl/incident/incdir_282223/orcl_ora_78738_i282223.trc

复制代码

上述ora-00600 错误，总的来都是常见的，几乎都可以定性为数据坏块。

对于file 3的坏块，处理相对简单，直接drop或者truncate相关对象即可，然后重建awr。

但是对于第11号数据文件，怎么办呢？这里我们的处理方法是当文件丢失处理（很久之前处理过Windows环境数据文件大小为 0 kb的问题，这几种情况都类似。）

这里我说一下简单的处理思路：

1. 通过odu 扫盘，分别以不同offset方式来扫（0和4096）。

2. 对比两次扫描结果，组合extent信息，抽取文件。

由于这里文件只有1个，大小20g左右，通过dd 拼接即可。

最终文件拼接完成后，由于这里是非归档环境，无法进行正常recover，因此还需要通过bbed来修改数据文件头的checkpoint信息。

对于文件系统来讲，目前已经不是Oracle存储的最佳选择了，建议使用ASM。当然，其他数据库比如MySQL、PostgreSQL除外。