# C4 修复方案:非尾段损坏必须硬错误 ## TL;DR > **目标**:让 WAL recovery 按 design §3.2 line 704 区分"尾段损坏(可截断)"和"中间段损坏(硬错误)"。当前 `ReplaySegmentFile` 不论在哪个 segment 都把 CollectingFragments / parse error 当 TailCorruption,导致中间段损坏时 `Recover` 错误地截断最后一段的有效数据。 > > **交付**: > - `ReplaySegmentFile` 加 `isLastSegment bool` 参数 > - `TailCorruptionError` 加 `SegmentPath string` 字段(诊断 + 防御) > - 非尾段遇到 corruption → 硬错误(不再是 TailCorruptionError) > - `RecoverFromSegments` 传递 `isLastSegment` 给 `ReplaySegmentFile` > - `Recover` 用 `tce.SegmentPath` 做截断目标(防御性 fallback 到 segments[last]) > - ~5 个新测试覆盖单/多 segment 场景 > - 单次 commit > > **预估工时**:2-3 小时 > **风险**:中。`ReplaySegmentFile` 签名变更是 breaking change(影响测试调用) --- ## Context ### Bug 摘要 `wal/recovery.go:133-138` 在 `ReplaySegmentFile` 末尾: ```go if collector.State() == FragmentCollecting { return nextSequence, &TailCorruptionError{...} } ``` **不区分 segment 是不是最后一个**。同样地,`ParseRecordsFromFile` / `ParseBlock` 返回的 TailCorruptionError 也不带 segment 位置信息。 `Recover` 拿到 TailCorruptionError 后,**总是对 `segments[len(segments)-1]` 调用 truncate**(C5+H8 修复后的 `wal/recover.go:60-71`)。如果 corruption 实际发生在中间段: - 真正损坏的中间段不动 - 最后一段(完全有效)被错误截掉 - 中间段之后、最后段之前的有效 batch 全部丢失 ### Oracle 验证(bg_ef425776) > C4 还有一个更严重的失败模式。`wal/recover.go:61` 总是对 `segments[len(segments)-1]` 调用 `truncateSegment`,但 `RecoverFromSegments` 的 `TailCorruptionError` 可能来自非尾段。结果:真正损坏的 segment 不动,最后一段的有效数据被错误截掉。 ### 设计依据(§3.2 line 704) > 尾部损坏通常来自进程崩溃或机器掉电时最后一次写入的 partial write。**若后面还有需要恢复的 segment,前一个 segment 的尾部异常不能按尾部损坏截断**,否则可能跳过已经持久化历史并破坏 sequence 连续性。中间损坏说明已持久化 WAL 文件被破坏,**默认不静默跳过,不默认 repair**。 错误分类表(line 697-704)明确:所有 4 类物理损坏(header 半写 / length 越界 / CRC 失败 / 非 0 padding)在"WAL 中间"位置都是**硬错误**。 ### 修复后的行为矩阵 | Corruption 位置 | isLastSegment | 行为 | |----------------|--------------|------| | 最后一段尾部 | true | TailCorruptionError → Recover 截断(现有行为)| | 中间段任何位置 | false | **硬错误**(新行为)→ Recover 返回错误 → DB.Open 失败 | | 干净 WAL | N/A | 正常 replay,无 corruption | ### 协同:C5+H8 / C8 关系 - **C5+H8 已修**:截断路径用 `findLastCompleteBatchEnd`(H8)+ 4 步协议(C5)。C4 修复后,截断路径只在"最后一段尾部损坏"时触发,行为正确。 - **C8 未修**:segment_manager.go:64-66 把字节偏移当 startSequence。但 `writeTestSegment` 直接构造 segment,可以指定正确 startSequence,所以 C4 测试不依赖 C8。 - **多 segment recovery** 在 C4 修复后行为正确(前提是 startSequence 正确,即 C8 也已修)。但 C4 本身的修复不依赖 C8。 --- ## 执行计划 ### Phase A:代码改动(45-60 分钟) #### A.1 扩展 `TailCorruptionError` 加 `SegmentPath` 文件:`wal/record_parser.go` ```go type TailCorruptionError struct { Offset int // offset within the segment file SegmentPath string // NEW: file path of the corrupted segment, for diagnostics + Recover's truncation target Err error } func (e *TailCorruptionError) Error() string { if e.SegmentPath != "" { return fmt.Sprintf("wal: tail corruption in %s at offset %d: %v", e.SegmentPath, e.Offset, e.Err) } return fmt.Sprintf("wal: tail corruption at offset %d: %v", e.Offset, e.Err) } ``` > **必要 docstring**:字段含义 + 用途(diagnostics + Recover 的截断目标)。 #### A.2 `ReplaySegmentFile` 加 `isLastSegment` 参数 文件:`wal/recovery.go` ```go // 改前: func ReplaySegmentFile(filePath string, startSequence uint64, replayer BatchReplayer) (nextSequence uint64, err error) // 改后: func ReplaySegmentFile(filePath string, startSequence uint64, isLastSegment bool, replayer BatchReplayer) (nextSequence uint64, err error) ``` 函数体改动: ```go nextSequence = startSequence records, parseErr := ParseRecordsFromFile(filePath) if parseErr != nil && !IsTailCorruption(parseErr) { return nextSequence, fmt.Errorf("wal: parse segment records: %w", parseErr) } // Attach SegmentPath to parseErr for downstream diagnostics + truncation target. // Do this BEFORE the non-last hard-error conversion so both paths benefit. if parseErr != nil { var tce *TailCorruptionError if errors.As(parseErr, &tce) { tce.SegmentPath = filePath } } // C4 fix: tail corruption in non-last segment is hard corruption per // design §3.2 line 704. Use %v (NOT %w) so IsTailCorruption returns false // for this wrapped error — otherwise errors.As would still find the // underlying *TailCorruptionError and Recover would treat it as truncatable. if parseErr != nil && !isLastSegment { return nextSequence, fmt.Errorf("wal: corruption in non-last segment %s (hard corruption): %v", filePath, parseErr) } collector := NewFragmentCollector() for _, record := range records { // ... (unchanged) } if parseErr != nil { return nextSequence, parseErr } if collector.State() == FragmentCollecting { if isLastSegment { return nextSequence, &TailCorruptionError{ Offset: 0, SegmentPath: filePath, Err: errors.New("incomplete fragmented batch at segment tail"), } } // Non-last segment with incomplete fragments = middle corruption. // Plain error (no TailCorruptionError wrapping) — IsTailCorruption is false. return nextSequence, fmt.Errorf("wal: incomplete fragmented batch in non-last segment %s (hard corruption)", filePath) } return nextSequence, nil ``` #### A.3 `RecoverFromSegments` 传递 `isLastSegment` 文件:`wal/recovery.go` ```go nextSequence = segments[0].StartSequence for i, segment := range segments { if segment.StartSequence != nextSequence { return nextSequence, fmt.Errorf("wal: segment start sequence %d does not match expected sequence %d", segment.StartSequence, nextSequence) } isLastSegment := i == len(segments)-1 nextSequence, err = ReplaySegmentFile(segment.FilePath, nextSequence, isLastSegment, replayer) if err != nil { if IsTailCorruption(err) { return nextSequence, err } return nextSequence, fmt.Errorf("wal: replay segment: %w", err) } } ``` #### A.4 `Recover` 使用 `tce.SegmentPath`(防御性) 文件:`wal/recover.go`,C5+H8 修复后的 truncation 块: ```go if len(segments) > 0 { // After C4 fix, TailCorruptionError is only returned for the last segment. // Use tce.SegmentPath as authoritative truncation target (defensive: // fall back to segments[last] if missing). corruptedPath := segments[len(segments)-1].FilePath var tce *TailCorruptionError if errors.As(err, &tce) && tce.SegmentPath != "" { corruptedPath = tce.SegmentPath } lastCompleteBatchEnd, findErr := findLastCompleteBatchEnd(corruptedPath) if findErr != nil { return nil, fmt.Errorf("wal: recover: find truncation offset: %w", findErr) } var emptyTrailing []string if err := truncateAndPersist(corruptedPath, lastCompleteBatchEnd, dir, emptyTrailing); err != nil { return nil, fmt.Errorf("wal: recover: persist tail truncation: %w", err) } } ``` **关键不变量**:C4 fix 后 TailCorruptionError 只来自最后一段,所以 `corruptedPath == segments[last].FilePath`。`tce.SegmentPath` 是防御性显式表达,未来如果不变量破坏能立即发现。 > **关于 `errors.As` 用法**:`err` 在 Recover 函数签名里是 outer 变量(`nextSequence, err := RecoverFromSegments(...)`),shadowing 后再 `errors.As(err, &tce)` 读的是当前 outer err。需要确认 Go 的语义在这里正确(应该是的,但加注释或测试覆盖)。 ### Phase B:测试(60-90 分钟) #### B.1 `ReplaySegmentFile` 单元测试(isLastSegment 参数) 新增到 `wal/recovery_test.go`: ```go // Regression guards for C4: isLastSegment controls whether CollectingFragments // at end is tail corruption (truncatable) or hard corruption. func TestReplaySegmentFile_LastSegmentCollectingFragmentsIsTailCorruption(t *testing.T) { // Build segment with: [Batch A][First][Middle* no Last]. // Call ReplaySegmentFile with isLastSegment=true. // Expect: TailCorruptionError (can be truncated). // Expect: tce.SegmentPath == filePath. } func TestReplaySegmentFile_NonLastSegmentCollectingFragmentsIsHardError(t *testing.T) { // Same segment content as above. // Call ReplaySegmentFile with isLastSegment=false. // Expect: hard error (NOT TailCorruptionError). // Expect: error mentions "non-last segment" or "hard corruption". } func TestReplaySegmentFile_LastSegmentParseErrorIsTailCorruption(t *testing.T) { // Build segment with: [Batch A][CRC-corrupted bytes]. // Call with isLastSegment=true. // Expect: TailCorruptionError. } func TestReplaySegmentFile_NonLastSegmentParseErrorIsHardError(t *testing.T) { // Same segment content. // Call with isLastSegment=false. // Expect: hard error. } ``` #### B.2 `RecoverFromSegments` / `Recover` 集成测试 新增到 `wal/recover_test.go`: ```go // Regression guard for C4: middle segment corruption must hard-fail Recover, // NOT truncate the (valid) last segment. This is the key bug Oracle flagged: // "wal/recover.go:61 总是对 segments[len(segments)-1] 调用 truncateSegment, // 但 RecoverFromSegments 的 TailCorruptionError 可能来自非尾段". func TestRecoverMiddleSegmentCorruptionHardFails(t *testing.T) { dir := t.TempDir() // Construct 3 segments with valid startSequences. // segment-0: [Batch seq 0-1] (2 entries → next=2) // segment-1: [Batch seq 2-3] + [First][Middle no Last] ← middle corruption // (2 complete entries → next=4 if recovery reached end) // segment-2: [Batch seq 4-5] (2 entries; valid; never reached) writeTestSegment(t, dir, 0, 0, [][]*WalEntry{ {makePutEntry("k0", "v0"), makePutEntry("k1", "v1")}, }) seg1Path := writeTestSegment(t, dir, 1, 2, [][]*WalEntry{ {makePutEntry("k2", "v2"), makePutEntry("k3", "v3")}, }) // Append First+Middle fragments to segment-1 (no Last). appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecFirst, []byte("first-frag"))) appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecMiddle, []byte("middle-frag"))) writeTestSegment(t, dir, 2, 4, [][]*WalEntry{ {makePutEntry("k4", "v4"), makePutEntry("k5", "v5")}, }) seg2Path := filepath.Join(dir, "segment-2.wal") fiBefore, _ := os.Stat(seg2Path) _, err := Recover(dir, &mockReplayer{}) if err == nil { t.Fatal("expected Recover to fail on middle segment corruption") } if IsTailCorruption(err) { t.Errorf("expected hard error (not tail corruption) for middle segment; got %v", err) } // CRITICAL: segment-2 must NOT be truncated (it's completely valid). fiAfter, _ := os.Stat(seg2Path) if fiAfter.Size() != fiBefore.Size() { t.Errorf("segment-2 was modified: before=%d after=%d (C4-2 regression)", fiBefore.Size(), fiAfter.Size()) } } // Regression guard for C4: single-segment tail corruption still truncates // correctly (existing behavior preserved). func TestRecoverLastSegmentCorruptionTruncatesCorrectly(t *testing.T) { dir := t.TempDir() filePath := writeTestSegment(t, dir, 0, 0, [][]*WalEntry{ {makePutEntry("k0", "v0")}, {makePutEntry("k1", "v1")}, }) encA, _ := EncodeWalBatch(0, []*WalEntry{makePutEntry("k0", "v0")}) endOfBatch1 := int64(WalFileHeaderSize) + int64(PhysicalRecordHeaderSize+len(encA)) + int64(PhysicalRecordHeaderSize+len(encA)) // Append partial fragments to last (only) segment. appendFileBytes(t, filePath, EncodePhysicalRecord(RecFirst, []byte("first"))) appendFileBytes(t, filePath, EncodePhysicalRecord(RecMiddle, []byte("middle"))) result, err := Recover(dir, &mockReplayer{}) if err != nil { t.Fatalf("Recover: %v", err) } if !result.Truncated { t.Fatal("Truncated = false, want true for last-segment corruption") } fi, _ := os.Stat(filePath) if fi.Size() != endOfBatch1 { t.Errorf("file size = %d, want %d", fi.Size(), endOfBatch1) } } // Regression guard for C4: multi-segment with last-segment corruption still // works correctly (the legitimate tail-truncation case). func TestRecoverMultiSegmentLastSegmentCorruptionTruncatesLast(t *testing.T) { dir := t.TempDir() // segment-0: valid batches // segment-1: valid batches + partial tail (corruption in LAST segment) writeTestSegment(t, dir, 0, 0, [][]*WalEntry{ {makePutEntry("k0", "v0")}, }) seg1Path := writeTestSegment(t, dir, 1, 1, [][]*WalEntry{ {makePutEntry("k1", "v1")}, {makePutEntry("k2", "v2")}, }) // Append partial fragments to segment-1 (the LAST segment). appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecFirst, []byte("first"))) appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecMiddle, []byte("middle"))) // Compute expected truncation point: end of Batch B in segment-1. encA, _ := EncodeWalBatch(1, []*WalEntry{makePutEntry("k1", "v1")}) encB, _ := EncodeWalBatch(2, []*WalEntry{makePutEntry("k2", "v2")}) endOfBatch2 := int64(WalFileHeaderSize) + int64(PhysicalRecordHeaderSize+len(encA)) + int64(PhysicalRecordHeaderSize+len(encB)) result, err := Recover(dir, &mockReplayer{}) if err != nil { t.Fatalf("Recover: %v", err) } if !result.Truncated { t.Fatal("Truncated = false, want true") } fi, _ := os.Stat(seg1Path) if fi.Size() != endOfBatch2 { t.Errorf("segment-1 size = %d, want %d (end of Batch 2)", fi.Size(), endOfBatch2) } } ``` #### B.3 现有测试更新 `wal/recover_test.go` 中的 `TestRecoverIdempotentAfterTruncation` 和 `TestRecoverPartialFragmentTailIdempotent` 都是单 segment 场景,行为不变。但它们间接调用了 `Recover` → `RecoverFromSegments` → `ReplaySegmentFile`,需要确认新签名 `isLastSegment=true` 在 RecoverFromSegments 内部正确传递。 `wal/recovery_test.go` 现有的直接调用 `ReplaySegmentFile` 的测试需要补 `isLastSegment` 参数。**3 个调用点**(Oracle bg_22edec11 提醒): ```bash grep -n "ReplaySegmentFile" wal/recovery_test.go wal/recover_test.go ``` - `wal/recovery_test.go:124`(TestReplaySegmentFile 之类)→ 加 `true` - `wal/recovery_test.go:150`(另一个 ReplaySegmentFile 测试)→ 加 `true` - `wal/recover_test.go:121`(TestRecoverWithTailCorruption 内部,验证截断后重放)→ 加 `true` 每个调用点加 `true`(默认按 last segment 处理,保留现有行为)。 #### B.4 补 middle-segment parser corruption 集成测试(Oracle 新增) > **Oracle 修订(bg_22edec11)NICE-TO-HAVE**:原计划只覆盖 middle-segment CollectingFragments,没覆盖 middle-segment parser corruption(CRC 失败)。需要加一个。 ```go // Regression guard for C4 (parser corruption half): middle segment CRC // corruption must hard-fail Recover, NOT truncate the (valid) last segment. // This tests the path where ParseBlock returns TailCorruptionError and // ReplaySegmentFile converts it to hard error for non-last segment. func TestRecoverMiddleSegmentCRCCorruptionHardFails(t *testing.T) { dir := t.TempDir() // segment-0: [Batch seq 0-1] (2 entries → next=2) // segment-1: [Batch seq 2-3] + [CRC-corrupted bytes] ← middle parser corruption // segment-2: [Batch seq 4-5] (valid; never reached) writeTestSegment(t, dir, 0, 0, [][]*WalEntry{ {makePutEntry("k0", "v0"), makePutEntry("k1", "v1")}, }) seg1Path := writeTestSegment(t, dir, 1, 2, [][]*WalEntry{ {makePutEntry("k2", "v2"), makePutEntry("k3", "v3")}, }) // Append CRC-corrupted bytes (will fail DecodePhysicalRecord's CRC check). // 10 bytes of 0xFF — looks like a record header but CRC won't match. appendFileBytes(t, seg1Path, []byte{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}) writeTestSegment(t, dir, 2, 4, [][]*WalEntry{ {makePutEntry("k4", "v4"), makePutEntry("k5", "v5")}, }) seg2Path := filepath.Join(dir, "segment-2.wal") fiBefore, _ := os.Stat(seg2Path) _, err := Recover(dir, &mockReplayer{}) if err == nil { t.Fatal("expected Recover to fail on middle segment CRC corruption") } if IsTailCorruption(err) { t.Errorf("expected hard error (not tail corruption) for middle segment CRC; got %v", err) } // CRITICAL: segment-2 must NOT be truncated. fiAfter, _ := os.Stat(seg2Path) if fiAfter.Size() != fiBefore.Size() { t.Errorf("segment-2 was modified: before=%d after=%d (C4-2 regression)", fiBefore.Size(), fiAfter.Size()) } } ``` ### Phase C:验证(15 分钟) ```bash # 1. 编译 go build ./... # 2. 重点测试 go test ./wal -run 'TestReplaySegmentFile|TestRecoverMiddleSegment|TestRecoverLastSegment|TestRecoverMultiSegment|TestRecoverIdempotent|TestRecoverPartialFragmentTail' -count=1 -v # 3. wal 包全量 go test ./wal/... -count=1 # 4. 全仓 go test ./... -count=1 # 5. race go test -race ./... -count=1 # 6. vet go vet ./... ``` ### Phase D:Commit message draft ``` fix: treat non-last segment corruption as hard error (C4) Per design §3.2 line 704, tail corruption in a non-last WAL segment is middle corruption, which must hard-fail recovery instead of being silently truncated. The previous code in wal/recovery.go always returned TailCorruptionError for CollectingFragments state or parse errors, regardless of segment position. Recover then always truncated segments[last], which could corrupt a valid last segment when the actual corruption was in a middle segment. Oracle bg_ef425776 flagged an additional failure mode: "wal/recover.go:61 总是对 segments[len(segments)-1] 调用 truncateSegment,但 RecoverFromSegments 的 TailCorruptionError 可能来自非尾段". Changes: - wal/record_parser.go: add SegmentPath field to TailCorruptionError for diagnostics and defensive truncation target identification. - wal/recovery.go: - ReplaySegmentFile now takes isLastSegment bool parameter. - When parse error or CollectingFragments occurs in non-last segment, return hard error instead of TailCorruptionError. - When in last segment, return TailCorruptionError with SegmentPath set. - RecoverFromSegments passes isLastSegment based on iteration index. - wal/recover.go: - Use tce.SegmentPath as authoritative truncation target (defensive fallback to segments[last] if missing). After C4 fix, TailCorruptionError is only returned for last segment, so this is always segments[last] in practice. Tests: - wal/recovery_test.go: 4 unit tests for ReplaySegmentFile covering last/non-last × CollectingFragments/parse-error matrix. Existing tests that directly call ReplaySegmentFile updated to pass isLastSegment=true. - wal/recover_test.go: 3 integration tests covering middle-segment corruption (must hard-fail), last-segment corruption in single-segment WAL (must truncate), last-segment corruption in multi-segment WAL (must truncate only last segment). Verified: each new test fails on pre-fix code (non-last corruption silently truncated valid last segment) and passes after the fix. Full suite green including go test -race ./... . Audit context: docs/audit-3.2.md C4 (Oracle-verified bg_ef425776). ``` --- ## 验收清单 - [ ] Phase A.1:`TailCorruptionError.SegmentPath` 字段存在 - [ ] Phase A.2:`ReplaySegmentFile` 签名包含 `isLastSegment bool` - [ ] Phase A.2:非尾段 CollectingFragments 返回硬错误(不是 TailCorruptionError) - [ ] Phase A.2:非尾段 parse error(TailCorruption from parser)转硬错误 - [ ] Phase A.3:`RecoverFromSegments` 传递正确 `isLastSegment` - [ ] Phase A.4:`Recover` 用 `tce.SegmentPath` 做截断目标 - [ ] Phase B.1:4 个 `ReplaySegmentFile` 单元测试存在 - [ ] Phase B.2:3 个集成测试存在(middle/last-single/last-multi) - [ ] Phase B.3:现有 `ReplaySegmentFile` 直接调用更新签名 - [ ] `go test ./wal/... -count=1` 全绿 - [ ] `go test ./... -count=1` 全绿 - [ ] `go test -race ./... -count=1` 全绿 - [ ] `go vet ./...` 无新增警告 - [ ] 单次 commit,message 引用 audit C4 --- ## 不在本次范围内(后续 issue) | 编号 | 为什么不放进来 | |------|---------------| | C8 | segment_manager.go:64-66 把字节偏移当 startSequence。本 patch 测试用 `writeTestSegment` 绕过,但生产环境多 segment recovery 仍受 C8 影响 | | C1 | CRC 多项式 IEEE → crc32c,独立 | | C7 | Put/Close 竞态,独立 | | H1-H7 | 其他 High,独立 | --- ## 修订记录 - **v1(原始)**:C4 修复方案初稿,送 Momus 审 - **v1.0(Momus 审核 bg_423473e7)**:[OKAY],无 blocking。思考过程敏锐发现 `TestRecoverMiddleSegmentCorruptionHardFails` 中 segment-2 的 startSequence=6 与 segment-1 的实际 complete batch 数不匹配(应该是 4)。修正为 4 - **v1.1(Oracle 修订 bg_22edec11)**: - **BLOCKING**:非尾段 parser corruption 用 `%v` 而不是 `%w` —— `IsTailCorruption` 用 `errors.As` 遍历链,`%w` 包装后仍能匹配 `*TailCorruptionError`,会被当 tail corruption 处理 - **BLOCKING**:漏了 `wal/recover_test.go:121` 的 `ReplaySegmentFile` 直接调用,需要更新签名。共 3 个直接调用点 - **NEW**:加 `TestRecoverMiddleSegmentCRCCorruptionHardFails` 集成测试,覆盖 parser corruption 中段损坏(CRC fail)的端到端路径 - **MINOR**:把 SegmentPath attachment 移到非尾段硬错误转换之前,让两条路径都能设置字段(虽然非尾段硬错误不需要,但保持顺序一致)