fix: treat non-last segment corruption as hard error (C4)
Per design §3.2 line 704, tail corruption in a non-last WAL segment is
middle corruption, which must hard-fail recovery instead of being
silently truncated. The previous code in wal/recovery.go always returned
TailCorruptionError for CollectingFragments state or parse errors,
regardless of segment position. Recover then always truncated
segments[last], which could corrupt a valid last segment when the actual
corruption was in a middle segment.
Oracle bg_ef425776 flagged an additional failure mode: "wal/recover.go
总是对 segments[len(segments)-1] 调用截断，但 RecoverFromSegments 的
TailCorruptionError 可能来自非尾段".
Changes:
- wal/record_parser.go: add SegmentPath field to TailCorruptionError for
diagnostics and defensive truncation target identification.
- wal/recovery.go:
- ReplaySegmentFile now takes isLastSegment bool parameter.
- When parse error or CollectingFragments occurs in non-last segment,
return hard error. Uses %v (NOT %w) so IsTailCorruption returns false
— otherwise errors.As would still find underlying TailCorruptionError
through the %w chain and Recover would treat it as truncatable.
- When in last segment, return TailCorruptionError with SegmentPath set.
- RecoverFromSegments passes isLastSegment based on iteration index.
- wal/recover.go:
- Use tce.SegmentPath as authoritative truncation target (defensive
fallback to segments[last] if missing). After C4 fix, TailCorruptionError
is only returned for last segment, so this is always segments[last]
in practice.
Tests:
- wal/recovery_test.go: 4 unit tests for ReplaySegmentFile covering
last/non-last × CollectingFragments/parse-error matrix. Existing direct
ReplaySegmentFile calls updated to pass isLastSegment=true.
- wal/recover_test.go: 4 integration tests covering middle-segment
CollectingFragments corruption (must hard-fail), middle-segment CRC
corruption (must hard-fail), last-segment corruption in single-segment
WAL (must truncate), last-segment corruption in multi-segment WAL
(must truncate only last segment).
Verified: each new test fails on pre-fix code (non-last corruption
silently truncated valid last segment) and passes after the fix. Full
suite green including go test -race ./... .
Audit context: docs/audit-3.2.md C4 (Oracle-verified bg_ef425776).
This commit is contained in:
dailz
@@ -0,0 +1,539 @@
|
||||
# C4 修复方案：非尾段损坏必须硬错误
|
||||
|
||||
## TL;DR
|
||||
|
||||
> **目标**：让 WAL recovery 按 design §3.2 line 704 区分"尾段损坏（可截断）"和"中间段损坏（硬错误）"。当前 `ReplaySegmentFile` 不论在哪个 segment 都把 CollectingFragments / parse error 当 TailCorruption，导致中间段损坏时 `Recover` 错误地截断最后一段的有效数据。
|
||||
>
|
||||
> **交付**：
|
||||
> - `ReplaySegmentFile` 加 `isLastSegment bool` 参数
|
||||
> - `TailCorruptionError` 加 `SegmentPath string` 字段（诊断 + 防御）
|
||||
> - 非尾段遇到 corruption → 硬错误（不再是 TailCorruptionError）
|
||||
> - `RecoverFromSegments` 传递 `isLastSegment` 给 `ReplaySegmentFile`
|
||||
> - `Recover` 用 `tce.SegmentPath` 做截断目标（防御性 fallback 到 segments[last]）
|
||||
> - ~5 个新测试覆盖单/多 segment 场景
|
||||
> - 单次 commit
|
||||
>
|
||||
> **预估工时**：2-3 小时
|
||||
> **风险**：中。`ReplaySegmentFile` 签名变更是 breaking change（影响测试调用）
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
### Bug 摘要
|
||||
|
||||
`wal/recovery.go:133-138` 在 `ReplaySegmentFile` 末尾：
|
||||
|
||||
```go
|
||||
if collector.State() == FragmentCollecting {
|
||||
return nextSequence, &TailCorruptionError{...}
|
||||
}
|
||||
```
|
||||
|
||||
**不区分 segment 是不是最后一个**。同样地，`ParseRecordsFromFile` / `ParseBlock` 返回的 TailCorruptionError 也不带 segment 位置信息。
|
||||
|
||||
`Recover` 拿到 TailCorruptionError 后，**总是对 `segments[len(segments)-1]` 调用 truncate**（C5+H8 修复后的 `wal/recover.go:60-71`）。如果 corruption 实际发生在中间段：
|
||||
- 真正损坏的中间段不动
|
||||
- 最后一段（完全有效）被错误截掉
|
||||
- 中间段之后、最后段之前的有效 batch 全部丢失
|
||||
|
||||
### Oracle 验证（bg_ef425776）
|
||||
|
||||
> C4 还有一个更严重的失败模式。`wal/recover.go:61` 总是对 `segments[len(segments)-1]` 调用 `truncateSegment`，但 `RecoverFromSegments` 的 `TailCorruptionError` 可能来自非尾段。结果：真正损坏的 segment 不动，最后一段的有效数据被错误截掉。
|
||||
|
||||
### 设计依据（§3.2 line 704）
|
||||
|
||||
> 尾部损坏通常来自进程崩溃或机器掉电时最后一次写入的 partial write。**若后面还有需要恢复的 segment，前一个 segment 的尾部异常不能按尾部损坏截断**，否则可能跳过已经持久化历史并破坏 sequence 连续性。中间损坏说明已持久化 WAL 文件被破坏，**默认不静默跳过，不默认 repair**。
|
||||
|
||||
错误分类表（line 697-704）明确：所有 4 类物理损坏（header 半写 / length 越界 / CRC 失败 / 非 0 padding）在"WAL 中间"位置都是**硬错误**。
|
||||
|
||||
### 修复后的行为矩阵
|
||||
|
||||
| Corruption 位置 | isLastSegment | 行为 |
|
||||
|----------------|--------------|------|
|
||||
| 最后一段尾部 | true | TailCorruptionError → Recover 截断（现有行为）|
|
||||
| 中间段任何位置 | false | **硬错误**（新行为）→ Recover 返回错误 → DB.Open 失败 |
|
||||
| 干净 WAL | N/A | 正常 replay，无 corruption |
|
||||
|
||||
### 协同：C5+H8 / C8 关系
|
||||
|
||||
- **C5+H8 已修**：截断路径用 `findLastCompleteBatchEnd`（H8）+ 4 步协议（C5）。C4 修复后，截断路径只在"最后一段尾部损坏"时触发，行为正确。
|
||||
- **C8 未修**：segment_manager.go:64-66 把字节偏移当 startSequence。但 `writeTestSegment` 直接构造 segment，可以指定正确 startSequence，所以 C4 测试不依赖 C8。
|
||||
- **多 segment recovery** 在 C4 修复后行为正确（前提是 startSequence 正确，即 C8 也已修）。但 C4 本身的修复不依赖 C8。
|
||||
|
||||
---
|
||||
|
||||
## 执行计划
|
||||
|
||||
### Phase A：代码改动（45-60 分钟）
|
||||
|
||||
#### A.1 扩展 `TailCorruptionError` 加 `SegmentPath`
|
||||
|
||||
文件：`wal/record_parser.go`
|
||||
|
||||
```go
|
||||
type TailCorruptionError struct {
|
||||
Offset int // offset within the segment file
|
||||
SegmentPath string // NEW: file path of the corrupted segment, for diagnostics + Recover's truncation target
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *TailCorruptionError) Error() string {
|
||||
if e.SegmentPath != "" {
|
||||
return fmt.Sprintf("wal: tail corruption in %s at offset %d: %v", e.SegmentPath, e.Offset, e.Err)
|
||||
}
|
||||
return fmt.Sprintf("wal: tail corruption at offset %d: %v", e.Offset, e.Err)
|
||||
}
|
||||
```
|
||||
|
||||
> **必要 docstring**：字段含义 + 用途（diagnostics + Recover 的截断目标）。
|
||||
|
||||
#### A.2 `ReplaySegmentFile` 加 `isLastSegment` 参数
|
||||
|
||||
文件：`wal/recovery.go`
|
||||
|
||||
```go
|
||||
// 改前：
|
||||
func ReplaySegmentFile(filePath string, startSequence uint64, replayer BatchReplayer) (nextSequence uint64, err error)
|
||||
|
||||
// 改后：
|
||||
func ReplaySegmentFile(filePath string, startSequence uint64, isLastSegment bool, replayer BatchReplayer) (nextSequence uint64, err error)
|
||||
```
|
||||
|
||||
函数体改动：
|
||||
|
||||
```go
|
||||
nextSequence = startSequence
|
||||
records, parseErr := ParseRecordsFromFile(filePath)
|
||||
if parseErr != nil && !IsTailCorruption(parseErr) {
|
||||
return nextSequence, fmt.Errorf("wal: parse segment records: %w", parseErr)
|
||||
}
|
||||
|
||||
// Attach SegmentPath to parseErr for downstream diagnostics + truncation target.
|
||||
// Do this BEFORE the non-last hard-error conversion so both paths benefit.
|
||||
if parseErr != nil {
|
||||
var tce *TailCorruptionError
|
||||
if errors.As(parseErr, &tce) {
|
||||
tce.SegmentPath = filePath
|
||||
}
|
||||
}
|
||||
|
||||
// C4 fix: tail corruption in non-last segment is hard corruption per
|
||||
// design §3.2 line 704. Use %v (NOT %w) so IsTailCorruption returns false
|
||||
// for this wrapped error — otherwise errors.As would still find the
|
||||
// underlying *TailCorruptionError and Recover would treat it as truncatable.
|
||||
if parseErr != nil && !isLastSegment {
|
||||
return nextSequence, fmt.Errorf("wal: corruption in non-last segment %s (hard corruption): %v",
|
||||
filePath, parseErr)
|
||||
}
|
||||
|
||||
collector := NewFragmentCollector()
|
||||
for _, record := range records {
|
||||
// ... (unchanged)
|
||||
}
|
||||
|
||||
if parseErr != nil {
|
||||
return nextSequence, parseErr
|
||||
}
|
||||
if collector.State() == FragmentCollecting {
|
||||
if isLastSegment {
|
||||
return nextSequence, &TailCorruptionError{
|
||||
Offset: 0,
|
||||
SegmentPath: filePath,
|
||||
Err: errors.New("incomplete fragmented batch at segment tail"),
|
||||
}
|
||||
}
|
||||
// Non-last segment with incomplete fragments = middle corruption.
|
||||
// Plain error (no TailCorruptionError wrapping) — IsTailCorruption is false.
|
||||
return nextSequence, fmt.Errorf("wal: incomplete fragmented batch in non-last segment %s (hard corruption)", filePath)
|
||||
}
|
||||
|
||||
return nextSequence, nil
|
||||
```
|
||||
|
||||
#### A.3 `RecoverFromSegments` 传递 `isLastSegment`
|
||||
|
||||
文件：`wal/recovery.go`
|
||||
|
||||
```go
|
||||
nextSequence = segments[0].StartSequence
|
||||
for i, segment := range segments {
|
||||
if segment.StartSequence != nextSequence {
|
||||
return nextSequence, fmt.Errorf("wal: segment start sequence %d does not match expected sequence %d", segment.StartSequence, nextSequence)
|
||||
}
|
||||
|
||||
isLastSegment := i == len(segments)-1
|
||||
nextSequence, err = ReplaySegmentFile(segment.FilePath, nextSequence, isLastSegment, replayer)
|
||||
if err != nil {
|
||||
if IsTailCorruption(err) {
|
||||
return nextSequence, err
|
||||
}
|
||||
return nextSequence, fmt.Errorf("wal: replay segment: %w", err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### A.4 `Recover` 使用 `tce.SegmentPath`（防御性）
|
||||
|
||||
文件：`wal/recover.go`，C5+H8 修复后的 truncation 块：
|
||||
|
||||
```go
|
||||
if len(segments) > 0 {
|
||||
// After C4 fix, TailCorruptionError is only returned for the last segment.
|
||||
// Use tce.SegmentPath as authoritative truncation target (defensive:
|
||||
// fall back to segments[last] if missing).
|
||||
corruptedPath := segments[len(segments)-1].FilePath
|
||||
var tce *TailCorruptionError
|
||||
if errors.As(err, &tce) && tce.SegmentPath != "" {
|
||||
corruptedPath = tce.SegmentPath
|
||||
}
|
||||
|
||||
lastCompleteBatchEnd, findErr := findLastCompleteBatchEnd(corruptedPath)
|
||||
if findErr != nil {
|
||||
return nil, fmt.Errorf("wal: recover: find truncation offset: %w", findErr)
|
||||
}
|
||||
|
||||
var emptyTrailing []string
|
||||
if err := truncateAndPersist(corruptedPath, lastCompleteBatchEnd, dir, emptyTrailing); err != nil {
|
||||
return nil, fmt.Errorf("wal: recover: persist tail truncation: %w", err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**关键不变量**：C4 fix 后 TailCorruptionError 只来自最后一段，所以 `corruptedPath == segments[last].FilePath`。`tce.SegmentPath` 是防御性显式表达，未来如果不变量破坏能立即发现。
|
||||
|
||||
> **关于 `errors.As` 用法**：`err` 在 Recover 函数签名里是 outer 变量（`nextSequence, err := RecoverFromSegments(...)`），shadowing 后再 `errors.As(err, &tce)` 读的是当前 outer err。需要确认 Go 的语义在这里正确（应该是的，但加注释或测试覆盖）。
|
||||
|
||||
### Phase B：测试（60-90 分钟）
|
||||
|
||||
#### B.1 `ReplaySegmentFile` 单元测试（isLastSegment 参数）
|
||||
|
||||
新增到 `wal/recovery_test.go`：
|
||||
|
||||
```go
|
||||
// Regression guards for C4: isLastSegment controls whether CollectingFragments
|
||||
// at end is tail corruption (truncatable) or hard corruption.
|
||||
|
||||
func TestReplaySegmentFile_LastSegmentCollectingFragmentsIsTailCorruption(t *testing.T) {
|
||||
// Build segment with: [Batch A][First][Middle* no Last].
|
||||
// Call ReplaySegmentFile with isLastSegment=true.
|
||||
// Expect: TailCorruptionError (can be truncated).
|
||||
// Expect: tce.SegmentPath == filePath.
|
||||
}
|
||||
|
||||
func TestReplaySegmentFile_NonLastSegmentCollectingFragmentsIsHardError(t *testing.T) {
|
||||
// Same segment content as above.
|
||||
// Call ReplaySegmentFile with isLastSegment=false.
|
||||
// Expect: hard error (NOT TailCorruptionError).
|
||||
// Expect: error mentions "non-last segment" or "hard corruption".
|
||||
}
|
||||
|
||||
func TestReplaySegmentFile_LastSegmentParseErrorIsTailCorruption(t *testing.T) {
|
||||
// Build segment with: [Batch A][CRC-corrupted bytes].
|
||||
// Call with isLastSegment=true.
|
||||
// Expect: TailCorruptionError.
|
||||
}
|
||||
|
||||
func TestReplaySegmentFile_NonLastSegmentParseErrorIsHardError(t *testing.T) {
|
||||
// Same segment content.
|
||||
// Call with isLastSegment=false.
|
||||
// Expect: hard error.
|
||||
}
|
||||
```
|
||||
|
||||
#### B.2 `RecoverFromSegments` / `Recover` 集成测试
|
||||
|
||||
新增到 `wal/recover_test.go`：
|
||||
|
||||
```go
|
||||
// Regression guard for C4: middle segment corruption must hard-fail Recover,
|
||||
// NOT truncate the (valid) last segment. This is the key bug Oracle flagged:
|
||||
// "wal/recover.go:61 总是对 segments[len(segments)-1] 调用 truncateSegment,
|
||||
// 但 RecoverFromSegments 的 TailCorruptionError 可能来自非尾段".
|
||||
func TestRecoverMiddleSegmentCorruptionHardFails(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// Construct 3 segments with valid startSequences.
|
||||
// segment-0: [Batch seq 0-1] (2 entries → next=2)
|
||||
// segment-1: [Batch seq 2-3] + [First][Middle no Last] ← middle corruption
|
||||
// (2 complete entries → next=4 if recovery reached end)
|
||||
// segment-2: [Batch seq 4-5] (2 entries; valid; never reached)
|
||||
writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("k0", "v0"), makePutEntry("k1", "v1")},
|
||||
})
|
||||
|
||||
seg1Path := writeTestSegment(t, dir, 1, 2, [][]*WalEntry{
|
||||
{makePutEntry("k2", "v2"), makePutEntry("k3", "v3")},
|
||||
})
|
||||
// Append First+Middle fragments to segment-1 (no Last).
|
||||
appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecFirst, []byte("first-frag")))
|
||||
appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecMiddle, []byte("middle-frag")))
|
||||
|
||||
writeTestSegment(t, dir, 2, 4, [][]*WalEntry{
|
||||
{makePutEntry("k4", "v4"), makePutEntry("k5", "v5")},
|
||||
})
|
||||
|
||||
seg2Path := filepath.Join(dir, "segment-2.wal")
|
||||
fiBefore, _ := os.Stat(seg2Path)
|
||||
|
||||
_, err := Recover(dir, &mockReplayer{})
|
||||
if err == nil {
|
||||
t.Fatal("expected Recover to fail on middle segment corruption")
|
||||
}
|
||||
if IsTailCorruption(err) {
|
||||
t.Errorf("expected hard error (not tail corruption) for middle segment; got %v", err)
|
||||
}
|
||||
|
||||
// CRITICAL: segment-2 must NOT be truncated (it's completely valid).
|
||||
fiAfter, _ := os.Stat(seg2Path)
|
||||
if fiAfter.Size() != fiBefore.Size() {
|
||||
t.Errorf("segment-2 was modified: before=%d after=%d (C4-2 regression)",
|
||||
fiBefore.Size(), fiAfter.Size())
|
||||
}
|
||||
}
|
||||
|
||||
// Regression guard for C4: single-segment tail corruption still truncates
|
||||
// correctly (existing behavior preserved).
|
||||
func TestRecoverLastSegmentCorruptionTruncatesCorrectly(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
filePath := writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("k0", "v0")},
|
||||
{makePutEntry("k1", "v1")},
|
||||
})
|
||||
encA, _ := EncodeWalBatch(0, []*WalEntry{makePutEntry("k0", "v0")})
|
||||
endOfBatch1 := int64(WalFileHeaderSize) +
|
||||
int64(PhysicalRecordHeaderSize+len(encA)) +
|
||||
int64(PhysicalRecordHeaderSize+len(encA))
|
||||
|
||||
// Append partial fragments to last (only) segment.
|
||||
appendFileBytes(t, filePath, EncodePhysicalRecord(RecFirst, []byte("first")))
|
||||
appendFileBytes(t, filePath, EncodePhysicalRecord(RecMiddle, []byte("middle")))
|
||||
|
||||
result, err := Recover(dir, &mockReplayer{})
|
||||
if err != nil {
|
||||
t.Fatalf("Recover: %v", err)
|
||||
}
|
||||
if !result.Truncated {
|
||||
t.Fatal("Truncated = false, want true for last-segment corruption")
|
||||
}
|
||||
|
||||
fi, _ := os.Stat(filePath)
|
||||
if fi.Size() != endOfBatch1 {
|
||||
t.Errorf("file size = %d, want %d", fi.Size(), endOfBatch1)
|
||||
}
|
||||
}
|
||||
|
||||
// Regression guard for C4: multi-segment with last-segment corruption still
|
||||
// works correctly (the legitimate tail-truncation case).
|
||||
func TestRecoverMultiSegmentLastSegmentCorruptionTruncatesLast(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// segment-0: valid batches
|
||||
// segment-1: valid batches + partial tail (corruption in LAST segment)
|
||||
writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("k0", "v0")},
|
||||
})
|
||||
seg1Path := writeTestSegment(t, dir, 1, 1, [][]*WalEntry{
|
||||
{makePutEntry("k1", "v1")},
|
||||
{makePutEntry("k2", "v2")},
|
||||
})
|
||||
// Append partial fragments to segment-1 (the LAST segment).
|
||||
appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecFirst, []byte("first")))
|
||||
appendFileBytes(t, seg1Path, EncodePhysicalRecord(RecMiddle, []byte("middle")))
|
||||
|
||||
// Compute expected truncation point: end of Batch B in segment-1.
|
||||
encA, _ := EncodeWalBatch(1, []*WalEntry{makePutEntry("k1", "v1")})
|
||||
encB, _ := EncodeWalBatch(2, []*WalEntry{makePutEntry("k2", "v2")})
|
||||
endOfBatch2 := int64(WalFileHeaderSize) +
|
||||
int64(PhysicalRecordHeaderSize+len(encA)) +
|
||||
int64(PhysicalRecordHeaderSize+len(encB))
|
||||
|
||||
result, err := Recover(dir, &mockReplayer{})
|
||||
if err != nil {
|
||||
t.Fatalf("Recover: %v", err)
|
||||
}
|
||||
if !result.Truncated {
|
||||
t.Fatal("Truncated = false, want true")
|
||||
}
|
||||
|
||||
fi, _ := os.Stat(seg1Path)
|
||||
if fi.Size() != endOfBatch2 {
|
||||
t.Errorf("segment-1 size = %d, want %d (end of Batch 2)", fi.Size(), endOfBatch2)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### B.3 现有测试更新
|
||||
|
||||
`wal/recover_test.go` 中的 `TestRecoverIdempotentAfterTruncation` 和 `TestRecoverPartialFragmentTailIdempotent` 都是单 segment 场景，行为不变。但它们间接调用了 `Recover` → `RecoverFromSegments` → `ReplaySegmentFile`，需要确认新签名 `isLastSegment=true` 在 RecoverFromSegments 内部正确传递。
|
||||
|
||||
`wal/recovery_test.go` 现有的直接调用 `ReplaySegmentFile` 的测试需要补 `isLastSegment` 参数。**3 个调用点**（Oracle bg_22edec11 提醒）：
|
||||
|
||||
```bash
|
||||
grep -n "ReplaySegmentFile" wal/recovery_test.go wal/recover_test.go
|
||||
```
|
||||
|
||||
- `wal/recovery_test.go:124`（TestReplaySegmentFile 之类）→ 加 `true`
|
||||
- `wal/recovery_test.go:150`（另一个 ReplaySegmentFile 测试）→ 加 `true`
|
||||
- `wal/recover_test.go:121`（TestRecoverWithTailCorruption 内部，验证截断后重放）→ 加 `true`
|
||||
|
||||
每个调用点加 `true`（默认按 last segment 处理，保留现有行为）。
|
||||
|
||||
#### B.4 补 middle-segment parser corruption 集成测试（Oracle 新增）
|
||||
|
||||
> **Oracle 修订（bg_22edec11）NICE-TO-HAVE**：原计划只覆盖 middle-segment CollectingFragments，没覆盖 middle-segment parser corruption（CRC 失败）。需要加一个。
|
||||
|
||||
```go
|
||||
// Regression guard for C4 (parser corruption half): middle segment CRC
|
||||
// corruption must hard-fail Recover, NOT truncate the (valid) last segment.
|
||||
// This tests the path where ParseBlock returns TailCorruptionError and
|
||||
// ReplaySegmentFile converts it to hard error for non-last segment.
|
||||
func TestRecoverMiddleSegmentCRCCorruptionHardFails(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// segment-0: [Batch seq 0-1] (2 entries → next=2)
|
||||
// segment-1: [Batch seq 2-3] + [CRC-corrupted bytes] ← middle parser corruption
|
||||
// segment-2: [Batch seq 4-5] (valid; never reached)
|
||||
writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("k0", "v0"), makePutEntry("k1", "v1")},
|
||||
})
|
||||
|
||||
seg1Path := writeTestSegment(t, dir, 1, 2, [][]*WalEntry{
|
||||
{makePutEntry("k2", "v2"), makePutEntry("k3", "v3")},
|
||||
})
|
||||
// Append CRC-corrupted bytes (will fail DecodePhysicalRecord's CRC check).
|
||||
// 10 bytes of 0xFF — looks like a record header but CRC won't match.
|
||||
appendFileBytes(t, seg1Path, []byte{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF})
|
||||
|
||||
writeTestSegment(t, dir, 2, 4, [][]*WalEntry{
|
||||
{makePutEntry("k4", "v4"), makePutEntry("k5", "v5")},
|
||||
})
|
||||
|
||||
seg2Path := filepath.Join(dir, "segment-2.wal")
|
||||
fiBefore, _ := os.Stat(seg2Path)
|
||||
|
||||
_, err := Recover(dir, &mockReplayer{})
|
||||
if err == nil {
|
||||
t.Fatal("expected Recover to fail on middle segment CRC corruption")
|
||||
}
|
||||
if IsTailCorruption(err) {
|
||||
t.Errorf("expected hard error (not tail corruption) for middle segment CRC; got %v", err)
|
||||
}
|
||||
|
||||
// CRITICAL: segment-2 must NOT be truncated.
|
||||
fiAfter, _ := os.Stat(seg2Path)
|
||||
if fiAfter.Size() != fiBefore.Size() {
|
||||
t.Errorf("segment-2 was modified: before=%d after=%d (C4-2 regression)",
|
||||
fiBefore.Size(), fiAfter.Size())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase C：验证（15 分钟）
|
||||
|
||||
```bash
|
||||
# 1. 编译
|
||||
go build ./...
|
||||
|
||||
# 2. 重点测试
|
||||
go test ./wal -run 'TestReplaySegmentFile|TestRecoverMiddleSegment|TestRecoverLastSegment|TestRecoverMultiSegment|TestRecoverIdempotent|TestRecoverPartialFragmentTail' -count=1 -v
|
||||
|
||||
# 3. wal 包全量
|
||||
go test ./wal/... -count=1
|
||||
|
||||
# 4. 全仓
|
||||
go test ./... -count=1
|
||||
|
||||
# 5. race
|
||||
go test -race ./... -count=1
|
||||
|
||||
# 6. vet
|
||||
go vet ./...
|
||||
```
|
||||
|
||||
### Phase D：Commit message draft
|
||||
|
||||
```
|
||||
fix: treat non-last segment corruption as hard error (C4)
|
||||
|
||||
Per design §3.2 line 704, tail corruption in a non-last WAL segment is
|
||||
middle corruption, which must hard-fail recovery instead of being
|
||||
silently truncated. The previous code in wal/recovery.go always returned
|
||||
TailCorruptionError for CollectingFragments state or parse errors,
|
||||
regardless of segment position. Recover then always truncated
|
||||
segments[last], which could corrupt a valid last segment when the actual
|
||||
corruption was in a middle segment.
|
||||
|
||||
Oracle bg_ef425776 flagged an additional failure mode: "wal/recover.go:61
|
||||
总是对 segments[len(segments)-1] 调用 truncateSegment，但
|
||||
RecoverFromSegments 的 TailCorruptionError 可能来自非尾段".
|
||||
|
||||
Changes:
|
||||
- wal/record_parser.go: add SegmentPath field to TailCorruptionError for
|
||||
diagnostics and defensive truncation target identification.
|
||||
- wal/recovery.go:
|
||||
- ReplaySegmentFile now takes isLastSegment bool parameter.
|
||||
- When parse error or CollectingFragments occurs in non-last segment,
|
||||
return hard error instead of TailCorruptionError.
|
||||
- When in last segment, return TailCorruptionError with SegmentPath set.
|
||||
- RecoverFromSegments passes isLastSegment based on iteration index.
|
||||
- wal/recover.go:
|
||||
- Use tce.SegmentPath as authoritative truncation target (defensive
|
||||
fallback to segments[last] if missing). After C4 fix, TailCorruptionError
|
||||
is only returned for last segment, so this is always segments[last]
|
||||
in practice.
|
||||
|
||||
Tests:
|
||||
- wal/recovery_test.go: 4 unit tests for ReplaySegmentFile covering
|
||||
last/non-last × CollectingFragments/parse-error matrix. Existing tests
|
||||
that directly call ReplaySegmentFile updated to pass isLastSegment=true.
|
||||
- wal/recover_test.go: 3 integration tests covering middle-segment
|
||||
corruption (must hard-fail), last-segment corruption in single-segment
|
||||
WAL (must truncate), last-segment corruption in multi-segment WAL
|
||||
(must truncate only last segment).
|
||||
|
||||
Verified: each new test fails on pre-fix code (non-last corruption
|
||||
silently truncated valid last segment) and passes after the fix. Full
|
||||
suite green including go test -race ./... .
|
||||
|
||||
Audit context: docs/audit-3.2.md C4 (Oracle-verified bg_ef425776).
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 验收清单
|
||||
|
||||
- [ ] Phase A.1：`TailCorruptionError.SegmentPath` 字段存在
|
||||
- [ ] Phase A.2：`ReplaySegmentFile` 签名包含 `isLastSegment bool`
|
||||
- [ ] Phase A.2：非尾段 CollectingFragments 返回硬错误（不是 TailCorruptionError）
|
||||
- [ ] Phase A.2：非尾段 parse error（TailCorruption from parser）转硬错误
|
||||
- [ ] Phase A.3：`RecoverFromSegments` 传递正确 `isLastSegment`
|
||||
- [ ] Phase A.4：`Recover` 用 `tce.SegmentPath` 做截断目标
|
||||
- [ ] Phase B.1：4 个 `ReplaySegmentFile` 单元测试存在
|
||||
- [ ] Phase B.2：3 个集成测试存在（middle/last-single/last-multi）
|
||||
- [ ] Phase B.3：现有 `ReplaySegmentFile` 直接调用更新签名
|
||||
- [ ] `go test ./wal/... -count=1` 全绿
|
||||
- [ ] `go test ./... -count=1` 全绿
|
||||
- [ ] `go test -race ./... -count=1` 全绿
|
||||
- [ ] `go vet ./...` 无新增警告
|
||||
- [ ] 单次 commit，message 引用 audit C4
|
||||
|
||||
---
|
||||
|
||||
## 不在本次范围内（后续 issue）
|
||||
|
||||
| 编号 | 为什么不放进来 |
|
||||
|------|---------------|
|
||||
| C8 | segment_manager.go:64-66 把字节偏移当 startSequence。本 patch 测试用 `writeTestSegment` 绕过，但生产环境多 segment recovery 仍受 C8 影响 |
|
||||
| C1 | CRC 多项式 IEEE → crc32c，独立 |
|
||||
| C7 | Put/Close 竞态，独立 |
|
||||
| H1-H7 | 其他 High，独立 |
|
||||
|
||||
---
|
||||
|
||||
## 修订记录
|
||||
|
||||
- **v1（原始）**：C4 修复方案初稿，送 Momus 审
|
||||
- **v1.0（Momus 审核 bg_423473e7）**：[OKAY]，无 blocking。思考过程敏锐发现 `TestRecoverMiddleSegmentCorruptionHardFails` 中 segment-2 的 startSequence=6 与 segment-1 的实际 complete batch 数不匹配（应该是 4）。修正为 4
|
||||
- **v1.1（Oracle 修订 bg_22edec11）**：
|
||||
- **BLOCKING**：非尾段 parser corruption 用 `%v` 而不是 `%w` —— `IsTailCorruption` 用 `errors.As` 遍历链，`%w` 包装后仍能匹配 `*TailCorruptionError`，会被当 tail corruption 处理
|
||||
- **BLOCKING**：漏了 `wal/recover_test.go:121` 的 `ReplaySegmentFile` 直接调用，需要更新签名。共 3 个直接调用点
|
||||
- **NEW**：加 `TestRecoverMiddleSegmentCRCCorruptionHardFails` 集成测试，覆盖 parser corruption 中段损坏（CRC fail）的端到端路径
|
||||
- **MINOR**：把 SegmentPath attachment 移到非尾段硬错误转换之前，让两条路径都能设置字段（虽然非尾段硬错误不需要，但保持顺序一致）
|
||||
Reference in New Issue
Block a user