fix: stop WAL recovery from advancing MANIFEST or using CURRENT (C2+C3)
Phase 1 default state had two data-loss paths in WAL recovery.
C2: resolveRecoverySegmentID fell back to CURRENT when MANIFEST=0.
Since segment_manager writes CURRENT on every segment create/rotate,
the first recovery in Phase 1 (MANIFEST always 0 without flush) would
start from the active segment, skipping earlier unflushed segments.
C3: Recover called manifest.Save after every recovery, advancing
recoverySegmentID past segments that were still the only durable copy
of their data (no SSTable flush yet). Next restart would filter those
segments out and permanently lose the data.
Per design §3.2 line 280, recovery must not update MANIFEST; per line
604-06, CURRENT must not be used as recovery start. Both fixes are
required together — fixing C3 alone leaves C2's data-loss window open.
Changes:
- wal/recover.go: remove manifest.Save calls on both success and
tail-repair paths; remove CURRENT fallback in resolveRecoverySegmentID.
RecoveryResult.NextSegmentID is now in-memory only (consumed by DB.Open
to seed the new WalWriter, but never persisted to MANIFEST).
- wal/recover_test.go: rewrite TestRecoverUpdatesManifest as
TestRecoverDoesNotUpdateManifest; add TestRecoverPreservesExistingManifest,
TestRecoverIdempotentClean, TestRecoverIdempotentAfterTruncation,
TestRecoverIgnoresCurrentFallback.
- db_test.go: add TestOpenThreeTimesKeepsData (three opens to catch C3's
second-restart data loss; single-segment to avoid unrelated C8 bug
where segment_manager passes byte offset as startSequence).
Verified: each new test fails on pre-fix code and passes after the fix.
Full suite green including -race.
Audit context: docs/audit-3.2.md (with Oracle revisions from bg_ef425776
and bg_2e86d33b; C8 added). Plan: .omo/plans/fix-c2-c3-wal-recovery.md
(Momus + Oracle reviewed v1.2).
This commit is contained in:
dailz
@@ -0,0 +1,641 @@
|
||||
# C→A 修复方案：C2+C3 数据安全 Patch
|
||||
|
||||
## TL;DR
|
||||
|
||||
> **目标**：消除 Phase 1 WAL recovery 的两个数据丢失路径（C2 + C3），按 Oracle 修订意见先更新审核报告，再统一修复。
|
||||
>
|
||||
> **交付**：
|
||||
> - 更新 `docs/audit-3.2.md`，加入 Oracle 的 4 条修订意见
|
||||
> - 删除 `wal/recover.go` 中 2 处 `manifest.Save` 调用 + 1 处 CURRENT fallback
|
||||
> - 重写 1 个测试 + 新增 3 个测试
|
||||
> - 单次 commit 提交
|
||||
>
|
||||
> **预估工时**：1.5-3 小时
|
||||
> **风险**：低（纯删除 + 测试调整，无新逻辑）
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
### 为什么 C2 和 C3 必须一起修
|
||||
|
||||
Oracle 验证（`bg_ef425776`）发现一个我原审核漏掉的关键事实：
|
||||
|
||||
- Phase 1 没有 flush，**MANIFEST 在正常运行中始终是 0**
|
||||
- `resolveRecoverySegmentID` 看到 MANIFEST=0 就 fallback 到 CURRENT
|
||||
- `segment_manager.go:47,89` 每次 create/rotate segment 都顺手写 CURRENT
|
||||
- 所以 Phase 1 默认状态下，首次 Recover 就会从 CURRENT 指向的 active segment 开始，跳过更早的 segment
|
||||
|
||||
→ 单独修 C3（删 `manifest.Save`）没用，C2 路径照样丢数据。
|
||||
|
||||
### Oracle 的其他修订
|
||||
|
||||
1. **C4** 多一个失败模式：`Recover` 总是 truncate `segments[last]`，但 corruption 可能出现在非尾段，导致截断错误的 segment
|
||||
2. **M1**（findValidOffset）应升级为 High：只校验物理记录不跟踪 batch 边界，对 `First+Middle*` 没 `Last` 的情况返回错误的截断点
|
||||
3. **测试方向**：连续两次 Recover 的"幂等"测试不能一刀切，要分干净 WAL 和尾部损坏两种场景
|
||||
|
||||
---
|
||||
|
||||
## 执行计划
|
||||
|
||||
### Phase C：更新审核报告（15 分钟）
|
||||
|
||||
**目标**：把 Oracle 的修订固化到 `docs/audit-3.2.md`，避免下次重新审计时遗漏。
|
||||
|
||||
#### C.1 修订 C2 条目
|
||||
|
||||
在 C2 的"后果"段后追加：
|
||||
|
||||
> **Oracle 修订（bg_ef425776）**：C2 在 Phase 1 比 C3 更严重。Phase 1 没有 flush，MANIFEST 一直保持 0；`segment_manager.go:47,89` 每次创建/轮转 segment 都会写 CURRENT 指向 active segment。**默认状态下首次 Recover 就会触发 fallback**，不需要"CURRENT 缺失/落后"这种特殊场景。修 C3 之前必须先修 C2，否则数据丢失窗口依然存在。
|
||||
|
||||
#### C.2 修订 C3 条目
|
||||
|
||||
在 C3 的"修复"段后追加：
|
||||
|
||||
> **Oracle 修订（bg_ef425776）**：单修 C3 不足以解决 Phase 1 数据丢失，必须和 C2 一起修。测试方向需要区分干净 WAL 和尾部损坏 WAL 两种幂等性。
|
||||
|
||||
#### C.3 修订 C4 条目
|
||||
|
||||
在 C4 的"后果"段后追加：
|
||||
|
||||
> **Oracle 修订（bg_ef425776）**：C4 还有一个更严重的失败模式。`wal/recover.go:61` 总是对 `segments[len(segments)-1]` 调用 `truncateSegment`，但 `RecoverFromSegments` 的 `TailCorruptionError` 可能来自非尾段。结果：真正损坏的 segment 不动，最后一段的有效数据被错误截掉。
|
||||
|
||||
#### C.4 升级 M1 → H8
|
||||
|
||||
把 M1（`findValidOffset` 与原始解析不一致）从 Medium 升到 High，改编号 H8，加入具体证据：
|
||||
|
||||
> **Oracle 修订（bg_ef425776）**：升级为 High。`findValidOffset` 只用 `DecodePhysicalRecord` 校验物理记录，**不跟踪完整 batch 边界**。对尾部 `First + Middle*` 没 `Last` 的情况，`recovery.go:133` 在最后一个完整 batch 结尾报尾部损坏，而 `findValidOffset` 可能返回不完整 fragment 之后的 EOF，导致截断点错位、残留半截 fragment、下次启动反复 repair。
|
||||
|
||||
#### C.5 修订优先级表
|
||||
|
||||
把"必须立刻修的"表格更新为：
|
||||
|
||||
| 编号 | 一句话 | 风险等级 | Phase 1 是否默认触发 |
|
||||
|------|--------|----------|---------------------|
|
||||
| C1 | CRC 多项式错 | 格式不符 | N/A |
|
||||
| **C2** | Recovery CURRENT 兜底 | **恢复起点错误** | **是（默认触发）** |
|
||||
| **C3** | Recovery 写 MANIFEST | 数据丢失 | 是（修了 C2 才能完全止血） |
|
||||
| C4 | 非尾段 fragment 当尾段损坏 | 截断错误 segment | 偶发 |
|
||||
| C5 | 截断无 fsync | DB 状态不一致 | 偶发 |
|
||||
| C6 | dir fsync 静默 | 已确认写入消失 | 偶发 |
|
||||
| C7 | Put/Close 竞态 | 偶发 panic | 偶发 |
|
||||
|
||||
并更新底部优先级：
|
||||
|
||||
> 修订后优先级：**C2 + C3（一起）→ C6 → C5 + H8（一起，同一文件）→ C4 → C1 → C7**
|
||||
|
||||
#### C.6 验证
|
||||
|
||||
```bash
|
||||
# 报告应该读起来前后一致，无矛盾
|
||||
grep -c "Oracle 修订" docs/audit-3.2.md # 期望 4
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase A：修复 C2+C3（45-90 分钟）
|
||||
|
||||
**目标**：消除 Phase 1 默认状态下的两条数据丢失路径。
|
||||
|
||||
#### A.1 修改 `wal/recover.go`
|
||||
|
||||
**变更 1：删除两处 `manifest.Save` 调用**
|
||||
|
||||
文件：`wal/recover.go`
|
||||
|
||||
删除尾部损坏路径（约 line 76-79）：
|
||||
```go
|
||||
// 删除：
|
||||
// Update MANIFEST with new recovery state.
|
||||
if saveErr := manifest.Save(dir, result.NextSegmentID); saveErr != nil {
|
||||
return nil, fmt.Errorf("wal: recover: save manifest after truncation: %w", saveErr)
|
||||
}
|
||||
```
|
||||
|
||||
删除成功路径（约 line 99-102）：
|
||||
```go
|
||||
// 删除：
|
||||
// Update MANIFEST with new recovery state.
|
||||
if saveErr := manifest.Save(dir, result.NextSegmentID); saveErr != nil {
|
||||
return nil, fmt.Errorf("wal: recover: save manifest: %w", saveErr)
|
||||
}
|
||||
```
|
||||
|
||||
**变更 2：删除 CURRENT fallback**
|
||||
|
||||
文件：`wal/recover.go`，函数 `resolveRecoverySegmentID`
|
||||
|
||||
修改前：
|
||||
```go
|
||||
func resolveRecoverySegmentID(dir string) (uint64, error) {
|
||||
mf, err := manifest.Load(dir)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("load manifest: %w", err)
|
||||
}
|
||||
if mf.RecoverySegmentID > 0 {
|
||||
return mf.RecoverySegmentID, nil
|
||||
}
|
||||
|
||||
// MANIFEST had 0 (fresh DB or not yet written). Try CURRENT.
|
||||
if segID, ok := manifest.ReadCurrent(dir); ok {
|
||||
return segID, nil
|
||||
}
|
||||
|
||||
return 0, nil
|
||||
}
|
||||
```
|
||||
|
||||
修改后：
|
||||
```go
|
||||
// resolveRecoverySegmentID returns the recovery start segment ID from MANIFEST.
|
||||
// MANIFEST is the only authoritative source of recovery start per design §3.2
|
||||
// line 600-06. CURRENT is a write-side hint and must NOT be used here.
|
||||
func resolveRecoverySegmentID(dir string) (uint64, error) {
|
||||
mf, err := manifest.Load(dir)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("load manifest: %w", err)
|
||||
}
|
||||
return mf.RecoverySegmentID, nil
|
||||
}
|
||||
```
|
||||
|
||||
**变更 3：更新 import 注释**
|
||||
|
||||
把 `recover.go:27` 注释从 `// Step 1: Determine recovery segment ID from MANIFEST or CURRENT.` 改成 `// Step 1: Determine recovery segment ID from MANIFEST.`
|
||||
|
||||
**变更 4：检查 manifest import 是否还需要**
|
||||
|
||||
`wal/recover.go` 还使用 `manifest.Load`，所以 import 保留。
|
||||
|
||||
#### A.2 检查 manifest.WriteCurrent 是否还有调用方
|
||||
|
||||
```bash
|
||||
grep -rn "manifest.WriteCurrent\|manifest.ReadCurrent" --include="*.go"
|
||||
```
|
||||
|
||||
预期：
|
||||
- `wal/segment_manager.go:48,90` 仍然调用 `WriteCurrent`（合法，CURRENT 作为写入侧 hint）
|
||||
- `manifest/current.go` 自身定义
|
||||
- `wal/recover.go` 之前调用 `ReadCurrent` 的地方已删除
|
||||
|
||||
→ CURRENT 文件保留，只是 recovery 不再读它。符合设计 §3.2 line 276。
|
||||
|
||||
#### A.3 修改 `wal/recover_test.go`
|
||||
|
||||
> **Oracle 修订（bg_2e86d33b）**：
|
||||
> - BLOCKING：原计划的可选 e2e `TestOpenTwiceKeepsData`（两次 open）不足以验证 C3。C3 数据丢失发生在**第二次 restart**，需要**三次 open**才能抓到。已升级为必选。
|
||||
> - NICE-TO-HAVE：注释中"NewSegmentWriter writes CURRENT"是错的，只有 SegmentManager 写。修正注释。
|
||||
> - NICE-TO-HAVE：MANIFEST 已存在的 subcase，防止"覆盖已有 MANIFEST"的回归。
|
||||
> - 注意：测试中避免强制多 segment 轮转，因为 `segment_manager.go:64-66` 有无关 bug（C8，见审核报告），多 segment recovery 会失败。
|
||||
|
||||
**变更 1：重写 `TestRecoverUpdatesManifest` → `TestRecoverDoesNotUpdateManifest`**
|
||||
|
||||
改名 `TestRecoverDoesNotUpdateManifest`，断言相反：
|
||||
|
||||
```go
|
||||
func TestRecoverDoesNotUpdateManifest(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// Write test data. NewSegmentWriter creates the .wal file via the
|
||||
// durable-ready protocol but does NOT write CURRENT (only SegmentManager
|
||||
// does) and does NOT write MANIFEST.
|
||||
writeTestSegment(t, dir, 0, 50, [][]*WalEntry{
|
||||
{makePutEntry("a", "b")},
|
||||
{makePutEntry("c", "d")},
|
||||
})
|
||||
|
||||
// Snapshot MANIFEST state before recovery. For a fresh DB, MANIFEST does
|
||||
// not exist on disk; manifest.Load returns a zero-value Manifest.
|
||||
beforeMF, err := manifest.Load(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("manifest.Load before recover: %v", err)
|
||||
}
|
||||
beforeExists := fileExists(t, filepath.Join(dir, "MANIFEST"))
|
||||
|
||||
replayer := &mockReplayer{}
|
||||
result, err := Recover(dir, replayer)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover: %v", err)
|
||||
}
|
||||
|
||||
// MANIFEST must be unchanged.
|
||||
afterMF, err := manifest.Load(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("manifest.Load after recover: %v", err)
|
||||
}
|
||||
if afterMF.RecoverySegmentID != beforeMF.RecoverySegmentID {
|
||||
t.Errorf("MANIFEST RecoverySegmentID changed: %d → %d",
|
||||
beforeMF.RecoverySegmentID, afterMF.RecoverySegmentID)
|
||||
}
|
||||
afterExists := fileExists(t, filepath.Join(dir, "MANIFEST"))
|
||||
if beforeExists != afterExists {
|
||||
t.Errorf("MANIFEST file existence changed: before=%v after=%v",
|
||||
beforeExists, afterExists)
|
||||
}
|
||||
|
||||
// RecoveryResult.NextSegmentID is in-memory only — not persisted.
|
||||
_ = result // NextSegmentID is allowed to differ from MANIFEST.RecoverySegmentID.
|
||||
}
|
||||
|
||||
// Subtest: MANIFEST already exists (e.g. from a prior flush in future phases).
|
||||
// Recovery must not overwrite or delete it. Catches accidental writes that
|
||||
// fresh-DB subtest above cannot detect (since fresh DB has no MANIFEST).
|
||||
func TestRecoverPreservesExistingManifest(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
writeTestSegment(t, dir, 0, 50, [][]*WalEntry{
|
||||
{makePutEntry("a", "b")},
|
||||
})
|
||||
|
||||
// Simulate a prior checkpoint having advanced MANIFEST to segment 0.
|
||||
// (Even though Phase 1 has no flush, future phases will. This test
|
||||
// guards the invariant going forward.)
|
||||
if err := manifest.Save(dir, 0); err != nil {
|
||||
t.Fatalf("manifest.Save setup: %v", err)
|
||||
}
|
||||
beforeBytes, err := os.ReadFile(filepath.Join(dir, "MANIFEST"))
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile MANIFEST: %v", err)
|
||||
}
|
||||
|
||||
replayer := &mockReplayer{}
|
||||
if _, err := Recover(dir, replayer); err != nil {
|
||||
t.Fatalf("Recover: %v", err)
|
||||
}
|
||||
|
||||
afterBytes, err := os.ReadFile(filepath.Join(dir, "MANIFEST"))
|
||||
if err != nil {
|
||||
t.Fatalf("ReadFile MANIFEST after recover: %v", err)
|
||||
}
|
||||
if !bytes.Equal(beforeBytes, afterBytes) {
|
||||
t.Errorf("MANIFEST bytes changed:\n before=%q\n after=%q",
|
||||
string(beforeBytes), string(afterBytes))
|
||||
}
|
||||
}
|
||||
|
||||
func fileExists(t *testing.T, path string) bool {
|
||||
t.Helper()
|
||||
_, err := os.Stat(path)
|
||||
if err == nil {
|
||||
return true
|
||||
}
|
||||
if os.IsNotExist(err) {
|
||||
return false
|
||||
}
|
||||
t.Fatalf("stat %s: %v", path, err)
|
||||
return false
|
||||
}
|
||||
```
|
||||
|
||||
**变更 2：新增 `TestRecoverIdempotentClean`**
|
||||
|
||||
```go
|
||||
// TestRecoverIdempotentClean verifies that recovering a clean WAL twice
|
||||
// produces identical results. This is the "no MANIFEST writes" guarantee
|
||||
// from design §3.2 line 280.
|
||||
func TestRecoverIdempotentClean(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("k1", "v1")},
|
||||
{makePutEntry("k2", "v2")},
|
||||
})
|
||||
|
||||
replayer1 := &mockReplayer{}
|
||||
result1, err := Recover(dir, replayer1)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover (1st): %v", err)
|
||||
}
|
||||
|
||||
replayer2 := &mockReplayer{}
|
||||
result2, err := Recover(dir, replayer2)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover (2nd): %v", err)
|
||||
}
|
||||
|
||||
if result1.NextSequence != result2.NextSequence {
|
||||
t.Errorf("NextSequence differs: %d vs %d", result1.NextSequence, result2.NextSequence)
|
||||
}
|
||||
if result1.NextSegmentID != result2.NextSegmentID {
|
||||
t.Errorf("NextSegmentID differs: %d vs %d", result1.NextSegmentID, result2.NextSegmentID)
|
||||
}
|
||||
if result1.Truncated || result2.Truncated {
|
||||
t.Errorf("Truncated should be false for clean WAL: r1=%v r2=%v",
|
||||
result1.Truncated, result2.Truncated)
|
||||
}
|
||||
if !reflect.DeepEqual(replayer1.puts, replayer2.puts) {
|
||||
t.Errorf("replayed puts differ:\n r1=%#v\n r2=%#v", replayer1.puts, replayer2.puts)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**变更 3：新增 `TestRecoverIdempotentAfterTruncation`**
|
||||
|
||||
```go
|
||||
// TestRecoverIdempotentAfterTruncation verifies that after the first recovery
|
||||
// truncates a corrupted tail, the second recovery sees a clean WAL with the
|
||||
// same NextSequence. Per Oracle: the first call reports Truncated=true, the
|
||||
// second reports Truncated=false but identical replay state.
|
||||
func TestRecoverIdempotentAfterTruncation(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
filePath := writeTestSegment(t, dir, 0, 100, [][]*WalEntry{
|
||||
{makePutEntry("good1", "before-corruption")},
|
||||
{makePutEntry("good2", "also-before")},
|
||||
})
|
||||
appendFileBytes(t, filePath, []byte{0xDE, 0xAD, 0xBE, 0xEF})
|
||||
|
||||
// First recovery: detects tail corruption, truncates, returns Truncated=true.
|
||||
replayer1 := &mockReplayer{}
|
||||
result1, err := Recover(dir, replayer1)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover (1st): %v", err)
|
||||
}
|
||||
if !result1.Truncated {
|
||||
t.Fatal("1st Recover: Truncated = false, want true")
|
||||
}
|
||||
|
||||
// Second recovery: file has been truncated, no corruption remains.
|
||||
replayer2 := &mockReplayer{}
|
||||
result2, err := Recover(dir, replayer2)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover (2nd): %v", err)
|
||||
}
|
||||
if result2.Truncated {
|
||||
t.Error("2nd Recover: Truncated = true, want false (tail already repaired)")
|
||||
}
|
||||
if result1.NextSequence != result2.NextSequence {
|
||||
t.Errorf("NextSequence differs: %d vs %d", result1.NextSequence, result2.NextSequence)
|
||||
}
|
||||
if !reflect.DeepEqual(replayer1.puts, replayer2.puts) {
|
||||
t.Errorf("replayed puts differ:\n r1=%#v\n r2=%#v", replayer1.puts, replayer2.puts)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**变更 4：新增 `TestRecoverIgnoresCurrentFallback`**（C2 回归测试）
|
||||
|
||||
> **Momus 修订（bg_3dfb54a9）**：`writeTestSegment` 走 `NewSegmentWriter`，**不写 CURRENT**（只有 `SegmentManager` 写）。测试必须**显式**调用 `manifest.WriteCurrent` 模拟 Phase 1 默认状态。
|
||||
|
||||
```go
|
||||
// TestRecoverIgnoresCurrentFallback verifies that CURRENT is not used as a
|
||||
// recovery start fallback when MANIFEST.RecoverySegmentID == 0. Per design
|
||||
// §3.2 line 604-06, CURRENT is only a write-side hint and must not affect
|
||||
// recovery start. Without this guarantee, Phase 1 default state (MANIFEST=0,
|
||||
// CURRENT pointing to active segment) causes recovery to skip older segments.
|
||||
//
|
||||
// Note: writeTestSegment uses NewSegmentWriter directly, which does NOT write
|
||||
// CURRENT (only SegmentManager does). We must write CURRENT explicitly to
|
||||
// simulate the Phase 1 default state where SegmentManager has been rotating
|
||||
// segments.
|
||||
func TestRecoverIgnoresCurrentFallback(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
// Write 3 segments with valid batches.
|
||||
writeTestSegment(t, dir, 0, 0, [][]*WalEntry{
|
||||
{makePutEntry("seg0-k1", "v1")},
|
||||
})
|
||||
writeTestSegment(t, dir, 1, 1, [][]*WalEntry{
|
||||
{makePutEntry("seg1-k1", "v1")},
|
||||
})
|
||||
writeTestSegment(t, dir, 2, 2, [][]*WalEntry{
|
||||
{makePutEntry("seg2-k1", "v1")},
|
||||
})
|
||||
|
||||
// Simulate Phase 1 default: SegmentManager has been rotating, so CURRENT
|
||||
// exists and points to the last segment (2). MANIFEST still does not exist
|
||||
// because no flush has happened yet.
|
||||
if err := manifest.WriteCurrent(dir, 2); err != nil {
|
||||
t.Fatalf("WriteCurrent: %v", err)
|
||||
}
|
||||
|
||||
// Sanity: CURRENT exists and points to segment 2.
|
||||
currentSegID, ok := manifest.ReadCurrent(dir)
|
||||
if !ok || currentSegID != 2 {
|
||||
t.Fatalf("CURRENT setup wrong: segID=%d ok=%v", currentSegID, ok)
|
||||
}
|
||||
// Sanity: MANIFEST.RecoverySegmentID == 0 (fresh DB).
|
||||
mf, err := manifest.Load(dir)
|
||||
if err != nil {
|
||||
t.Fatalf("Load: %v", err)
|
||||
}
|
||||
if mf.RecoverySegmentID != 0 {
|
||||
t.Fatalf("MANIFEST.RecoverySegmentID = %d, want 0", mf.RecoverySegmentID)
|
||||
}
|
||||
|
||||
// Recover must start from segment 0, not 2.
|
||||
replayer := &mockReplayer{}
|
||||
result, err := Recover(dir, replayer)
|
||||
if err != nil {
|
||||
t.Fatalf("Recover: %v", err)
|
||||
}
|
||||
|
||||
// All 3 entries must be replayed.
|
||||
wantPuts := []replayPut{
|
||||
{key: "seg0-k1", value: "v1", seq: 0},
|
||||
{key: "seg1-k1", value: "v1", seq: 1},
|
||||
{key: "seg2-k1", value: "v1", seq: 2},
|
||||
}
|
||||
if !reflect.DeepEqual(replayer.puts, wantPuts) {
|
||||
t.Errorf("puts = %#v, want %#v", replayer.puts, wantPuts)
|
||||
}
|
||||
if result.NextSequence != 3 {
|
||||
t.Errorf("NextSequence = %d, want 3", result.NextSequence)
|
||||
}
|
||||
if result.NextSegmentID != 3 {
|
||||
t.Errorf("NextSegmentID = %d, want 3", result.NextSegmentID)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### A.4 验证步骤
|
||||
|
||||
按顺序执行：
|
||||
|
||||
```bash
|
||||
# 1. 编译通过
|
||||
go build ./...
|
||||
|
||||
# 2. wal 包测试全绿
|
||||
go test ./wal/... -count=1 -v
|
||||
|
||||
# 3. 全仓测试不回归
|
||||
go test ./... -count=1
|
||||
|
||||
# 4. vet
|
||||
go vet ./...
|
||||
|
||||
# 5. lint（如果环境装了 golangci-lint；没装就跳过，不阻塞）
|
||||
if command -v golangci-lint >/dev/null 2>&1; then
|
||||
golangci-lint run ./wal/... ./manifest/...
|
||||
else
|
||||
echo "golangci-lint not installed, skipping"
|
||||
fi
|
||||
|
||||
# 6. 重点跑这次新增的 4+1 个测试（Oracle 建议）
|
||||
go test ./wal -run 'TestRecover(DoesNotUpdateManifest|PreservesExistingManifest|IdempotentClean|IdempotentAfterTruncation|IgnoresCurrentFallback)$' -count=1 -v
|
||||
go test . -run 'TestOpenThreeTimesKeepsData$' -count=1 -v
|
||||
|
||||
# 7. race 检测（Oracle 建议，wal 包有 goroutine）
|
||||
go test -race ./wal/... -count=1
|
||||
```
|
||||
|
||||
#### A.5 必选 e2e：`TestOpenThreeTimesKeepsData`（Oracle 升级）
|
||||
|
||||
> **Oracle 修订（bg_2e86d33b）BLOCKING**：原计划 `TestOpenTwiceKeepsData` 两次 open 不足以验证 C3。C3 的数据丢失发生在**第二次 restart**（第一次 recovery 写了 MANIFEST，第二次 recovery 才会跳过 segment）。必须用**三次 open**。
|
||||
|
||||
放在 `db_test.go`（package `go_kv`）：
|
||||
|
||||
```go
|
||||
// TestOpenThreeTimesKeepsData is the end-to-end regression for C2+C3.
|
||||
//
|
||||
// C3's data-loss bug manifests on the SECOND restart after writes:
|
||||
// - Open 1: write data, close. (Phase 1: no MANIFEST write yet from flush.)
|
||||
// - Open 2: recovery (buggy code) writes MANIFEST=NextSegmentID, then
|
||||
// opens new writer. Data still visible because the in-memory memtable
|
||||
// was rebuilt from WAL.
|
||||
// - Open 3: recovery reads advanced MANIFEST, skips old segments,
|
||||
// data NOT replayed → data permanently invisible.
|
||||
//
|
||||
// Two opens cannot catch this; three opens can.
|
||||
//
|
||||
// Single-segment only: do NOT force rotation, because segment_manager
|
||||
// has an unrelated C8 bug (passes byte offset as startSequence) that
|
||||
// breaks multi-segment recovery. That bug is tracked separately.
|
||||
func TestOpenThreeTimesKeepsData(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
cfg := config.Defaults()
|
||||
// Default MaxSegmentSize=64MB is plenty for these writes; no rotation.
|
||||
|
||||
// Open 1: write keys.
|
||||
db1, err := Open(dir, &cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Open 1: %v", err)
|
||||
}
|
||||
keys := []string{"k1", "k2", "k3", "k4", "k5"}
|
||||
for _, k := range keys {
|
||||
if err := db1.Put([]byte(k), []byte("v-"+k)); err != nil {
|
||||
t.Fatalf("Put %s: %v", k, err)
|
||||
}
|
||||
}
|
||||
if err := db1.Close(); err != nil {
|
||||
t.Fatalf("Close 1: %v", err)
|
||||
}
|
||||
|
||||
// Open 2: verify, then close. (Buggy code: this is where MANIFEST
|
||||
// gets advanced. Fixed code: MANIFEST stays unchanged.)
|
||||
db2, err := Open(dir, &cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Open 2: %v", err)
|
||||
}
|
||||
for _, k := range keys {
|
||||
r := db2.Get([]byte(k))
|
||||
if !r.Found {
|
||||
t.Errorf("Open 2: key %s not found", k)
|
||||
}
|
||||
}
|
||||
if err := db2.Close(); err != nil {
|
||||
t.Fatalf("Close 2: %v", err)
|
||||
}
|
||||
|
||||
// Open 3: verify again. This is where C3's data loss would manifest
|
||||
// on buggy code (MANIFEST was advanced in Open 2, recovery now skips
|
||||
// the original segments).
|
||||
db3, err := Open(dir, &cfg)
|
||||
if err != nil {
|
||||
t.Fatalf("Open 3: %v", err)
|
||||
}
|
||||
defer db3.Close()
|
||||
for _, k := range keys {
|
||||
r := db3.Get([]byte(k))
|
||||
if !r.Found {
|
||||
t.Errorf("Open 3: key %s not found (C3 regression: data lost)", k)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### A.6 Commit
|
||||
|
||||
单次 commit，message：
|
||||
|
||||
```
|
||||
fix(wal): C2+C3 stop recovery from advancing MANIFEST or using CURRENT
|
||||
|
||||
Phase 1 default state had two data-loss paths in WAL recovery:
|
||||
|
||||
1. (C2) resolveRecoverySegmentID fell back to CURRENT when MANIFEST=0.
|
||||
Since segment_manager writes CURRENT on every segment create/rotate,
|
||||
the first recovery in Phase 1 (MANIFEST always 0 without flush) would
|
||||
start from the active segment, skipping earlier unflushed segments.
|
||||
|
||||
2. (C3) Recover called manifest.Save after every recovery, advancing
|
||||
recoverySegmentID past segments that were still the only durable copy
|
||||
of their data (no SSTable flush yet). Next restart would filter those
|
||||
segments out and permanently lose the data.
|
||||
|
||||
Per design §3.2 line 280, recovery must not update MANIFEST; per line 604-06,
|
||||
CURRENT must not be used as recovery start. Both fixes are required together
|
||||
— fixing C3 alone leaves C2's data-loss window open.
|
||||
|
||||
Changes:
|
||||
- wal/recover.go: remove manifest.Save calls on both success and tail-repair
|
||||
paths; remove CURRENT fallback in resolveRecoverySegmentID.
|
||||
RecoveryResult.NextSegmentID is now in-memory only (consumed by DB.Open
|
||||
to seed the new writer, but never persisted to MANIFEST).
|
||||
- wal/recover_test.go: rewrite TestRecoverUpdatesManifest as
|
||||
TestRecoverDoesNotUpdateManifest; add TestRecoverPreservesExistingManifest,
|
||||
TestRecoverIdempotentClean, TestRecoverIdempotentAfterTruncation,
|
||||
TestRecoverIgnoresCurrentFallback.
|
||||
- db_test.go: add TestOpenThreeTimesKeepsData (three opens to catch C3's
|
||||
second-restart data loss).
|
||||
|
||||
Refs docs/audit-3.2.md C2+C3 (with Oracle revisions from bg_ef425776 and
|
||||
bg_2e86d33b).
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 验收清单
|
||||
|
||||
- [ ] Phase C：`docs/audit-3.2.md` 包含 4 处"Oracle 修订"标注
|
||||
- [ ] Phase A.1：`wal/recover.go` 中 `manifest.Save` 出现次数 = 0
|
||||
- [ ] Phase A.1：`wal/recover.go` 中 `manifest.ReadCurrent` 出现次数 = 0
|
||||
- [ ] Phase A.3：5 个测试名（1 改 + 4 新）全部存在
|
||||
- [ ] Phase A.5：`TestOpenThreeTimesKeepsData`（必选）通过
|
||||
- [ ] `go test ./wal/... -count=1` 全绿
|
||||
- [ ] `go test . -count=1` 全绿（含 db_test.go）
|
||||
- [ ] `go test ./... -count=1` 全绿
|
||||
- [ ] `go test -race ./wal/... -count=1` 全绿
|
||||
- [ ] `go vet ./...` 无新增警告
|
||||
- [ ] 单次 commit，message 引用 audit C2+C3
|
||||
|
||||
---
|
||||
|
||||
## 不在本次范围内（后续 issue）
|
||||
|
||||
| 编号 | 为什么不放进来 |
|
||||
|------|---------------|
|
||||
| **C8（新）** | `segment_manager.go:64-66` 把 `active.CurrentOffset()`（字节偏移）当 `newStartSequence` 传给 rotate。多 segment recovery 会因 startSequence 不匹配而失败。和 C2+C3 完全独立，但相关测试必须避免强制轮转 |
|
||||
| C4 | 涉及 `RecoverFromSegments` 接口变更（isLast 参数），改动面更大，独立做 |
|
||||
| C5 + H8 | 都在 truncation 路径，应该一起做（fsync + 删空 segment + dir fsync + findValidOffset batch 边界），但和 C2+C3 不耦合 |
|
||||
| C6 | 涉及 segment_writer + segment_manager，独立做 |
|
||||
| C1 | CRC 改动跨多个文件，需要测试向量，独立做 |
|
||||
| C7 | 并发竞态，独立做 |
|
||||
|
||||
---
|
||||
|
||||
## 修订记录
|
||||
|
||||
- **v1（原始）**：C→A 方案初稿，送 Momus 审
|
||||
- **v1.1（Momus 修订 bg_3dfb54a9）**：
|
||||
- Blocking：`TestRecoverIgnoresCurrentFallback` setup 假设 `writeTestSegment` 会写 CURRENT，但实际只有 `SegmentManager` 写。修订：测试中显式调用 `manifest.WriteCurrent(dir, 2)` 模拟 Phase 1 默认状态
|
||||
- Minor：`golangci-lint run ... 2>/dev/null || true` 太宽容，改为 `command -v` 探测，没装就跳过
|
||||
- **v1.2（Oracle 修订 bg_2e86d33b）**：
|
||||
- Blocking：`TestOpenTwiceKeepsData`（两次 open）不足以验证 C3，C3 数据丢失发生在第二次 restart。升级为必选 `TestOpenThreeTimesKeepsData`（三次 open）。测试中避免强制多 segment 轮转，因为 C8 会让多 segment recovery 失败
|
||||
- 新增 `TestRecoverPreservesExistingManifest` 子测试，覆盖"覆盖已有 MANIFEST"的回归
|
||||
- 修正 `TestRecoverDoesNotUpdateManifest` 的注释（NewSegmentWriter 不写 CURRENT）
|
||||
- 加重点测试命令 + `-race`
|
||||
- commit message 补一句 `RecoveryResult.NextSegmentID` 现在是 in-memory
|
||||
- 发现无关 bug C8（segment_manager 轮转 startSequence 错），加入"不在本次范围"表
|
||||
Reference in New Issue
Block a user