Files

T

dailz fe2d4fc5f0 feat(wal): implement WAL writer with group commit and recovery batch replay

- wal/commit_queue.go: bounded buffered channel for write requests
- wal/writer.go: single-goroutine main loop implementing 11-step write flow
  with group commit, sequence allocation, MemTable publish/abort, write-stopped
- wal/recovery.go: BatchReplayer interface, ReplayBatch, ReplaySegmentFile,
  RecoverFromSegments with fragment reassembly and tail corruption handling
- Comprehensive tests for all modules, all pass with -race

2026-06-12 13:57:31 +08:00

68 KiB

Raw Blame History

Phase 1: WAL 子系统 — Go KV 存储引擎

TL;DR

Quick Summary: 基于 docs/design.md §3.2 实现 WAL 子系统，配合 MemTable（§3.3）、读路径（§3.4）和最小 MANIFEST，构建可嵌入的单 key autocommit KV 存储引擎。

Deliverables:

WAL 写入路径（group commit、sequence 管理、segment 轮转、fsync）

WAL 恢复路径（segment 扫描、fragment 重组、batch 校验、尾部截断）

MemTable（Arena + SkipList + 原子发布）

嵌入式 API（Open/Close/Put/Delete/Get）

MANIFEST 最小 stub + CURRENT 文件

完整测试套件（单元 + 集成 + 故障注入）

Estimated Effort: XL（~30-50 个任务） Parallel Execution: YES — 5 waves Critical Path: Task 1 → Task 6 → Task 10 → Task 11 → Task 13 → Task 19 → Task 20/21 → F1-F4

Context

Original Request

用户指向 docs/design.md §3.2（WAL），要求制定第一阶段开发方案。

Interview Summary

Key Discussions:

设计文档是唯一权威来源（1189 行，§3.2 WAL 约 730 行）
项目为 greenfield — 无任何 Go 源码
Phase 1 范围：WAL + MemTable + 读路径 + MANIFEST stub + 嵌入式 API
不含 SSTable / Compaction / Value Log / MVCC / 网络层

Research Findings:

项目完全空白（仅有 README.md、.omo/、docs/）
无 go.mod、无依赖
设计文档定义了极其精确的崩溃语义和错误分类

Metis Review

Identified Gaps (addressed):

MANIFEST 完整设计不在文档中 → Phase 1 用最小 stub（仅存 recoverySegmentID）
MemTable 无 SSTable 可刷 → 冻结为 Immutable 后持有在内存，不刷盘；3×64MB 上限后阻塞写入
Periodic/Never sync 策略 → Phase 1 只实现 Always
Group commit 定时器 → time.AfterFunc + reset
API surface → Put/Delete/Get/Open/Close + GetDurableSequence + IsWriteStopped
Graceful shutdown → Close() 等待 pending batch、sync WAL、写 MANIFEST
文件锁 → Phase 1 不实现，单进程假设
Varint → Go 标准库 encoding/binary.PutUvarint，key/value 上限 4KB 保证 varint ≤ 5 bytes

Work Objectives

Core Objective

实现一个功能完整的 WAL 子系统，支持 group commit 写入、崩溃恢复、和单 key autocommit 语义，作为后续 SSTable / MVCC 的基础。

Concrete Deliverables

wal/ — WAL 编解码、segment 管理、写入器、恢复器
memtable/ — Arena + SkipList + 原子发布
db.go — DB 入口（Open/Close/Put/Delete/Get）
errors.go — 公共错误类型（ErrCommitUnknown 等）
manifest/ — 最小 MANIFEST stub
测试覆盖所有关键路径

Definition of Done

go build ./... 编译通过
go test ./... -race 全部通过，无 data race
写入 N 条数据 → Close → Open → Get 全部正确
WAL 尾部损坏 → 恢复成功，完整 batch 不丢失
ErrCommitUnknown 后引擎进入 write-stopped

Must Have

WAL 写入路径严格遵循设计文档步骤 ①-⑪
Per-batch private encode buffer + direct write（无 bufio.Writer）
atomic.Uint64 用于 publishedSequence / durableSequence / nextSequence
atomic.Pointer 用于 skiplist next 指针
MemTable Arena 容量预留发生在 WAL write 之前
Sequence 分配前完成所有可失败校验
WAL Batch 不跨 segment
Segment rotation 只在 batch 边界
Get 返回 GetResult{Found bool, Value []byte} 区分 key-not-found 与 empty-value
Config 不变量在 Open 时校验（checked arithmetic）
Recovery 严格遵循 fragment 状态机
错误分类严格遵循设计文档表格

Must NOT Have (Guardrails)

❌ SSTable / Compaction / Bloom Filter / Block Cache
❌ Value Log / ValueLogPointer 写入（Entry 类型定义可预留，但 Phase 1 不写入 ValueLogPointer）
❌ MVCC / SSI 事务
❌ Periodic / Never sync 策略（仅 Always）
❌ Range scan / Iterator API
❌ 网络层
❌ bufio.Writer 在 WAL append 路径上
❌ MemTable 写入失败发生在 WAL write 成功之后
❌ CRC 校验跳过
❌ 共享 buffered writer 模糊 WAL 副作用边界

Verification Strategy

ZERO HUMAN INTERVENTION — ALL verification is agent-executed. No exceptions.

Test Decision

Infrastructure exists: NO (greenfield)
Automated tests: Tests-after（先实现后测试，每个模块实现完即写测试）
Framework: Go 标准 testing + testify（断言库）
Agent-Executed QA: 每个任务包含 go test 命令和 go test -race 验证

QA Policy

每个 task 的 QA 通过 go test 和 go vet 执行
Evidence: test output saved to .omo/evidence/task-{N}-{scenario}.txt

Execution Strategy

Parallel Execution Waves

Wave 1a (Foundation — 1 task):
└── 1. Project scaffolding + go.mod + config types [quick]

Wave 1b (Codec + Types — 4 parallel + 2 independent, after Wave 1a):
├── 2. Error types + constants [quick]
├── 3. WAL File Header encode/decode [quick]
├── 4. Physical Record encode/decode + Block boundary [quick]
├── 5. WAL Entry encode/decode [quick]
├── 12. Sequence manager [quick]
└── 18. MANIFEST stub + CURRENT file [quick]

Wave 2a (Batch + Arena + Validation — 3 parallel, after Wave 1b):
├── 6. WAL Batch encode/decode + fragment collector [unspecified-high]  (needs 3,4,5)
├── 7. Arena allocator [unspecified-high]  (needs 1)
└── 9. WAL Batch resource validation [quick]  (needs 1, 5)

Wave 2b (SkipList — after Arena):
└── 8. SkipList (mutex write + lock-free read) [deep]  (needs 7)

Wave 3a (Segment writer — after Batch codec):
└── 10. Segment writer (file write + block writer) [unspecified-high]  (needs 3,4,6)

Wave 3b (Segment rotation — after Segment writer):
└── 11. Segment rotation + durable-ready protocol [unspecified-high]  (needs 10)

Wave 3c (MemTable — after Arena + SkipList):
└── 14. MemTable (wrap skiplist + arena + publish/abort) [unspecified-high]  (needs 7,8)

Wave 3d (WAL Writer — after MemTable API available):
└── 13. Commit queue + WAL writer main loop [deep]  (needs 2,9,10,11,12,14)

Wave 4a (Recovery scanner — after Segment writer):
└── 15. WAL recovery: scanner + record parser [unspecified-high]  (needs 4,10)

Wave 4b (Recovery batch replay — after scanner + Batch codec):
└── 16. WAL recovery: fragment collector + batch replay [deep]  (needs 6,15)

Wave 4c (Recovery tail truncation — after batch replay):
└── 17. WAL recovery: tail truncation + main flow [unspecified-high]  (needs 15,16)

Wave 4d (DB integration — after all above):
└── 19. DB integration: Open/Close/Put/Delete/Get [deep]  (needs 13,14,17,18)

Wave 5 (Tests + Benchmark — after DB integration):
├── 20. End-to-end + crash recovery tests [deep]
└── 21. Benchmark suite [unspecified-high]

Wave FINAL (4 parallel reviews):
├── F1. Plan compliance audit (oracle)
├── F2. Code quality review (unspecified-high)
├── F3. Real manual QA (unspecified-high)
└── F4. Scope fidelity check (deep)

Critical Path: 1 → 6 → 10 → 11 → 13 → 19 → 20/21 → F1-F4
Parallel Speedup: ~50% faster than sequential
Max Concurrent: 6 (Wave 1b)

Dependency Matrix

Task	Depends On	Blocks	Sub-Wave
1	—	2-5, 7, 9, 12, 18	1a
2	1	13	1b
3	1	6, 10	1b
4	1	6, 10, 15	1b
5	1	6, 9	1b
6	3, 4, 5	10, 16	2a
7	1	8, 14	2a
8	7	14	2b
9	1, 5	13	2a
10	3, 4, 6	11, 15	3a
11	10	13	3b
12	1	13	1b
13	2, 9, 10, 11, 12, 14	19	3d
14	7, 8	19	3c
15	4, 10	16	4a
16	6, 15	17	4b
17	15, 16	19	4c
18	1	19	1b
19	13, 14, 17, 18	20, 21	4d
20	19	F1-F4	5
21	19	F1-F4	5

Agent Dispatch Summary

1a: 1 — T1 → quick
1b: 6 — T2 → quick, T3 → quick, T4 → quick, T5 → quick, T12 → quick, T18 → quick
2a: 3 — T6 → unspecified-high, T7 → unspecified-high, T9 → quick
2b: 1 — T8 → deep
3a: 1 — T10 → unspecified-high
3b: 1 — T11 → unspecified-high
3c: 1 — T14 → unspecified-high
3d: 1 — T13 → deep
4a: 1 — T15 → unspecified-high
4b: 1 — T16 → deep
4c: 1 — T17 → unspecified-high
4d: 1 — T19 → deep
5: 2 — T20 → deep, T21 → unspecified-high
FINAL: 4 — F1 → oracle, F2 → unspecified-high, F3 → unspecified-high, F4 → deep

TODOs

1. Project scaffolding + go.mod + config types

What to do:
- 初始化 go.mod（模块名 github.com/dailz/go-kv，Go 1.22+）
- 创建目录结构：wal/、memtable/、manifest/、config/
- 创建 config/config.go：WalConfig 结构体（MaxSegmentSize, BlockSize, SyncMode, MaxBatchEntries, MaxBatchSize, MaxKeyBytes, MaxInlineValue, MemTableSize, MaxImmutableCount）
- 创建 config/config.go：Validate() error 函数，用 checked arithmetic 校验不变量：maxWalSegmentPayload >= maxEncodedWalBatchSize + worstCasePhysicalRecordOverhead + worstCaseBlockPadding
- 计算公式参照设计文档 lines 402-453：walFileHeaderSize=32, walBlockSize=32768, physicalRecordHeaderSize=7, walBatchHeaderSize=18, maxWalBatchEntriesSize=4MB
- 创建 .golangci.yml
Must NOT do:
- 不引入外部依赖（仅标准库 + testify）
- 不定义 SSTable / Compaction 相关配置
Recommended Agent Profile:
- Category: quick
- Skills: [golang-project-layout]
  - golang-project-layout: Go 项目结构最佳实践
- Skills Evaluated but Omitted:
  - golang-cli: 不是 CLI 工具
Parallelization:
- Can Run In Parallel: NO (foundation task)
- Parallel Group: Sub-Wave 1a (solo — blocks all others)
- Blocks: 2, 3, 4, 5, 7, 9, 12, 18
- Blocked By: None
References: Pattern References:
- docs/design.md:286-289 — 设计决策表：批量窗口、日志格式、文件管理、segment 大小
- docs/design.md:402-453 — 配置不变量计算公式和最小 segment 大小推导
API/Type References:
- docs/design.md:501-513 — WAL Batch 资源上限默认值
External References:
- Go project layout: https://go.dev/doc/modules/layout
WHY Each Reference Matters:
- 设计决策表定义了所有默认值，config 结构体必须映射这些值
- 配置不变量公式是 Open 时必须校验的核心约束
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Config validation accepts valid defaults
  Tool: Bash
  Preconditions: Go toolchain installed
  Steps:
    1. cd /home/dailz/workspace/src/go-kv
    2. Create a test file config/config_test.go with TestValidateDefaults
    3. Test that WalConfig with default values passes Validate()
  Expected Result: go test ./config/... passes
  Evidence: .omo/evidence/task-1-config-defaults.txt

Scenario: Config validation rejects invalid segment size
  Tool: Bash
  Preconditions: go.mod exists
  Steps:
    1. Create TestValidateSegmentTooSmall in config/config_test.go
    2. Set MaxSegmentSize = 1024 (far below minimum 4,195,264)
    3. Verify Validate() returns error containing "segment"
  Expected Result: Test fails validation with descriptive error
  Evidence: .omo/evidence/task-1-config-invalid.txt
```
Commit: YES (group with Task 2)
- Message: feat: initialize project structure and error types
- Files: go.mod, config/, .golangci.yml
2. Error types + constants

What to do:
- 创建 errors.go：定义 ErrCommitUnknown、ErrWriteStopped、ErrSequenceExhausted、ErrWALCorrupted、ErrInvalidConfig
- 创建 wal/constants.go：所有 WAL 常量（WalMagic, WalFormatVersion, WalFileHeaderSize, WalBlockSize, PhysicalRecordHeaderSize, WalBatchHeaderSize, MaxWalBatchEntryCount, MaxWalBatchEntriesSize, MaxWalKeyBytes, MaxWalInlineValueBytes, MaxWalVarintBytes, DefaultMaxWalSegmentSize）
- 创建 wal/constants.go：Fragment types (RecInvalid=0, RecFull=1, RecFirst=2, RecMiddle=3, RecLast=4)
- 创建 wal/constants.go：OpType (OpInvalid=0, OpPut=1, OpDelete=2) 和 ValueKind (VKNone=0, VKInline=1, VKValueLogPointer=2)
Must NOT do:
- 不实现具体逻辑，仅类型和常量定义
Recommended Agent Profile:
- Category: quick
- Skills: []
- Skills Evaluated but Omitted:
  - golang-naming: 常量命名简单，不需要完整命名 skill
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 1b (with Tasks 3, 4, 5, 12, 18)
- Blocks: 13 (WAL writer references error types)
- Blocked By: 1 (needs go.mod)
References: Pattern References:
- docs/design.md:129-135 — 错误分类表（5 种失败场景 + ErrCommitUnknown）
- docs/design.md:155 — ErrCommitUnknown 的 maybe committed 语义
- docs/design.md:496-499 — sequence exhausted 终态
API/Type References:
- docs/design.md:501-513 — 所有资源上限默认值
- docs/design.md:364-373 — Fragment type 枚举
- docs/design.md:571-577 — OpType 枚举
- docs/design.md:580-584 — ValueKind 枚举
WHY Each Reference Matters:
- 错误类型直接映射设计文档的错误分类表
- 常量值必须与设计文档精确匹配（block size=32KB, header=32 bytes 等）
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Error types are distinguishable via errors.Is
  Tool: Bash
  Preconditions: Task 1 complete
  Steps:
    1. Create errors_test.go with TestErrorTypes
    2. Verify errors.Is(ErrCommitUnknown, ErrCommitUnknown) == true
    3. Verify errors.Is(ErrWriteStopped, ErrCommitUnknown) == false
  Expected Result: All error type identity checks pass
  Evidence: .omo/evidence/task-2-error-types.txt

Scenario: Constants match design doc values
  Tool: Bash
  Preconditions: wal/constants.go exists
  Steps:
    1. Create wal/constants_test.go with TestConstantValues
    2. Assert WalBlockSize == 32*1024
    3. Assert WalFileHeaderSize == 32
    4. Assert MaxWalBatchEntriesSize == 4*1024*1024
    5. Assert DefaultMaxWalSegmentSize == 64*1024*1024
  Expected Result: All assertions pass
  Evidence: .omo/evidence/task-2-constants.txt
```
Commit: YES (group with Task 1)
- Message: feat: initialize project structure and error types
- Files: errors.go, wal/constants.go
3. WAL File Header encode/decode

What to do:
- 创建 wal/header.go：WalFileHeader 结构体（Magic uint32, FormatVersion uint16, HeaderSize uint16, BlockSize uint32, SegmentID uint64, StartSequence uint64, HeaderCRC uint32）
- EncodeWalHeader(h *WalFileHeader) [WalFileHeaderSize]byte — 编码为 little-endian 32 bytes，headerCRC 覆盖 magic 到 startSequence（不包含自身）
- DecodeWalHeader(data []byte) (*WalFileHeader, error) — 校验 magic、formatVersion、headerSize、headerCRC
- CRC 使用 hash/crc32 with IEEE table（或 CRC32C 如果标准库支持）
- 创建 table-driven tests
Must NOT do:
- 不使用外部 CRC 库
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 1b (with Tasks 2, 4, 5, 12, 18)
- Blocks: 6 (batch codec), 10 (segment writer)
- Blocked By: 1
References: Pattern References:
- docs/design.md:316-341 — WAL File Header 格式定义和字段说明
WHY Each Reference Matters:
- Header 格式是 WAL segment 文件的基础，所有后续 segment 操作都依赖正确编码
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Header encode/decode round-trip
  Tool: Bash
  Preconditions: wal/header.go exists
  Steps:
    1. Create TestHeaderRoundtrip in wal/header_test.go
    2. Encode a header with SegmentID=5, StartSequence=1000
    3. Decode the encoded bytes
    4. Assert all fields match original
  Expected Result: Round-trip preserves all fields
  Evidence: .omo/evidence/task-3-header-roundtrip.txt

Scenario: Header CRC detects corruption
  Tool: Bash
  Steps:
    1. Encode a valid header
    2. Flip one byte in the magic field
    3. Attempt DecodeWalHeader
    4. Assert error returned containing "crc" or "checksum"
  Expected Result: Corrupted header rejected
  Evidence: .omo/evidence/task-3-header-crc.txt
```
Commit: YES (group with Tasks 4, 5)
- Message: feat(wal): implement WAL entry/record/header codec
4. Physical Record encode/decode + Block boundary

What to do:
- 创建 wal/record.go：PhysicalRecord 结构体（CRC uint32, Length uint16, Type uint8, Payload []byte）
- EncodePhysicalRecord(recType uint8, payload []byte) []byte — 编码为 7-byte header + payload
- DecodePhysicalRecord(data []byte) (rec *PhysicalRecord, consumed int, err error) — 解析并校验 CRC
- CRC 覆盖 length + type + payload（不含 CRC 自身）
- Block 边界辅助函数：
  - PaddingNeeded(blockOffset uint32) int — 剩余 <= 7 bytes 时返回 padding 大小
  - CanFitRecord(blockOffset uint32, payloadLen uint32) bool
- SplitIntoRecords(encodedBatch []byte) [][]byte — 将编码后的 batch 按 block 边界拆分为 Physical Record payloads
- Table-driven tests 覆盖：单 record、跨 block 拆分、padding
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 1b (with Tasks 2, 3, 5, 12, 18)
- Blocks: 6, 10, 15
- Blocked By: 1
References: Pattern References:
- docs/design.md:345-381 — Physical Record 格式、fragment 类型、合法组合
- docs/design.md:383-391 — Block 边界处理规则
WHY Each Reference Matters:
- Physical Record 是 WAL I/O 的最小单元，block 边界处理是拆分/重组的关键
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Physical Record round-trip
  Tool: Bash
  Steps:
    1. Create TestRecordRoundtrip in wal/record_test.go
    2. Encode RecFull with payload "hello world"
    3. Decode the encoded bytes
    4. Assert Type==RecFull, Payload=="hello world"
  Expected Result: Round-trip correct
  Evidence: .omo/evidence/task-4-record-roundtrip.txt

Scenario: Batch split into fragments across blocks
  Tool: Bash
  Steps:
    1. Create TestSplitIntoRecords in wal/record_test.go
    2. Create a 40KB payload (larger than 32KB block)
    3. Call SplitIntoRecords with blockSize=32KB
    4. Assert result has First + Last (or First + Middle + Last)
    5. Concatenate all payloads → equals original
  Expected Result: Split produces valid fragment sequence, reassembly equals original
  Evidence: .omo/evidence/task-4-split-records.txt

Scenario: Block padding when remaining <= 7 bytes
  Tool: Bash
  Steps:
    1. Create TestBlockPadding in wal/record_test.go
    2. blockOffset = 32*1024 - 5 (only 5 bytes remaining)
    3. Assert PaddingNeeded(blockOffset) == 5
    4. Assert CanFitRecord(blockOffset, 1) == false
  Expected Result: Padding and fit check correct
  Evidence: .omo/evidence/task-4-padding.txt
```
Commit: YES (group with Tasks 3, 5)
5. WAL Entry encode/decode

What to do:
- 创建 wal/entry.go：WalEntry 结构体（OpType uint8, ValueKind uint8, Key []byte, Value []byte）
- EncodeEntry(e *WalEntry) ([]byte, error) — 编码 opType(u8) + valueKind(u8) + keyLen(varint) + valLen(varint) + key + value
- DecodeEntry(data []byte) (entry *WalEntry, consumed int, err error) — 解码返回 entry 和 consumed bytes
- Varint 使用 encoding/binary.PutUvarint / binary.ReadUvarint
- 校验规则（参照 design.md lines 586-594）：
  - keyLen > 0 && keyLen <= MaxWalKeyBytes
  - Put 要求 valueKind ∈ {VKInline, VKValueLogPointer}
  - Put + VKInline: valLen <= MaxWalInlineValueBytes (允许 valLen = 0)
  - Put + VKValueLogPointer: valLen > 0
  - Delete: valueKind == VKNone, valLen == 0
- Table-driven tests 覆盖所有合法/非法组合
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 1b (with Tasks 2, 3, 4, 12, 18)
- Blocks: 6, 9
- Blocked By: 1
References: Pattern References:
- docs/design.md:558-594 — Entry 格式、OpType、ValueKind、合法性规则
WHY Each Reference Matters:
- Entry 是 WAL 的最小逻辑单元，编解码正确性直接影响数据完整性
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Entry encode/decode round-trip for all valid types
  Tool: Bash
  Steps:
    1. Create TestEntryRoundtrip in wal/entry_test.go
    2. Test cases: Put+Inline("key1","val1"), Put+Inline("key2",[]byte{}), Delete("key3")
    3. Encode → Decode → assert all fields match
  Expected Result: All round-trips correct
  Evidence: .omo/evidence/task-5-entry-roundtrip.txt

Scenario: Entry validation rejects invalid entries
  Tool: Bash
  Steps:
    1. Create TestEntryValidation in wal/entry_test.go
    2. Test: OpInvalid → error
    3. Test: Put+VKNone → error
    4. Test: Delete+VKInline → error
    5. Test: keyLen=0 → error
    6. Test: keyLen > MaxWalKeyBytes → error
  Expected Result: All invalid entries rejected with descriptive errors
  Evidence: .omo/evidence/task-5-entry-validation.txt
```
Commit: YES (group with Tasks 3, 4)
6. WAL Batch encode/decode + fragment collector

What to do:
- 创建 wal/batch.go：WalBatch 结构体（Flags uint16, BaseSequence uint64, EntryCount uint32, EntriesSize uint32, Entries []byte）
- EncodeWalBatch(baseSequence uint64, entries []*WalEntry) ([]byte, error) — 编码 Batch Header + 序列化 Entries
- DecodeWalBatch(data []byte) (*WalBatch, error) — 校验 flags、entryCount（0 < N <= max）、entriesSize（与实际 bytes 匹配）
- FragmentCollector 结构体 — 实现 Idle / CollectingFragments 状态机
  - Reset()
  - Append(recType uint8, payload []byte) error — 按 fragment type 追加，校验 buffer 大小上限
  - IsComplete() bool — 是否拼出完整 batch
  - BatchData() []byte — 返回重组后的完整 batch bytes
  - State() FragmentState — 当前状态（Idle / CollectingFragments）
- 状态转移严格遵循 design.md lines 706-731 的状态机
- Buffer 大小限制：Batch Header 长度 + MaxWalBatchEntriesSize
- Varint 长度限制：MaxWalVarintBytes = 5
Must NOT do:
- FragmentCollector 不做 CRC 校验（由 Physical Record 层负责）
Recommended Agent Profile:
- Category: unspecified-high
- Skills: []
Parallelization:
- Can Run In Parallel: YES (with Tasks 7, 9 — all depend only on Wave 1b outputs)
- Parallel Group: Sub-Wave 2a (with Tasks 7, 9)
- Blocks: 10, 16
- Blocked By: 3, 4, 5
References: Pattern References:
- docs/design.md:455-530 — WAL Batch 格式、Batch Header、Entry sequence 推导
- docs/design.md:706-731 — Fragment 状态机定义
WHY Each Reference Matters:
- Batch 是 WAL 的核心持久化单元，FragmentCollector 是 recovery 的核心组件
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Batch encode/decode round-trip with multiple entries
  Tool: Bash
  Steps:
    1. Create TestBatchRoundtrip in wal/batch_test.go
    2. Create 10 entries: 5 Put+Inline, 3 Delete, 2 Put+Inline with empty value
    3. Encode with baseSequence=42
    4. Decode → assert Flags, BaseSequence, EntryCount, EntriesSize match
    5. Parse all entries from Entries bytes → assert each matches original
  Expected Result: Full round-trip preserves all data
  Evidence: .omo/evidence/task-6-batch-roundtrip.txt

Scenario: Fragment collector state machine
  Tool: Bash
  Steps:
    1. Create TestFragmentCollector in wal/batch_test.go
    2. Encode a 100KB batch, split into records (First + Middle* + Last)
    3. Feed fragments to collector one by one
    4. Assert state transitions: Idle → CollectingFragments → ... → Idle
    5. Assert collected batch bytes equal original encoded batch
  Expected Result: Fragment reassembly produces original batch
  Evidence: .omo/evidence/task-6-fragment-collector.txt

Scenario: Fragment collector rejects illegal transitions
  Tool: Bash
  Steps:
    1. Feed RecMiddle to Idle state → expect error
    2. Feed RecLast to Idle state → expect error
    3. Feed RecFull to CollectingFragments state → expect error
    4. Feed RecFirst to CollectingFragments state → expect error
  Expected Result: All illegal transitions rejected
  Evidence: .omo/evidence/task-6-fragment-illegal.txt
```
Commit: YES (group with Task 9)
- Message: feat(wal): implement WAL batch codec and resource validation
7. Arena allocator

What to do:
- 创建 memtable/arena.go：Arena 结构体
- 固定大小 byte slice（默认 64MB，由 config.MemTableSize 配置）
- Allocate(size uint32) (offset uint32, err error) — 分配对齐的内存块，返回 offset；满时返回 error
- GetBytes(offset uint32, size uint32) []byte — 从 offset 获取 bytes slice
- Remaining() uint32 — 剩余可用 bytes
- Capacity() uint32 — 总容量
- Reserve(totalSize uint32) error — 检查剩余容量是否足够（不实际分配）
- 对齐：8-byte 对齐分配
- 线程安全：分配使用 sync.Mutex（写路径已被 WAL writer 串行化）
- 测试：分配、对齐、满时错误、并发安全
Recommended Agent Profile:
- Category: unspecified-high
- Skills: [golang-concurrency]
  - golang-concurrency: Arena 需要线程安全分配
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 2a (with Tasks 6, 9 — all depend only on Task 1 or Wave 1b outputs)
- Blocks: 8, 14
- Blocked By: 1
References: Pattern References:
- docs/design.md:864-879 — Arena 容量预留规则
WHY Each Reference Matters:
- Arena 是 MemTable 的内存基础，容量预留是 WAL 写入路径的前置约束
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Arena allocate and read back
  Tool: Bash
  Steps:
    1. Create TestArenaAllocate in memtable/arena_test.go
    2. Create Arena with 1024 bytes
    3. Allocate 100 bytes → write "test data" at offset 0
    4. GetBytes(offset, 9) → assert equals "test data"
    5. Allocate another 200 bytes → assert offset is 8-byte aligned
  Expected Result: Allocation and readback correct, alignment enforced
  Evidence: .omo/evidence/task-7-arena-alloc.txt

Scenario: Arena full returns error
  Tool: Bash
  Steps:
    1. Create Arena with 64 bytes
    2. Allocate 60 bytes → success
    3. Allocate 10 bytes → error (not enough remaining)
  Expected Result: Full arena rejects allocation
  Evidence: .omo/evidence/task-7-arena-full.txt
```
Commit: YES (with Task 8)
- Message: feat(memtable): implement Arena allocator and SkipList
8. SkipList (mutex write + lock-free read)

What to do:
- 创建 memtable/skiplist.go：SkipList 结构体
- 最大 20 层，p=0.25 的概率晋升
- Put(key []byte, value []byte, sequence uint64, nodeData interface{}) error — Mutex 保护下的有序插入
- Get(key []byte) (found bool, value []byte, sequence uint64) — 无锁读
- NewIterator() *Iterator — 有序遍历
- 原子发布：所有 next 指针使用 atomic.Pointer[skipNode]，store 使用 release 语义
- 节点包含：key、value、sequence、pending 标记（用 atomic.Bool 或存入 nodeData）
- Key 比较使用 bytes.Compare
- Arena 分配：节点在 Arena 中分配（offset-based 引用，通过 Arena.GetBytes 访问）
- 测试：有序插入、查找、遍历、并发读写（-race）
Must NOT do:
- 不使用普通指针（必须 atomic.Pointer）
- 不依赖 Mutex 向读者发布内存（必须 atomic store-release）
Recommended Agent Profile:
- Category: deep
- Skills: [golang-concurrency]
  - golang-concurrency: 原子操作和内存序是核心难点
Parallelization:
- Can Run In Parallel: NO (depends on Task 7)
- Parallel Group: Sub-Wave 2b (solo — after Arena complete)
- Blocks: 14
- Blocked By: 7
References: Pattern References:
- docs/design.md:834-840 — SkipList 设计决策（20 层、64MB Arena、Mutex 写 + 无锁读）
- docs/design.md:841-850 — 发布与内存序约束（atomic.Pointer, release store）
WHY Each Reference Matters:
- 内存序约束是整个读路径正确性的关键，ARM 等弱内存序平台必须严格遵守
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: SkipList ordered insertion and retrieval
  Tool: Bash
  Steps:
    1. Create TestSkipListOrdered in memtable/skiplist_test.go
    2. Insert keys "c", "a", "e", "b", "d" with values "val-C" etc.
    3. Get each key → assert correct value
    4. Iterate → assert order is a, b, c, d, e
  Expected Result: Keys stored and retrieved in correct order
  Evidence: .omo/evidence/task-8-skiplist-ordered.txt

Scenario: SkipList concurrent read/write no data race
  Tool: Bash
  Steps:
    1. Create TestSkipListConcurrent in memtable/skiplist_test.go
    2. Start 4 writer goroutines inserting 100 keys each
    3. Start 4 reader goroutines reading random keys
    4. Run with go test -race
  Expected Result: No data race detected
  Evidence: .omo/evidence/task-8-skiplist-race.txt
```
Commit: YES (with Task 7)
9. WAL Batch resource validation

What to do:
- 创建 wal/validate.go：ValidateBatchLimits(entries []*WalEntry, cfg *config.WalConfig) error
- 在 sequence 分配之前检查所有可失败条件：
  - entryCount > 0 && entryCount <= cfg.MaxBatchEntries
  - 每个 entry 的 keyLen > 0 && keyLen <= cfg.MaxKeyBytes
  - 每个 entry 如果是 Put+Inline 则 valLen <= cfg.MaxInlineValue
  - 所有 entries 编码后总大小 <= cfg.MaxBatchSize
  - worstCase Physical Record overhead + batch encoded size <= maxWalSegmentPayload
- checked arithmetic：计算中不溢出
- 返回的具体错误类型允许区分不同违规
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Sub-Wave 2a (with Tasks 6, 7 — all depend on Task 1/Wave 1b)
- Blocks: 13
- Blocked By: 1, 5
References: Pattern References:
- docs/design.md:501-513 — WAL Batch 资源上限
- docs/design.md:514 — "写入侧必须在 sequence 分配、MemTable 预留和 WAL append 之前完成这些校验"
WHY Each Reference Matters:
- 这些校验必须在 sequence 分配之前完成，否则失败会触发 write-stopped
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Valid batch passes validation
  Tool: Bash
  Steps:
    1. Create TestValidateBatchLimits in wal/validate_test.go
    2. Create 100 Put+Inline entries with 10-byte keys and 10-byte values
    3. Call ValidateBatchLimits → expect nil error
  Expected Result: Valid batch accepted
  Evidence: .omo/evidence/task-9-validate-pass.txt

Scenario: Batch exceeding entry count limit rejected
  Tool: Bash
  Steps:
    1. Create 10,001 entries (exceeds default MaxBatchEntries=10000)
    2. Call ValidateBatchLimits → expect error containing "entry count"
  Expected Result: Over-limit batch rejected before sequence allocation
  Evidence: .omo/evidence/task-9-validate-entrycount.txt

Scenario: Key exceeding size limit rejected
  Tool: Bash
  Steps:
    1. Create entry with keyLen = 5KB (exceeds MaxWalKeyBytes=4KB)
    2. Call ValidateBatchLimits → expect error containing "key"
  Expected Result: Oversized key rejected
  Evidence: .omo/evidence/task-9-validate-keysize.txt
```
Commit: YES (group with Task 6)
10. Segment writer (file write + block writer)

What to do:
- 创建 wal/block_writer.go：BlockWriter — 管理 32KB block 的 Physical Record 写入
  - 内部维护当前 block 的 byte buffer
  - 写入 Physical Record 时检查 block 边界
  - 剩余 <= 7 bytes 时自动 padding（全 0）
  - 跨 block 的 batch fragment 自动拆分
  - WriteRecord(recType uint8, payload []byte) error
  - Flush(io.Writer) error — 将当前 block 写入 fd
  - Reset() — 重置为空 block
- 创建 wal/segment_writer.go：SegmentWriter
  - 封装 WAL segment 文件的追加写入
  - NewSegmentWriter(dir string, segmentID uint64, startSequence uint64, cfg *config.WalConfig) (*SegmentWriter, error) — 创建 segment-N.wal.tmp → 写 header → fsync → rename → fsync dir → durable-ready
  - AppendBatch(encodedBatch []byte) error — 按 block 边界写入 batch（使用 BlockWriter）
  - Sync() error — fsync 当前文件
  - Close() error
  - RemainingPayload() uint64 — 当前 segment 剩余 payload
  - CurrentOffset() uint64
  - SegmentID() uint64
  - EndState() SegmentEndState — 返回 (segmentID, endOffset, endSequence)
Recommended Agent Profile:
- Category: unspecified-high
- Skills: []
Parallelization:
- Can Run In Parallel: NO (depends on 3, 4, 6)
- Parallel Group: Sub-Wave 3a (solo — after batch codec complete)
- Blocks: 11, 15
- Blocked By: 3, 4, 6
References: Pattern References:
- docs/design.md:247-278 — WAL 元数据持久化协议（durable-ready）
- docs/design.md:383-391 — Block 边界处理规则
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Segment creation follows durable-ready protocol
  Tool: Bash
  Steps:
    1. Create TestSegmentCreation in wal/segment_writer_test.go
    2. Create a temp directory, create SegmentWriter with segmentID=0
    3. Verify segment-0.wal exists in directory
    4. Read file → verify header is correct (magic, segmentID, startSequence)
    5. Verify no .tmp file remains
  Expected Result: Segment file created with correct header, no temp file
  Evidence: .omo/evidence/task-10-segment-create.txt

Scenario: Batch written to segment correctly
  Tool: Bash
  Steps:
    1. Create SegmentWriter in temp dir
    2. Encode a batch with 5 entries
    3. Call AppendBatch(encodedBatch)
    4. Sync + Close
    5. Read file back → verify batch can be recovered
  Expected Result: Written batch recoverable from segment file
  Evidence: .omo/evidence/task-10-segment-write.txt
```
Commit: YES (with Task 11)
- Message: feat(wal): implement segment writer with rotation

11. Segment rotation + durable-ready protocol

What to do:

创建 wal/segment_manager.go：管理 segment 的轮转和生命周期
写入 batch 前检查 RemainingPayload() 是否足够容纳整个 batch
不足时：
1. 当前 segment 完成（在 batch 边界）
2. 创建新 segment：segment-N+1.wal.tmp → header → fsync → rename → fsync dir
3. 新 segment 的 startSequence = 下一个 expected sequence
4. 切换 active segment
不允许 batch 跨 segment
CURRENT 文件 best-effort 更新（temp + rename）
CURRENT 更新失败不影响已 durable-ready 的 segment

Recommended Agent Profile:

Category: unspecified-high
Skills: []

Parallelization:

Can Run In Parallel: NO (depends on Task 10)
Parallel Group: Sub-Wave 3b (solo — after segment writer)
Blocks: 13
Blocked By: 10

References: Pattern References:

docs/design.md:392-453 — Segment Rotation 约束
docs/design.md:247-278 — WAL 元数据持久化协议

Acceptance Criteria:

QA Scenarios (MANDATORY):

Scenario: Segment rotation triggers when payload exceeds remaining space
  Tool: Bash
  Steps:
    1. Create a test WalConfig with: MaxBatchSize=256, MaxBatchEntries=10,
       BlockSize=256, MaxSegmentSize=4096 (satisfies invariant: 4096 ≥ maxEncodedWalBatchSize
       since MaxBatchSize=256 → maxEncoded ≈ 256 + 4*10 + overhead < 512)
    2. Create SegmentManager with this config
    3. Write small batches until segment-0 is nearly full (~3800 bytes written)
    4. Write one more batch that would exceed remaining → triggers rotation
    5. Verify segment-1.wal exists with correct header
    6. Verify batch was written to segment-1, not segment-0
  Expected Result: Rotation creates new segment, batch not split across segments
  Failure Indicators: segment-1.wal does not exist, or batch data spans both segment files
  Evidence: .omo/evidence/task-11-rotation.txt

Scenario: CURRENT file updated after rotation
  Tool: Bash
  Steps:
    1. Trigger rotation using same test config as above
    2. Read CURRENT file → verify it points to latest segment filename
  Expected Result: CURRENT reflects latest active segment filename
  Failure Indicators: CURRENT still points to segment-0, or file content is stale
  Evidence: .omo/evidence/task-11-current.txt

Commit: YES (with Task 10)

12. Sequence manager

What to do:
- 创建 wal/sequence.go：SequenceManager
- nextSequence atomic.Uint64 — 下一个待分配 sequence
- publishedSequence atomic.Uint64 — 已发布的连续 high-water mark
- durableSequence atomic.Uint64 — 已满足 durable 条件的 high-water mark
- AllocateBatch(count uint32) (baseSequence uint64, err error) — checked arithmetic：baseSequence + count - 1 不溢出 uint64。溢出 → 返回 ErrSequenceExhausted
- Publish(seq uint64) — Store publishedSequence（在所有 entry 节点发布后调用）
- MarkDurable(endState SegmentEndState) — 推进 durableSequence
- Published() uint64 — Load publishedSequence
- Durable() uint64 — Load durableSequence
- NextSequence() uint64 — Load nextSequence
- 恢复后初始化：NewSequenceManager(recoveredSequence uint64)
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1b (with Tasks 2, 3, 4, 5, 18 — all depend only on Task 1)
- Blocks: 13
- Blocked By: 1
References: Pattern References:
- docs/design.md:486-499 — Sequence 运算和溢出检查
- docs/design.md:205-223 — durableSequence 推进逻辑
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Sequence allocation returns consecutive bases
  Tool: Bash
  Steps:
    1. Create TestSequenceAllocation in wal/sequence_test.go
    2. NewSequenceManager(0)
    3. AllocateBatch(5) → base=0
    4. AllocateBatch(3) → base=5
    5. AllocateBatch(1) → base=8
  Expected Result: Consecutive, no gaps
  Evidence: .omo/evidence/task-12-seq-alloc.txt

Scenario: Sequence overflow returns ErrSequenceExhausted
  Tool: Bash
  Steps:
    1. NewSequenceManager(math.MaxUint64 - 2)
    2. AllocateBatch(5) → err (would overflow)
  Expected Result: ErrSequenceExhausted returned
  Evidence: .omo/evidence/task-12-seq-overflow.txt
```
Commit: YES (with Task 13)
13. Commit queue + WAL writer main loop

What to do:
- 创建 wal/commit_queue.go：CommitQueue — 写请求队列
  - 每个 request 包含 entries、result channel
  - Submit(entries []*WalEntry) chan WriteResult — 非阻塞提交
  - Collect(timeout time.Duration) []*CommitRequest — 收集当前队列中的请求
- 创建 wal/writer.go：WalWriter — 核心写入循环
  - 单 goroutine 运行 main loop
  - Group commit：收集 queue 中的请求 → 组装 batch → 触发条件（500µs 或 32KB）
  - 严格遵循 design.md steps ①-⑪：
    1. 从 commit queue 收集写入
    2. 组装 WAL Batch
    3. ValidateBatchLimits → 失败则普通错误（不分配 sequence）
    4. 计算 MemTable Arena 预留量 → 不足则 freeze/switch 或等待
    5. AllocateBatch → 分配 baseSequence（此步之后任何失败 → write-stopped）
    6. 私有缓冲编码 WAL Batch
    7. 检查 segment rotation
    8. segmentWriter.AppendBatch(encoded) → 失败 → ErrCommitUnknown + write-stopped
    9. 写入 MemTable pending entries
    10. segmentWriter.Sync() (Always 模式) → 失败 → ErrCommitUnknown + write-stopped
    11. 原子发布 MemTable entries (atomic Pointer store-release)
    12. Publish(baseSequence + entryCount - 1)
    13. 唤醒所有等待的调用方
  - writeStopped atomic.Bool — write-stopped 状态
  - Put(key, value []byte) error — 提交到 commit queue 并等待结果
  - Delete(key []byte) error — 同上
  - Close() error — 等待 pending batch 完成
Must NOT do:
- 不使用 bufio.Writer（直接 os.File.Write）
- 不在 sequence 分配后做可失败的校验
Recommended Agent Profile:
- Category: deep
- Skills: [golang-concurrency]
  - golang-concurrency: commit queue、group commit、goroutine 生命周期管理
Parallelization:
- Can Run In Parallel: NO (depends on MemTable API from Task 14)
- Parallel Group: Sub-Wave 3d (solo — after MemTable complete)
- Blocks: 19
- Blocked By: 2, 9, 10, 11, 12, 14
References: Pattern References:
- docs/design.md:103-115 — 写入流程 11 步
- docs/design.md:119 — 资源前置校验约束
- docs/design.md:127-153 — ErrCommitUnknown 和 WAL 副作用边界
- docs/design.md:139-153 — per-batch private encode buffer
WHY Each Reference Matters:
- 这是整个 WAL 子系统最复杂的组件，必须严格遵循设计文档的步骤顺序和错误分类
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Single Put writes and syncs correctly
  Tool: Bash
  Steps:
    1. Create TestWalWriterSinglePut in wal/writer_test.go
    2. Start WalWriter in temp dir
    3. Put("key1", "value1")
    4. Close writer
    5. Read WAL segment → verify batch with 1 entry exists
  Expected Result: Entry persisted in WAL
  Evidence: .omo/evidence/task-13-writer-single.txt

Scenario: Multiple concurrent writes group-committed
  Tool: Bash
  Steps:
    1. Create TestWalWriterGroupCommit in wal/writer_test.go
    2. Start WalWriter
    3. Launch 10 goroutines each doing Put
    4. Wait for all to complete
    5. Close writer
    6. Read WAL → verify entries batched together
  Expected Result: Entries grouped in batches, all sequences assigned
  Evidence: .omo/evidence/task-13-writer-group.txt

Scenario: Write-stopped after simulated WAL write failure
  Tool: Bash
  Steps:
    1. Create a WalWriter with a mock segment writer that fails on 3rd AppendBatch
    2. Write 5 entries sequentially
    3. 3rd write should return ErrCommitUnknown
    4. 4th write should be rejected (write-stopped)
  Expected Result: ErrCommitUnknown returned, subsequent writes rejected
  Evidence: .omo/evidence/task-13-writer-stopped.txt
```
Commit: YES (with Task 12)
- Message: feat(wal): implement WAL writer with group commit
14. MemTable (wrap skiplist + arena + publish/abort)

What to do:
- 创建 memtable/memtable.go：MemTable 结构体
- 封装 Arena + SkipList
- Reserve(entries []ReserveEntry) (totalSize uint32, err error) — 容量预留（最坏情况：最大层高 next 指针数组 + 对齐 padding）。单个 batch 超过空 MemTable → 返回错误
- PutPending(key []byte, value []byte, sequence uint64) error — 写入 pending entry（Arena 已预留）
- DeletePending(key []byte, sequence uint64) error — 写入 pending tombstone
- Publish(upToSequence uint64) — 批量发布 sequence <= upToSequence 的 pending entries（atomic store-release）
- Abort(fromSequence uint64) — 标记 aborted
- Get(key []byte, publishedSequence uint64) *GetResult — 无锁读：先 Load publishedSequence，遍历 skiplist，只返回 sequence <= publishedSequence 且非 aborted 的 entry
- NewIterator(publishedSequence uint64) *MemTableIterator — 有序遍历（同样过滤 pending/aborted）
- ApproximateSize() uint64
- UsableCapacity() uint32 — 可用容量（扣除固定元数据）
Must NOT do:
- 不允许 PutPending 在 Reserve 成功后因 Arena 满失败
Recommended Agent Profile:
- Category: unspecified-high
- Skills: [golang-concurrency]
  - golang-concurrency: 原子发布、无锁读、内存序
Parallelization:
- Can Run In Parallel: NO (solo in sub-wave, Task 13 depends on this)
- Parallel Group: Sub-Wave 3c (solo)
- Blocks: 13, 19
- Blocked By: 7, 8
References: Pattern References:
- docs/design.md:864-879 — Arena 容量预留协议
- docs/design.md:881-889 — Flush 过滤规则（pending/aborted 不进入 SSTable）
- docs/design.md:841-850 — 发布与内存序
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Pending entry invisible to readers
  Tool: Bash
  Steps:
    1. Create TestMemTablePending in memtable/memtable_test.go
    2. Create MemTable, write PutPending("key1","val1", seq=10)
    3. Get("key1", publishedSequence=5) → Found=false
    4. Publish(upToSequence=10)
    5. Get("key1", publishedSequence=10) → Found=true, Value="val1"
  Expected Result: Entry invisible before publish, visible after
  Evidence: .omo/evidence/task-14-memtable-pending.txt

Scenario: Aborted entry invisible to readers
  Tool: Bash
  Steps:
    1. PutPending("key2","val2", seq=20)
    2. Abort(20)
    3. Publish(upToSequence=30) — publish others but not aborted
    4. Get("key2", publishedSequence=30) → Found=false
  Expected Result: Aborted entry never visible
  Evidence: .omo/evidence/task-14-memtable-aborted.txt

Scenario: Reserve accuracy
  Tool: Bash
  Steps:
    1. Create MemTable with 1024 bytes usable
    2. Reserve 10 entries with 10-byte keys + 10-byte values
    3. Assert reserved size includes node overhead + alignment
    4. Attempt to reserve more than remaining → error
  Expected Result: Reserve accounts for full overhead
  Evidence: .omo/evidence/task-14-memtable-reserve.txt
```
Commit: YES
- Message: feat(memtable): integrate arena+skiplist with publish/abort
15. WAL recovery: scanner + record parser

What to do:
- 创建 wal/scanner.go：SegmentScanner — 从 WAL 目录扫描 segment 文件
  - ScanSegments(dir string, recoverySegmentID uint64) ([]*SegmentInfo, error) — 按 segmentID 排序，过滤 < recoverySegmentID
  - SegmentInfo 结构体：FilePath, SegmentID, StartSequence（从 header 解析）
  - 连续性校验：segmentID 连续、startSequence 衔接
- 创建 wal/record_parser.go：RecordParser — Block 级别 Physical Record 解析
  - ParseBlock(data []byte) ([]*PhysicalRecord, error) — 解析 block 内所有 records
  - Padding 校验：剩余 < 7 bytes 时全 0
  - Short block 处理：最后 segment 的最后 block 可以短于 32KB
  - 错误分类辅助：IsTailCorruption(err error) bool
Recommended Agent Profile:
- Category: unspecified-high
- Skills: []
Parallelization:
- Can Run In Parallel: NO (depends on 4, 10)
- Parallel Group: Sub-Wave 4a (solo)
- Blocks: 16
- Blocked By: 4, 10
References: Pattern References:
- docs/design.md:597-665 — Recovery 扫描流程、segment 连续性校验
- docs/design.md:669-691 — Physical Record 解析规则
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Segment scanner finds and orders segments correctly
  Tool: Bash
  Steps:
    1. Create temp dir with segment-0.wal, segment-2.wal, segment-1.wal
    2. ScanSegments(dir, 0) → returns 3 segments ordered by ID
    3. ScanSegments(dir, 1) → returns segments 1,2 only
  Expected Result: Correct ordering and filtering
  Evidence: .omo/evidence/task-15-scanner.txt

Scenario: Record parser handles valid blocks
  Tool: Bash
  Steps:
    1. Create a 32KB block with 3 Physical Records
    2. ParseBlock → verify 3 records parsed with correct payloads
  Expected Result: All records extracted
  Evidence: .omo/evidence/task-15-parser.txt
```
Commit: YES (with Tasks 16, 17)
16. WAL recovery: fragment collector + batch replay

What to do:
- 创建 wal/recovery.go：恢复核心逻辑
- 使用 Task 6 的 FragmentCollector 进行 fragment 重组
- BatchReplayer 接口：ReplayPut(key, value []byte, sequence uint64) / ReplayDelete(key []byte, sequence uint64)
- ReplayBatch(batch *WalBatch, expectedSequence uint64, replayer BatchReplayer) (nextSequence uint64, err error)
  - 校验 Batch Header：flags、entryCount (0 < N <= max)、entriesSize、baseSequence == expectedSequence
  - 逐条解析 Entry，校验 opType、valueKind、keyLen、valLen
  - 调用 replayer 回调
  - 返回 nextSequence = expectedSequence + entryCount
- Sequence 溢出检查：baseSequence + entryCount - 1 不溢出
Recommended Agent Profile:
- Category: deep
- Skills: []
Parallelization:
- Can Run In Parallel: NO (depends on 6, 15)
- Parallel Group: Sub-Wave 4b (solo — after scanner)
- Blocks: 17
- Blocked By: 6, 15
References: Pattern References:
- docs/design.md:733-781 — WAL Batch 校验与重放规则
- docs/design.md:486-499 — Sequence 溢出检查
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Valid batch replayed correctly
  Tool: Bash
  Steps:
    1. Create TestBatchReplay in wal/recovery_test.go
    2. Encode a batch: baseSeq=0, 3 entries (Put a, Put b, Delete c)
    3. ReplayBatch with expectedSequence=0 and mock replayer
    4. Assert replayer received 3 calls in order with correct args
  Expected Result: All entries replayed with correct sequences
  Evidence: .omo/evidence/task-16-batch-replay.txt

Scenario: Batch with wrong baseSequence rejected
  Tool: Bash
  Steps:
    1. Encode batch with baseSeq=100
    2. ReplayBatch with expectedSequence=50 → error
  Expected Result: Sequence mismatch error
  Evidence: .omo/evidence/task-16-seq-mismatch.txt
```
Commit: YES (with Tasks 15, 17)
17. WAL recovery: tail truncation + main flow

What to do:
- 创建 wal/truncation.go：尾部截断持久化
  - TruncateTail(segmentPath string, truncateOffset int64) error — ftruncate + fsync
  - DeleteEmptySegments(dir string, fromSegmentID uint64, expectedSequence uint64) error — 删除空 segment + fsync dir
- 复用 Task 16 定义的 BatchReplayer 接口和 ReplayBatch 函数（同在 wal/recovery.go）
- Recover(dir string, manifest *manifest.Manifest, cfg *config.WalConfig, replayer BatchReplayer) (*RecoveryResult, error)
  - BatchReplayer 由 Task 16 定义（interface{ ReplayPut/ReplayDelete }），Task 19 传入 MemTable 适配实现
  - 主流程严格遵循 design.md lines 622-636：
    1. 读取 MANIFEST → recoverySegmentID
    2. ScanSegments → 过滤
    3. 校验连续性
    4. 逐 segment：校验 header → 逐 block 解析 → fragment 重组 → 调用 Task 16 的 ReplayBatch(batch, expectedSeq, replayer) 重放
    5. 处理尾部异常（最后 segment → truncation，中间 segment → 报错）
    6. 持久化截断
  - 返回 RecoveryResult：RecoveredSequence, NextSequence, PublishedSequence
- 错误分类：
  - 尾部（最后 segment）：header 半写、length 越界、CRC 失败、非 0 padding → 截断
  - 中间（非最后 segment）：同样情况 → 报错
  - CRC-valid batch 内部错误（entryCount 越界等）→ 报错（不可截断）
Recommended Agent Profile:
- Category: unspecified-high
- Skills: []
Parallelization:
- Can Run In Parallel: NO (depends on 15, 16)
- Parallel Group: Sub-Wave 4c (solo — after batch replay)
- Blocks: 19
- Blocked By: 15, 16
References: Pattern References:
- docs/design.md:622-636 — Recovery 扫描流程
- docs/design.md:640-665 — Segment 连续性校验
- docs/design.md:783-800 — 尾部截断持久化
- docs/design.md:694-703 — 错误分类（尾部 vs 中间）
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Normal recovery from clean WAL
  Tool: Bash
  Steps:
    1. Create temp dir, write 2 segments with valid batches
    2. Recover(dir, manifest) → success
    3. Assert RecoveredSequence == last batch end sequence
    4. Assert all entries replayed via mock replayer
  Expected Result: Full recovery, no data loss
  Evidence: .omo/evidence/task-17-recovery-normal.txt

Scenario: Recovery with tail corruption truncates
  Tool: Bash
  Steps:
    1. Write valid segment + partial write at end (corrupt last record)
    2. Recover → success (truncated to last complete batch)
    3. Verify segment file was truncated
  Expected Result: Tail corruption handled, complete batches preserved
  Evidence: .omo/evidence/task-17-recovery-tail.txt

Scenario: Recovery with middle corruption reports error
  Tool: Bash
  Steps:
    1. Write 3 segments, corrupt the middle one (bad CRC)
    2. Recover → error containing "corrupt" or "damaged"
  Expected Result: Middle corruption detected, recovery fails
  Evidence: .omo/evidence/task-17-recovery-middle.txt
```
Commit: YES (with Tasks 15, 16)
18. MANIFEST stub + CURRENT file

What to do:
- 创建 manifest/manifest.go：Manifest 最小 stub
  - 仅存储 recoverySegmentID uint64
  - Load(dir string) (*Manifest, error) — 读取 MANIFEST 文件
  - Save(dir string, recoverySegmentID uint64) error — temp + rename 原子写入
  - 首次创建：如果 MANIFEST 不存在，创建初始文件（recoverySegmentID=0）
- 创建 manifest/current.go：CURRENT 文件
  - WriteCurrent(dir string, segmentID uint64) error — temp + rename，best-effort
  - ReadCurrent(dir string) (segmentID uint64, ok bool) — 读取，不存在或损坏返回 ok=false
- 格式：简单的文本文件或二进制（Phase 1 选择简单文本）
Recommended Agent Profile:
- Category: quick
- Skills: []
Parallelization:
- Can Run In Parallel: YES
- Parallel Group: Wave 1b (with Tasks 2, 3, 4, 5, 12 — all depend only on Task 1)
- Blocks: 19
- Blocked By: 1
References: Pattern References:
- docs/design.md:603-619 — CURRENT / MANIFEST 权威性
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: MANIFEST save and load round-trip
  Tool: Bash
  Steps:
    1. Save(dir, 42)
    2. Load(dir) → assert recoverySegmentID == 42
  Expected Result: Round-trip correct
  Evidence: .omo/evidence/task-18-manifest.txt

Scenario: CURRENT best-effort handles missing file
  Tool: Bash
  Steps:
    1. ReadCurrent on empty dir → ok=false, no error
  Expected Result: Missing CURRENT handled gracefully
  Evidence: .omo/evidence/task-18-current-missing.txt
```
Commit: YES
- Message: feat: implement MANIFEST stub and CURRENT file
19. DB integration: Open/Close/Put/Delete/Get

What to do:
- 创建 db.go：DB 结构体和核心 API
- DB 字段：walWriter, memTable, immutableTables, seqMgr, manifest, config, dir, writeStopped
- Open(dir string, opts ...Option) (*DB, error):
  1. Load or create MANIFEST
  2. Recover WAL → 重建 MemTable（恢复的 entries 标记为 published）
  3. 设置 nextSequence、publishedSequence
  4. 启动 WAL writer goroutine
- Close() error:
  1. 停止 WAL writer（等待 pending batch 完成）
  2. Sync WAL
  3. Save MANIFEST（recoverySegmentID stays 0 in Phase 1 — no SSTable flush means all segments must be replayed）
  4. Close segment writer
- Put(key, value []byte) error:
  1. ValidateBatchLimits (key/value size)
  2. Submit to WAL writer → wait for result
  3. 返回 nil 或 ErrCommitUnknown 或 write-stopped error
- Delete(key []byte) error — 同上，opType=Delete
- Get(key []byte) (GetResult, error):
  1. Load publishedSequence
  2. 查 active MemTable → Immutable #1 → Immutable #2
  3. 只返回 sequence <= publishedSequence 且非 aborted
  4. Tombstone → Found=false
  5. 未找到 → Found=false
- GetResult 结构体：Value []byte, Found bool（参照 design.md §3.4）
- GetDurableSequence() uint64
- IsWriteStopped() bool
- MemTable freeze + switch：
  - Put 前检查容量，不足时 freeze
  - Immutable 队列满时阻塞
  - Phase 1：Immutable 不 flush 到 SSTable，仅持有在内存
- Options pattern：type Option func(*config.WalConfig)
Recommended Agent Profile:
- Category: deep
- Skills: [golang-design-patterns]
  - golang-design-patterns: Options pattern for configuration
Parallelization:
- Can Run In Parallel: NO (integrates all deps)
- Parallel Group: Sub-Wave 4d (solo — after recovery + WAL writer + MemTable + MANIFEST)
- Blocks: 20, 21
- Blocked By: 13, 14, 17, 18
References: Pattern References:
- docs/design.md:901-921 — Get API 和 GetResult 定义
- docs/design.md:928-941 — 读路径优先级
- docs/design.md:852-862 — MemTable 生命周期
WHY Each Reference Matters:
- 这是所有模块的集成点，必须正确协调 WAL writer、MemTable、recovery 和 MANIFEST
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Open creates fresh DB, Put+Get round-trip
  Tool: Bash
  Steps:
    1. Create temp dir
    2. Open(dir) → success
    3. Put("hello", "world") → nil
    4. Get("hello") → {Found:true, Value:"world"}
    5. Close() → nil
  Expected Result: Full round-trip works
  Evidence: .omo/evidence/task-19-open-put-get.txt

Scenario: Close and reopen preserves data
  Tool: Bash
  Steps:
    1. Open(dir), Put("k1","v1"), Put("k2","v2"), Close()
    2. Open(dir) → recovery runs
    3. Get("k1") → {Found:true, Value:"v1"}
    4. Get("k2") → {Found:true, Value:"v2"}
    5. Close()
  Expected Result: Data survives close/reopen
  Evidence: .omo/evidence/task-19-reopen.txt

Scenario: Delete makes key not found
  Tool: Bash
  Steps:
    1. Open, Put("k","v"), Delete("k")
    2. Get("k") → {Found:false}
  Expected Result: Delete works correctly
  Evidence: .omo/evidence/task-19-delete.txt

Scenario: Empty value distinguished from not found
  Tool: Bash
  Steps:
    1. Put("empty", []byte{})
    2. Get("empty") → {Found:true, Value:[]byte{}}
    3. Get("nonexist") → {Found:false}
  Expected Result: Found distinguishes empty value from missing key
  Evidence: .omo/evidence/task-19-empty-value.txt

Scenario: Concurrent Put and Get work correctly
  Tool: Bash
  Steps:
    1. Open
    2. Launch 10 writers (Put "key-{i}", "val-{i}")
    3. Launch 10 readers (Get "key-{i}")
    4. go test -race
  Expected Result: No race conditions
  Evidence: .omo/evidence/task-19-concurrent.txt
```
Commit: YES
- Message: feat: integrate DB with Open/Close/Put/Delete/Get
20. End-to-end + crash recovery tests

What to do:
- 创建 db_test.go：集成测试
- TestWALRoundtrip — 写 N 条 → Close → Open → 验证全部
- TestRecoveryPartialWrite — 模拟崩溃：写入数据 → 在 WAL 写入中途截断文件 → Recover → 验证完整 batch 保留
- TestRecoveryMultiSegment — 多 segment 恢复
- TestRecoveryTailCorruption — 尾部损坏修复
- TestWriteStoppedAfterFailure — 模拟写入失败 → ErrCommitUnknown → write-stopped
- TestGetSemantics — Found vs empty value vs not found
- TestSequenceExhaustion — 虽然不可达，但验证 sequence 溢出返回正确错误
- TestConcurrentReadWrite — 多 goroutine 并发读写
- TestLargeBatchSegmentRotation — 大量写入触发多次 segment 轮转
- TestMemTableFreezeNoSSTable — 验证 MemTable 满时冻结，不刷盘
Recommended Agent Profile:
- Category: deep
- Skills: [golang-testing]
  - golang-testing: Go testing patterns
Parallelization:
- Can Run In Parallel: YES (with Task 21)
- Parallel Group: Wave 5 (with Task 21)
- Blocks: F1-F4
- Blocked By: 19
References: Pattern References:
- docs/design.md:694-703 — 错误分类
- docs/design.md:783-800 — 尾部截断
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Full WAL roundtrip with crash simulation
  Tool: Bash
  Steps:
    1. Open DB, write 1000 keys
    2. Close DB (simulate clean shutdown)
    3. Open DB → verify all 1000 keys readable
  Expected Result: All 1000 keys recovered
  Evidence: .omo/evidence/task-20-roundtrip.txt

Scenario: Crash recovery with tail truncation
  Tool: Bash
  Steps:
    1. Open DB, write 500 keys, Close
    2. Open DB, write 200 more keys
    3. Manually truncate last WAL segment by 100 bytes (simulate crash)
    4. Open DB → verify 500 original keys present, some of 200 may be lost
  Expected Result: Complete batches preserved, tail truncated
  Evidence: .omo/evidence/task-20-crash.txt

Scenario: go test -race passes all integration tests
  Tool: Bash
  Steps:
    1. go test ./... -race -count=1
  Expected Result: Zero race conditions
  Evidence: .omo/evidence/task-20-race.txt
```
Commit: YES
- Message: test: add end-to-end and crash recovery tests
21. Benchmark suite

What to do:
- 创建 bench_test.go：性能基准
- BenchmarkSinglePut — 单线程 Put 吞吐
- BenchmarkConcurrentPut — 多线程 Put 吞吐（2/4/8 goroutines）
- BenchmarkGet — Get 延迟
- BenchmarkConcurrentGet — 并发 Get
- BenchmarkMixedReadWrite — 混合读写
- BenchmarkWALRecovery — 不同数据量的恢复时间
- 使用 b.ReportAllocs() 跟踪内存分配
Recommended Agent Profile:
- Category: unspecified-high
- Skills: [golang-benchmark]
  - golang-benchmark: Go benchmarking best practices
Parallelization:
- Can Run In Parallel: YES (with Task 20)
- Parallel Group: Wave 5 (with Task 20)
- Blocks: F1-F4
- Blocked By: 19
References:
- 无特定设计文档引用（benchmark 是验证手段，不是设计要求）
Acceptance Criteria:

QA Scenarios (MANDATORY):
```
Scenario: Benchmark runs without errors
  Tool: Bash
  Steps:
    1. go test -bench=. -benchmem ./...
  Expected Result: All benchmarks complete, results printed
  Evidence: .omo/evidence/task-21-bench.txt
```
Commit: YES
- Message: bench: add WAL performance benchmarks

Final Verification Wave

F1. Plan Compliance Audit — oracle Read the plan end-to-end. For each "Must Have": verify implementation exists (grep for atomic.Uint64, atomic.Pointer, no bufio.Writer on WAL path, GetResult with Found field, etc). For each "Must NOT Have": search codebase for forbidden patterns (SSTable, Compaction, ValueLogPointer write, Periodic sync, bufio.Writer in wal/). Check evidence files exist in .omo/evidence/. Compare deliverables against plan. Acceptance Criteria:
- All "Must Have" items verified present in codebase
- All "Must NOT Have" items verified absent
- Evidence files exist for tasks 1-21
- Output: Must Have [N/N] | Must NOT Have [N/N] | Tasks [21/21] | VERDICT: APPROVE/REJECT
F2. Code Quality Review — unspecified-high Run go vet ./... + golangci-lint run + go test ./... -race. Review all .go files for: any type casts, empty catches, fmt.Println in prod, commented-out code, unused imports. Check AI slop: excessive comments, over-abstraction, generic names. Acceptance Criteria:
- go vet ./... passes with zero warnings
- golangci-lint run passes with zero warnings
- go test ./... -race passes with zero races
- No fmt.Println or log.Println in non-test files
- Output: Build [PASS/FAIL] | Lint [PASS/FAIL] | Tests [N pass/0 fail] | Files [N clean/0 issues] | VERDICT
F3. Real Manual QA — unspecified-high Build the project. Run ALL QA scenarios from tasks 1-21. Test cross-module integration: write data with Put → Close → Open → Get verifies data. Test edge cases: empty value, large batch (9999 entries), concurrent writes from 10 goroutines, segment rotation after filling segment. Save output to .omo/evidence/final-qa/. Acceptance Criteria:
- Every QA scenario from tasks 1-21 executed and passing
- Cross-module integration: Put → Close → Open → Get round-trip
- Edge case: empty value returns Found=true
- Edge case: key not found returns Found=false
- Edge case: concurrent 10-goroutine write/read with -race clean
- Output: Scenarios [N/N pass] | Integration [3/3] | Edge Cases [4 tested] | VERDICT
F4. Scope Fidelity Check — deep For each task: read "What to do", read actual diff (git diff). Verify 1:1 — everything in spec was built (no missing), nothing beyond spec was built (no creep). Check "Must NOT do" compliance per task. Detect cross-task contamination: Task N touching Task M's files without justification. Flag unaccounted files. Acceptance Criteria:
- Each task's "What to do" matches its actual implementation
- No SSTable/Compaction/ValueLog/BloomFilter code found
- No bufio.Writer import in wal/ package
- No Periodic/Never sync code (only Always)
- No cross-task file contamination
- Output: Tasks [21/21 compliant] | Contamination [CLEAN] | Unaccounted [CLEAN] | VERDICT

Commit Strategy

Tasks	Commit Message	Pre-commit Check
1-2	`feat: initialize project structure and error types`	`go build ./...`
3-5	`feat(wal): implement WAL entry/record/header codec`	`go test ./wal/... -run TestCodec`
6, 9	`feat(wal): implement WAL batch codec and resource validation`	`go test ./wal/... -run TestBatch`
7-8	`feat(memtable): implement Arena allocator and SkipList`	`go test ./memtable/... -race`
10-11	`feat(wal): implement segment writer with rotation`	`go test ./wal/... -run TestSegment`
12-13	`feat(wal): implement WAL writer with group commit`	`go test ./wal/... -run TestWriter`
14	`feat(memtable): integrate arena+skiplist with publish/abort`	`go test ./memtable/... -race`
15-17	`feat(wal): implement WAL recovery`	`go test ./wal/... -run TestRecovery`
18	`feat: implement MANIFEST stub and CURRENT file`	`go test ./manifest/...`
19	`feat: integrate DB with Open/Close/Put/Delete/Get`	`go test ./... -race`
20	`test: add end-to-end and crash recovery tests`	`go test ./... -race -count=1`
21	`bench: add WAL performance benchmarks`	`go test -bench=. ./...`

Success Criteria

Verification Commands

go build ./...                          # Expected: success, no errors
go vet ./...                            # Expected: no issues
go test ./... -race -count=1            # Expected: all pass, no races
go test ./... -run TestWALRoundtrip     # Expected: write→close→open→read roundtrip
go test ./... -run TestRecovery         # Expected: crash recovery correct
go test ./... -run TestGetSemantics     # Expected: Found/empty-value distinction

Final Checklist

All "Must Have" present
All "Must NOT Have" absent
All tests pass with -race
WAL roundtrip test passes
Crash recovery test passes
No bufio.Writer on WAL append path
All atomics are typed (atomic.Uint64, atomic.Pointer)
Config invariant checked at Open

68 KiB Raw Blame History Unescape Escape