# Phase 1: WAL 子系统 — Go KV 存储引擎 ## TL;DR > **Quick Summary**: 基于 `docs/design.md` §3.2 实现 WAL 子系统,配合 MemTable(§3.3)、读路径(§3.4)和最小 MANIFEST,构建可嵌入的单 key autocommit KV 存储引擎。 > > **Deliverables**: > - WAL 写入路径(group commit、sequence 管理、segment 轮转、fsync) > - WAL 恢复路径(segment 扫描、fragment 重组、batch 校验、尾部截断) > - MemTable(Arena + SkipList + 原子发布) > - 嵌入式 API(Open/Close/Put/Delete/Get) > - MANIFEST 最小 stub + CURRENT 文件 > - 完整测试套件(单元 + 集成 + 故障注入) > > **Estimated Effort**: XL(~30-50 个任务) > **Parallel Execution**: YES — 5 waves > **Critical Path**: Task 1 → Task 6 → Task 10 → Task 11 → Task 13 → Task 19 → Task 20/21 → F1-F4 --- ## Context ### Original Request 用户指向 `docs/design.md` §3.2(WAL),要求制定第一阶段开发方案。 ### Interview Summary **Key Discussions**: - 设计文档是唯一权威来源(1189 行,§3.2 WAL 约 730 行) - 项目为 greenfield — 无任何 Go 源码 - Phase 1 范围:WAL + MemTable + 读路径 + MANIFEST stub + 嵌入式 API - 不含 SSTable / Compaction / Value Log / MVCC / 网络层 **Research Findings**: - 项目完全空白(仅有 README.md、.omo/、docs/) - 无 go.mod、无依赖 - 设计文档定义了极其精确的崩溃语义和错误分类 ### Metis Review **Identified Gaps** (addressed): - MANIFEST 完整设计不在文档中 → Phase 1 用最小 stub(仅存 recoverySegmentID) - MemTable 无 SSTable 可刷 → 冻结为 Immutable 后持有在内存,不刷盘;3×64MB 上限后阻塞写入 - Periodic/Never sync 策略 → Phase 1 只实现 Always - Group commit 定时器 → time.AfterFunc + reset - API surface → Put/Delete/Get/Open/Close + GetDurableSequence + IsWriteStopped - Graceful shutdown → Close() 等待 pending batch、sync WAL、写 MANIFEST - 文件锁 → Phase 1 不实现,单进程假设 - Varint → Go 标准库 encoding/binary.PutUvarint,key/value 上限 4KB 保证 varint ≤ 5 bytes --- ## Work Objectives ### Core Objective 实现一个功能完整的 WAL 子系统,支持 group commit 写入、崩溃恢复、和单 key autocommit 语义,作为后续 SSTable / MVCC 的基础。 ### Concrete Deliverables - `wal/` — WAL 编解码、segment 管理、写入器、恢复器 - `memtable/` — Arena + SkipList + 原子发布 - `db.go` — DB 入口(Open/Close/Put/Delete/Get) - `errors.go` — 公共错误类型(ErrCommitUnknown 等) - `manifest/` — 最小 MANIFEST stub - 测试覆盖所有关键路径 ### Definition of Done - [ ] `go build ./...` 编译通过 - [ ] `go test ./... -race` 全部通过,无 data race - [ ] 写入 N 条数据 → Close → Open → Get 全部正确 - [ ] WAL 尾部损坏 → 恢复成功,完整 batch 不丢失 - [ ] ErrCommitUnknown 后引擎进入 write-stopped ### Must Have - WAL 写入路径严格遵循设计文档步骤 ①-⑪ - Per-batch private encode buffer + direct write(无 bufio.Writer) - `atomic.Uint64` 用于 publishedSequence / durableSequence / nextSequence - `atomic.Pointer` 用于 skiplist next 指针 - MemTable Arena 容量预留发生在 WAL write 之前 - Sequence 分配前完成所有可失败校验 - WAL Batch 不跨 segment - Segment rotation 只在 batch 边界 - `Get` 返回 `GetResult{Found bool, Value []byte}` 区分 key-not-found 与 empty-value - Config 不变量在 Open 时校验(checked arithmetic) - Recovery 严格遵循 fragment 状态机 - 错误分类严格遵循设计文档表格 ### Must NOT Have (Guardrails) - ❌ SSTable / Compaction / Bloom Filter / Block Cache - ❌ Value Log / ValueLogPointer 写入(Entry 类型定义可预留,但 Phase 1 不写入 ValueLogPointer) - ❌ MVCC / SSI 事务 - ❌ Periodic / Never sync 策略(仅 Always) - ❌ Range scan / Iterator API - ❌ 网络层 - ❌ `bufio.Writer` 在 WAL append 路径上 - ❌ MemTable 写入失败发生在 WAL write 成功之后 - ❌ CRC 校验跳过 - ❌ 共享 buffered writer 模糊 WAL 副作用边界 --- ## Verification Strategy > **ZERO HUMAN INTERVENTION** — ALL verification is agent-executed. No exceptions. ### Test Decision - **Infrastructure exists**: NO (greenfield) - **Automated tests**: Tests-after(先实现后测试,每个模块实现完即写测试) - **Framework**: Go 标准 `testing` + `testify`(断言库) - **Agent-Executed QA**: 每个任务包含 `go test` 命令和 `go test -race` 验证 ### QA Policy - 每个 task 的 QA 通过 `go test` 和 `go vet` 执行 - Evidence: test output saved to `.omo/evidence/task-{N}-{scenario}.txt` --- ## Execution Strategy ### Parallel Execution Waves ``` Wave 1a (Foundation — 1 task): └── 1. Project scaffolding + go.mod + config types [quick] Wave 1b (Codec + Types — 4 parallel + 2 independent, after Wave 1a): ├── 2. Error types + constants [quick] ├── 3. WAL File Header encode/decode [quick] ├── 4. Physical Record encode/decode + Block boundary [quick] ├── 5. WAL Entry encode/decode [quick] ├── 12. Sequence manager [quick] └── 18. MANIFEST stub + CURRENT file [quick] Wave 2a (Batch + Arena + Validation — 3 parallel, after Wave 1b): ├── 6. WAL Batch encode/decode + fragment collector [unspecified-high] (needs 3,4,5) ├── 7. Arena allocator [unspecified-high] (needs 1) └── 9. WAL Batch resource validation [quick] (needs 1, 5) Wave 2b (SkipList — after Arena): └── 8. SkipList (mutex write + lock-free read) [deep] (needs 7) Wave 3a (Segment writer — after Batch codec): └── 10. Segment writer (file write + block writer) [unspecified-high] (needs 3,4,6) Wave 3b (Segment rotation — after Segment writer): └── 11. Segment rotation + durable-ready protocol [unspecified-high] (needs 10) Wave 3c (MemTable — after Arena + SkipList): └── 14. MemTable (wrap skiplist + arena + publish/abort) [unspecified-high] (needs 7,8) Wave 3d (WAL Writer — after MemTable API available): └── 13. Commit queue + WAL writer main loop [deep] (needs 2,9,10,11,12,14) Wave 4a (Recovery scanner — after Segment writer): └── 15. WAL recovery: scanner + record parser [unspecified-high] (needs 4,10) Wave 4b (Recovery batch replay — after scanner + Batch codec): └── 16. WAL recovery: fragment collector + batch replay [deep] (needs 6,15) Wave 4c (Recovery tail truncation — after batch replay): └── 17. WAL recovery: tail truncation + main flow [unspecified-high] (needs 15,16) Wave 4d (DB integration — after all above): └── 19. DB integration: Open/Close/Put/Delete/Get [deep] (needs 13,14,17,18) Wave 5 (Tests + Benchmark — after DB integration): ├── 20. End-to-end + crash recovery tests [deep] └── 21. Benchmark suite [unspecified-high] Wave FINAL (4 parallel reviews): ├── F1. Plan compliance audit (oracle) ├── F2. Code quality review (unspecified-high) ├── F3. Real manual QA (unspecified-high) └── F4. Scope fidelity check (deep) Critical Path: 1 → 6 → 10 → 11 → 13 → 19 → 20/21 → F1-F4 Parallel Speedup: ~50% faster than sequential Max Concurrent: 6 (Wave 1b) ``` ### Dependency Matrix | Task | Depends On | Blocks | Sub-Wave | |------|-----------|--------|----------| | 1 | — | 2-5, 7, 9, 12, 18 | 1a | | 2 | 1 | 13 | 1b | | 3 | 1 | 6, 10 | 1b | | 4 | 1 | 6, 10, 15 | 1b | | 5 | 1 | 6, 9 | 1b | | 6 | 3, 4, 5 | 10, 16 | 2a | | 7 | 1 | 8, 14 | 2a | | 8 | 7 | 14 | 2b | | 9 | 1, 5 | 13 | 2a | | 10 | 3, 4, 6 | 11, 15 | 3a | | 11 | 10 | 13 | 3b | | 12 | 1 | 13 | 1b | | 13 | 2, 9, 10, 11, 12, 14 | 19 | 3d | | 14 | 7, 8 | 19 | 3c | | 15 | 4, 10 | 16 | 4a | | 16 | 6, 15 | 17 | 4b | | 17 | 15, 16 | 19 | 4c | | 18 | 1 | 19 | 1b | | 19 | 13, 14, 17, 18 | 20, 21 | 4d | | 20 | 19 | F1-F4 | 5 | | 21 | 19 | F1-F4 | 5 | ### Agent Dispatch Summary - **1a**: **1** — T1 → `quick` - **1b**: **6** — T2 → `quick`, T3 → `quick`, T4 → `quick`, T5 → `quick`, T12 → `quick`, T18 → `quick` - **2a**: **3** — T6 → `unspecified-high`, T7 → `unspecified-high`, T9 → `quick` - **2b**: **1** — T8 → `deep` - **3a**: **1** — T10 → `unspecified-high` - **3b**: **1** — T11 → `unspecified-high` - **3c**: **1** — T14 → `unspecified-high` - **3d**: **1** — T13 → `deep` - **4a**: **1** — T15 → `unspecified-high` - **4b**: **1** — T16 → `deep` - **4c**: **1** — T17 → `unspecified-high` - **4d**: **1** — T19 → `deep` - **5**: **2** — T20 → `deep`, T21 → `unspecified-high` - **FINAL**: **4** — F1 → `oracle`, F2 → `unspecified-high`, F3 → `unspecified-high`, F4 → `deep` --- ## TODOs - [x] 1. Project scaffolding + go.mod + config types **What to do**: - 初始化 `go.mod`(模块名 `github.com/dailz/go-kv`,Go 1.22+) - 创建目录结构:`wal/`、`memtable/`、`manifest/`、`config/` - 创建 `config/config.go`:`WalConfig` 结构体(MaxSegmentSize, BlockSize, SyncMode, MaxBatchEntries, MaxBatchSize, MaxKeyBytes, MaxInlineValue, MemTableSize, MaxImmutableCount) - 创建 `config/config.go`:`Validate() error` 函数,用 checked arithmetic 校验不变量:`maxWalSegmentPayload >= maxEncodedWalBatchSize + worstCasePhysicalRecordOverhead + worstCaseBlockPadding` - 计算公式参照设计文档 lines 402-453:walFileHeaderSize=32, walBlockSize=32768, physicalRecordHeaderSize=7, walBatchHeaderSize=18, maxWalBatchEntriesSize=4MB - 创建 `.golangci.yml` **Must NOT do**: - 不引入外部依赖(仅标准库 + testify) - 不定义 SSTable / Compaction 相关配置 **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [`golang-project-layout`] - `golang-project-layout`: Go 项目结构最佳实践 - **Skills Evaluated but Omitted**: - `golang-cli`: 不是 CLI 工具 **Parallelization**: - **Can Run In Parallel**: NO (foundation task) - **Parallel Group**: Sub-Wave 1a (solo — blocks all others) - **Blocks**: 2, 3, 4, 5, 7, 9, 12, 18 - **Blocked By**: None **References**: **Pattern References**: - `docs/design.md:286-289` — 设计决策表:批量窗口、日志格式、文件管理、segment 大小 - `docs/design.md:402-453` — 配置不变量计算公式和最小 segment 大小推导 **API/Type References**: - `docs/design.md:501-513` — WAL Batch 资源上限默认值 **External References**: - Go project layout: https://go.dev/doc/modules/layout **WHY Each Reference Matters**: - 设计决策表定义了所有默认值,config 结构体必须映射这些值 - 配置不变量公式是 Open 时必须校验的核心约束 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Config validation accepts valid defaults Tool: Bash Preconditions: Go toolchain installed Steps: 1. cd /home/dailz/workspace/src/go-kv 2. Create a test file config/config_test.go with TestValidateDefaults 3. Test that WalConfig with default values passes Validate() Expected Result: go test ./config/... passes Evidence: .omo/evidence/task-1-config-defaults.txt Scenario: Config validation rejects invalid segment size Tool: Bash Preconditions: go.mod exists Steps: 1. Create TestValidateSegmentTooSmall in config/config_test.go 2. Set MaxSegmentSize = 1024 (far below minimum 4,195,264) 3. Verify Validate() returns error containing "segment" Expected Result: Test fails validation with descriptive error Evidence: .omo/evidence/task-1-config-invalid.txt ``` **Commit**: YES (group with Task 2) - Message: `feat: initialize project structure and error types` - Files: `go.mod, config/, .golangci.yml` - [x] 2. Error types + constants **What to do**: - 创建 `errors.go`:定义 `ErrCommitUnknown`、`ErrWriteStopped`、`ErrSequenceExhausted`、`ErrWALCorrupted`、`ErrInvalidConfig` - 创建 `wal/constants.go`:所有 WAL 常量(WalMagic, WalFormatVersion, WalFileHeaderSize, WalBlockSize, PhysicalRecordHeaderSize, WalBatchHeaderSize, MaxWalBatchEntryCount, MaxWalBatchEntriesSize, MaxWalKeyBytes, MaxWalInlineValueBytes, MaxWalVarintBytes, DefaultMaxWalSegmentSize) - 创建 `wal/constants.go`:Fragment types (RecInvalid=0, RecFull=1, RecFirst=2, RecMiddle=3, RecLast=4) - 创建 `wal/constants.go`:OpType (OpInvalid=0, OpPut=1, OpDelete=2) 和 ValueKind (VKNone=0, VKInline=1, VKValueLogPointer=2) **Must NOT do**: - 不实现具体逻辑,仅类型和常量定义 **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] - **Skills Evaluated but Omitted**: - `golang-naming`: 常量命名简单,不需要完整命名 skill **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 1b (with Tasks 3, 4, 5, 12, 18) - **Blocks**: 13 (WAL writer references error types) - **Blocked By**: 1 (needs go.mod) **References**: **Pattern References**: - `docs/design.md:129-135` — 错误分类表(5 种失败场景 + ErrCommitUnknown) - `docs/design.md:155` — ErrCommitUnknown 的 maybe committed 语义 - `docs/design.md:496-499` — sequence exhausted 终态 **API/Type References**: - `docs/design.md:501-513` — 所有资源上限默认值 - `docs/design.md:364-373` — Fragment type 枚举 - `docs/design.md:571-577` — OpType 枚举 - `docs/design.md:580-584` — ValueKind 枚举 **WHY Each Reference Matters**: - 错误类型直接映射设计文档的错误分类表 - 常量值必须与设计文档精确匹配(block size=32KB, header=32 bytes 等) **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Error types are distinguishable via errors.Is Tool: Bash Preconditions: Task 1 complete Steps: 1. Create errors_test.go with TestErrorTypes 2. Verify errors.Is(ErrCommitUnknown, ErrCommitUnknown) == true 3. Verify errors.Is(ErrWriteStopped, ErrCommitUnknown) == false Expected Result: All error type identity checks pass Evidence: .omo/evidence/task-2-error-types.txt Scenario: Constants match design doc values Tool: Bash Preconditions: wal/constants.go exists Steps: 1. Create wal/constants_test.go with TestConstantValues 2. Assert WalBlockSize == 32*1024 3. Assert WalFileHeaderSize == 32 4. Assert MaxWalBatchEntriesSize == 4*1024*1024 5. Assert DefaultMaxWalSegmentSize == 64*1024*1024 Expected Result: All assertions pass Evidence: .omo/evidence/task-2-constants.txt ``` **Commit**: YES (group with Task 1) - Message: `feat: initialize project structure and error types` - Files: `errors.go, wal/constants.go` - [x] 3. WAL File Header encode/decode **What to do**: - 创建 `wal/header.go`:`WalFileHeader` 结构体(Magic uint32, FormatVersion uint16, HeaderSize uint16, BlockSize uint32, SegmentID uint64, StartSequence uint64, HeaderCRC uint32) - `EncodeWalHeader(h *WalFileHeader) [WalFileHeaderSize]byte` — 编码为 little-endian 32 bytes,headerCRC 覆盖 magic 到 startSequence(不包含自身) - `DecodeWalHeader(data []byte) (*WalFileHeader, error)` — 校验 magic、formatVersion、headerSize、headerCRC - CRC 使用 `hash/crc32` with IEEE table(或 CRC32C 如果标准库支持) - 创建 table-driven tests **Must NOT do**: - 不使用外部 CRC 库 **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 1b (with Tasks 2, 4, 5, 12, 18) - **Blocks**: 6 (batch codec), 10 (segment writer) - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:316-341` — WAL File Header 格式定义和字段说明 **WHY Each Reference Matters**: - Header 格式是 WAL segment 文件的基础,所有后续 segment 操作都依赖正确编码 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Header encode/decode round-trip Tool: Bash Preconditions: wal/header.go exists Steps: 1. Create TestHeaderRoundtrip in wal/header_test.go 2. Encode a header with SegmentID=5, StartSequence=1000 3. Decode the encoded bytes 4. Assert all fields match original Expected Result: Round-trip preserves all fields Evidence: .omo/evidence/task-3-header-roundtrip.txt Scenario: Header CRC detects corruption Tool: Bash Steps: 1. Encode a valid header 2. Flip one byte in the magic field 3. Attempt DecodeWalHeader 4. Assert error returned containing "crc" or "checksum" Expected Result: Corrupted header rejected Evidence: .omo/evidence/task-3-header-crc.txt ``` **Commit**: YES (group with Tasks 4, 5) - Message: `feat(wal): implement WAL entry/record/header codec` - [x] 4. Physical Record encode/decode + Block boundary **What to do**: - 创建 `wal/record.go`:`PhysicalRecord` 结构体(CRC uint32, Length uint16, Type uint8, Payload []byte) - `EncodePhysicalRecord(recType uint8, payload []byte) []byte` — 编码为 7-byte header + payload - `DecodePhysicalRecord(data []byte) (rec *PhysicalRecord, consumed int, err error)` — 解析并校验 CRC - CRC 覆盖 length + type + payload(不含 CRC 自身) - Block 边界辅助函数: - `PaddingNeeded(blockOffset uint32) int` — 剩余 <= 7 bytes 时返回 padding 大小 - `CanFitRecord(blockOffset uint32, payloadLen uint32) bool` - `SplitIntoRecords(encodedBatch []byte) [][]byte` — 将编码后的 batch 按 block 边界拆分为 Physical Record payloads - Table-driven tests 覆盖:单 record、跨 block 拆分、padding **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 1b (with Tasks 2, 3, 5, 12, 18) - **Blocks**: 6, 10, 15 - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:345-381` — Physical Record 格式、fragment 类型、合法组合 - `docs/design.md:383-391` — Block 边界处理规则 **WHY Each Reference Matters**: - Physical Record 是 WAL I/O 的最小单元,block 边界处理是拆分/重组的关键 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Physical Record round-trip Tool: Bash Steps: 1. Create TestRecordRoundtrip in wal/record_test.go 2. Encode RecFull with payload "hello world" 3. Decode the encoded bytes 4. Assert Type==RecFull, Payload=="hello world" Expected Result: Round-trip correct Evidence: .omo/evidence/task-4-record-roundtrip.txt Scenario: Batch split into fragments across blocks Tool: Bash Steps: 1. Create TestSplitIntoRecords in wal/record_test.go 2. Create a 40KB payload (larger than 32KB block) 3. Call SplitIntoRecords with blockSize=32KB 4. Assert result has First + Last (or First + Middle + Last) 5. Concatenate all payloads → equals original Expected Result: Split produces valid fragment sequence, reassembly equals original Evidence: .omo/evidence/task-4-split-records.txt Scenario: Block padding when remaining <= 7 bytes Tool: Bash Steps: 1. Create TestBlockPadding in wal/record_test.go 2. blockOffset = 32*1024 - 5 (only 5 bytes remaining) 3. Assert PaddingNeeded(blockOffset) == 5 4. Assert CanFitRecord(blockOffset, 1) == false Expected Result: Padding and fit check correct Evidence: .omo/evidence/task-4-padding.txt ``` **Commit**: YES (group with Tasks 3, 5) - [x] 5. WAL Entry encode/decode **What to do**: - 创建 `wal/entry.go`:`WalEntry` 结构体(OpType uint8, ValueKind uint8, Key []byte, Value []byte) - `EncodeEntry(e *WalEntry) ([]byte, error)` — 编码 opType(u8) + valueKind(u8) + keyLen(varint) + valLen(varint) + key + value - `DecodeEntry(data []byte) (entry *WalEntry, consumed int, err error)` — 解码返回 entry 和 consumed bytes - Varint 使用 `encoding/binary.PutUvarint` / `binary.ReadUvarint` - 校验规则(参照 design.md lines 586-594): - keyLen > 0 && keyLen <= MaxWalKeyBytes - Put 要求 valueKind ∈ {VKInline, VKValueLogPointer} - Put + VKInline: valLen <= MaxWalInlineValueBytes (允许 valLen = 0) - Put + VKValueLogPointer: valLen > 0 - Delete: valueKind == VKNone, valLen == 0 - Table-driven tests 覆盖所有合法/非法组合 **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 1b (with Tasks 2, 3, 4, 12, 18) - **Blocks**: 6, 9 - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:558-594` — Entry 格式、OpType、ValueKind、合法性规则 **WHY Each Reference Matters**: - Entry 是 WAL 的最小逻辑单元,编解码正确性直接影响数据完整性 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Entry encode/decode round-trip for all valid types Tool: Bash Steps: 1. Create TestEntryRoundtrip in wal/entry_test.go 2. Test cases: Put+Inline("key1","val1"), Put+Inline("key2",[]byte{}), Delete("key3") 3. Encode → Decode → assert all fields match Expected Result: All round-trips correct Evidence: .omo/evidence/task-5-entry-roundtrip.txt Scenario: Entry validation rejects invalid entries Tool: Bash Steps: 1. Create TestEntryValidation in wal/entry_test.go 2. Test: OpInvalid → error 3. Test: Put+VKNone → error 4. Test: Delete+VKInline → error 5. Test: keyLen=0 → error 6. Test: keyLen > MaxWalKeyBytes → error Expected Result: All invalid entries rejected with descriptive errors Evidence: .omo/evidence/task-5-entry-validation.txt ``` **Commit**: YES (group with Tasks 3, 4) - [x] 6. WAL Batch encode/decode + fragment collector **What to do**: - 创建 `wal/batch.go`:`WalBatch` 结构体(Flags uint16, BaseSequence uint64, EntryCount uint32, EntriesSize uint32, Entries []byte) - `EncodeWalBatch(baseSequence uint64, entries []*WalEntry) ([]byte, error)` — 编码 Batch Header + 序列化 Entries - `DecodeWalBatch(data []byte) (*WalBatch, error)` — 校验 flags、entryCount(0 < N <= max)、entriesSize(与实际 bytes 匹配) - `FragmentCollector` 结构体 — 实现 Idle / CollectingFragments 状态机 - `Reset()` - `Append(recType uint8, payload []byte) error` — 按 fragment type 追加,校验 buffer 大小上限 - `IsComplete() bool` — 是否拼出完整 batch - `BatchData() []byte` — 返回重组后的完整 batch bytes - `State() FragmentState` — 当前状态(Idle / CollectingFragments) - 状态转移严格遵循 design.md lines 706-731 的状态机 - Buffer 大小限制:Batch Header 长度 + MaxWalBatchEntriesSize - Varint 长度限制:MaxWalVarintBytes = 5 **Must NOT do**: - FragmentCollector 不做 CRC 校验(由 Physical Record 层负责) **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES (with Tasks 7, 9 — all depend only on Wave 1b outputs) - **Parallel Group**: Sub-Wave 2a (with Tasks 7, 9) - **Blocks**: 10, 16 - **Blocked By**: 3, 4, 5 **References**: **Pattern References**: - `docs/design.md:455-530` — WAL Batch 格式、Batch Header、Entry sequence 推导 - `docs/design.md:706-731` — Fragment 状态机定义 **WHY Each Reference Matters**: - Batch 是 WAL 的核心持久化单元,FragmentCollector 是 recovery 的核心组件 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Batch encode/decode round-trip with multiple entries Tool: Bash Steps: 1. Create TestBatchRoundtrip in wal/batch_test.go 2. Create 10 entries: 5 Put+Inline, 3 Delete, 2 Put+Inline with empty value 3. Encode with baseSequence=42 4. Decode → assert Flags, BaseSequence, EntryCount, EntriesSize match 5. Parse all entries from Entries bytes → assert each matches original Expected Result: Full round-trip preserves all data Evidence: .omo/evidence/task-6-batch-roundtrip.txt Scenario: Fragment collector state machine Tool: Bash Steps: 1. Create TestFragmentCollector in wal/batch_test.go 2. Encode a 100KB batch, split into records (First + Middle* + Last) 3. Feed fragments to collector one by one 4. Assert state transitions: Idle → CollectingFragments → ... → Idle 5. Assert collected batch bytes equal original encoded batch Expected Result: Fragment reassembly produces original batch Evidence: .omo/evidence/task-6-fragment-collector.txt Scenario: Fragment collector rejects illegal transitions Tool: Bash Steps: 1. Feed RecMiddle to Idle state → expect error 2. Feed RecLast to Idle state → expect error 3. Feed RecFull to CollectingFragments state → expect error 4. Feed RecFirst to CollectingFragments state → expect error Expected Result: All illegal transitions rejected Evidence: .omo/evidence/task-6-fragment-illegal.txt ``` **Commit**: YES (group with Task 9) - Message: `feat(wal): implement WAL batch codec and resource validation` - [x] 7. Arena allocator **What to do**: - 创建 `memtable/arena.go`:`Arena` 结构体 - 固定大小 byte slice(默认 64MB,由 config.MemTableSize 配置) - `Allocate(size uint32) (offset uint32, err error)` — 分配对齐的内存块,返回 offset;满时返回 error - `GetBytes(offset uint32, size uint32) []byte` — 从 offset 获取 bytes slice - `Remaining() uint32` — 剩余可用 bytes - `Capacity() uint32` — 总容量 - `Reserve(totalSize uint32) error` — 检查剩余容量是否足够(不实际分配) - 对齐:8-byte 对齐分配 - 线程安全:分配使用 sync.Mutex(写路径已被 WAL writer 串行化) - 测试:分配、对齐、满时错误、并发安全 **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [`golang-concurrency`] - `golang-concurrency`: Arena 需要线程安全分配 **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 2a (with Tasks 6, 9 — all depend only on Task 1 or Wave 1b outputs) - **Blocks**: 8, 14 - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:864-879` — Arena 容量预留规则 **WHY Each Reference Matters**: - Arena 是 MemTable 的内存基础,容量预留是 WAL 写入路径的前置约束 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Arena allocate and read back Tool: Bash Steps: 1. Create TestArenaAllocate in memtable/arena_test.go 2. Create Arena with 1024 bytes 3. Allocate 100 bytes → write "test data" at offset 0 4. GetBytes(offset, 9) → assert equals "test data" 5. Allocate another 200 bytes → assert offset is 8-byte aligned Expected Result: Allocation and readback correct, alignment enforced Evidence: .omo/evidence/task-7-arena-alloc.txt Scenario: Arena full returns error Tool: Bash Steps: 1. Create Arena with 64 bytes 2. Allocate 60 bytes → success 3. Allocate 10 bytes → error (not enough remaining) Expected Result: Full arena rejects allocation Evidence: .omo/evidence/task-7-arena-full.txt ``` **Commit**: YES (with Task 8) - Message: `feat(memtable): implement Arena allocator and SkipList` - [x] 8. SkipList (mutex write + lock-free read) **What to do**: - 创建 `memtable/skiplist.go`:`SkipList` 结构体 - 最大 20 层,p=0.25 的概率晋升 - `Put(key []byte, value []byte, sequence uint64, nodeData interface{}) error` — Mutex 保护下的有序插入 - `Get(key []byte) (found bool, value []byte, sequence uint64)` — 无锁读 - `NewIterator() *Iterator` — 有序遍历 - **原子发布**:所有 next 指针使用 `atomic.Pointer[skipNode]`,store 使用 release 语义 - 节点包含:key、value、sequence、pending 标记(用 atomic.Bool 或存入 nodeData) - Key 比较使用 bytes.Compare - Arena 分配:节点在 Arena 中分配(offset-based 引用,通过 Arena.GetBytes 访问) - 测试:有序插入、查找、遍历、并发读写(`-race`) **Must NOT do**: - 不使用普通指针(必须 atomic.Pointer) - 不依赖 Mutex 向读者发布内存(必须 atomic store-release) **Recommended Agent Profile**: - **Category**: `deep` - **Skills**: [`golang-concurrency`] - `golang-concurrency`: 原子操作和内存序是核心难点 **Parallelization**: - **Can Run In Parallel**: NO (depends on Task 7) - **Parallel Group**: Sub-Wave 2b (solo — after Arena complete) - **Blocks**: 14 - **Blocked By**: 7 **References**: **Pattern References**: - `docs/design.md:834-840` — SkipList 设计决策(20 层、64MB Arena、Mutex 写 + 无锁读) - `docs/design.md:841-850` — 发布与内存序约束(atomic.Pointer, release store) **WHY Each Reference Matters**: - 内存序约束是整个读路径正确性的关键,ARM 等弱内存序平台必须严格遵守 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: SkipList ordered insertion and retrieval Tool: Bash Steps: 1. Create TestSkipListOrdered in memtable/skiplist_test.go 2. Insert keys "c", "a", "e", "b", "d" with values "val-C" etc. 3. Get each key → assert correct value 4. Iterate → assert order is a, b, c, d, e Expected Result: Keys stored and retrieved in correct order Evidence: .omo/evidence/task-8-skiplist-ordered.txt Scenario: SkipList concurrent read/write no data race Tool: Bash Steps: 1. Create TestSkipListConcurrent in memtable/skiplist_test.go 2. Start 4 writer goroutines inserting 100 keys each 3. Start 4 reader goroutines reading random keys 4. Run with go test -race Expected Result: No data race detected Evidence: .omo/evidence/task-8-skiplist-race.txt ``` **Commit**: YES (with Task 7) - [x] 9. WAL Batch resource validation **What to do**: - 创建 `wal/validate.go`:`ValidateBatchLimits(entries []*WalEntry, cfg *config.WalConfig) error` - 在 sequence 分配之前检查所有可失败条件: - entryCount > 0 && entryCount <= cfg.MaxBatchEntries - 每个 entry 的 keyLen > 0 && keyLen <= cfg.MaxKeyBytes - 每个 entry 如果是 Put+Inline 则 valLen <= cfg.MaxInlineValue - 所有 entries 编码后总大小 <= cfg.MaxBatchSize - worstCase Physical Record overhead + batch encoded size <= maxWalSegmentPayload - checked arithmetic:计算中不溢出 - 返回的具体错误类型允许区分不同违规 **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Sub-Wave 2a (with Tasks 6, 7 — all depend on Task 1/Wave 1b) - **Blocks**: 13 - **Blocked By**: 1, 5 **References**: **Pattern References**: - `docs/design.md:501-513` — WAL Batch 资源上限 - `docs/design.md:514` — "写入侧必须在 sequence 分配、MemTable 预留和 WAL append 之前完成这些校验" **WHY Each Reference Matters**: - 这些校验必须在 sequence 分配之前完成,否则失败会触发 write-stopped **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Valid batch passes validation Tool: Bash Steps: 1. Create TestValidateBatchLimits in wal/validate_test.go 2. Create 100 Put+Inline entries with 10-byte keys and 10-byte values 3. Call ValidateBatchLimits → expect nil error Expected Result: Valid batch accepted Evidence: .omo/evidence/task-9-validate-pass.txt Scenario: Batch exceeding entry count limit rejected Tool: Bash Steps: 1. Create 10,001 entries (exceeds default MaxBatchEntries=10000) 2. Call ValidateBatchLimits → expect error containing "entry count" Expected Result: Over-limit batch rejected before sequence allocation Evidence: .omo/evidence/task-9-validate-entrycount.txt Scenario: Key exceeding size limit rejected Tool: Bash Steps: 1. Create entry with keyLen = 5KB (exceeds MaxWalKeyBytes=4KB) 2. Call ValidateBatchLimits → expect error containing "key" Expected Result: Oversized key rejected Evidence: .omo/evidence/task-9-validate-keysize.txt ``` **Commit**: YES (group with Task 6) - [x] 10. Segment writer (file write + block writer) **What to do**: - 创建 `wal/block_writer.go`:`BlockWriter` — 管理 32KB block 的 Physical Record 写入 - 内部维护当前 block 的 byte buffer - 写入 Physical Record 时检查 block 边界 - 剩余 <= 7 bytes 时自动 padding(全 0) - 跨 block 的 batch fragment 自动拆分 - `WriteRecord(recType uint8, payload []byte) error` - `Flush(io.Writer) error` — 将当前 block 写入 fd - `Reset()` — 重置为空 block - 创建 `wal/segment_writer.go`:`SegmentWriter` - 封装 WAL segment 文件的追加写入 - `NewSegmentWriter(dir string, segmentID uint64, startSequence uint64, cfg *config.WalConfig) (*SegmentWriter, error)` — 创建 segment-N.wal.tmp → 写 header → fsync → rename → fsync dir → durable-ready - `AppendBatch(encodedBatch []byte) error` — 按 block 边界写入 batch(使用 BlockWriter) - `Sync() error` — fsync 当前文件 - `Close() error` - `RemainingPayload() uint64` — 当前 segment 剩余 payload - `CurrentOffset() uint64` - `SegmentID() uint64` - `EndState() SegmentEndState` — 返回 (segmentID, endOffset, endSequence) **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: NO (depends on 3, 4, 6) - **Parallel Group**: Sub-Wave 3a (solo — after batch codec complete) - **Blocks**: 11, 15 - **Blocked By**: 3, 4, 6 **References**: **Pattern References**: - `docs/design.md:247-278` — WAL 元数据持久化协议(durable-ready) - `docs/design.md:383-391` — Block 边界处理规则 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Segment creation follows durable-ready protocol Tool: Bash Steps: 1. Create TestSegmentCreation in wal/segment_writer_test.go 2. Create a temp directory, create SegmentWriter with segmentID=0 3. Verify segment-0.wal exists in directory 4. Read file → verify header is correct (magic, segmentID, startSequence) 5. Verify no .tmp file remains Expected Result: Segment file created with correct header, no temp file Evidence: .omo/evidence/task-10-segment-create.txt Scenario: Batch written to segment correctly Tool: Bash Steps: 1. Create SegmentWriter in temp dir 2. Encode a batch with 5 entries 3. Call AppendBatch(encodedBatch) 4. Sync + Close 5. Read file back → verify batch can be recovered Expected Result: Written batch recoverable from segment file Evidence: .omo/evidence/task-10-segment-write.txt ``` **Commit**: YES (with Task 11) - Message: `feat(wal): implement segment writer with rotation` - [x] 11. Segment rotation + durable-ready protocol **What to do**: - 创建 `wal/segment_manager.go`:管理 segment 的轮转和生命周期 - 写入 batch 前检查 `RemainingPayload()` 是否足够容纳整个 batch - 不足时: 1. 当前 segment 完成(在 batch 边界) 2. 创建新 segment:segment-N+1.wal.tmp → header → fsync → rename → fsync dir 3. 新 segment 的 startSequence = 下一个 expected sequence 4. 切换 active segment - 不允许 batch 跨 segment - CURRENT 文件 best-effort 更新(temp + rename) - CURRENT 更新失败不影响已 durable-ready 的 segment **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: NO (depends on Task 10) - **Parallel Group**: Sub-Wave 3b (solo — after segment writer) - **Blocks**: 13 - **Blocked By**: 10 **References**: **Pattern References**: - `docs/design.md:392-453` — Segment Rotation 约束 - `docs/design.md:247-278` — WAL 元数据持久化协议 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Segment rotation triggers when payload exceeds remaining space Tool: Bash Steps: 1. Create a test WalConfig with: MaxBatchSize=256, MaxBatchEntries=10, BlockSize=256, MaxSegmentSize=4096 (satisfies invariant: 4096 ≥ maxEncodedWalBatchSize since MaxBatchSize=256 → maxEncoded ≈ 256 + 4*10 + overhead < 512) 2. Create SegmentManager with this config 3. Write small batches until segment-0 is nearly full (~3800 bytes written) 4. Write one more batch that would exceed remaining → triggers rotation 5. Verify segment-1.wal exists with correct header 6. Verify batch was written to segment-1, not segment-0 Expected Result: Rotation creates new segment, batch not split across segments Failure Indicators: segment-1.wal does not exist, or batch data spans both segment files Evidence: .omo/evidence/task-11-rotation.txt Scenario: CURRENT file updated after rotation Tool: Bash Steps: 1. Trigger rotation using same test config as above 2. Read CURRENT file → verify it points to latest segment filename Expected Result: CURRENT reflects latest active segment filename Failure Indicators: CURRENT still points to segment-0, or file content is stale Evidence: .omo/evidence/task-11-current.txt ``` **Commit**: YES (with Task 10) - [x] 12. Sequence manager **What to do**: - 创建 `wal/sequence.go`:`SequenceManager` - `nextSequence atomic.Uint64` — 下一个待分配 sequence - `publishedSequence atomic.Uint64` — 已发布的连续 high-water mark - `durableSequence atomic.Uint64` — 已满足 durable 条件的 high-water mark - `AllocateBatch(count uint32) (baseSequence uint64, err error)` — checked arithmetic:`baseSequence + count - 1` 不溢出 uint64。溢出 → 返回 ErrSequenceExhausted - `Publish(seq uint64)` — Store publishedSequence(在所有 entry 节点发布后调用) - `MarkDurable(endState SegmentEndState)` — 推进 durableSequence - `Published() uint64` — Load publishedSequence - `Durable() uint64` — Load durableSequence - `NextSequence() uint64` — Load nextSequence - 恢复后初始化:`NewSequenceManager(recoveredSequence uint64)` **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Wave 1b (with Tasks 2, 3, 4, 5, 18 — all depend only on Task 1) - **Blocks**: 13 - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:486-499` — Sequence 运算和溢出检查 - `docs/design.md:205-223` — durableSequence 推进逻辑 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Sequence allocation returns consecutive bases Tool: Bash Steps: 1. Create TestSequenceAllocation in wal/sequence_test.go 2. NewSequenceManager(0) 3. AllocateBatch(5) → base=0 4. AllocateBatch(3) → base=5 5. AllocateBatch(1) → base=8 Expected Result: Consecutive, no gaps Evidence: .omo/evidence/task-12-seq-alloc.txt Scenario: Sequence overflow returns ErrSequenceExhausted Tool: Bash Steps: 1. NewSequenceManager(math.MaxUint64 - 2) 2. AllocateBatch(5) → err (would overflow) Expected Result: ErrSequenceExhausted returned Evidence: .omo/evidence/task-12-seq-overflow.txt ``` **Commit**: YES (with Task 13) - [x] 13. Commit queue + WAL writer main loop **What to do**: - 创建 `wal/commit_queue.go`:`CommitQueue` — 写请求队列 - 每个 request 包含 entries、result channel - `Submit(entries []*WalEntry) chan WriteResult` — 非阻塞提交 - `Collect(timeout time.Duration) []*CommitRequest` — 收集当前队列中的请求 - 创建 `wal/writer.go`:`WalWriter` — 核心写入循环 - 单 goroutine 运行 main loop - Group commit:收集 queue 中的请求 → 组装 batch → 触发条件(500µs 或 32KB) - 严格遵循 design.md steps ①-⑪: 1. 从 commit queue 收集写入 2. 组装 WAL Batch 3. ValidateBatchLimits → 失败则普通错误(不分配 sequence) 4. 计算 MemTable Arena 预留量 → 不足则 freeze/switch 或等待 5. AllocateBatch → 分配 baseSequence(此步之后任何失败 → write-stopped) 6. 私有缓冲编码 WAL Batch 7. 检查 segment rotation 8. segmentWriter.AppendBatch(encoded) → 失败 → ErrCommitUnknown + write-stopped 9. 写入 MemTable pending entries 10. segmentWriter.Sync() (Always 模式) → 失败 → ErrCommitUnknown + write-stopped 11. 原子发布 MemTable entries (atomic Pointer store-release) 12. Publish(baseSequence + entryCount - 1) 13. 唤醒所有等待的调用方 - `writeStopped atomic.Bool` — write-stopped 状态 - `Put(key, value []byte) error` — 提交到 commit queue 并等待结果 - `Delete(key []byte) error` — 同上 - `Close() error` — 等待 pending batch 完成 **Must NOT do**: - 不使用 bufio.Writer(直接 os.File.Write) - 不在 sequence 分配后做可失败的校验 **Recommended Agent Profile**: - **Category**: `deep` - **Skills**: [`golang-concurrency`] - `golang-concurrency`: commit queue、group commit、goroutine 生命周期管理 **Parallelization**: - **Can Run In Parallel**: NO (depends on MemTable API from Task 14) - **Parallel Group**: Sub-Wave 3d (solo — after MemTable complete) - **Blocks**: 19 - **Blocked By**: 2, 9, 10, 11, 12, 14 **References**: **Pattern References**: - `docs/design.md:103-115` — 写入流程 11 步 - `docs/design.md:119` — 资源前置校验约束 - `docs/design.md:127-153` — ErrCommitUnknown 和 WAL 副作用边界 - `docs/design.md:139-153` — per-batch private encode buffer **WHY Each Reference Matters**: - 这是整个 WAL 子系统最复杂的组件,必须严格遵循设计文档的步骤顺序和错误分类 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Single Put writes and syncs correctly Tool: Bash Steps: 1. Create TestWalWriterSinglePut in wal/writer_test.go 2. Start WalWriter in temp dir 3. Put("key1", "value1") 4. Close writer 5. Read WAL segment → verify batch with 1 entry exists Expected Result: Entry persisted in WAL Evidence: .omo/evidence/task-13-writer-single.txt Scenario: Multiple concurrent writes group-committed Tool: Bash Steps: 1. Create TestWalWriterGroupCommit in wal/writer_test.go 2. Start WalWriter 3. Launch 10 goroutines each doing Put 4. Wait for all to complete 5. Close writer 6. Read WAL → verify entries batched together Expected Result: Entries grouped in batches, all sequences assigned Evidence: .omo/evidence/task-13-writer-group.txt Scenario: Write-stopped after simulated WAL write failure Tool: Bash Steps: 1. Create a WalWriter with a mock segment writer that fails on 3rd AppendBatch 2. Write 5 entries sequentially 3. 3rd write should return ErrCommitUnknown 4. 4th write should be rejected (write-stopped) Expected Result: ErrCommitUnknown returned, subsequent writes rejected Evidence: .omo/evidence/task-13-writer-stopped.txt ``` **Commit**: YES (with Task 12) - Message: `feat(wal): implement WAL writer with group commit` - [x] 14. MemTable (wrap skiplist + arena + publish/abort) **What to do**: - 创建 `memtable/memtable.go`:`MemTable` 结构体 - 封装 Arena + SkipList - `Reserve(entries []ReserveEntry) (totalSize uint32, err error)` — 容量预留(最坏情况:最大层高 next 指针数组 + 对齐 padding)。单个 batch 超过空 MemTable → 返回错误 - `PutPending(key []byte, value []byte, sequence uint64) error` — 写入 pending entry(Arena 已预留) - `DeletePending(key []byte, sequence uint64) error` — 写入 pending tombstone - `Publish(upToSequence uint64)` — 批量发布 sequence <= upToSequence 的 pending entries(atomic store-release) - `Abort(fromSequence uint64)` — 标记 aborted - `Get(key []byte, publishedSequence uint64) *GetResult` — 无锁读:先 Load publishedSequence,遍历 skiplist,只返回 sequence <= publishedSequence 且非 aborted 的 entry - `NewIterator(publishedSequence uint64) *MemTableIterator` — 有序遍历(同样过滤 pending/aborted) - `ApproximateSize() uint64` - `UsableCapacity() uint32` — 可用容量(扣除固定元数据) **Must NOT do**: - 不允许 PutPending 在 Reserve 成功后因 Arena 满失败 **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [`golang-concurrency`] - `golang-concurrency`: 原子发布、无锁读、内存序 **Parallelization**: - **Can Run In Parallel**: NO (solo in sub-wave, Task 13 depends on this) - **Parallel Group**: Sub-Wave 3c (solo) - **Blocks**: 13, 19 - **Blocked By**: 7, 8 **References**: **Pattern References**: - `docs/design.md:864-879` — Arena 容量预留协议 - `docs/design.md:881-889` — Flush 过滤规则(pending/aborted 不进入 SSTable) - `docs/design.md:841-850` — 发布与内存序 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Pending entry invisible to readers Tool: Bash Steps: 1. Create TestMemTablePending in memtable/memtable_test.go 2. Create MemTable, write PutPending("key1","val1", seq=10) 3. Get("key1", publishedSequence=5) → Found=false 4. Publish(upToSequence=10) 5. Get("key1", publishedSequence=10) → Found=true, Value="val1" Expected Result: Entry invisible before publish, visible after Evidence: .omo/evidence/task-14-memtable-pending.txt Scenario: Aborted entry invisible to readers Tool: Bash Steps: 1. PutPending("key2","val2", seq=20) 2. Abort(20) 3. Publish(upToSequence=30) — publish others but not aborted 4. Get("key2", publishedSequence=30) → Found=false Expected Result: Aborted entry never visible Evidence: .omo/evidence/task-14-memtable-aborted.txt Scenario: Reserve accuracy Tool: Bash Steps: 1. Create MemTable with 1024 bytes usable 2. Reserve 10 entries with 10-byte keys + 10-byte values 3. Assert reserved size includes node overhead + alignment 4. Attempt to reserve more than remaining → error Expected Result: Reserve accounts for full overhead Evidence: .omo/evidence/task-14-memtable-reserve.txt ``` **Commit**: YES - Message: `feat(memtable): integrate arena+skiplist with publish/abort` - [x] 15. WAL recovery: scanner + record parser **What to do**: - 创建 `wal/scanner.go`:`SegmentScanner` — 从 WAL 目录扫描 segment 文件 - `ScanSegments(dir string, recoverySegmentID uint64) ([]*SegmentInfo, error)` — 按 segmentID 排序,过滤 < recoverySegmentID - `SegmentInfo` 结构体:FilePath, SegmentID, StartSequence(从 header 解析) - 连续性校验:segmentID 连续、startSequence 衔接 - 创建 `wal/record_parser.go`:`RecordParser` — Block 级别 Physical Record 解析 - `ParseBlock(data []byte) ([]*PhysicalRecord, error)` — 解析 block 内所有 records - Padding 校验:剩余 < 7 bytes 时全 0 - Short block 处理:最后 segment 的最后 block 可以短于 32KB - 错误分类辅助:`IsTailCorruption(err error) bool` **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: NO (depends on 4, 10) - **Parallel Group**: Sub-Wave 4a (solo) - **Blocks**: 16 - **Blocked By**: 4, 10 **References**: **Pattern References**: - `docs/design.md:597-665` — Recovery 扫描流程、segment 连续性校验 - `docs/design.md:669-691` — Physical Record 解析规则 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Segment scanner finds and orders segments correctly Tool: Bash Steps: 1. Create temp dir with segment-0.wal, segment-2.wal, segment-1.wal 2. ScanSegments(dir, 0) → returns 3 segments ordered by ID 3. ScanSegments(dir, 1) → returns segments 1,2 only Expected Result: Correct ordering and filtering Evidence: .omo/evidence/task-15-scanner.txt Scenario: Record parser handles valid blocks Tool: Bash Steps: 1. Create a 32KB block with 3 Physical Records 2. ParseBlock → verify 3 records parsed with correct payloads Expected Result: All records extracted Evidence: .omo/evidence/task-15-parser.txt ``` **Commit**: YES (with Tasks 16, 17) - [x] 16. WAL recovery: fragment collector + batch replay **What to do**: - 创建 `wal/recovery.go`:恢复核心逻辑 - 使用 Task 6 的 `FragmentCollector` 进行 fragment 重组 - `BatchReplayer` 接口:`ReplayPut(key, value []byte, sequence uint64)` / `ReplayDelete(key []byte, sequence uint64)` - `ReplayBatch(batch *WalBatch, expectedSequence uint64, replayer BatchReplayer) (nextSequence uint64, err error)` - 校验 Batch Header:flags、entryCount (0 < N <= max)、entriesSize、baseSequence == expectedSequence - 逐条解析 Entry,校验 opType、valueKind、keyLen、valLen - 调用 replayer 回调 - 返回 nextSequence = expectedSequence + entryCount - Sequence 溢出检查:`baseSequence + entryCount - 1` 不溢出 **Recommended Agent Profile**: - **Category**: `deep` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: NO (depends on 6, 15) - **Parallel Group**: Sub-Wave 4b (solo — after scanner) - **Blocks**: 17 - **Blocked By**: 6, 15 **References**: **Pattern References**: - `docs/design.md:733-781` — WAL Batch 校验与重放规则 - `docs/design.md:486-499` — Sequence 溢出检查 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Valid batch replayed correctly Tool: Bash Steps: 1. Create TestBatchReplay in wal/recovery_test.go 2. Encode a batch: baseSeq=0, 3 entries (Put a, Put b, Delete c) 3. ReplayBatch with expectedSequence=0 and mock replayer 4. Assert replayer received 3 calls in order with correct args Expected Result: All entries replayed with correct sequences Evidence: .omo/evidence/task-16-batch-replay.txt Scenario: Batch with wrong baseSequence rejected Tool: Bash Steps: 1. Encode batch with baseSeq=100 2. ReplayBatch with expectedSequence=50 → error Expected Result: Sequence mismatch error Evidence: .omo/evidence/task-16-seq-mismatch.txt ``` **Commit**: YES (with Tasks 15, 17) - [x] 17. WAL recovery: tail truncation + main flow **What to do**: - 创建 `wal/truncation.go`:尾部截断持久化 - `TruncateTail(segmentPath string, truncateOffset int64) error` — ftruncate + fsync - `DeleteEmptySegments(dir string, fromSegmentID uint64, expectedSequence uint64) error` — 删除空 segment + fsync dir - 复用 Task 16 定义的 `BatchReplayer` 接口和 `ReplayBatch` 函数(同在 `wal/recovery.go`) - `Recover(dir string, manifest *manifest.Manifest, cfg *config.WalConfig, replayer BatchReplayer) (*RecoveryResult, error)` - `BatchReplayer` 由 Task 16 定义(interface{ ReplayPut/ReplayDelete }),Task 19 传入 MemTable 适配实现 - 主流程严格遵循 design.md lines 622-636: 1. 读取 MANIFEST → recoverySegmentID 2. ScanSegments → 过滤 3. 校验连续性 4. 逐 segment:校验 header → 逐 block 解析 → fragment 重组 → 调用 Task 16 的 `ReplayBatch(batch, expectedSeq, replayer)` 重放 5. 处理尾部异常(最后 segment → truncation,中间 segment → 报错) 6. 持久化截断 - 返回 `RecoveryResult`:RecoveredSequence, NextSequence, PublishedSequence - 错误分类: - 尾部(最后 segment):header 半写、length 越界、CRC 失败、非 0 padding → 截断 - 中间(非最后 segment):同样情况 → 报错 - CRC-valid batch 内部错误(entryCount 越界等)→ 报错(不可截断) **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: NO (depends on 15, 16) - **Parallel Group**: Sub-Wave 4c (solo — after batch replay) - **Blocks**: 19 - **Blocked By**: 15, 16 **References**: **Pattern References**: - `docs/design.md:622-636` — Recovery 扫描流程 - `docs/design.md:640-665` — Segment 连续性校验 - `docs/design.md:783-800` — 尾部截断持久化 - `docs/design.md:694-703` — 错误分类(尾部 vs 中间) **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Normal recovery from clean WAL Tool: Bash Steps: 1. Create temp dir, write 2 segments with valid batches 2. Recover(dir, manifest) → success 3. Assert RecoveredSequence == last batch end sequence 4. Assert all entries replayed via mock replayer Expected Result: Full recovery, no data loss Evidence: .omo/evidence/task-17-recovery-normal.txt Scenario: Recovery with tail corruption truncates Tool: Bash Steps: 1. Write valid segment + partial write at end (corrupt last record) 2. Recover → success (truncated to last complete batch) 3. Verify segment file was truncated Expected Result: Tail corruption handled, complete batches preserved Evidence: .omo/evidence/task-17-recovery-tail.txt Scenario: Recovery with middle corruption reports error Tool: Bash Steps: 1. Write 3 segments, corrupt the middle one (bad CRC) 2. Recover → error containing "corrupt" or "damaged" Expected Result: Middle corruption detected, recovery fails Evidence: .omo/evidence/task-17-recovery-middle.txt ``` **Commit**: YES (with Tasks 15, 16) - [x] 18. MANIFEST stub + CURRENT file **What to do**: - 创建 `manifest/manifest.go`:`Manifest` 最小 stub - 仅存储 `recoverySegmentID uint64` - `Load(dir string) (*Manifest, error)` — 读取 MANIFEST 文件 - `Save(dir string, recoverySegmentID uint64) error` — temp + rename 原子写入 - 首次创建:如果 MANIFEST 不存在,创建初始文件(recoverySegmentID=0) - 创建 `manifest/current.go`:CURRENT 文件 - `WriteCurrent(dir string, segmentID uint64) error` — temp + rename,best-effort - `ReadCurrent(dir string) (segmentID uint64, ok bool)` — 读取,不存在或损坏返回 ok=false - 格式:简单的文本文件或二进制(Phase 1 选择简单文本) **Recommended Agent Profile**: - **Category**: `quick` - **Skills**: [] **Parallelization**: - **Can Run In Parallel**: YES - **Parallel Group**: Wave 1b (with Tasks 2, 3, 4, 5, 12 — all depend only on Task 1) - **Blocks**: 19 - **Blocked By**: 1 **References**: **Pattern References**: - `docs/design.md:603-619` — CURRENT / MANIFEST 权威性 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: MANIFEST save and load round-trip Tool: Bash Steps: 1. Save(dir, 42) 2. Load(dir) → assert recoverySegmentID == 42 Expected Result: Round-trip correct Evidence: .omo/evidence/task-18-manifest.txt Scenario: CURRENT best-effort handles missing file Tool: Bash Steps: 1. ReadCurrent on empty dir → ok=false, no error Expected Result: Missing CURRENT handled gracefully Evidence: .omo/evidence/task-18-current-missing.txt ``` **Commit**: YES - Message: `feat: implement MANIFEST stub and CURRENT file` - [ ] 19. DB integration: Open/Close/Put/Delete/Get **What to do**: - 创建 `db.go`:`DB` 结构体和核心 API - `DB` 字段:walWriter, memTable, immutableTables, seqMgr, manifest, config, dir, writeStopped - `Open(dir string, opts ...Option) (*DB, error)`: 1. Load or create MANIFEST 2. Recover WAL → 重建 MemTable(恢复的 entries 标记为 published) 3. 设置 nextSequence、publishedSequence 4. 启动 WAL writer goroutine - `Close() error`: 1. 停止 WAL writer(等待 pending batch 完成) 2. Sync WAL 3. Save MANIFEST(recoverySegmentID stays 0 in Phase 1 — no SSTable flush means all segments must be replayed) 4. Close segment writer - `Put(key, value []byte) error`: 1. ValidateBatchLimits (key/value size) 2. Submit to WAL writer → wait for result 3. 返回 nil 或 ErrCommitUnknown 或 write-stopped error - `Delete(key []byte) error` — 同上,opType=Delete - `Get(key []byte) (GetResult, error)`: 1. Load publishedSequence 2. 查 active MemTable → Immutable #1 → Immutable #2 3. 只返回 sequence <= publishedSequence 且非 aborted 4. Tombstone → Found=false 5. 未找到 → Found=false - `GetResult` 结构体:Value []byte, Found bool(参照 design.md §3.4) - `GetDurableSequence() uint64` - `IsWriteStopped() bool` - MemTable freeze + switch: - Put 前检查容量,不足时 freeze - Immutable 队列满时阻塞 - Phase 1:Immutable 不 flush 到 SSTable,仅持有在内存 - Options pattern:`type Option func(*config.WalConfig)` **Recommended Agent Profile**: - **Category**: `deep` - **Skills**: [`golang-design-patterns`] - `golang-design-patterns`: Options pattern for configuration **Parallelization**: - **Can Run In Parallel**: NO (integrates all deps) - **Parallel Group**: Sub-Wave 4d (solo — after recovery + WAL writer + MemTable + MANIFEST) - **Blocks**: 20, 21 - **Blocked By**: 13, 14, 17, 18 **References**: **Pattern References**: - `docs/design.md:901-921` — Get API 和 GetResult 定义 - `docs/design.md:928-941` — 读路径优先级 - `docs/design.md:852-862` — MemTable 生命周期 **WHY Each Reference Matters**: - 这是所有模块的集成点,必须正确协调 WAL writer、MemTable、recovery 和 MANIFEST **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Open creates fresh DB, Put+Get round-trip Tool: Bash Steps: 1. Create temp dir 2. Open(dir) → success 3. Put("hello", "world") → nil 4. Get("hello") → {Found:true, Value:"world"} 5. Close() → nil Expected Result: Full round-trip works Evidence: .omo/evidence/task-19-open-put-get.txt Scenario: Close and reopen preserves data Tool: Bash Steps: 1. Open(dir), Put("k1","v1"), Put("k2","v2"), Close() 2. Open(dir) → recovery runs 3. Get("k1") → {Found:true, Value:"v1"} 4. Get("k2") → {Found:true, Value:"v2"} 5. Close() Expected Result: Data survives close/reopen Evidence: .omo/evidence/task-19-reopen.txt Scenario: Delete makes key not found Tool: Bash Steps: 1. Open, Put("k","v"), Delete("k") 2. Get("k") → {Found:false} Expected Result: Delete works correctly Evidence: .omo/evidence/task-19-delete.txt Scenario: Empty value distinguished from not found Tool: Bash Steps: 1. Put("empty", []byte{}) 2. Get("empty") → {Found:true, Value:[]byte{}} 3. Get("nonexist") → {Found:false} Expected Result: Found distinguishes empty value from missing key Evidence: .omo/evidence/task-19-empty-value.txt Scenario: Concurrent Put and Get work correctly Tool: Bash Steps: 1. Open 2. Launch 10 writers (Put "key-{i}", "val-{i}") 3. Launch 10 readers (Get "key-{i}") 4. go test -race Expected Result: No race conditions Evidence: .omo/evidence/task-19-concurrent.txt ``` **Commit**: YES - Message: `feat: integrate DB with Open/Close/Put/Delete/Get` - [ ] 20. End-to-end + crash recovery tests **What to do**: - 创建 `db_test.go`:集成测试 - `TestWALRoundtrip` — 写 N 条 → Close → Open → 验证全部 - `TestRecoveryPartialWrite` — 模拟崩溃:写入数据 → 在 WAL 写入中途截断文件 → Recover → 验证完整 batch 保留 - `TestRecoveryMultiSegment` — 多 segment 恢复 - `TestRecoveryTailCorruption` — 尾部损坏修复 - `TestWriteStoppedAfterFailure` — 模拟写入失败 → ErrCommitUnknown → write-stopped - `TestGetSemantics` — Found vs empty value vs not found - `TestSequenceExhaustion` — 虽然不可达,但验证 sequence 溢出返回正确错误 - `TestConcurrentReadWrite` — 多 goroutine 并发读写 - `TestLargeBatchSegmentRotation` — 大量写入触发多次 segment 轮转 - `TestMemTableFreezeNoSSTable` — 验证 MemTable 满时冻结,不刷盘 **Recommended Agent Profile**: - **Category**: `deep` - **Skills**: [`golang-testing`] - `golang-testing`: Go testing patterns **Parallelization**: - **Can Run In Parallel**: YES (with Task 21) - **Parallel Group**: Wave 5 (with Task 21) - **Blocks**: F1-F4 - **Blocked By**: 19 **References**: **Pattern References**: - `docs/design.md:694-703` — 错误分类 - `docs/design.md:783-800` — 尾部截断 **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Full WAL roundtrip with crash simulation Tool: Bash Steps: 1. Open DB, write 1000 keys 2. Close DB (simulate clean shutdown) 3. Open DB → verify all 1000 keys readable Expected Result: All 1000 keys recovered Evidence: .omo/evidence/task-20-roundtrip.txt Scenario: Crash recovery with tail truncation Tool: Bash Steps: 1. Open DB, write 500 keys, Close 2. Open DB, write 200 more keys 3. Manually truncate last WAL segment by 100 bytes (simulate crash) 4. Open DB → verify 500 original keys present, some of 200 may be lost Expected Result: Complete batches preserved, tail truncated Evidence: .omo/evidence/task-20-crash.txt Scenario: go test -race passes all integration tests Tool: Bash Steps: 1. go test ./... -race -count=1 Expected Result: Zero race conditions Evidence: .omo/evidence/task-20-race.txt ``` **Commit**: YES - Message: `test: add end-to-end and crash recovery tests` - [ ] 21. Benchmark suite **What to do**: - 创建 `bench_test.go`:性能基准 - `BenchmarkSinglePut` — 单线程 Put 吞吐 - `BenchmarkConcurrentPut` — 多线程 Put 吞吐(2/4/8 goroutines) - `BenchmarkGet` — Get 延迟 - `BenchmarkConcurrentGet` — 并发 Get - `BenchmarkMixedReadWrite` — 混合读写 - `BenchmarkWALRecovery` — 不同数据量的恢复时间 - 使用 `b.ReportAllocs()` 跟踪内存分配 **Recommended Agent Profile**: - **Category**: `unspecified-high` - **Skills**: [`golang-benchmark`] - `golang-benchmark`: Go benchmarking best practices **Parallelization**: - **Can Run In Parallel**: YES (with Task 20) - **Parallel Group**: Wave 5 (with Task 20) - **Blocks**: F1-F4 - **Blocked By**: 19 **References**: - 无特定设计文档引用(benchmark 是验证手段,不是设计要求) **Acceptance Criteria**: **QA Scenarios (MANDATORY):** ``` Scenario: Benchmark runs without errors Tool: Bash Steps: 1. go test -bench=. -benchmem ./... Expected Result: All benchmarks complete, results printed Evidence: .omo/evidence/task-21-bench.txt ``` **Commit**: YES - Message: `bench: add WAL performance benchmarks` --- ## Final Verification Wave - [ ] F1. **Plan Compliance Audit** — `oracle` Read the plan end-to-end. For each "Must Have": verify implementation exists (grep for atomic.Uint64, atomic.Pointer, no bufio.Writer on WAL path, GetResult with Found field, etc). For each "Must NOT Have": search codebase for forbidden patterns (SSTable, Compaction, ValueLogPointer write, Periodic sync, bufio.Writer in wal/). Check evidence files exist in .omo/evidence/. Compare deliverables against plan. **Acceptance Criteria**: - [ ] All "Must Have" items verified present in codebase - [ ] All "Must NOT Have" items verified absent - [ ] Evidence files exist for tasks 1-21 - [ ] Output: `Must Have [N/N] | Must NOT Have [N/N] | Tasks [21/21] | VERDICT: APPROVE/REJECT` - [ ] F2. **Code Quality Review** — `unspecified-high` Run `go vet ./...` + `golangci-lint run` + `go test ./... -race`. Review all .go files for: `any` type casts, empty catches, fmt.Println in prod, commented-out code, unused imports. Check AI slop: excessive comments, over-abstraction, generic names. **Acceptance Criteria**: - [ ] `go vet ./...` passes with zero warnings - [ ] `golangci-lint run` passes with zero warnings - [ ] `go test ./... -race` passes with zero races - [ ] No `fmt.Println` or `log.Println` in non-test files - [ ] Output: `Build [PASS/FAIL] | Lint [PASS/FAIL] | Tests [N pass/0 fail] | Files [N clean/0 issues] | VERDICT` - [ ] F3. **Real Manual QA** — `unspecified-high` Build the project. Run ALL QA scenarios from tasks 1-21. Test cross-module integration: write data with Put → Close → Open → Get verifies data. Test edge cases: empty value, large batch (9999 entries), concurrent writes from 10 goroutines, segment rotation after filling segment. Save output to `.omo/evidence/final-qa/`. **Acceptance Criteria**: - [ ] Every QA scenario from tasks 1-21 executed and passing - [ ] Cross-module integration: Put → Close → Open → Get round-trip - [ ] Edge case: empty value returns Found=true - [ ] Edge case: key not found returns Found=false - [ ] Edge case: concurrent 10-goroutine write/read with -race clean - [ ] Output: `Scenarios [N/N pass] | Integration [3/3] | Edge Cases [4 tested] | VERDICT` - [ ] F4. **Scope Fidelity Check** — `deep` For each task: read "What to do", read actual diff (git diff). Verify 1:1 — everything in spec was built (no missing), nothing beyond spec was built (no creep). Check "Must NOT do" compliance per task. Detect cross-task contamination: Task N touching Task M's files without justification. Flag unaccounted files. **Acceptance Criteria**: - [ ] Each task's "What to do" matches its actual implementation - [ ] No SSTable/Compaction/ValueLog/BloomFilter code found - [ ] No bufio.Writer import in wal/ package - [ ] No Periodic/Never sync code (only Always) - [ ] No cross-task file contamination - [ ] Output: `Tasks [21/21 compliant] | Contamination [CLEAN] | Unaccounted [CLEAN] | VERDICT` --- ## Commit Strategy | Tasks | Commit Message | Pre-commit Check | |-------|---------------|-----------------| | 1-2 | `feat: initialize project structure and error types` | `go build ./...` | | 3-5 | `feat(wal): implement WAL entry/record/header codec` | `go test ./wal/... -run TestCodec` | | 6, 9 | `feat(wal): implement WAL batch codec and resource validation` | `go test ./wal/... -run TestBatch` | | 7-8 | `feat(memtable): implement Arena allocator and SkipList` | `go test ./memtable/... -race` | | 10-11 | `feat(wal): implement segment writer with rotation` | `go test ./wal/... -run TestSegment` | | 12-13 | `feat(wal): implement WAL writer with group commit` | `go test ./wal/... -run TestWriter` | | 14 | `feat(memtable): integrate arena+skiplist with publish/abort` | `go test ./memtable/... -race` | | 15-17 | `feat(wal): implement WAL recovery` | `go test ./wal/... -run TestRecovery` | | 18 | `feat: implement MANIFEST stub and CURRENT file` | `go test ./manifest/...` | | 19 | `feat: integrate DB with Open/Close/Put/Delete/Get` | `go test ./... -race` | | 20 | `test: add end-to-end and crash recovery tests` | `go test ./... -race -count=1` | | 21 | `bench: add WAL performance benchmarks` | `go test -bench=. ./...` | --- ## Success Criteria ### Verification Commands ```bash go build ./... # Expected: success, no errors go vet ./... # Expected: no issues go test ./... -race -count=1 # Expected: all pass, no races go test ./... -run TestWALRoundtrip # Expected: write→close→open→read roundtrip go test ./... -run TestRecovery # Expected: crash recovery correct go test ./... -run TestGetSemantics # Expected: Found/empty-value distinction ``` ### Final Checklist - [ ] All "Must Have" present - [ ] All "Must NOT Have" absent - [ ] All tests pass with `-race` - [ ] WAL roundtrip test passes - [ ] Crash recovery test passes - [ ] No bufio.Writer on WAL append path - [ ] All atomics are typed (atomic.Uint64, atomic.Pointer) - [ ] Config invariant checked at Open