dailz
fede839eb6
- wal/batch.go: WalBatch encode/decode with FragmentCollector state machine - wal/validate.go: ValidateBatchLimits with checked arithmetic - memtable/arena.go: Arena allocator with 8-byte alignment and mutex - Comprehensive tests for all modules, all pass with -race
1719 lines
68 KiB
Markdown
1719 lines
68 KiB
Markdown
# Phase 1: WAL 子系统 — Go KV 存储引擎
|
||
|
||
## TL;DR
|
||
|
||
> **Quick Summary**: 基于 `docs/design.md` §3.2 实现 WAL 子系统，配合 MemTable（§3.3）、读路径（§3.4）和最小 MANIFEST，构建可嵌入的单 key autocommit KV 存储引擎。
|
||
>
|
||
> **Deliverables**:
|
||
> - WAL 写入路径（group commit、sequence 管理、segment 轮转、fsync）
|
||
> - WAL 恢复路径（segment 扫描、fragment 重组、batch 校验、尾部截断）
|
||
> - MemTable（Arena + SkipList + 原子发布）
|
||
> - 嵌入式 API（Open/Close/Put/Delete/Get）
|
||
> - MANIFEST 最小 stub + CURRENT 文件
|
||
> - 完整测试套件（单元 + 集成 + 故障注入）
|
||
>
|
||
> **Estimated Effort**: XL（~30-50 个任务）
|
||
> **Parallel Execution**: YES — 5 waves
|
||
> **Critical Path**: Task 1 → Task 6 → Task 10 → Task 11 → Task 13 → Task 19 → Task 20/21 → F1-F4
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
### Original Request
|
||
用户指向 `docs/design.md` §3.2（WAL），要求制定第一阶段开发方案。
|
||
|
||
### Interview Summary
|
||
**Key Discussions**:
|
||
- 设计文档是唯一权威来源（1189 行，§3.2 WAL 约 730 行）
|
||
- 项目为 greenfield — 无任何 Go 源码
|
||
- Phase 1 范围：WAL + MemTable + 读路径 + MANIFEST stub + 嵌入式 API
|
||
- 不含 SSTable / Compaction / Value Log / MVCC / 网络层
|
||
|
||
**Research Findings**:
|
||
- 项目完全空白（仅有 README.md、.omo/、docs/）
|
||
- 无 go.mod、无依赖
|
||
- 设计文档定义了极其精确的崩溃语义和错误分类
|
||
|
||
### Metis Review
|
||
**Identified Gaps** (addressed):
|
||
- MANIFEST 完整设计不在文档中 → Phase 1 用最小 stub（仅存 recoverySegmentID）
|
||
- MemTable 无 SSTable 可刷 → 冻结为 Immutable 后持有在内存，不刷盘；3×64MB 上限后阻塞写入
|
||
- Periodic/Never sync 策略 → Phase 1 只实现 Always
|
||
- Group commit 定时器 → time.AfterFunc + reset
|
||
- API surface → Put/Delete/Get/Open/Close + GetDurableSequence + IsWriteStopped
|
||
- Graceful shutdown → Close() 等待 pending batch、sync WAL、写 MANIFEST
|
||
- 文件锁 → Phase 1 不实现，单进程假设
|
||
- Varint → Go 标准库 encoding/binary.PutUvarint，key/value 上限 4KB 保证 varint ≤ 5 bytes
|
||
|
||
---
|
||
|
||
## Work Objectives
|
||
|
||
### Core Objective
|
||
实现一个功能完整的 WAL 子系统，支持 group commit 写入、崩溃恢复、和单 key autocommit 语义，作为后续 SSTable / MVCC 的基础。
|
||
|
||
### Concrete Deliverables
|
||
- `wal/` — WAL 编解码、segment 管理、写入器、恢复器
|
||
- `memtable/` — Arena + SkipList + 原子发布
|
||
- `db.go` — DB 入口（Open/Close/Put/Delete/Get）
|
||
- `errors.go` — 公共错误类型（ErrCommitUnknown 等）
|
||
- `manifest/` — 最小 MANIFEST stub
|
||
- 测试覆盖所有关键路径
|
||
|
||
### Definition of Done
|
||
- [ ] `go build ./...` 编译通过
|
||
- [ ] `go test ./... -race` 全部通过，无 data race
|
||
- [ ] 写入 N 条数据 → Close → Open → Get 全部正确
|
||
- [ ] WAL 尾部损坏 → 恢复成功，完整 batch 不丢失
|
||
- [ ] ErrCommitUnknown 后引擎进入 write-stopped
|
||
|
||
### Must Have
|
||
- WAL 写入路径严格遵循设计文档步骤 ①-⑪
|
||
- Per-batch private encode buffer + direct write（无 bufio.Writer）
|
||
- `atomic.Uint64` 用于 publishedSequence / durableSequence / nextSequence
|
||
- `atomic.Pointer` 用于 skiplist next 指针
|
||
- MemTable Arena 容量预留发生在 WAL write 之前
|
||
- Sequence 分配前完成所有可失败校验
|
||
- WAL Batch 不跨 segment
|
||
- Segment rotation 只在 batch 边界
|
||
- `Get` 返回 `GetResult{Found bool, Value []byte}` 区分 key-not-found 与 empty-value
|
||
- Config 不变量在 Open 时校验（checked arithmetic）
|
||
- Recovery 严格遵循 fragment 状态机
|
||
- 错误分类严格遵循设计文档表格
|
||
|
||
### Must NOT Have (Guardrails)
|
||
- ❌ SSTable / Compaction / Bloom Filter / Block Cache
|
||
- ❌ Value Log / ValueLogPointer 写入（Entry 类型定义可预留，但 Phase 1 不写入 ValueLogPointer）
|
||
- ❌ MVCC / SSI 事务
|
||
- ❌ Periodic / Never sync 策略（仅 Always）
|
||
- ❌ Range scan / Iterator API
|
||
- ❌ 网络层
|
||
- ❌ `bufio.Writer` 在 WAL append 路径上
|
||
- ❌ MemTable 写入失败发生在 WAL write 成功之后
|
||
- ❌ CRC 校验跳过
|
||
- ❌ 共享 buffered writer 模糊 WAL 副作用边界
|
||
|
||
---
|
||
|
||
## Verification Strategy
|
||
|
||
> **ZERO HUMAN INTERVENTION** — ALL verification is agent-executed. No exceptions.
|
||
|
||
### Test Decision
|
||
- **Infrastructure exists**: NO (greenfield)
|
||
- **Automated tests**: Tests-after（先实现后测试，每个模块实现完即写测试）
|
||
- **Framework**: Go 标准 `testing` + `testify`（断言库）
|
||
- **Agent-Executed QA**: 每个任务包含 `go test` 命令和 `go test -race` 验证
|
||
|
||
### QA Policy
|
||
- 每个 task 的 QA 通过 `go test` 和 `go vet` 执行
|
||
- Evidence: test output saved to `.omo/evidence/task-{N}-{scenario}.txt`
|
||
|
||
---
|
||
|
||
## Execution Strategy
|
||
|
||
### Parallel Execution Waves
|
||
|
||
```
|
||
Wave 1a (Foundation — 1 task):
|
||
└── 1. Project scaffolding + go.mod + config types [quick]
|
||
|
||
Wave 1b (Codec + Types — 4 parallel + 2 independent, after Wave 1a):
|
||
├── 2. Error types + constants [quick]
|
||
├── 3. WAL File Header encode/decode [quick]
|
||
├── 4. Physical Record encode/decode + Block boundary [quick]
|
||
├── 5. WAL Entry encode/decode [quick]
|
||
├── 12. Sequence manager [quick]
|
||
└── 18. MANIFEST stub + CURRENT file [quick]
|
||
|
||
Wave 2a (Batch + Arena + Validation — 3 parallel, after Wave 1b):
|
||
├── 6. WAL Batch encode/decode + fragment collector [unspecified-high] (needs 3,4,5)
|
||
├── 7. Arena allocator [unspecified-high] (needs 1)
|
||
└── 9. WAL Batch resource validation [quick] (needs 1, 5)
|
||
|
||
Wave 2b (SkipList — after Arena):
|
||
└── 8. SkipList (mutex write + lock-free read) [deep] (needs 7)
|
||
|
||
Wave 3a (Segment writer — after Batch codec):
|
||
└── 10. Segment writer (file write + block writer) [unspecified-high] (needs 3,4,6)
|
||
|
||
Wave 3b (Segment rotation — after Segment writer):
|
||
└── 11. Segment rotation + durable-ready protocol [unspecified-high] (needs 10)
|
||
|
||
Wave 3c (MemTable — after Arena + SkipList):
|
||
└── 14. MemTable (wrap skiplist + arena + publish/abort) [unspecified-high] (needs 7,8)
|
||
|
||
Wave 3d (WAL Writer — after MemTable API available):
|
||
└── 13. Commit queue + WAL writer main loop [deep] (needs 2,9,10,11,12,14)
|
||
|
||
Wave 4a (Recovery scanner — after Segment writer):
|
||
└── 15. WAL recovery: scanner + record parser [unspecified-high] (needs 4,10)
|
||
|
||
Wave 4b (Recovery batch replay — after scanner + Batch codec):
|
||
└── 16. WAL recovery: fragment collector + batch replay [deep] (needs 6,15)
|
||
|
||
Wave 4c (Recovery tail truncation — after batch replay):
|
||
└── 17. WAL recovery: tail truncation + main flow [unspecified-high] (needs 15,16)
|
||
|
||
Wave 4d (DB integration — after all above):
|
||
└── 19. DB integration: Open/Close/Put/Delete/Get [deep] (needs 13,14,17,18)
|
||
|
||
Wave 5 (Tests + Benchmark — after DB integration):
|
||
├── 20. End-to-end + crash recovery tests [deep]
|
||
└── 21. Benchmark suite [unspecified-high]
|
||
|
||
Wave FINAL (4 parallel reviews):
|
||
├── F1. Plan compliance audit (oracle)
|
||
├── F2. Code quality review (unspecified-high)
|
||
├── F3. Real manual QA (unspecified-high)
|
||
└── F4. Scope fidelity check (deep)
|
||
|
||
Critical Path: 1 → 6 → 10 → 11 → 13 → 19 → 20/21 → F1-F4
|
||
Parallel Speedup: ~50% faster than sequential
|
||
Max Concurrent: 6 (Wave 1b)
|
||
```
|
||
|
||
### Dependency Matrix
|
||
|
||
| Task | Depends On | Blocks | Sub-Wave |
|
||
|------|-----------|--------|----------|
|
||
| 1 | — | 2-5, 7, 9, 12, 18 | 1a |
|
||
| 2 | 1 | 13 | 1b |
|
||
| 3 | 1 | 6, 10 | 1b |
|
||
| 4 | 1 | 6, 10, 15 | 1b |
|
||
| 5 | 1 | 6, 9 | 1b |
|
||
| 6 | 3, 4, 5 | 10, 16 | 2a |
|
||
| 7 | 1 | 8, 14 | 2a |
|
||
| 8 | 7 | 14 | 2b |
|
||
| 9 | 1, 5 | 13 | 2a |
|
||
| 10 | 3, 4, 6 | 11, 15 | 3a |
|
||
| 11 | 10 | 13 | 3b |
|
||
| 12 | 1 | 13 | 1b |
|
||
| 13 | 2, 9, 10, 11, 12, 14 | 19 | 3d |
|
||
| 14 | 7, 8 | 19 | 3c |
|
||
| 15 | 4, 10 | 16 | 4a |
|
||
| 16 | 6, 15 | 17 | 4b |
|
||
| 17 | 15, 16 | 19 | 4c |
|
||
| 18 | 1 | 19 | 1b |
|
||
| 19 | 13, 14, 17, 18 | 20, 21 | 4d |
|
||
| 20 | 19 | F1-F4 | 5 |
|
||
| 21 | 19 | F1-F4 | 5 |
|
||
|
||
### Agent Dispatch Summary
|
||
|
||
- **1a**: **1** — T1 → `quick`
|
||
- **1b**: **6** — T2 → `quick`, T3 → `quick`, T4 → `quick`, T5 → `quick`, T12 → `quick`, T18 → `quick`
|
||
- **2a**: **3** — T6 → `unspecified-high`, T7 → `unspecified-high`, T9 → `quick`
|
||
- **2b**: **1** — T8 → `deep`
|
||
- **3a**: **1** — T10 → `unspecified-high`
|
||
- **3b**: **1** — T11 → `unspecified-high`
|
||
- **3c**: **1** — T14 → `unspecified-high`
|
||
- **3d**: **1** — T13 → `deep`
|
||
- **4a**: **1** — T15 → `unspecified-high`
|
||
- **4b**: **1** — T16 → `deep`
|
||
- **4c**: **1** — T17 → `unspecified-high`
|
||
- **4d**: **1** — T19 → `deep`
|
||
- **5**: **2** — T20 → `deep`, T21 → `unspecified-high`
|
||
- **FINAL**: **4** — F1 → `oracle`, F2 → `unspecified-high`, F3 → `unspecified-high`, F4 → `deep`
|
||
|
||
---
|
||
|
||
## TODOs
|
||
|
||
- [x] 1. Project scaffolding + go.mod + config types
|
||
|
||
**What to do**:
|
||
- 初始化 `go.mod`（模块名 `github.com/dailz/go-kv`，Go 1.22+）
|
||
- 创建目录结构：`wal/`、`memtable/`、`manifest/`、`config/`
|
||
- 创建 `config/config.go`：`WalConfig` 结构体（MaxSegmentSize, BlockSize, SyncMode, MaxBatchEntries, MaxBatchSize, MaxKeyBytes, MaxInlineValue, MemTableSize, MaxImmutableCount）
|
||
- 创建 `config/config.go`：`Validate() error` 函数，用 checked arithmetic 校验不变量：`maxWalSegmentPayload >= maxEncodedWalBatchSize + worstCasePhysicalRecordOverhead + worstCaseBlockPadding`
|
||
- 计算公式参照设计文档 lines 402-453：walFileHeaderSize=32, walBlockSize=32768, physicalRecordHeaderSize=7, walBatchHeaderSize=18, maxWalBatchEntriesSize=4MB
|
||
- 创建 `.golangci.yml`
|
||
|
||
**Must NOT do**:
|
||
- 不引入外部依赖（仅标准库 + testify）
|
||
- 不定义 SSTable / Compaction 相关配置
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: [`golang-project-layout`]
|
||
- `golang-project-layout`: Go 项目结构最佳实践
|
||
- **Skills Evaluated but Omitted**:
|
||
- `golang-cli`: 不是 CLI 工具
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (foundation task)
|
||
- **Parallel Group**: Sub-Wave 1a (solo — blocks all others)
|
||
- **Blocks**: 2, 3, 4, 5, 7, 9, 12, 18
|
||
- **Blocked By**: None
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:286-289` — 设计决策表：批量窗口、日志格式、文件管理、segment 大小
|
||
- `docs/design.md:402-453` — 配置不变量计算公式和最小 segment 大小推导
|
||
|
||
**API/Type References**:
|
||
- `docs/design.md:501-513` — WAL Batch 资源上限默认值
|
||
|
||
**External References**:
|
||
- Go project layout: https://go.dev/doc/modules/layout
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 设计决策表定义了所有默认值，config 结构体必须映射这些值
|
||
- 配置不变量公式是 Open 时必须校验的核心约束
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Config validation accepts valid defaults
|
||
Tool: Bash
|
||
Preconditions: Go toolchain installed
|
||
Steps:
|
||
1. cd /home/dailz/workspace/src/go-kv
|
||
2. Create a test file config/config_test.go with TestValidateDefaults
|
||
3. Test that WalConfig with default values passes Validate()
|
||
Expected Result: go test ./config/... passes
|
||
Evidence: .omo/evidence/task-1-config-defaults.txt
|
||
|
||
Scenario: Config validation rejects invalid segment size
|
||
Tool: Bash
|
||
Preconditions: go.mod exists
|
||
Steps:
|
||
1. Create TestValidateSegmentTooSmall in config/config_test.go
|
||
2. Set MaxSegmentSize = 1024 (far below minimum 4,195,264)
|
||
3. Verify Validate() returns error containing "segment"
|
||
Expected Result: Test fails validation with descriptive error
|
||
Evidence: .omo/evidence/task-1-config-invalid.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Task 2)
|
||
- Message: `feat: initialize project structure and error types`
|
||
- Files: `go.mod, config/, .golangci.yml`
|
||
|
||
- [x] 2. Error types + constants
|
||
|
||
**What to do**:
|
||
- 创建 `errors.go`：定义 `ErrCommitUnknown`、`ErrWriteStopped`、`ErrSequenceExhausted`、`ErrWALCorrupted`、`ErrInvalidConfig`
|
||
- 创建 `wal/constants.go`：所有 WAL 常量（WalMagic, WalFormatVersion, WalFileHeaderSize, WalBlockSize, PhysicalRecordHeaderSize, WalBatchHeaderSize, MaxWalBatchEntryCount, MaxWalBatchEntriesSize, MaxWalKeyBytes, MaxWalInlineValueBytes, MaxWalVarintBytes, DefaultMaxWalSegmentSize）
|
||
- 创建 `wal/constants.go`：Fragment types (RecInvalid=0, RecFull=1, RecFirst=2, RecMiddle=3, RecLast=4)
|
||
- 创建 `wal/constants.go`：OpType (OpInvalid=0, OpPut=1, OpDelete=2) 和 ValueKind (VKNone=0, VKInline=1, VKValueLogPointer=2)
|
||
|
||
**Must NOT do**:
|
||
- 不实现具体逻辑，仅类型和常量定义
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
- **Skills Evaluated but Omitted**:
|
||
- `golang-naming`: 常量命名简单，不需要完整命名 skill
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 1b (with Tasks 3, 4, 5, 12, 18)
|
||
- **Blocks**: 13 (WAL writer references error types)
|
||
- **Blocked By**: 1 (needs go.mod)
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:129-135` — 错误分类表（5 种失败场景 + ErrCommitUnknown）
|
||
- `docs/design.md:155` — ErrCommitUnknown 的 maybe committed 语义
|
||
- `docs/design.md:496-499` — sequence exhausted 终态
|
||
|
||
**API/Type References**:
|
||
- `docs/design.md:501-513` — 所有资源上限默认值
|
||
- `docs/design.md:364-373` — Fragment type 枚举
|
||
- `docs/design.md:571-577` — OpType 枚举
|
||
- `docs/design.md:580-584` — ValueKind 枚举
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 错误类型直接映射设计文档的错误分类表
|
||
- 常量值必须与设计文档精确匹配（block size=32KB, header=32 bytes 等）
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Error types are distinguishable via errors.Is
|
||
Tool: Bash
|
||
Preconditions: Task 1 complete
|
||
Steps:
|
||
1. Create errors_test.go with TestErrorTypes
|
||
2. Verify errors.Is(ErrCommitUnknown, ErrCommitUnknown) == true
|
||
3. Verify errors.Is(ErrWriteStopped, ErrCommitUnknown) == false
|
||
Expected Result: All error type identity checks pass
|
||
Evidence: .omo/evidence/task-2-error-types.txt
|
||
|
||
Scenario: Constants match design doc values
|
||
Tool: Bash
|
||
Preconditions: wal/constants.go exists
|
||
Steps:
|
||
1. Create wal/constants_test.go with TestConstantValues
|
||
2. Assert WalBlockSize == 32*1024
|
||
3. Assert WalFileHeaderSize == 32
|
||
4. Assert MaxWalBatchEntriesSize == 4*1024*1024
|
||
5. Assert DefaultMaxWalSegmentSize == 64*1024*1024
|
||
Expected Result: All assertions pass
|
||
Evidence: .omo/evidence/task-2-constants.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Task 1)
|
||
- Message: `feat: initialize project structure and error types`
|
||
- Files: `errors.go, wal/constants.go`
|
||
|
||
- [x] 3. WAL File Header encode/decode
|
||
|
||
**What to do**:
|
||
- 创建 `wal/header.go`：`WalFileHeader` 结构体（Magic uint32, FormatVersion uint16, HeaderSize uint16, BlockSize uint32, SegmentID uint64, StartSequence uint64, HeaderCRC uint32）
|
||
- `EncodeWalHeader(h *WalFileHeader) [WalFileHeaderSize]byte` — 编码为 little-endian 32 bytes，headerCRC 覆盖 magic 到 startSequence（不包含自身）
|
||
- `DecodeWalHeader(data []byte) (*WalFileHeader, error)` — 校验 magic、formatVersion、headerSize、headerCRC
|
||
- CRC 使用 `hash/crc32` with IEEE table（或 CRC32C 如果标准库支持）
|
||
- 创建 table-driven tests
|
||
|
||
**Must NOT do**:
|
||
- 不使用外部 CRC 库
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 1b (with Tasks 2, 4, 5, 12, 18)
|
||
- **Blocks**: 6 (batch codec), 10 (segment writer)
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:316-341` — WAL File Header 格式定义和字段说明
|
||
|
||
**WHY Each Reference Matters**:
|
||
- Header 格式是 WAL segment 文件的基础，所有后续 segment 操作都依赖正确编码
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Header encode/decode round-trip
|
||
Tool: Bash
|
||
Preconditions: wal/header.go exists
|
||
Steps:
|
||
1. Create TestHeaderRoundtrip in wal/header_test.go
|
||
2. Encode a header with SegmentID=5, StartSequence=1000
|
||
3. Decode the encoded bytes
|
||
4. Assert all fields match original
|
||
Expected Result: Round-trip preserves all fields
|
||
Evidence: .omo/evidence/task-3-header-roundtrip.txt
|
||
|
||
Scenario: Header CRC detects corruption
|
||
Tool: Bash
|
||
Steps:
|
||
1. Encode a valid header
|
||
2. Flip one byte in the magic field
|
||
3. Attempt DecodeWalHeader
|
||
4. Assert error returned containing "crc" or "checksum"
|
||
Expected Result: Corrupted header rejected
|
||
Evidence: .omo/evidence/task-3-header-crc.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Tasks 4, 5)
|
||
- Message: `feat(wal): implement WAL entry/record/header codec`
|
||
|
||
- [x] 4. Physical Record encode/decode + Block boundary
|
||
|
||
**What to do**:
|
||
- 创建 `wal/record.go`：`PhysicalRecord` 结构体（CRC uint32, Length uint16, Type uint8, Payload []byte）
|
||
- `EncodePhysicalRecord(recType uint8, payload []byte) []byte` — 编码为 7-byte header + payload
|
||
- `DecodePhysicalRecord(data []byte) (rec *PhysicalRecord, consumed int, err error)` — 解析并校验 CRC
|
||
- CRC 覆盖 length + type + payload（不含 CRC 自身）
|
||
- Block 边界辅助函数：
|
||
- `PaddingNeeded(blockOffset uint32) int` — 剩余 <= 7 bytes 时返回 padding 大小
|
||
- `CanFitRecord(blockOffset uint32, payloadLen uint32) bool`
|
||
- `SplitIntoRecords(encodedBatch []byte) [][]byte` — 将编码后的 batch 按 block 边界拆分为 Physical Record payloads
|
||
- Table-driven tests 覆盖：单 record、跨 block 拆分、padding
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 1b (with Tasks 2, 3, 5, 12, 18)
|
||
- **Blocks**: 6, 10, 15
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:345-381` — Physical Record 格式、fragment 类型、合法组合
|
||
- `docs/design.md:383-391` — Block 边界处理规则
|
||
|
||
**WHY Each Reference Matters**:
|
||
- Physical Record 是 WAL I/O 的最小单元，block 边界处理是拆分/重组的关键
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Physical Record round-trip
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestRecordRoundtrip in wal/record_test.go
|
||
2. Encode RecFull with payload "hello world"
|
||
3. Decode the encoded bytes
|
||
4. Assert Type==RecFull, Payload=="hello world"
|
||
Expected Result: Round-trip correct
|
||
Evidence: .omo/evidence/task-4-record-roundtrip.txt
|
||
|
||
Scenario: Batch split into fragments across blocks
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestSplitIntoRecords in wal/record_test.go
|
||
2. Create a 40KB payload (larger than 32KB block)
|
||
3. Call SplitIntoRecords with blockSize=32KB
|
||
4. Assert result has First + Last (or First + Middle + Last)
|
||
5. Concatenate all payloads → equals original
|
||
Expected Result: Split produces valid fragment sequence, reassembly equals original
|
||
Evidence: .omo/evidence/task-4-split-records.txt
|
||
|
||
Scenario: Block padding when remaining <= 7 bytes
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestBlockPadding in wal/record_test.go
|
||
2. blockOffset = 32*1024 - 5 (only 5 bytes remaining)
|
||
3. Assert PaddingNeeded(blockOffset) == 5
|
||
4. Assert CanFitRecord(blockOffset, 1) == false
|
||
Expected Result: Padding and fit check correct
|
||
Evidence: .omo/evidence/task-4-padding.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Tasks 3, 5)
|
||
|
||
- [x] 5. WAL Entry encode/decode
|
||
|
||
**What to do**:
|
||
- 创建 `wal/entry.go`：`WalEntry` 结构体（OpType uint8, ValueKind uint8, Key []byte, Value []byte）
|
||
- `EncodeEntry(e *WalEntry) ([]byte, error)` — 编码 opType(u8) + valueKind(u8) + keyLen(varint) + valLen(varint) + key + value
|
||
- `DecodeEntry(data []byte) (entry *WalEntry, consumed int, err error)` — 解码返回 entry 和 consumed bytes
|
||
- Varint 使用 `encoding/binary.PutUvarint` / `binary.ReadUvarint`
|
||
- 校验规则（参照 design.md lines 586-594）：
|
||
- keyLen > 0 && keyLen <= MaxWalKeyBytes
|
||
- Put 要求 valueKind ∈ {VKInline, VKValueLogPointer}
|
||
- Put + VKInline: valLen <= MaxWalInlineValueBytes (允许 valLen = 0)
|
||
- Put + VKValueLogPointer: valLen > 0
|
||
- Delete: valueKind == VKNone, valLen == 0
|
||
- Table-driven tests 覆盖所有合法/非法组合
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 1b (with Tasks 2, 3, 4, 12, 18)
|
||
- **Blocks**: 6, 9
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:558-594` — Entry 格式、OpType、ValueKind、合法性规则
|
||
|
||
**WHY Each Reference Matters**:
|
||
- Entry 是 WAL 的最小逻辑单元，编解码正确性直接影响数据完整性
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Entry encode/decode round-trip for all valid types
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestEntryRoundtrip in wal/entry_test.go
|
||
2. Test cases: Put+Inline("key1","val1"), Put+Inline("key2",[]byte{}), Delete("key3")
|
||
3. Encode → Decode → assert all fields match
|
||
Expected Result: All round-trips correct
|
||
Evidence: .omo/evidence/task-5-entry-roundtrip.txt
|
||
|
||
Scenario: Entry validation rejects invalid entries
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestEntryValidation in wal/entry_test.go
|
||
2. Test: OpInvalid → error
|
||
3. Test: Put+VKNone → error
|
||
4. Test: Delete+VKInline → error
|
||
5. Test: keyLen=0 → error
|
||
6. Test: keyLen > MaxWalKeyBytes → error
|
||
Expected Result: All invalid entries rejected with descriptive errors
|
||
Evidence: .omo/evidence/task-5-entry-validation.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Tasks 3, 4)
|
||
|
||
- [x] 6. WAL Batch encode/decode + fragment collector
|
||
|
||
**What to do**:
|
||
- 创建 `wal/batch.go`：`WalBatch` 结构体（Flags uint16, BaseSequence uint64, EntryCount uint32, EntriesSize uint32, Entries []byte）
|
||
- `EncodeWalBatch(baseSequence uint64, entries []*WalEntry) ([]byte, error)` — 编码 Batch Header + 序列化 Entries
|
||
- `DecodeWalBatch(data []byte) (*WalBatch, error)` — 校验 flags、entryCount（0 < N <= max）、entriesSize（与实际 bytes 匹配）
|
||
- `FragmentCollector` 结构体 — 实现 Idle / CollectingFragments 状态机
|
||
- `Reset()`
|
||
- `Append(recType uint8, payload []byte) error` — 按 fragment type 追加，校验 buffer 大小上限
|
||
- `IsComplete() bool` — 是否拼出完整 batch
|
||
- `BatchData() []byte` — 返回重组后的完整 batch bytes
|
||
- `State() FragmentState` — 当前状态（Idle / CollectingFragments）
|
||
- 状态转移严格遵循 design.md lines 706-731 的状态机
|
||
- Buffer 大小限制：Batch Header 长度 + MaxWalBatchEntriesSize
|
||
- Varint 长度限制：MaxWalVarintBytes = 5
|
||
|
||
**Must NOT do**:
|
||
- FragmentCollector 不做 CRC 校验（由 Physical Record 层负责）
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES (with Tasks 7, 9 — all depend only on Wave 1b outputs)
|
||
- **Parallel Group**: Sub-Wave 2a (with Tasks 7, 9)
|
||
- **Blocks**: 10, 16
|
||
- **Blocked By**: 3, 4, 5
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:455-530` — WAL Batch 格式、Batch Header、Entry sequence 推导
|
||
- `docs/design.md:706-731` — Fragment 状态机定义
|
||
|
||
**WHY Each Reference Matters**:
|
||
- Batch 是 WAL 的核心持久化单元，FragmentCollector 是 recovery 的核心组件
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Batch encode/decode round-trip with multiple entries
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestBatchRoundtrip in wal/batch_test.go
|
||
2. Create 10 entries: 5 Put+Inline, 3 Delete, 2 Put+Inline with empty value
|
||
3. Encode with baseSequence=42
|
||
4. Decode → assert Flags, BaseSequence, EntryCount, EntriesSize match
|
||
5. Parse all entries from Entries bytes → assert each matches original
|
||
Expected Result: Full round-trip preserves all data
|
||
Evidence: .omo/evidence/task-6-batch-roundtrip.txt
|
||
|
||
Scenario: Fragment collector state machine
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestFragmentCollector in wal/batch_test.go
|
||
2. Encode a 100KB batch, split into records (First + Middle* + Last)
|
||
3. Feed fragments to collector one by one
|
||
4. Assert state transitions: Idle → CollectingFragments → ... → Idle
|
||
5. Assert collected batch bytes equal original encoded batch
|
||
Expected Result: Fragment reassembly produces original batch
|
||
Evidence: .omo/evidence/task-6-fragment-collector.txt
|
||
|
||
Scenario: Fragment collector rejects illegal transitions
|
||
Tool: Bash
|
||
Steps:
|
||
1. Feed RecMiddle to Idle state → expect error
|
||
2. Feed RecLast to Idle state → expect error
|
||
3. Feed RecFull to CollectingFragments state → expect error
|
||
4. Feed RecFirst to CollectingFragments state → expect error
|
||
Expected Result: All illegal transitions rejected
|
||
Evidence: .omo/evidence/task-6-fragment-illegal.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Task 9)
|
||
- Message: `feat(wal): implement WAL batch codec and resource validation`
|
||
|
||
- [x] 7. Arena allocator
|
||
|
||
**What to do**:
|
||
- 创建 `memtable/arena.go`：`Arena` 结构体
|
||
- 固定大小 byte slice（默认 64MB，由 config.MemTableSize 配置）
|
||
- `Allocate(size uint32) (offset uint32, err error)` — 分配对齐的内存块，返回 offset；满时返回 error
|
||
- `GetBytes(offset uint32, size uint32) []byte` — 从 offset 获取 bytes slice
|
||
- `Remaining() uint32` — 剩余可用 bytes
|
||
- `Capacity() uint32` — 总容量
|
||
- `Reserve(totalSize uint32) error` — 检查剩余容量是否足够（不实际分配）
|
||
- 对齐：8-byte 对齐分配
|
||
- 线程安全：分配使用 sync.Mutex（写路径已被 WAL writer 串行化）
|
||
- 测试：分配、对齐、满时错误、并发安全
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: [`golang-concurrency`]
|
||
- `golang-concurrency`: Arena 需要线程安全分配
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 2a (with Tasks 6, 9 — all depend only on Task 1 or Wave 1b outputs)
|
||
- **Blocks**: 8, 14
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:864-879` — Arena 容量预留规则
|
||
|
||
**WHY Each Reference Matters**:
|
||
- Arena 是 MemTable 的内存基础，容量预留是 WAL 写入路径的前置约束
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Arena allocate and read back
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestArenaAllocate in memtable/arena_test.go
|
||
2. Create Arena with 1024 bytes
|
||
3. Allocate 100 bytes → write "test data" at offset 0
|
||
4. GetBytes(offset, 9) → assert equals "test data"
|
||
5. Allocate another 200 bytes → assert offset is 8-byte aligned
|
||
Expected Result: Allocation and readback correct, alignment enforced
|
||
Evidence: .omo/evidence/task-7-arena-alloc.txt
|
||
|
||
Scenario: Arena full returns error
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create Arena with 64 bytes
|
||
2. Allocate 60 bytes → success
|
||
3. Allocate 10 bytes → error (not enough remaining)
|
||
Expected Result: Full arena rejects allocation
|
||
Evidence: .omo/evidence/task-7-arena-full.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 8)
|
||
- Message: `feat(memtable): implement Arena allocator and SkipList`
|
||
|
||
- [ ] 8. SkipList (mutex write + lock-free read)
|
||
|
||
**What to do**:
|
||
- 创建 `memtable/skiplist.go`：`SkipList` 结构体
|
||
- 最大 20 层，p=0.25 的概率晋升
|
||
- `Put(key []byte, value []byte, sequence uint64, nodeData interface{}) error` — Mutex 保护下的有序插入
|
||
- `Get(key []byte) (found bool, value []byte, sequence uint64)` — 无锁读
|
||
- `NewIterator() *Iterator` — 有序遍历
|
||
- **原子发布**：所有 next 指针使用 `atomic.Pointer[skipNode]`，store 使用 release 语义
|
||
- 节点包含：key、value、sequence、pending 标记（用 atomic.Bool 或存入 nodeData）
|
||
- Key 比较使用 bytes.Compare
|
||
- Arena 分配：节点在 Arena 中分配（offset-based 引用，通过 Arena.GetBytes 访问）
|
||
- 测试：有序插入、查找、遍历、并发读写（`-race`）
|
||
|
||
**Must NOT do**:
|
||
- 不使用普通指针（必须 atomic.Pointer）
|
||
- 不依赖 Mutex 向读者发布内存（必须 atomic store-release）
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `deep`
|
||
- **Skills**: [`golang-concurrency`]
|
||
- `golang-concurrency`: 原子操作和内存序是核心难点
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on Task 7)
|
||
- **Parallel Group**: Sub-Wave 2b (solo — after Arena complete)
|
||
- **Blocks**: 14
|
||
- **Blocked By**: 7
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:834-840` — SkipList 设计决策（20 层、64MB Arena、Mutex 写 + 无锁读）
|
||
- `docs/design.md:841-850` — 发布与内存序约束（atomic.Pointer, release store）
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 内存序约束是整个读路径正确性的关键，ARM 等弱内存序平台必须严格遵守
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: SkipList ordered insertion and retrieval
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestSkipListOrdered in memtable/skiplist_test.go
|
||
2. Insert keys "c", "a", "e", "b", "d" with values "val-C" etc.
|
||
3. Get each key → assert correct value
|
||
4. Iterate → assert order is a, b, c, d, e
|
||
Expected Result: Keys stored and retrieved in correct order
|
||
Evidence: .omo/evidence/task-8-skiplist-ordered.txt
|
||
|
||
Scenario: SkipList concurrent read/write no data race
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestSkipListConcurrent in memtable/skiplist_test.go
|
||
2. Start 4 writer goroutines inserting 100 keys each
|
||
3. Start 4 reader goroutines reading random keys
|
||
4. Run with go test -race
|
||
Expected Result: No data race detected
|
||
Evidence: .omo/evidence/task-8-skiplist-race.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 7)
|
||
|
||
- [x] 9. WAL Batch resource validation
|
||
|
||
**What to do**:
|
||
- 创建 `wal/validate.go`：`ValidateBatchLimits(entries []*WalEntry, cfg *config.WalConfig) error`
|
||
- 在 sequence 分配之前检查所有可失败条件：
|
||
- entryCount > 0 && entryCount <= cfg.MaxBatchEntries
|
||
- 每个 entry 的 keyLen > 0 && keyLen <= cfg.MaxKeyBytes
|
||
- 每个 entry 如果是 Put+Inline 则 valLen <= cfg.MaxInlineValue
|
||
- 所有 entries 编码后总大小 <= cfg.MaxBatchSize
|
||
- worstCase Physical Record overhead + batch encoded size <= maxWalSegmentPayload
|
||
- checked arithmetic：计算中不溢出
|
||
- 返回的具体错误类型允许区分不同违规
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Sub-Wave 2a (with Tasks 6, 7 — all depend on Task 1/Wave 1b)
|
||
- **Blocks**: 13
|
||
- **Blocked By**: 1, 5
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:501-513` — WAL Batch 资源上限
|
||
- `docs/design.md:514` — "写入侧必须在 sequence 分配、MemTable 预留和 WAL append 之前完成这些校验"
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 这些校验必须在 sequence 分配之前完成，否则失败会触发 write-stopped
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Valid batch passes validation
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestValidateBatchLimits in wal/validate_test.go
|
||
2. Create 100 Put+Inline entries with 10-byte keys and 10-byte values
|
||
3. Call ValidateBatchLimits → expect nil error
|
||
Expected Result: Valid batch accepted
|
||
Evidence: .omo/evidence/task-9-validate-pass.txt
|
||
|
||
Scenario: Batch exceeding entry count limit rejected
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create 10,001 entries (exceeds default MaxBatchEntries=10000)
|
||
2. Call ValidateBatchLimits → expect error containing "entry count"
|
||
Expected Result: Over-limit batch rejected before sequence allocation
|
||
Evidence: .omo/evidence/task-9-validate-entrycount.txt
|
||
|
||
Scenario: Key exceeding size limit rejected
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create entry with keyLen = 5KB (exceeds MaxWalKeyBytes=4KB)
|
||
2. Call ValidateBatchLimits → expect error containing "key"
|
||
Expected Result: Oversized key rejected
|
||
Evidence: .omo/evidence/task-9-validate-keysize.txt
|
||
```
|
||
|
||
**Commit**: YES (group with Task 6)
|
||
|
||
- [ ] 10. Segment writer (file write + block writer)
|
||
|
||
**What to do**:
|
||
- 创建 `wal/block_writer.go`：`BlockWriter` — 管理 32KB block 的 Physical Record 写入
|
||
- 内部维护当前 block 的 byte buffer
|
||
- 写入 Physical Record 时检查 block 边界
|
||
- 剩余 <= 7 bytes 时自动 padding（全 0）
|
||
- 跨 block 的 batch fragment 自动拆分
|
||
- `WriteRecord(recType uint8, payload []byte) error`
|
||
- `Flush(io.Writer) error` — 将当前 block 写入 fd
|
||
- `Reset()` — 重置为空 block
|
||
- 创建 `wal/segment_writer.go`：`SegmentWriter`
|
||
- 封装 WAL segment 文件的追加写入
|
||
- `NewSegmentWriter(dir string, segmentID uint64, startSequence uint64, cfg *config.WalConfig) (*SegmentWriter, error)` — 创建 segment-N.wal.tmp → 写 header → fsync → rename → fsync dir → durable-ready
|
||
- `AppendBatch(encodedBatch []byte) error` — 按 block 边界写入 batch（使用 BlockWriter）
|
||
- `Sync() error` — fsync 当前文件
|
||
- `Close() error`
|
||
- `RemainingPayload() uint64` — 当前 segment 剩余 payload
|
||
- `CurrentOffset() uint64`
|
||
- `SegmentID() uint64`
|
||
- `EndState() SegmentEndState` — 返回 (segmentID, endOffset, endSequence)
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on 3, 4, 6)
|
||
- **Parallel Group**: Sub-Wave 3a (solo — after batch codec complete)
|
||
- **Blocks**: 11, 15
|
||
- **Blocked By**: 3, 4, 6
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:247-278` — WAL 元数据持久化协议（durable-ready）
|
||
- `docs/design.md:383-391` — Block 边界处理规则
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Segment creation follows durable-ready protocol
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestSegmentCreation in wal/segment_writer_test.go
|
||
2. Create a temp directory, create SegmentWriter with segmentID=0
|
||
3. Verify segment-0.wal exists in directory
|
||
4. Read file → verify header is correct (magic, segmentID, startSequence)
|
||
5. Verify no .tmp file remains
|
||
Expected Result: Segment file created with correct header, no temp file
|
||
Evidence: .omo/evidence/task-10-segment-create.txt
|
||
|
||
Scenario: Batch written to segment correctly
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create SegmentWriter in temp dir
|
||
2. Encode a batch with 5 entries
|
||
3. Call AppendBatch(encodedBatch)
|
||
4. Sync + Close
|
||
5. Read file back → verify batch can be recovered
|
||
Expected Result: Written batch recoverable from segment file
|
||
Evidence: .omo/evidence/task-10-segment-write.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 11)
|
||
- Message: `feat(wal): implement segment writer with rotation`
|
||
|
||
- [ ] 11. Segment rotation + durable-ready protocol
|
||
|
||
**What to do**:
|
||
- 创建 `wal/segment_manager.go`：管理 segment 的轮转和生命周期
|
||
- 写入 batch 前检查 `RemainingPayload()` 是否足够容纳整个 batch
|
||
- 不足时：
|
||
1. 当前 segment 完成（在 batch 边界）
|
||
2. 创建新 segment：segment-N+1.wal.tmp → header → fsync → rename → fsync dir
|
||
3. 新 segment 的 startSequence = 下一个 expected sequence
|
||
4. 切换 active segment
|
||
- 不允许 batch 跨 segment
|
||
- CURRENT 文件 best-effort 更新（temp + rename）
|
||
- CURRENT 更新失败不影响已 durable-ready 的 segment
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on Task 10)
|
||
- **Parallel Group**: Sub-Wave 3b (solo — after segment writer)
|
||
- **Blocks**: 13
|
||
- **Blocked By**: 10
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:392-453` — Segment Rotation 约束
|
||
- `docs/design.md:247-278` — WAL 元数据持久化协议
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Segment rotation triggers when payload exceeds remaining space
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create a test WalConfig with: MaxBatchSize=256, MaxBatchEntries=10,
|
||
BlockSize=256, MaxSegmentSize=4096 (satisfies invariant: 4096 ≥ maxEncodedWalBatchSize
|
||
since MaxBatchSize=256 → maxEncoded ≈ 256 + 4*10 + overhead < 512)
|
||
2. Create SegmentManager with this config
|
||
3. Write small batches until segment-0 is nearly full (~3800 bytes written)
|
||
4. Write one more batch that would exceed remaining → triggers rotation
|
||
5. Verify segment-1.wal exists with correct header
|
||
6. Verify batch was written to segment-1, not segment-0
|
||
Expected Result: Rotation creates new segment, batch not split across segments
|
||
Failure Indicators: segment-1.wal does not exist, or batch data spans both segment files
|
||
Evidence: .omo/evidence/task-11-rotation.txt
|
||
|
||
Scenario: CURRENT file updated after rotation
|
||
Tool: Bash
|
||
Steps:
|
||
1. Trigger rotation using same test config as above
|
||
2. Read CURRENT file → verify it points to latest segment filename
|
||
Expected Result: CURRENT reflects latest active segment filename
|
||
Failure Indicators: CURRENT still points to segment-0, or file content is stale
|
||
Evidence: .omo/evidence/task-11-current.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 10)
|
||
|
||
- [x] 12. Sequence manager
|
||
|
||
**What to do**:
|
||
- 创建 `wal/sequence.go`：`SequenceManager`
|
||
- `nextSequence atomic.Uint64` — 下一个待分配 sequence
|
||
- `publishedSequence atomic.Uint64` — 已发布的连续 high-water mark
|
||
- `durableSequence atomic.Uint64` — 已满足 durable 条件的 high-water mark
|
||
- `AllocateBatch(count uint32) (baseSequence uint64, err error)` — checked arithmetic：`baseSequence + count - 1` 不溢出 uint64。溢出 → 返回 ErrSequenceExhausted
|
||
- `Publish(seq uint64)` — Store publishedSequence（在所有 entry 节点发布后调用）
|
||
- `MarkDurable(endState SegmentEndState)` — 推进 durableSequence
|
||
- `Published() uint64` — Load publishedSequence
|
||
- `Durable() uint64` — Load durableSequence
|
||
- `NextSequence() uint64` — Load nextSequence
|
||
- 恢复后初始化：`NewSequenceManager(recoveredSequence uint64)`
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Wave 1b (with Tasks 2, 3, 4, 5, 18 — all depend only on Task 1)
|
||
- **Blocks**: 13
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:486-499` — Sequence 运算和溢出检查
|
||
- `docs/design.md:205-223` — durableSequence 推进逻辑
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Sequence allocation returns consecutive bases
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestSequenceAllocation in wal/sequence_test.go
|
||
2. NewSequenceManager(0)
|
||
3. AllocateBatch(5) → base=0
|
||
4. AllocateBatch(3) → base=5
|
||
5. AllocateBatch(1) → base=8
|
||
Expected Result: Consecutive, no gaps
|
||
Evidence: .omo/evidence/task-12-seq-alloc.txt
|
||
|
||
Scenario: Sequence overflow returns ErrSequenceExhausted
|
||
Tool: Bash
|
||
Steps:
|
||
1. NewSequenceManager(math.MaxUint64 - 2)
|
||
2. AllocateBatch(5) → err (would overflow)
|
||
Expected Result: ErrSequenceExhausted returned
|
||
Evidence: .omo/evidence/task-12-seq-overflow.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 13)
|
||
|
||
- [ ] 13. Commit queue + WAL writer main loop
|
||
|
||
**What to do**:
|
||
- 创建 `wal/commit_queue.go`：`CommitQueue` — 写请求队列
|
||
- 每个 request 包含 entries、result channel
|
||
- `Submit(entries []*WalEntry) chan WriteResult` — 非阻塞提交
|
||
- `Collect(timeout time.Duration) []*CommitRequest` — 收集当前队列中的请求
|
||
- 创建 `wal/writer.go`：`WalWriter` — 核心写入循环
|
||
- 单 goroutine 运行 main loop
|
||
- Group commit：收集 queue 中的请求 → 组装 batch → 触发条件（500µs 或 32KB）
|
||
- 严格遵循 design.md steps ①-⑪：
|
||
1. 从 commit queue 收集写入
|
||
2. 组装 WAL Batch
|
||
3. ValidateBatchLimits → 失败则普通错误（不分配 sequence）
|
||
4. 计算 MemTable Arena 预留量 → 不足则 freeze/switch 或等待
|
||
5. AllocateBatch → 分配 baseSequence（此步之后任何失败 → write-stopped）
|
||
6. 私有缓冲编码 WAL Batch
|
||
7. 检查 segment rotation
|
||
8. segmentWriter.AppendBatch(encoded) → 失败 → ErrCommitUnknown + write-stopped
|
||
9. 写入 MemTable pending entries
|
||
10. segmentWriter.Sync() (Always 模式) → 失败 → ErrCommitUnknown + write-stopped
|
||
11. 原子发布 MemTable entries (atomic Pointer store-release)
|
||
12. Publish(baseSequence + entryCount - 1)
|
||
13. 唤醒所有等待的调用方
|
||
- `writeStopped atomic.Bool` — write-stopped 状态
|
||
- `Put(key, value []byte) error` — 提交到 commit queue 并等待结果
|
||
- `Delete(key []byte) error` — 同上
|
||
- `Close() error` — 等待 pending batch 完成
|
||
|
||
**Must NOT do**:
|
||
- 不使用 bufio.Writer（直接 os.File.Write）
|
||
- 不在 sequence 分配后做可失败的校验
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `deep`
|
||
- **Skills**: [`golang-concurrency`]
|
||
- `golang-concurrency`: commit queue、group commit、goroutine 生命周期管理
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on MemTable API from Task 14)
|
||
- **Parallel Group**: Sub-Wave 3d (solo — after MemTable complete)
|
||
- **Blocks**: 19
|
||
- **Blocked By**: 2, 9, 10, 11, 12, 14
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:103-115` — 写入流程 11 步
|
||
- `docs/design.md:119` — 资源前置校验约束
|
||
- `docs/design.md:127-153` — ErrCommitUnknown 和 WAL 副作用边界
|
||
- `docs/design.md:139-153` — per-batch private encode buffer
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 这是整个 WAL 子系统最复杂的组件，必须严格遵循设计文档的步骤顺序和错误分类
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Single Put writes and syncs correctly
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestWalWriterSinglePut in wal/writer_test.go
|
||
2. Start WalWriter in temp dir
|
||
3. Put("key1", "value1")
|
||
4. Close writer
|
||
5. Read WAL segment → verify batch with 1 entry exists
|
||
Expected Result: Entry persisted in WAL
|
||
Evidence: .omo/evidence/task-13-writer-single.txt
|
||
|
||
Scenario: Multiple concurrent writes group-committed
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestWalWriterGroupCommit in wal/writer_test.go
|
||
2. Start WalWriter
|
||
3. Launch 10 goroutines each doing Put
|
||
4. Wait for all to complete
|
||
5. Close writer
|
||
6. Read WAL → verify entries batched together
|
||
Expected Result: Entries grouped in batches, all sequences assigned
|
||
Evidence: .omo/evidence/task-13-writer-group.txt
|
||
|
||
Scenario: Write-stopped after simulated WAL write failure
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create a WalWriter with a mock segment writer that fails on 3rd AppendBatch
|
||
2. Write 5 entries sequentially
|
||
3. 3rd write should return ErrCommitUnknown
|
||
4. 4th write should be rejected (write-stopped)
|
||
Expected Result: ErrCommitUnknown returned, subsequent writes rejected
|
||
Evidence: .omo/evidence/task-13-writer-stopped.txt
|
||
```
|
||
|
||
**Commit**: YES (with Task 12)
|
||
- Message: `feat(wal): implement WAL writer with group commit`
|
||
|
||
- [ ] 14. MemTable (wrap skiplist + arena + publish/abort)
|
||
|
||
**What to do**:
|
||
- 创建 `memtable/memtable.go`：`MemTable` 结构体
|
||
- 封装 Arena + SkipList
|
||
- `Reserve(entries []ReserveEntry) (totalSize uint32, err error)` — 容量预留（最坏情况：最大层高 next 指针数组 + 对齐 padding）。单个 batch 超过空 MemTable → 返回错误
|
||
- `PutPending(key []byte, value []byte, sequence uint64) error` — 写入 pending entry（Arena 已预留）
|
||
- `DeletePending(key []byte, sequence uint64) error` — 写入 pending tombstone
|
||
- `Publish(upToSequence uint64)` — 批量发布 sequence <= upToSequence 的 pending entries（atomic store-release）
|
||
- `Abort(fromSequence uint64)` — 标记 aborted
|
||
- `Get(key []byte, publishedSequence uint64) *GetResult` — 无锁读：先 Load publishedSequence，遍历 skiplist，只返回 sequence <= publishedSequence 且非 aborted 的 entry
|
||
- `NewIterator(publishedSequence uint64) *MemTableIterator` — 有序遍历（同样过滤 pending/aborted）
|
||
- `ApproximateSize() uint64`
|
||
- `UsableCapacity() uint32` — 可用容量（扣除固定元数据）
|
||
|
||
**Must NOT do**:
|
||
- 不允许 PutPending 在 Reserve 成功后因 Arena 满失败
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: [`golang-concurrency`]
|
||
- `golang-concurrency`: 原子发布、无锁读、内存序
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (solo in sub-wave, Task 13 depends on this)
|
||
- **Parallel Group**: Sub-Wave 3c (solo)
|
||
- **Blocks**: 13, 19
|
||
- **Blocked By**: 7, 8
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:864-879` — Arena 容量预留协议
|
||
- `docs/design.md:881-889` — Flush 过滤规则（pending/aborted 不进入 SSTable）
|
||
- `docs/design.md:841-850` — 发布与内存序
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Pending entry invisible to readers
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestMemTablePending in memtable/memtable_test.go
|
||
2. Create MemTable, write PutPending("key1","val1", seq=10)
|
||
3. Get("key1", publishedSequence=5) → Found=false
|
||
4. Publish(upToSequence=10)
|
||
5. Get("key1", publishedSequence=10) → Found=true, Value="val1"
|
||
Expected Result: Entry invisible before publish, visible after
|
||
Evidence: .omo/evidence/task-14-memtable-pending.txt
|
||
|
||
Scenario: Aborted entry invisible to readers
|
||
Tool: Bash
|
||
Steps:
|
||
1. PutPending("key2","val2", seq=20)
|
||
2. Abort(20)
|
||
3. Publish(upToSequence=30) — publish others but not aborted
|
||
4. Get("key2", publishedSequence=30) → Found=false
|
||
Expected Result: Aborted entry never visible
|
||
Evidence: .omo/evidence/task-14-memtable-aborted.txt
|
||
|
||
Scenario: Reserve accuracy
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create MemTable with 1024 bytes usable
|
||
2. Reserve 10 entries with 10-byte keys + 10-byte values
|
||
3. Assert reserved size includes node overhead + alignment
|
||
4. Attempt to reserve more than remaining → error
|
||
Expected Result: Reserve accounts for full overhead
|
||
Evidence: .omo/evidence/task-14-memtable-reserve.txt
|
||
```
|
||
|
||
**Commit**: YES
|
||
- Message: `feat(memtable): integrate arena+skiplist with publish/abort`
|
||
|
||
- [ ] 15. WAL recovery: scanner + record parser
|
||
|
||
**What to do**:
|
||
- 创建 `wal/scanner.go`：`SegmentScanner` — 从 WAL 目录扫描 segment 文件
|
||
- `ScanSegments(dir string, recoverySegmentID uint64) ([]*SegmentInfo, error)` — 按 segmentID 排序，过滤 < recoverySegmentID
|
||
- `SegmentInfo` 结构体：FilePath, SegmentID, StartSequence（从 header 解析）
|
||
- 连续性校验：segmentID 连续、startSequence 衔接
|
||
- 创建 `wal/record_parser.go`：`RecordParser` — Block 级别 Physical Record 解析
|
||
- `ParseBlock(data []byte) ([]*PhysicalRecord, error)` — 解析 block 内所有 records
|
||
- Padding 校验：剩余 < 7 bytes 时全 0
|
||
- Short block 处理：最后 segment 的最后 block 可以短于 32KB
|
||
- 错误分类辅助：`IsTailCorruption(err error) bool`
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on 4, 10)
|
||
- **Parallel Group**: Sub-Wave 4a (solo)
|
||
- **Blocks**: 16
|
||
- **Blocked By**: 4, 10
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:597-665` — Recovery 扫描流程、segment 连续性校验
|
||
- `docs/design.md:669-691` — Physical Record 解析规则
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Segment scanner finds and orders segments correctly
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create temp dir with segment-0.wal, segment-2.wal, segment-1.wal
|
||
2. ScanSegments(dir, 0) → returns 3 segments ordered by ID
|
||
3. ScanSegments(dir, 1) → returns segments 1,2 only
|
||
Expected Result: Correct ordering and filtering
|
||
Evidence: .omo/evidence/task-15-scanner.txt
|
||
|
||
Scenario: Record parser handles valid blocks
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create a 32KB block with 3 Physical Records
|
||
2. ParseBlock → verify 3 records parsed with correct payloads
|
||
Expected Result: All records extracted
|
||
Evidence: .omo/evidence/task-15-parser.txt
|
||
```
|
||
|
||
**Commit**: YES (with Tasks 16, 17)
|
||
|
||
- [ ] 16. WAL recovery: fragment collector + batch replay
|
||
|
||
**What to do**:
|
||
- 创建 `wal/recovery.go`：恢复核心逻辑
|
||
- 使用 Task 6 的 `FragmentCollector` 进行 fragment 重组
|
||
- `BatchReplayer` 接口：`ReplayPut(key, value []byte, sequence uint64)` / `ReplayDelete(key []byte, sequence uint64)`
|
||
- `ReplayBatch(batch *WalBatch, expectedSequence uint64, replayer BatchReplayer) (nextSequence uint64, err error)`
|
||
- 校验 Batch Header：flags、entryCount (0 < N <= max)、entriesSize、baseSequence == expectedSequence
|
||
- 逐条解析 Entry，校验 opType、valueKind、keyLen、valLen
|
||
- 调用 replayer 回调
|
||
- 返回 nextSequence = expectedSequence + entryCount
|
||
- Sequence 溢出检查：`baseSequence + entryCount - 1` 不溢出
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `deep`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on 6, 15)
|
||
- **Parallel Group**: Sub-Wave 4b (solo — after scanner)
|
||
- **Blocks**: 17
|
||
- **Blocked By**: 6, 15
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:733-781` — WAL Batch 校验与重放规则
|
||
- `docs/design.md:486-499` — Sequence 溢出检查
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Valid batch replayed correctly
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create TestBatchReplay in wal/recovery_test.go
|
||
2. Encode a batch: baseSeq=0, 3 entries (Put a, Put b, Delete c)
|
||
3. ReplayBatch with expectedSequence=0 and mock replayer
|
||
4. Assert replayer received 3 calls in order with correct args
|
||
Expected Result: All entries replayed with correct sequences
|
||
Evidence: .omo/evidence/task-16-batch-replay.txt
|
||
|
||
Scenario: Batch with wrong baseSequence rejected
|
||
Tool: Bash
|
||
Steps:
|
||
1. Encode batch with baseSeq=100
|
||
2. ReplayBatch with expectedSequence=50 → error
|
||
Expected Result: Sequence mismatch error
|
||
Evidence: .omo/evidence/task-16-seq-mismatch.txt
|
||
```
|
||
|
||
**Commit**: YES (with Tasks 15, 17)
|
||
|
||
- [ ] 17. WAL recovery: tail truncation + main flow
|
||
|
||
**What to do**:
|
||
- 创建 `wal/truncation.go`：尾部截断持久化
|
||
- `TruncateTail(segmentPath string, truncateOffset int64) error` — ftruncate + fsync
|
||
- `DeleteEmptySegments(dir string, fromSegmentID uint64, expectedSequence uint64) error` — 删除空 segment + fsync dir
|
||
- 复用 Task 16 定义的 `BatchReplayer` 接口和 `ReplayBatch` 函数（同在 `wal/recovery.go`）
|
||
- `Recover(dir string, manifest *manifest.Manifest, cfg *config.WalConfig, replayer BatchReplayer) (*RecoveryResult, error)`
|
||
- `BatchReplayer` 由 Task 16 定义（interface{ ReplayPut/ReplayDelete }），Task 19 传入 MemTable 适配实现
|
||
- 主流程严格遵循 design.md lines 622-636：
|
||
1. 读取 MANIFEST → recoverySegmentID
|
||
2. ScanSegments → 过滤
|
||
3. 校验连续性
|
||
4. 逐 segment：校验 header → 逐 block 解析 → fragment 重组 → 调用 Task 16 的 `ReplayBatch(batch, expectedSeq, replayer)` 重放
|
||
5. 处理尾部异常（最后 segment → truncation，中间 segment → 报错）
|
||
6. 持久化截断
|
||
- 返回 `RecoveryResult`：RecoveredSequence, NextSequence, PublishedSequence
|
||
- 错误分类：
|
||
- 尾部（最后 segment）：header 半写、length 越界、CRC 失败、非 0 padding → 截断
|
||
- 中间（非最后 segment）：同样情况 → 报错
|
||
- CRC-valid batch 内部错误（entryCount 越界等）→ 报错（不可截断）
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (depends on 15, 16)
|
||
- **Parallel Group**: Sub-Wave 4c (solo — after batch replay)
|
||
- **Blocks**: 19
|
||
- **Blocked By**: 15, 16
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:622-636` — Recovery 扫描流程
|
||
- `docs/design.md:640-665` — Segment 连续性校验
|
||
- `docs/design.md:783-800` — 尾部截断持久化
|
||
- `docs/design.md:694-703` — 错误分类（尾部 vs 中间）
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Normal recovery from clean WAL
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create temp dir, write 2 segments with valid batches
|
||
2. Recover(dir, manifest) → success
|
||
3. Assert RecoveredSequence == last batch end sequence
|
||
4. Assert all entries replayed via mock replayer
|
||
Expected Result: Full recovery, no data loss
|
||
Evidence: .omo/evidence/task-17-recovery-normal.txt
|
||
|
||
Scenario: Recovery with tail corruption truncates
|
||
Tool: Bash
|
||
Steps:
|
||
1. Write valid segment + partial write at end (corrupt last record)
|
||
2. Recover → success (truncated to last complete batch)
|
||
3. Verify segment file was truncated
|
||
Expected Result: Tail corruption handled, complete batches preserved
|
||
Evidence: .omo/evidence/task-17-recovery-tail.txt
|
||
|
||
Scenario: Recovery with middle corruption reports error
|
||
Tool: Bash
|
||
Steps:
|
||
1. Write 3 segments, corrupt the middle one (bad CRC)
|
||
2. Recover → error containing "corrupt" or "damaged"
|
||
Expected Result: Middle corruption detected, recovery fails
|
||
Evidence: .omo/evidence/task-17-recovery-middle.txt
|
||
```
|
||
|
||
**Commit**: YES (with Tasks 15, 16)
|
||
|
||
- [x] 18. MANIFEST stub + CURRENT file
|
||
|
||
**What to do**:
|
||
- 创建 `manifest/manifest.go`：`Manifest` 最小 stub
|
||
- 仅存储 `recoverySegmentID uint64`
|
||
- `Load(dir string) (*Manifest, error)` — 读取 MANIFEST 文件
|
||
- `Save(dir string, recoverySegmentID uint64) error` — temp + rename 原子写入
|
||
- 首次创建：如果 MANIFEST 不存在，创建初始文件（recoverySegmentID=0）
|
||
- 创建 `manifest/current.go`：CURRENT 文件
|
||
- `WriteCurrent(dir string, segmentID uint64) error` — temp + rename，best-effort
|
||
- `ReadCurrent(dir string) (segmentID uint64, ok bool)` — 读取，不存在或损坏返回 ok=false
|
||
- 格式：简单的文本文件或二进制（Phase 1 选择简单文本）
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `quick`
|
||
- **Skills**: []
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES
|
||
- **Parallel Group**: Wave 1b (with Tasks 2, 3, 4, 5, 12 — all depend only on Task 1)
|
||
- **Blocks**: 19
|
||
- **Blocked By**: 1
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:603-619` — CURRENT / MANIFEST 权威性
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: MANIFEST save and load round-trip
|
||
Tool: Bash
|
||
Steps:
|
||
1. Save(dir, 42)
|
||
2. Load(dir) → assert recoverySegmentID == 42
|
||
Expected Result: Round-trip correct
|
||
Evidence: .omo/evidence/task-18-manifest.txt
|
||
|
||
Scenario: CURRENT best-effort handles missing file
|
||
Tool: Bash
|
||
Steps:
|
||
1. ReadCurrent on empty dir → ok=false, no error
|
||
Expected Result: Missing CURRENT handled gracefully
|
||
Evidence: .omo/evidence/task-18-current-missing.txt
|
||
```
|
||
|
||
**Commit**: YES
|
||
- Message: `feat: implement MANIFEST stub and CURRENT file`
|
||
|
||
- [ ] 19. DB integration: Open/Close/Put/Delete/Get
|
||
|
||
**What to do**:
|
||
- 创建 `db.go`：`DB` 结构体和核心 API
|
||
- `DB` 字段：walWriter, memTable, immutableTables, seqMgr, manifest, config, dir, writeStopped
|
||
- `Open(dir string, opts ...Option) (*DB, error)`:
|
||
1. Load or create MANIFEST
|
||
2. Recover WAL → 重建 MemTable（恢复的 entries 标记为 published）
|
||
3. 设置 nextSequence、publishedSequence
|
||
4. 启动 WAL writer goroutine
|
||
- `Close() error`:
|
||
1. 停止 WAL writer（等待 pending batch 完成）
|
||
2. Sync WAL
|
||
3. Save MANIFEST（recoverySegmentID stays 0 in Phase 1 — no SSTable flush means all segments must be replayed）
|
||
4. Close segment writer
|
||
- `Put(key, value []byte) error`:
|
||
1. ValidateBatchLimits (key/value size)
|
||
2. Submit to WAL writer → wait for result
|
||
3. 返回 nil 或 ErrCommitUnknown 或 write-stopped error
|
||
- `Delete(key []byte) error` — 同上，opType=Delete
|
||
- `Get(key []byte) (GetResult, error)`:
|
||
1. Load publishedSequence
|
||
2. 查 active MemTable → Immutable #1 → Immutable #2
|
||
3. 只返回 sequence <= publishedSequence 且非 aborted
|
||
4. Tombstone → Found=false
|
||
5. 未找到 → Found=false
|
||
- `GetResult` 结构体：Value []byte, Found bool（参照 design.md §3.4）
|
||
- `GetDurableSequence() uint64`
|
||
- `IsWriteStopped() bool`
|
||
- MemTable freeze + switch：
|
||
- Put 前检查容量，不足时 freeze
|
||
- Immutable 队列满时阻塞
|
||
- Phase 1：Immutable 不 flush 到 SSTable，仅持有在内存
|
||
- Options pattern：`type Option func(*config.WalConfig)`
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `deep`
|
||
- **Skills**: [`golang-design-patterns`]
|
||
- `golang-design-patterns`: Options pattern for configuration
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: NO (integrates all deps)
|
||
- **Parallel Group**: Sub-Wave 4d (solo — after recovery + WAL writer + MemTable + MANIFEST)
|
||
- **Blocks**: 20, 21
|
||
- **Blocked By**: 13, 14, 17, 18
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:901-921` — Get API 和 GetResult 定义
|
||
- `docs/design.md:928-941` — 读路径优先级
|
||
- `docs/design.md:852-862` — MemTable 生命周期
|
||
|
||
**WHY Each Reference Matters**:
|
||
- 这是所有模块的集成点，必须正确协调 WAL writer、MemTable、recovery 和 MANIFEST
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Open creates fresh DB, Put+Get round-trip
|
||
Tool: Bash
|
||
Steps:
|
||
1. Create temp dir
|
||
2. Open(dir) → success
|
||
3. Put("hello", "world") → nil
|
||
4. Get("hello") → {Found:true, Value:"world"}
|
||
5. Close() → nil
|
||
Expected Result: Full round-trip works
|
||
Evidence: .omo/evidence/task-19-open-put-get.txt
|
||
|
||
Scenario: Close and reopen preserves data
|
||
Tool: Bash
|
||
Steps:
|
||
1. Open(dir), Put("k1","v1"), Put("k2","v2"), Close()
|
||
2. Open(dir) → recovery runs
|
||
3. Get("k1") → {Found:true, Value:"v1"}
|
||
4. Get("k2") → {Found:true, Value:"v2"}
|
||
5. Close()
|
||
Expected Result: Data survives close/reopen
|
||
Evidence: .omo/evidence/task-19-reopen.txt
|
||
|
||
Scenario: Delete makes key not found
|
||
Tool: Bash
|
||
Steps:
|
||
1. Open, Put("k","v"), Delete("k")
|
||
2. Get("k") → {Found:false}
|
||
Expected Result: Delete works correctly
|
||
Evidence: .omo/evidence/task-19-delete.txt
|
||
|
||
Scenario: Empty value distinguished from not found
|
||
Tool: Bash
|
||
Steps:
|
||
1. Put("empty", []byte{})
|
||
2. Get("empty") → {Found:true, Value:[]byte{}}
|
||
3. Get("nonexist") → {Found:false}
|
||
Expected Result: Found distinguishes empty value from missing key
|
||
Evidence: .omo/evidence/task-19-empty-value.txt
|
||
|
||
Scenario: Concurrent Put and Get work correctly
|
||
Tool: Bash
|
||
Steps:
|
||
1. Open
|
||
2. Launch 10 writers (Put "key-{i}", "val-{i}")
|
||
3. Launch 10 readers (Get "key-{i}")
|
||
4. go test -race
|
||
Expected Result: No race conditions
|
||
Evidence: .omo/evidence/task-19-concurrent.txt
|
||
```
|
||
|
||
**Commit**: YES
|
||
- Message: `feat: integrate DB with Open/Close/Put/Delete/Get`
|
||
|
||
- [ ] 20. End-to-end + crash recovery tests
|
||
|
||
**What to do**:
|
||
- 创建 `db_test.go`：集成测试
|
||
- `TestWALRoundtrip` — 写 N 条 → Close → Open → 验证全部
|
||
- `TestRecoveryPartialWrite` — 模拟崩溃：写入数据 → 在 WAL 写入中途截断文件 → Recover → 验证完整 batch 保留
|
||
- `TestRecoveryMultiSegment` — 多 segment 恢复
|
||
- `TestRecoveryTailCorruption` — 尾部损坏修复
|
||
- `TestWriteStoppedAfterFailure` — 模拟写入失败 → ErrCommitUnknown → write-stopped
|
||
- `TestGetSemantics` — Found vs empty value vs not found
|
||
- `TestSequenceExhaustion` — 虽然不可达，但验证 sequence 溢出返回正确错误
|
||
- `TestConcurrentReadWrite` — 多 goroutine 并发读写
|
||
- `TestLargeBatchSegmentRotation` — 大量写入触发多次 segment 轮转
|
||
- `TestMemTableFreezeNoSSTable` — 验证 MemTable 满时冻结，不刷盘
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `deep`
|
||
- **Skills**: [`golang-testing`]
|
||
- `golang-testing`: Go testing patterns
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES (with Task 21)
|
||
- **Parallel Group**: Wave 5 (with Task 21)
|
||
- **Blocks**: F1-F4
|
||
- **Blocked By**: 19
|
||
|
||
**References**:
|
||
**Pattern References**:
|
||
- `docs/design.md:694-703` — 错误分类
|
||
- `docs/design.md:783-800` — 尾部截断
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Full WAL roundtrip with crash simulation
|
||
Tool: Bash
|
||
Steps:
|
||
1. Open DB, write 1000 keys
|
||
2. Close DB (simulate clean shutdown)
|
||
3. Open DB → verify all 1000 keys readable
|
||
Expected Result: All 1000 keys recovered
|
||
Evidence: .omo/evidence/task-20-roundtrip.txt
|
||
|
||
Scenario: Crash recovery with tail truncation
|
||
Tool: Bash
|
||
Steps:
|
||
1. Open DB, write 500 keys, Close
|
||
2. Open DB, write 200 more keys
|
||
3. Manually truncate last WAL segment by 100 bytes (simulate crash)
|
||
4. Open DB → verify 500 original keys present, some of 200 may be lost
|
||
Expected Result: Complete batches preserved, tail truncated
|
||
Evidence: .omo/evidence/task-20-crash.txt
|
||
|
||
Scenario: go test -race passes all integration tests
|
||
Tool: Bash
|
||
Steps:
|
||
1. go test ./... -race -count=1
|
||
Expected Result: Zero race conditions
|
||
Evidence: .omo/evidence/task-20-race.txt
|
||
```
|
||
|
||
**Commit**: YES
|
||
- Message: `test: add end-to-end and crash recovery tests`
|
||
|
||
- [ ] 21. Benchmark suite
|
||
|
||
**What to do**:
|
||
- 创建 `bench_test.go`：性能基准
|
||
- `BenchmarkSinglePut` — 单线程 Put 吞吐
|
||
- `BenchmarkConcurrentPut` — 多线程 Put 吞吐（2/4/8 goroutines）
|
||
- `BenchmarkGet` — Get 延迟
|
||
- `BenchmarkConcurrentGet` — 并发 Get
|
||
- `BenchmarkMixedReadWrite` — 混合读写
|
||
- `BenchmarkWALRecovery` — 不同数据量的恢复时间
|
||
- 使用 `b.ReportAllocs()` 跟踪内存分配
|
||
|
||
**Recommended Agent Profile**:
|
||
- **Category**: `unspecified-high`
|
||
- **Skills**: [`golang-benchmark`]
|
||
- `golang-benchmark`: Go benchmarking best practices
|
||
|
||
**Parallelization**:
|
||
- **Can Run In Parallel**: YES (with Task 20)
|
||
- **Parallel Group**: Wave 5 (with Task 20)
|
||
- **Blocks**: F1-F4
|
||
- **Blocked By**: 19
|
||
|
||
**References**:
|
||
- 无特定设计文档引用（benchmark 是验证手段，不是设计要求）
|
||
|
||
**Acceptance Criteria**:
|
||
|
||
**QA Scenarios (MANDATORY):**
|
||
|
||
```
|
||
Scenario: Benchmark runs without errors
|
||
Tool: Bash
|
||
Steps:
|
||
1. go test -bench=. -benchmem ./...
|
||
Expected Result: All benchmarks complete, results printed
|
||
Evidence: .omo/evidence/task-21-bench.txt
|
||
```
|
||
|
||
**Commit**: YES
|
||
- Message: `bench: add WAL performance benchmarks`
|
||
|
||
---
|
||
|
||
## Final Verification Wave
|
||
|
||
- [ ] F1. **Plan Compliance Audit** — `oracle`
|
||
Read the plan end-to-end. For each "Must Have": verify implementation exists (grep for atomic.Uint64, atomic.Pointer, no bufio.Writer on WAL path, GetResult with Found field, etc). For each "Must NOT Have": search codebase for forbidden patterns (SSTable, Compaction, ValueLogPointer write, Periodic sync, bufio.Writer in wal/). Check evidence files exist in .omo/evidence/. Compare deliverables against plan.
|
||
**Acceptance Criteria**:
|
||
- [ ] All "Must Have" items verified present in codebase
|
||
- [ ] All "Must NOT Have" items verified absent
|
||
- [ ] Evidence files exist for tasks 1-21
|
||
- [ ] Output: `Must Have [N/N] | Must NOT Have [N/N] | Tasks [21/21] | VERDICT: APPROVE/REJECT`
|
||
|
||
- [ ] F2. **Code Quality Review** — `unspecified-high`
|
||
Run `go vet ./...` + `golangci-lint run` + `go test ./... -race`. Review all .go files for: `any` type casts, empty catches, fmt.Println in prod, commented-out code, unused imports. Check AI slop: excessive comments, over-abstraction, generic names.
|
||
**Acceptance Criteria**:
|
||
- [ ] `go vet ./...` passes with zero warnings
|
||
- [ ] `golangci-lint run` passes with zero warnings
|
||
- [ ] `go test ./... -race` passes with zero races
|
||
- [ ] No `fmt.Println` or `log.Println` in non-test files
|
||
- [ ] Output: `Build [PASS/FAIL] | Lint [PASS/FAIL] | Tests [N pass/0 fail] | Files [N clean/0 issues] | VERDICT`
|
||
|
||
- [ ] F3. **Real Manual QA** — `unspecified-high`
|
||
Build the project. Run ALL QA scenarios from tasks 1-21. Test cross-module integration: write data with Put → Close → Open → Get verifies data. Test edge cases: empty value, large batch (9999 entries), concurrent writes from 10 goroutines, segment rotation after filling segment. Save output to `.omo/evidence/final-qa/`.
|
||
**Acceptance Criteria**:
|
||
- [ ] Every QA scenario from tasks 1-21 executed and passing
|
||
- [ ] Cross-module integration: Put → Close → Open → Get round-trip
|
||
- [ ] Edge case: empty value returns Found=true
|
||
- [ ] Edge case: key not found returns Found=false
|
||
- [ ] Edge case: concurrent 10-goroutine write/read with -race clean
|
||
- [ ] Output: `Scenarios [N/N pass] | Integration [3/3] | Edge Cases [4 tested] | VERDICT`
|
||
|
||
- [ ] F4. **Scope Fidelity Check** — `deep`
|
||
For each task: read "What to do", read actual diff (git diff). Verify 1:1 — everything in spec was built (no missing), nothing beyond spec was built (no creep). Check "Must NOT do" compliance per task. Detect cross-task contamination: Task N touching Task M's files without justification. Flag unaccounted files.
|
||
**Acceptance Criteria**:
|
||
- [ ] Each task's "What to do" matches its actual implementation
|
||
- [ ] No SSTable/Compaction/ValueLog/BloomFilter code found
|
||
- [ ] No bufio.Writer import in wal/ package
|
||
- [ ] No Periodic/Never sync code (only Always)
|
||
- [ ] No cross-task file contamination
|
||
- [ ] Output: `Tasks [21/21 compliant] | Contamination [CLEAN] | Unaccounted [CLEAN] | VERDICT`
|
||
|
||
---
|
||
|
||
## Commit Strategy
|
||
|
||
| Tasks | Commit Message | Pre-commit Check |
|
||
|-------|---------------|-----------------|
|
||
| 1-2 | `feat: initialize project structure and error types` | `go build ./...` |
|
||
| 3-5 | `feat(wal): implement WAL entry/record/header codec` | `go test ./wal/... -run TestCodec` |
|
||
| 6, 9 | `feat(wal): implement WAL batch codec and resource validation` | `go test ./wal/... -run TestBatch` |
|
||
| 7-8 | `feat(memtable): implement Arena allocator and SkipList` | `go test ./memtable/... -race` |
|
||
| 10-11 | `feat(wal): implement segment writer with rotation` | `go test ./wal/... -run TestSegment` |
|
||
| 12-13 | `feat(wal): implement WAL writer with group commit` | `go test ./wal/... -run TestWriter` |
|
||
| 14 | `feat(memtable): integrate arena+skiplist with publish/abort` | `go test ./memtable/... -race` |
|
||
| 15-17 | `feat(wal): implement WAL recovery` | `go test ./wal/... -run TestRecovery` |
|
||
| 18 | `feat: implement MANIFEST stub and CURRENT file` | `go test ./manifest/...` |
|
||
| 19 | `feat: integrate DB with Open/Close/Put/Delete/Get` | `go test ./... -race` |
|
||
| 20 | `test: add end-to-end and crash recovery tests` | `go test ./... -race -count=1` |
|
||
| 21 | `bench: add WAL performance benchmarks` | `go test -bench=. ./...` |
|
||
|
||
---
|
||
|
||
## Success Criteria
|
||
|
||
### Verification Commands
|
||
```bash
|
||
go build ./... # Expected: success, no errors
|
||
go vet ./... # Expected: no issues
|
||
go test ./... -race -count=1 # Expected: all pass, no races
|
||
go test ./... -run TestWALRoundtrip # Expected: write→close→open→read roundtrip
|
||
go test ./... -run TestRecovery # Expected: crash recovery correct
|
||
go test ./... -run TestGetSemantics # Expected: Found/empty-value distinction
|
||
```
|
||
|
||
### Final Checklist
|
||
- [ ] All "Must Have" present
|
||
- [ ] All "Must NOT Have" absent
|
||
- [ ] All tests pass with `-race`
|
||
- [ ] WAL roundtrip test passes
|
||
- [ ] Crash recovery test passes
|
||
- [ ] No bufio.Writer on WAL append path
|
||
- [ ] All atomics are typed (atomic.Uint64, atomic.Pointer)
|
||
- [ ] Config invariant checked at Open
|