Per design §3.2 line 248-272, segment directory fsync is a hard
requirement for durable-ready state, not best-effort. rename is atomic
in memory but not guaranteed to survive power loss without a directory
fsync. The previous code silently swallowed both os.Open(dir) and
dirFD.Sync() errors, leaving WAL writer to confirm batches as durable
when their segment might not exist after a crash.
Failure propagation:
- Initial segment creation: NewSegmentWriter fails -> NewSegmentManager
fails -> DB.Open fails (user sees error, no data promise violated).
- Rotation during AppendBatch: NewSegmentWriter fails -> AppendBatch
fails -> WalWriter.stopWithError(ErrCommitUnknown) -> write-stopped
(per design line 272).
Changes:
- wal/segment_writer.go: extract dirFsync helper (Open -> f.Stat ->
IsDir -> f.Sync, avoiding TOCTOU window), replace silent swallow with
fatal error; on failure clean up resources (fd.Close + os.Remove) and
surface cleanup errors via errors.Join so nothing is silently lost.
- wal/dir_fsync_test.go (new): unit test the helper with valid dir,
non-existent dir (fails at os.Open), and not-a-dir (fails at IsDir).
- wal/segment_writer_test.go: add TestNewSegmentWriterDirFsyncFailure
(injects failure via package-level dirFsyncFn override; documents the
not-parallel-safe constraint), TestNewSegmentWriterNormalPathStillWorks
(regression), and TestNewSegmentWriterRetryAfterDirFsyncFailure
(verifies cleanup is effective for retry).
- wal/segment_manager_test.go: add TestSegmentManagerRotateFailsOnDirFsyncFailure
(fills segment until rotation triggers, injects failure, verifies
propagation through AppendBatch path) and TestNewSegmentManagerFailsOnDirFsyncFailure
(covers the DB.Open failure path).
dirFsyncFn injection note: tests that override this package-level var
must not use t.Parallel(). All existing wal tests run serially within
the package; this is the lightest mechanism that doesn't require
interface indirection in production code.
Verified: each new test fails on pre-fix code (silent swallow returned
nil error) and passes after the fix. Full suite green including
go test -race ./... .
Audit context: docs/audit-3.2.md C6 (Oracle-verified bg_ef425776).
Phase 1 default state had two data-loss paths in WAL recovery.
C2: resolveRecoverySegmentID fell back to CURRENT when MANIFEST=0.
Since segment_manager writes CURRENT on every segment create/rotate,
the first recovery in Phase 1 (MANIFEST always 0 without flush) would
start from the active segment, skipping earlier unflushed segments.
C3: Recover called manifest.Save after every recovery, advancing
recoverySegmentID past segments that were still the only durable copy
of their data (no SSTable flush yet). Next restart would filter those
segments out and permanently lose the data.
Per design §3.2 line 280, recovery must not update MANIFEST; per line
604-06, CURRENT must not be used as recovery start. Both fixes are
required together — fixing C3 alone leaves C2's data-loss window open.
Changes:
- wal/recover.go: remove manifest.Save calls on both success and
tail-repair paths; remove CURRENT fallback in resolveRecoverySegmentID.
RecoveryResult.NextSegmentID is now in-memory only (consumed by DB.Open
to seed the new WalWriter, but never persisted to MANIFEST).
- wal/recover_test.go: rewrite TestRecoverUpdatesManifest as
TestRecoverDoesNotUpdateManifest; add TestRecoverPreservesExistingManifest,
TestRecoverIdempotentClean, TestRecoverIdempotentAfterTruncation,
TestRecoverIgnoresCurrentFallback.
- db_test.go: add TestOpenThreeTimesKeepsData (three opens to catch C3's
second-restart data loss; single-segment to avoid unrelated C8 bug
where segment_manager passes byte offset as startSequence).
Verified: each new test fails on pre-fix code and passes after the fix.
Full suite green including -race.
Audit context: docs/audit-3.2.md (with Oracle revisions from bg_ef425776
and bg_2e86d33b; C8 added). Plan: .omo/plans/fix-c2-c3-wal-recovery.md
(Momus + Oracle reviewed v1.2).
Replace polling loop (Collect + Sleep) with proper batch collection:
- Block on first request via channel receive
- Start 500µs timer for collection window
- Accumulate requests until timer fires or batch reaches 32KB
- Shutdown handling at all blocking points
Design doc §3.2.5: '等待组提交触发（500µs 或 32KB，先到者触发）'
Add GroupCommitDelay config field (default 500µs, must be > 0 and < 10ms).
- Add TestWriteStoppedAfterIOError verifying ErrCommitUnknown → write-stopped
transition after I/O failure (DoD requirement)
- Replace VKValueLogPointer with VKInline in segment_writer_test.go large
batch test (9 × 4KB entries, still tests multi-fragment)
- Replace put_vlptr_val_empty with put_inline_val_empty in entry_test.go
- VKValueLogPointer remains only in constants_test.go (enum value check)
- Sync() now flushes BlockWriter before fd.Sync() for durability
- Pre-validate encoding before sequence allocation (design doc compliance)
- Remove dead _ = rec assignment in recover.go
- Remove unused maxPayload field from SegmentWriter
- Handle Put error in MemTable.Publish with panic on invariant violation
- wal/recover.go: top-level Recover() function that orchestrates scan + replay + truncation
Reads MANIFEST/CURRENT for recovery segment ID, handles tail corruption by truncating
- Comprehensive tests for empty dir, full flow, tail corruption, and MANIFEST updates
- All tests pass with -race
- wal/commit_queue.go: bounded buffered channel for write requests
- wal/writer.go: single-goroutine main loop implementing 11-step write flow
with group commit, sequence allocation, MemTable publish/abort, write-stopped
- wal/recovery.go: BatchReplayer interface, ReplayBatch, ReplaySegmentFile,
RecoverFromSegments with fragment reassembly and tail corruption handling
- Comprehensive tests for all modules, all pass with -race
- wal/segment_manager.go: segment lifecycle with rotation at batch boundaries
- wal/scanner.go: segment discovery, ordering, and continuity validation
- wal/record_parser.go: block-level physical record parsing with tail corruption detection
- Comprehensive tests for all modules, all pass with -race
- wal/block_writer.go: 32KB block buffer with padding and flush
- wal/segment_writer.go: WAL segment file with durable-ready protocol
- memtable/memtable.go: Arena+SkipList wrapper with publish/abort semantics
- Comprehensive tests for all modules, all pass with -race
- wal/batch.go: WalBatch encode/decode with FragmentCollector state machine
- wal/validate.go: ValidateBatchLimits with checked arithmetic
- memtable/arena.go: Arena allocator with 8-byte alignment and mutex
- Comprehensive tests for all modules, all pass with -race
- go.mod with github.com/dailz/go-kv, Go 1.26.3, testify
- config/config.go with WalConfig, Validate() with checked arithmetic
- errors.go with sentinel errors (ErrCommitUnknown, ErrWriteStopped, etc.)
- wal/constants.go with all WAL format constants and enums
- wal/header.go with WAL File Header encode/decode (CRC32 IEEE)
- wal/record.go with Physical Record codec, block boundary, SplitIntoRecords
- wal/entry.go with WAL Entry codec (varint keys/values, OpType, ValueKind)
- wal/sequence.go with SequenceManager (atomic, CAS, overflow-safe)
- manifest/manifest.go with MANIFEST stub (Load/Save atomic)
- manifest/current.go with CURRENT file (WriteCurrent/ReadCurrent)
- Comprehensive tests for all modules
- .golangci.yml configuration
dailz