56 Commits
Author SHA1 Message Date
dailzandSisyphus c9aa50f932 docs: refresh AGENTS.md for app/ module split and updated test counts
CI / ci (windows-latest) (push) Has been cancelled
CI / ci (ubuntu-latest) (push) Failing after 49m21s
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-07-09 13:58:38 +08:00
dailzandSisyphus 9854ef13b6 refactor(tui): split remaining app.rs into per-concern submodules
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-07-09 13:53:36 +08:00
dailzandSisyphus 58ac4a65c7 refactor(tui): split app query and viewport cache helpers
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-06-30 16:55:34 +08:00
dailzandSisyphus 0910d4e85f docs: clarify benchmark test file setup
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-06-30 16:54:46 +08:00
dailz ef26481976 style: apply rustfmt across bench and core crates
CI / ci (ubuntu-latest) (push) Has been cancelled
CI / ci (windows-latest) (push) Has been cancelled
Pure formatting pass - no semantic changes. Mostly long-line reflow,
import statement reordering (alphabetical), and trailing-newline fixes
that had been accumulating in the working tree.

Affected files:
* crates/bench/src/{data_gen,main,mmap_reader,pread_reader,report,runner}.rs
* crates/bench/src/suites/{concurrent,growth,jump,memory,render,rotation,startup}.rs
* crates/core/src/io/{file_reader,index_cache,read_cache}.rs
* crates/core/src/{types.rs, watcher/file_watcher.rs}
2026-06-22 16:38:28 +08:00
dailz c4ba196016 chore: fix CI branch trigger and add project READMEs
CI / ci (ubuntu-latest) (push) Has been cancelled
CI / ci (windows-latest) (push) Has been cancelled
* ci.yml: trigger on pushes/PRs to master instead of main. The repo's
  default branch is master, so the previous config never fired and CI
  silently passed on every commit.

* Add README.md (English) and README-zh.md (Chinese), the bilingual
  project documentation referenced by AGENTS.md.
2026-06-22 16:32:52 +08:00
dailz ec5a163f05 chore: expand .gitignore for per-crate target dirs and local artifacts
* crates/*/target/ - per-crate build output (root /target/ was already
  covered, but workspace members' own target/ dirs were leaking through)
* .omo/ .sisyphus/ - local agent tooling state, not project content
* benchmark-report.md - output of cargo run -p log-viewer-bench,
  regenerated per run so should not be tracked
2026-06-22 15:14:52 +08:00
dailz 95f259a2bb fix(core): silence three clippy warnings blocking CI gate
* line_index.rs: replace manual `x % BLOCK_SIZE == 0` with
  `.is_multiple_of(BLOCK_SIZE)` (Rust 1.81+).

* line_index.rs: gate three pub(crate) accessor methods
  (`sampled_offsets`, `total_lines`, `has_trailing_newline`) behind
  `#[cfg(test)]`. They are only consumed by tests in file_reader.rs;
  marking them test-only removes them from production builds entirely,
  eliminating the dead_code warning without suppressing it.

* json.rs: drop redundant `Some(... .ok()?)` wrapper in
  parse_json_object_with_duplicates. `.ok()` already returns Option,
  so wrapping it in Some and unwrapping with ? was a no-op.

No behavior change. Unblocks `cargo clippy --workspace -- -D warnings`.
2026-06-22 15:02:33 +08:00
dailz e69b7af32a fix(tui): decouple cursor from viewport + correct width calc in no-VHI paths
Three intertwined fixes for the no-VHI scroll path that together eliminate
the 'switching to a new entry scrolls it to viewport top' behavior observed
during Loading state, Loading->Ready transition window, and Tab-toggle
VHI rebuild window.

Cursor/viewport decoupling (the user-visible bug):
* scroll_down_line / scroll_up_line no-VHI branches now advance only the
  cursor (cursor_line, cursor_sub_offset), then call ensure_cursor_visible_no_vhi
  to adjust the viewport minimally if needed. Previously they locked
  v_offset = cursor_line, forcing the cursor's entry to the top of the
  viewport on every keypress.
* New helpers: advance_visual_pos, retreat_visual_pos (bounded visual-row
  walks using compute_visual_height), ensure_cursor_visible_no_vhi (matches
  the VHI-path semantics - top-align when cursor is above viewport,
  bottom-align with minimal scroll when below, no movement when within).
* Same pattern applied to scroll_down/up_half_page and scroll_down/up_page.
* scroll_to_bottom sets cursor_sub_offset to the last visual row of the
  last line in no-VHI state (was 0).
* ensure_viewport_cache hidden anchors removed: the params-change recenter
  and the Loading post-check that did v_offset = cursor_line are replaced
  with the visibility helper.
* 4 transition sites (Tab handler, ensure_visual_height_index, reload,
  poll VHI complete) now clamp cursor_sub_offset against the current
  line height instead of blindly zeroing.

Width calculation correctness (root cause of the residual j/k bug):
* gutter_width() pub(crate) method extracted from ui.rs's inline formula,
  shared between renderer and App to prevent drift.
* get_content_width() now returns effective text-wrap width (area minus
  gutter), matching the renderer's actual_content_width. Previously it
  returned raw area width, causing compute_visual_height to under-count
  wrap rows for lines that wrapped only after gutter subtraction.
* Free function gutter_width_for(total_lines, is_loading) added so
  handle_file_appended and reload_ready_reader can recompute width
  after line-count changes without conflicting borrows on self.
* handle_file_appended: width now captured AFTER update_for_append
  (was before), so digit-boundary crossings (9->10, 99->100) don't
  leave stale gutter widths.

ui.rs render_content now calls app.gutter_width() instead of inlining
the formula (DRY).

Tests: 7 existing tests that locked in the old buggy v_offset = cursor_line
coupling have been rewritten to assert the new correct behavior. New
property tests verify wrap_line_count matches wrap_line_chars across
CJK/emoji/tab/combining-mark/control-char inputs.
2026-06-22 14:52:32 +08:00
dailz 4421da35f4 perf(core): linear mmap VHI rebuild + zero-alloc wrap helpers
Four changes that compound to a 2-5x speedup on VHI rebuild for large
files, plus eliminating one full wasted VHI pass on initial open.

* spawn_indexer no longer computes VHI inline (progressive_reader.rs).
  The inline VHI used a hardcoded width=80, was invalidated immediately
  on Loading->Ready transition (gutter digit-boundary), and triggered
  a second full rebuild. Now returns visual_height_index: None and lets
  the UI trigger one rebuild with the correct post-layout width.

* spawn_visual_height_rebuild rewritten as a single linear pass over
  mmap bytes using memchr::memchr (SIMD) to find newlines, replacing
  the byte-by-byte BufRead::read_until scan. Cancel check throttled
  to every 1MB instead of per-line.

* wrap_line_count(&str, width) -> usize added as a zero-allocation
  counterpart to wrap_line_chars. Property tests cover ASCII, CJK,
  emoji, combining marks, tabs, zero-width chars, control chars, and
  real log lines across many widths - all match wrap_line_chars().len()
  exactly. compute_text_visual_height now uses wrap_line_count, removing
  10M Vec allocations on a 10M-line file.

* format_json_line returns Cow<'_, str> instead of String. Non-JSON
  lines borrow the input (zero allocation); only pretty-printed JSON
  objects allocate. Removes another 10M allocations when json_format=true.

Removes now-unused compute_visual_heights (only caller was the deleted
inline VHI path).
2026-06-22 14:52:01 +08:00
dailz e765f8967f docs: add AGENTS.md with repo-specific agent guidance
Compact instruction file covering workspace layout, the non-obvious
default-members behavior (bare cargo run builds core, not tui), the
12 CoreError variants, the CRITICAL TerminalGuard Drop caveat, per-module
temp-file test helpers, and the benchmark test-file generation requirement.
2026-06-22 14:51:27 +08:00
dailz 967c11fea9 fix(tui): preserve old state on load_file failure (issue #32)
load_file previously cleared file_watcher before attempting to open the
new file. If open failed, the watcher was lost while loading_state and
file_path still pointed to the old file, breaking change detection.

Refactor to build-then-swap pattern: all fallible work runs on local
variables first, then self is updated only on success. On failure, the
old file's watcher, loading_state and file_path remain intact.

Update test to verify all three fields are preserved on failure.

Closes #32
2026-06-11 17:23:16 +08:00
dailz 10323ce814 fix(tui): add area offset to settings popup positioning (closes #31) 2026-06-11 16:49:37 +08:00
dailz c1a931551b fix(tui): handle settings save errors and rewrite truncate_to_columns (closes #30)
- Report save failures in settings instead of silently discarding errors
- Save draft first, commit color_config only on success (prevents split-brain)
- Show error message in red on status bar, stay in Settings mode on failure
- Clear error on Esc, entering settings, or any settings edit
- Rewrite truncate_to_columns as dedicated O(prefix) truncator
- Tab: stop before tab if expansion would exceed width
- Fixes pre-existing test_truncate_to_columns_tab failure
2026-06-11 16:08:57 +08:00
dailz dfc016c348 fix(tui): wrap color backward cycle at index 0 instead of clamping (closes #29) 2026-06-11 15:05:42 +08:00
dailz 19a3b877f9 fix(core): tighten word boundary to reject digits and underscores in level detection (closes #28) 2026-06-11 14:28:13 +08:00
dailz 5cb56dafd8 fix(core): correct tab-stop alignment and width overflow in wrap_line_chars
- Extract TAB_WIDTH constant (4) replacing magic numbers
- Calculate tab stop as TAB_WIDTH - (col % TAB_WIDTH) for proper alignment
- Split tab expansion across rows when width < TAB_WIDTH
- Update test_wrap_with_tab expected value for new behavior
- Add tests: narrow width, stop alignment, line boundary, regression

Fixes: #27
2026-06-11 13:37:14 +08:00
dailz e99861c76d fix(tui): add MAX_WRAP_INPUT_LEN guard to prevent UI freeze on oversized lines (closes #26)
- compute_line_entry/compute_visual_height: double guard (raw + post-format)
- Skip detect_level on oversized raw input to avoid O(n) JSON parsing
- Add post-format guard for JSON lines that expand beyond 10MB
- progressive_reader: add post-format guard to compute_line_visual_height
- Add truncate_to_columns helper using existing wrap_line_chars
- Fix misleading docstring on MAX_WRAP_INPUT_LEN constant
- Add 6 regression tests covering all guard paths
2026-06-11 13:15:11 +08:00
dailz a43ef673b0 fix(tui): rebase v_offset before VHI invalidation to prevent viewport jump (closes #25)
Ready state can have VHI=None during async rebuild (Tab toggle, resize,
file append, Loading→Ready transition). Without rebase, v_offset retains
a visual-row value that gets treated as a logical-line offset, causing
viewport jumps and cursor drift.

Changes:
- Add rebase_offset_for_invalidate() to convert v_offset from visual-row
  to logical-line + sub-row before VHI invalidation
- Call it at all 6 invalidation sites in Ready state
- Recalibrate v_offset from viewport top when VHI rebuild completes
- Clamp preserved sub_row to new line height after JSON/width changes

Regression tests: 6 new tests covering rebase conversion, Tab toggle,
VHI rebuild recalibration, sub-row clamping, and Loading→Ready transition.
2026-06-11 09:49:10 +08:00
dailz 70f930eef7 fix(tui): use updated v_offset after params_changed in ensure_viewport_cache (#24)
The loading branch of ensure_viewport_cache captured v_offset before the
params_changed block, which could reassign self.v_offset. This caused the
viewport to use a stale offset when loading + width/format changed together.

Remove the stale local variable and read self.v_offset directly, consistent
with the non-loading branch. Add regression test.
2026-06-11 08:58:10 +08:00
dailz 463c53148b fix(tui): filter KeyEventKind to prevent Release/Repeat from triggering commands (closes #23) 2026-06-10 17:34:17 +08:00
dailz e9f75ce3b1 fix(parser): detect duplicate JSON keys via custom Visitor instead of silent last-wins (closes #22) 2026-06-10 15:22:55 +08:00
dailz ef1889767a fix(parser): trim whitespace in LogLevel::from_str to prevent misclassification (closes #21)
Before this fix, level strings with surrounding whitespace (e.g. " WARN ")
were incorrectly parsed as Unknown instead of the matching variant.
This primarily affected the JSON log path where serde preserves the raw
field value including whitespace.

Changes:
- Add s.trim() before case-insensitive matching in FromStr impl
- Store trimmed value in Unknown variant to avoid whitespace noise
- Add unit tests for whitespace-padded known levels
- Add unit tests for Unknown trimming semantics (empty, internal ws)
- Add JSON regression test for level field with surrounding whitespace
2026-06-10 13:37:11 +08:00
dailz eedab3ac96 fix(parser): strip UTF-8 BOM before JSON parsing
serde_json rejects BOM (U+FEFF) prefixed input with 'expected value'
error, causing BOM-prefixed JSON log lines to be silently dropped.

Add strip_bom() helper that strips exactly one leading BOM character,
apply it in detect_json_log() and parse_line().

Closes #20
2026-06-10 11:37:00 +08:00
dailzandSisyphus 8e9600dda2 fix(parser): preserve non-string timestamp/level fields instead of silently dropping them (closes #19)
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-06-10 10:47:04 +08:00
dailz 2cebbd94c4 fix: concurrent cache save uses unique temp files (#17)
Replace deterministic .index.tmp path with per-save unique temp files
via tempfile::Builder. Eliminates race condition where background
indexer thread and TUI main thread could collide on the same temp path,
causing data truncation or corrupt cache writes.

Changes:
- Add write_cache_atomically() helper using tempfile::Builder
- Refactor save_with_hash() and save() to use the helper
- Extract encode_cache() to deduplicate serialization logic
- Move tempfile from [dev-dependencies] to [dependencies]
- Add 2 concurrent tests validating no corruption under parallel writes

Fixes #17
2026-06-09 16:13:39 +08:00
dailz 0d88e933e6 fix(io): replace blocking channel sends with cancel-aware alternatives (closes #16)
Background worker threads used blocking tx.send() on bounded channels.
If the consumer stopped draining, threads hung forever with no way to
reach the cancel check. Drop-issued cancellation was ineffective.

Changes:
- Progress messages: tx.try_send() (discard if full, never blocks loop)
- Terminal messages (Complete/Error): new send_cancelable<T>() helper
  using crossbeam select! — sleeps efficiently until send succeeds or
  cancel arrives
- Drop cancellation: tx.try_send() — Drop must never block
- spawn_visual_height_rebuild: same fix for its bounded(1) channel
- 5 new tests covering full-channel + cancel scenarios
2026-06-09 15:14:37 +08:00
dailz 420b853cb9 fix(watcher): filter Remove events by path to prevent false removed reports (closes #15) 2026-06-09 13:18:23 +08:00
dailz 7852e92ecc fix(watcher): forward notify backend errors instead of silently discarding
Previously Err(_) => return in the notify callback silently dropped all
backend errors (inotify exhaustion, fs unmount, permission loss), leaving
the application unaware that file monitoring had stopped working.

Add FileEvent::WatcherError { message: String } variant to propagate
backend errors through the existing bounded channel. The TUI consumer
receives the event without disrupting the UI for transient errors.

Closes #14
2026-06-09 11:26:54 +08:00
dailz d37ed6df68 fix(io): harden read_cache against zero-length false hits and overflow (closes #13)
- Add early return for len==0 (Ok(&[])) matching std::io semantics
- Add slot.len > 0 guard to cache hit predicate to prevent empty-slot
  false matches
- Replace unchecked arithmetic with checked_add/saturating_add for
  request_end, block_end, and post-read coverage check
- Fix misleading comment about get(file,0,0) behavior on miss path
- Strengthen clear() to fully reset block_offset and last_access
- Register read_cache module in io/mod.rs
- Add 4 regression tests: zero-len on fresh/populated cache,
  zero-len at u64::MAX, overflow error on nonzero read at u64::MAX
2026-06-09 10:48:34 +08:00
dailz b58d66f2aa fix(io): use unicode-width for correct CJK/emoji/zero-width display width (closes #12) 2026-06-07 12:50:17 +08:00
dailz d4679a7543 fix(io): update visual height of last line on append without trailing newline (closes #11)
When a file does not end with a newline, appending content extends the
last logical line's text and thus its visual height. The incremental
extend path in handle_file_appended only computed heights for newly
created logical lines, missing the old last line whose content changed.

Add VisualHeightIndex::replace_last_line_height() — an O(1) method that
rewrites the final prefix sum entry and total. Called before
extend_from_heights so the correct line is targeted.

Changes:
- progressive_reader.rs: add replace_last_line_height, pub with_params,
  7 VHI unit tests
- app.rs: save old_reader_line_count before update, recompute last old
  line height in extend path, 2 integration regression tests
2026-06-07 09:46:24 +08:00
dailz 8844e58cb4 Merge fix/m20-append-lines-error-handling: fix append_lines I/O error swallowing (closes #45) + clippy cleanup 2026-06-07 09:17:41 +08:00
dailz 6a2f8ecb66 fix(bench): resolve pre-existing clippy warnings in report.rs and mmap_reader.rs 2026-06-07 09:15:34 +08:00
dailz f6081b9fe9 fix(bench): propagate I/O errors in append_lines instead of silently defaulting to 0 (closes #45) 2026-06-07 09:13:37 +08:00
dailz 97a2c6a925 fix(bench): regenerate growable file each iteration in truncate safety benchmarks (closes #44) 2026-06-07 09:02:19 +08:00
dailz e6e0e2cc90 fix(bench): correct lines_read to actual successful reads in bench_scroll_rss
lines_read was incorrectly set to max_lines.min(total) (the loop upper bound)
instead of the actual number of successfully read lines. Now tracks lines_read
via get_line(i).is_some() counter and uses max_lines.min(total) as the correct
loop upper bound to handle empty file edge case.

Fixes #43
2026-06-07 08:50:20 +08:00
dailz ffaf462bae Merge fix/m23-single-frame-tail-overlap: [M23] small file single_frame_tail overlap fix 2026-06-07 08:32:58 +08:00
dailz a8dc067cd4 fix(bench): [M23] prevent single_frame_tail/head overlap for small files
- Extract shared FRAME_LINES constant into suites/mod.rs
- Add select_frame_positions() helper with 3*FRAME_LINES threshold
  to guarantee non-overlapping head/middle/tail ranges
- Guard bench_reverse_scan against total <= FRAME_LINES
- Add 9 boundary tests for position selection (0..1M lines)

Closes #42
2026-06-07 08:31:00 +08:00
dailz 502479677b fix(bench): warn when clear_file_cache fails instead of silently skipping cold benchmarks (closes #41) 2026-06-05 17:28:57 +08:00
dailz 5656b26d7b refactor(bench): unify line counting in get_file_info to use count_existing_lines
Reuse the existing count_existing_lines() (reader.lines().count())
instead of a manual read_until loop, eliminating duplicate line-counting
logic in data_gen.rs.

Closes #40
2026-06-05 17:04:46 +08:00
dailz a8b64e78bd fix(bench): stabilize report column/row ordering across input permutations (#39)
- variants: use direct tuple comparison instead of format! string, add sort() after dedup
- Memory section: sort rows by (test_name, backend, variant) before output
- Extra Metrics section: sort rows by (test_name, variant_label) before output
- add [lib] target to Cargo.toml to enable unit tests
- add regression test: same data in different input order produces identical report
2026-06-05 16:24:41 +08:00
dailz e945a357f7 fix(bench): warn on all reset_vm_hwm errors, not just PermissionDenied
Issue #38: warn_reset_hwm() silently swallowed non-permission I/O errors
from /proc/self/clear_refs (e.g. missing /proc, read-only procfs, kernel
incompatibility). This left users unaware that VmHWM reset failed and
memory peak data could be contaminated across suites.

Changes:
- runner.rs: all errors now produce a warning with specific failure reason;
  PermissionDenied retains 'try running as root' hint; AtomicBool warn-once
  prevents duplicate output across 7 suite runs
- main.rs: preflight check now uses warn_reset_hwm() instead of the vague
  can_reset_vm_hwm(), sharing the same warn-once mechanism
- metrics.rs: remove dead can_reset_vm_hwm() (no callers remaining)
- tests: add hwm_warned_flag_prevents_reentry and warn_reset_hwm_does_not_panic
2026-06-05 15:52:01 +08:00
dailz fb57584546 fix(bench): validate --suites names, reject unknown suites at CLI boundary
Introduce Suite enum (runner.rs) replacing stringly-typed suite matching.
BenchConfig.suites is now Option<Vec<Suite>>, making invalid states
unrepresentable. Unknown suite names produce a clear error listing all
valid values.

Fixes: #37
2026-06-05 15:20:24 +08:00
dailz 9baec5ab69 fix(bench): refresh PreadReader index periodically in scroll_during_append (closes #36)
The reader's line_index and file_size were frozen at open time.
After current_line exceeded the initial 150K lines, get_line_impl
returned None for all subsequent reads. With the background thread
appending ~10K lines/sec, ~40% of measured frame latencies were
actually the cost of a None return, not real I/O.

- Add PreadReaderCore::refresh_index(&mut self): seek to start,
  rebuild LineIndex, update file_size, invalidate read cache
- Add PreadReaderPlain::refresh_index forwarding method
- Add ReadCache::invalidate to force cache miss after reindex
- Rewrite bench_scroll_during_append: time-based refresh (250ms),
  only record latencies for successful reads, assert max_line > initial
- Add regression tests for refresh_index with appended lines
2026-06-05 14:40:32 +08:00
dailz 6dd87d2872 fix(bench): wrap file writes with BufWriter to reduce syscall overhead
Add BufWriter::with_capacity(64KB) to generate_test_file,
generate_growable_file, and append_lines in data_gen.rs.

Previously each writeln! triggered an individual write syscall,
making 5GB/74M-line benchmark data generation extremely slow.
BufWriter batches writes into 64KB chunks, reducing syscalls
by ~1000x.

Explicit flush()? + drop before subsequent reads ensures data
visibility and propagates flush errors (BufWriter::drop swallows
them).

Closes #35
2026-06-05 14:01:35 +08:00
dailz 83f633a562 fix(bench): make can_reset_vm_hwm side-effect-free with open probe (closes #34) 2026-06-05 13:34:01 +08:00
dailz dad5f5a635 fix(bench): eliminate SIGBUS handler static mut UB with Once + raw atomics (closes #33)
Replace `static mut OLD_SIGBUS_HANDLER` with AtomicU8 + AtomicPtr to
remove data race UB when concurrent benchmarks call open() from multiple
threads.

Key changes:
- Use `Once::call_once` to guarantee single handler installation
- Publish old handler to atomics BEFORE installing new handler (closes
  the handler-active-but-state-unpublished race window)
- Read atomics with Acquire in signal handler (async-signal-safe)
- Align si_addr to page boundary before mmap(MAP_FIXED)
- Add concurrent test: 8 threads open all 5 variants simultaneously
2026-06-05 13:22:02 +08:00
dailz 534a089b58 fix(tui): defer file-change events during Loading state to prevent stale reader (closes #10)
Loading state silently dropped FileEvent::Appended/Truncated via _ => {}.
After Loading→Ready transition the FileReader was based on a stale snapshot.

- Add reload_after_loading flag to defer reload until Ready state
- Extract reload_ready_reader() from handle_file_truncated
- Explicit 3-branch match: Ready handles, Loading sets flag, rest ignores
- Clear flag on IndexerMessage::Error to prevent stale dirty bit
- 4 regression tests covering append/truncate/collapse/error paths
2026-06-04 17:32:58 +08:00
dailz b7938e069d fix(tui): RAII TerminalGuard prevents terminal corruption on error exit (closes #8)
The previous code called std::process::exit(1) on file load failure,
bypassing all terminal restoration (disable_raw_mode, LeaveAlternateScreen,
show_cursor). This left the user's shell in a broken state with no echo
and no visible cursor.

Introduce TerminalGuard with Drop-based cleanup that fires on every exit
path: normal return, ? error propagation, and panic unwind. The guard
also handles partial initialization rollback (e.g. raw mode enabled but
alternate screen fails). Remove all process::exit calls from the guarded
scope.
2026-06-04 16:11:16 +08:00
dailz d40d70c600 fix(watcher): use try_send to prevent blocking notify callback
Replace tx.send() with tx.try_send() in file watcher notify callback
to avoid blocking the notify thread when the bounded channel (cap 100)
is full. Events are silently dropped instead of blocking, preventing
shutdown delays and event loss during consumer stalls.

Closes #7
2026-06-04 14:33:40 +08:00
dailz 1350f659fa fix(io): eliminate SIGBUS risk in background indexer threads (closes #6)
Background threads (spawn_indexer, spawn_visual_height_rebuild) previously
held mmap during entire file scan, risking SIGBUS if file was truncated
externally. Now uses BufReader streaming scan with mmap created only
after scan completes, plus stat validation.

Changes:
- spawn_indexer: replace mmap scan with BufReader fill_buf/consume loop,
  create mmap post-scan with fd stat validation
- spawn_visual_height_rebuild: replace mmap/FileReader with sequential
  BufReader scan, discard results on line count mismatch
- FileReader::open/reload/update_for_append: add stat-after-mmap check
- LineIndex: make fields pub(crate) for direct construction from scan loop
- Add 3 regression tests for truncation scenarios
2026-06-04 14:10:51 +08:00
dailz 1bb6b2e9f3 fix: eliminate TOCTOU race in IndexCache::save (closes #5)
spawn_indexer builds LineIndex from mmap snapshot but IndexCache::save()
re-opened the file to compute the hash. If the file changed between those
two steps, the cached index would be stored under the wrong hash.

- Add IndexCache::save_with_hash() that computes hash from in-memory data
- Add compute_data_hash() public function (same algorithm as compute_file_hash)
- Update spawn_indexer, FileReader::save_cache, and test callers
2026-06-04 13:30:16 +08:00
dailz bef0b44e91 fix(io): guard sampled_offsets index in get_line() to prevent panic on corrupt cache
Replace direct indexing with .get()? to return None instead of panicking
when sampled_offsets is shorter than expected (e.g. corrupt bincode cache).

Closes #3
2026-06-03 15:39:05 +08:00
dailz 24fe97a457 fix(io): eliminate TOCTOU race in FileReader open/reload
Replace LineIndex::from_reader(BufReader) with LineIndex::from_bytes(&mmap)
in both open() and reload(), ensuring the line index is always built from
the same mmap snapshot rather than a separate read through the file descriptor.

This closes the race window where an external file modification between
mmap() and from_reader() could cause line_index offsets to disagree with
the mmap data, leading to get_line() returning wrong content or panicking.

Closes #2
2026-06-03 15:08:22 +08:00
dailz b6e655bff6 fix(io): handle file shrink in update_for_append to prevent SIGBUS
File truncation/rotation during mmap lifetime caused SIGBUS crash
because update_for_append() ignored new_size < old_size, leaving
a stale mapping that would fault on access.

Introduce AppendStatus enum (Unchanged/Appended/Reloaded) so the
caller can distinguish shrink events from normal appends. On shrink,
reload() rebuilds the mmap and line index. The TUI layer clamps
cursor and invalidates viewport cache on Reloaded, matching the
existing handle_file_truncated() behavior.

Fixes #1
2026-06-03 14:47:23 +08:00
50 changed files with 9339 additions and 1453 deletions
+2 -2
View File
@@ -2,9 +2,9 @@ name: CI
on: on:
push: push:
branches: [main] branches: [master]
pull_request: pull_request:
branches: [main] branches: [master]
env: env:
CARGO_TERM_COLOR: always CARGO_TERM_COLOR: always
+8
View File
@@ -1,5 +1,13 @@
/target/ /target/
crates/*/target/
*.swp *.swp
*.swo *.swo
.idea/ .idea/
.vscode/ .vscode/
# Agent tooling state
.omo/
.sisyphus/
# Benchmark output
benchmark-report.md
+122
View File
@@ -0,0 +1,122 @@
# logViewer
High-performance log file viewer. Rust workspace with 4 crates, edition 2024, MSRV 1.92 (pinned in `rust-toolchain.toml`).
## Workspace Layout
```
crates/core — log-viewer-core (library) — I/O, parsing, types, config, file watching
crates/tui — log-viewer-tui (binary) — terminal UI (ratatui + crossterm)
crates/gui — log-viewer-gui (binary) — GUI (egui + eframe) — STUB, not yet functional
crates/bench — log-viewer-bench (binary) — mmap vs pread benchmark harness
```
`core` owns all I/O, parsing, data types, and file watching. TUI/GUI consume core's public API. GUI is a placeholder with no core integration yet.
**`default-members = ["crates/core"]`** — bare `cargo run` / `cargo test` target *core*, not the TUI. Always pass `-p log-viewer-tui` (or `--workspace`) explicitly.
## Commands
```bash
# Run the TUI (primary interface)
cargo run -p log-viewer-tui -- <log-file>
# Build/check/test/lint the whole workspace
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# Single crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
# Benchmarks (test file is NOT auto-generated — see "Benchmarks" section below)
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
```
## CI Gate (must pass before merge)
Runs on ubuntu-latest + windows-latest (`.github/workflows/ci.yml`):
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
Toolchain installed via `dtolnay/rust-toolchain@stable` reading `rust-toolchain.toml` (channel `1.92`). No rustfmt.toml or clippy.toml — uses toolchain defaults.
## Core Architecture
All ten modules are declared `pub` in `lib.rs` and re-exported as `log_viewer_core::<module>`.
### Key Modules
- **`types`** — `LogLevel`, `LogEntry`, `SearchQuery`, `SearchFilter`, `Bookmark`, `FileSession`, `DuplicateKey`, `SearchResult`
- **`error`** — `CoreError` (**12 variants** via thiserror: Io, Parse, Search, Index, Config, TomlSerialize, Encoding, Watch, Mmap, Cache, FileNotFound, Other) + `Result<T>` alias. `From` impls for `io::Error` / `serde_json::Error` / `toml::de::Error` / `toml::ser::Error` / `notify::Error` mean `?` works freely.
- **`io::file_reader`** — mmap-backed `FileReader` with TOCTOU mitigation (post-mmap stat check), append/reload detection
- **`io::progressive_reader`** — `ProgressiveFileReader` state machine (`ReaderState::Sampling { .. } | Ready { .. }`), background indexer via crossbeam-channel, `VisualHeightIndex` for wrapped-line scroll
- **`io::line_index`** — Sparse index (every `BLOCK_SIZE = 256` lines, memchr SIMD). Serializable to disk.
- **`io::index_cache`** — Persistent cache with xxh3 content hash, atomic writes via temp files
- **`io::line_sampler`** / **`io::cache_util`** — shared helpers used by the index path
- **`io::wrap`** — Unicode-aware line wrapping (CJK/emoji/tab), JSON pretty-printing; enforces `MAX_WRAP_INPUT_LEN`
- **`io::read_cache`** — LRU read cache (not yet integrated into the live reader; future pread backend)
- **`parser::json`** — NDJSON parser with BOM handling, duplicate key detection
- **`parser::level`** — JSON-first level detection (`detect_level`); falls back to bounded keyword scan with word-boundary check on non-JSON lines
- **`watcher::file_watcher`** — File event watcher (notify + crossbeam) with append/truncate/rotation detection
- **`config`** — `ColorConfig` TOML load/save with per-field serde defaults
### Stubs (single-line TODOs — planned, not yet built)
- `filter::Filter`
- `bookmark::BookmarkManager`
- `session::SessionManager`
- `search::engine::SearchEngine`
## TUI Architecture
- **`main.rs`** — clap CLI (`files: Vec<String>`), `TerminalGuard` RAII for raw mode + alternate screen. **CRITICAL**: `Drop` does not run on `std::process::exit` or `panic = "abort"` — never call `process::exit` inside the guarded scope; return `Err` and use `?` instead. Event loop: poll indexer → poll watcher → draw → poll keys, 100ms timeout.
- **`app/`** — module directory (split from the former monolithic `app.rs`). `mod.rs` (~3000 lines, inline `tests` mod at `#[cfg(test)] mod tests`) holds the `App` struct, `AppLoadingState: Empty | Loading { reader, estimated_lines, progress_percent } | Ready { reader } | Error(String)`, `AppMode: Normal | Settings`, `ViewportRenderRow`, and the main impl blocks. Per-concern helpers are split into submodules:
- `input.rs``handle_key` + Normal-mode key dispatch
- `settings.rs` — Settings-mode key dispatch (←/→ color cycle, j/k level select, Enter/Esc)
- `scroll.rs` — line/half/full-page scroll + VHI-aware sub-offset walk
- `viewport.rs` — viewport computation helpers
- `viewport_cache.rs` — on-demand viewport-sized render cache
- `loading.rs` — loading-state polling/progress helpers
- `watcher.rs` — file-watcher event polling + reload/reindex glue
- `query.rs` — query helpers
- **`ui.rs`** — ratatui rendering: title bar, content area, status bar, settings popup
- **`color.rs`** — `LogLevel` → ratatui `Color` via `ColorConfig`
### TUI Keybindings
j/k scroll, Ctrl+d/u half-page, Ctrl+f/b full-page, G/gg jump end/top, Tab toggle JSON formatting, S settings panel, q/Esc quit. Settings: j/k select level, ←/→ cycle colors, Enter save, Esc cancel.
## Testing
All tests are inline (`#[cfg(test)] mod tests` blocks). No `tests/` directories, no integration tests, no async tests.
- **~470 tests total**: core 262, tui 142, bench 63, gui 0
- Temp-file helpers vary per module — check the local `tests` mod before assuming an API:
- `make_temp_file(content) -> PathBuf``tui/app/mod.rs`, `tui/ui.rs`
- `make_test_file(lines) -> NamedTempFile``core/io/index_cache.rs`
- `make_file(data) -> NamedTempFile``core/io/read_cache.rs`
- `struct TempFile { ... }``core/io/line_sampler.rs`
- `insta` is a declared dev-dependency but currently **unused** (no `insta::` calls anywhere in the workspace)
- TUI render tests use ratatui `TestBackend` (see `tui/ui.rs`)
- Bench crate has its own unit tests for mmap/pread backends, metrics, and data_gen helpers
## Benchmarks
Custom harness (not criterion). Wall-clock + `/proc/self/` RSS/page fault metrics. 7 suites: startup, render, jump, memory, growth, rotation, concurrent. Tests mmap (plain/sequential/random/populate/phase_aware) vs pread (plain/random/sequential) backends. Output: markdown tables in `benchmark-report.md`.
**The harness does NOT auto-generate the test file.** `main.rs` requires `/tmp/test-logviewer/extreme.log` to already exist; if missing it prints the `dd` command and exits. Generate it manually first:
```bash
mkdir -p /tmp/test-logviewer
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000 # ~5GB
```
(`crates/bench/src/data_gen.rs` has `generate_test_file` / `ensure_test_file` helpers and is used by the `growth` and `rotation` suites for per-run growable files, but it is **not** wired into `main.rs` for the primary test file.)
Note: `log-viewer-bench` depends directly on `memmap2` / `nix` / `libc` / `memchr` — it does **not** depend on `log-viewer-core`, so it can be built standalone with `cargo build -p log-viewer-bench`.
Generated
+2
View File
@@ -2333,6 +2333,7 @@ dependencies = [
"tempfile", "tempfile",
"thiserror 2.0.18", "thiserror 2.0.18",
"toml", "toml",
"unicode-width",
"xxhash-rust", "xxhash-rust",
] ]
@@ -2357,6 +2358,7 @@ dependencies = [
"ratatui", "ratatui",
"serde_json", "serde_json",
"tempfile", "tempfile",
"unicode-width",
] ]
[[package]] [[package]]
+1
View File
@@ -27,3 +27,4 @@ textwrap = "0.16"
tempfile = "3" tempfile = "3"
xxhash-rust = { version = "0.8", features = ["xxh3"] } xxhash-rust = { version = "0.8", features = ["xxh3"] }
bincode = "1" bincode = "1"
unicode-width = "0.2"
+109
View File
@@ -0,0 +1,109 @@
# logViewer
**[English](README.md)**
用 Rust 构建的高性能终端日志查看器,专为处理超大日志文件(GB 级别)而设计,内存占用极低。
基于 mmap 内存映射、稀疏行索引和后台渐进式加载,打开文件几乎瞬时完成——即使是 5GB 以上的日志文件,也能在索引完成前就开始滚动浏览。
## 特性
- **瞬时打开** — mmap 读取 + 后台渐进式索引,索引未完成即可开始浏览
- **超大文件支持** — 稀疏行索引(每 256 行采样一次),5GB 文件仅占用约 8MB 内存
- **实时文件追踪** — 通过 `notify` 监控文件追加、截断和日志轮转
- **JSON 日志解析** — 支持 NDJSON 格式,自动处理 BOM、检测重复键,可切换格式化显示
- **Unicode 感知的换行** — 正确处理中文、emoji 和制表符的行宽计算
- **持久化索引缓存** — 行索引保存到磁盘,使用 xxh3 哈希校验内容变更,再次打开几乎无需等待
- **Vim 风格快捷键** — 终端用户无学习成本
- **可自定义配色** — 每个日志级别独立配色,通过 TOML 配置文件管理,TUI 内直接调整
## 快速开始
```bash
# 直接编译运行
cargo run -p log-viewer-tui -- path/to/logfile.log
# 或先编译再运行
cargo build --release -p log-viewer-tui
./target/release/log-viewer-tui path/to/logfile.log
```
需要 Rust 1.92 及以上版本(见 `rust-toolchain.toml`)。
## 快捷键
| 按键 | 操作 |
|------|------|
| `j` / `↓` | 向下滚动一行 |
| `k` / `↑` | 向上滚动一行 |
| `Ctrl+d` | 向下滚动半页 |
| `Ctrl+u` | 向上滚动半页 |
| `Ctrl+f` / `PgDn` | 向下滚动一页 |
| `Ctrl+b` / `PgUp` | 向上滚动一页 |
| `G` / `End` | 跳转到文件末尾 |
| `gg` / `Home` | 跳转到文件开头 |
| `Tab` | 切换 JSON 格式化显示 |
| `S` | 打开配色设置 |
| `q` / `Esc` | 退出 |
设置面板:`j`/`k` 选择日志级别,`←`/`→` 切换颜色,`Enter` 保存,`Esc` 取消。
## 项目结构
```
crates/core — 核心库:I/O、解析、数据类型、配置、文件监控
crates/tui — 终端界面(ratatui + crossterm
crates/gui — 图形界面(egui + eframe)— 占位模块,尚未实现
crates/bench — mmap 与 pread 性能对比基准测试
```
## 开发
```bash
# 检查、测试、代码规范
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# 运行基准测试(需要先生成约 5GB 测试文件)
mkdir -p /tmp/test-logviewer
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
# 单独测试某个 crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
```
### CI
CI 在 ubuntu-latest 和 windows-latest 上运行以下检查:
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
无自定义 rustfmt 或 clippy 配置——使用工具链默认值。
## 架构
核心库 `log-viewer-core` 负责所有 I/O、解析和数据类型。关键设计决策:
- **稀疏行索引** — 每 256 行采样一次,使用 memchr SIMD 加速。100 万行的文件仅产生约 32KB 的索引,而非 8MB。
- **渐进式加载** — `ProgressiveFileReader` 先读取文件头尾进行快速行数估算,再通过 crossbeam-channel 在后台线程构建完整索引。
- **mmap 与 TOCTOU 防护** — mmap 映射后进行 stat 校验以检测文件变更。`read_cache` 模块为未来基于 pread 的实现预留,可彻底消除 SIGBUS 风险。
- **持久化缓存** — 行索引序列化到磁盘,使用 xxh3 内容哈希进行失效校验,通过临时文件实现原子写入。
- **视觉高度索引** — 基于换行后行高的前缀和数组,支持 O(log n) 的滚动定位,适用于长行换行场景。
计划中但尚未实现的功能:过滤、书签、会话管理、搜索引擎(stub 模块已创建)。
## 基准测试
自研基准测试框架,对比 mmap 与 pread 两种后端在 7 个测试场景(启动、渲染、跳转、内存、增长、轮转、并发)下的表现。使用挂钟计时和 `/proc/self/` 的 RSS 及页错误指标。结果以 Markdown 表格形式输出到 `benchmark-report.md`
基准测试框架要求 `/tmp/test-logviewer/extreme.log` 已经存在。它不会自动生成主用的 5GB 测试文件;请先使用上方的 `dd` 命令创建。
基准测试二进制包含约 75 个单元测试,覆盖读取器后端和指标采集逻辑。
+108
View File
@@ -0,0 +1,108 @@
# logViewer
**[中文文档](README-zh.md)**
A high-performance terminal log file viewer built in Rust, designed to handle multi-gigabyte files with minimal memory overhead.
Uses memory-mapped I/O with a sparse line index and progressive background loading to open files instantly — even 5GB+ logs scroll smoothly from the first keystroke.
## Features
- **Instant file open** — mmap-backed reader with background progressive indexing; start scrolling before indexing finishes
- **Handles huge files** — sparse line index (1 entry per 256 lines) keeps memory usage at ~8MB even for 5GB files
- **Live file tracking** — watches for appends, truncations, and log rotation via `notify`
- **JSON log support** — NDJSON parsing with BOM handling, duplicate key detection, and toggleable pretty-printing
- **Unicode-aware wrapping** — correct line wrapping for CJK, emoji, and tabs
- **Persistent index cache** — line indexes saved to disk with xxh3 content hashing; re-opens are near-instant
- **Vim-like keybindings** — familiar navigation for terminal users
- **Customizable colors** — per-log-level colors via TOML config, adjustable from within the TUI
## Quick Start
```bash
# Build and run
cargo run -p log-viewer-tui -- path/to/logfile.log
# Or build first
cargo build --release -p log-viewer-tui
./target/release/log-viewer-tui path/to/logfile.log
```
Requires Rust 1.92+ (see `rust-toolchain.toml`).
## Keybindings
| Key | Action |
|-----|--------|
| `j` / `↓` | Scroll down one line |
| `k` / `↑` | Scroll up one line |
| `Ctrl+d` | Scroll down half page |
| `Ctrl+u` | Scroll up half page |
| `Ctrl+f` / `PgDn` | Scroll down full page |
| `Ctrl+b` / `PgUp` | Scroll up full page |
| `G` / `End` | Jump to end of file |
| `gg` / `Home` | Jump to start of file |
| `Tab` | Toggle JSON pretty-printing |
| `S` | Open color settings |
| `q` / `Esc` | Quit |
Settings panel: use `j`/`k` to select a log level, `←`/`→` to cycle colors, `Enter` to save, `Esc` to cancel.
## Workspace Structure
```
crates/core — Shared library: I/O, parsing, types, config, file watching
crates/tui — Terminal UI (ratatui + crossterm)
crates/gui — GUI (egui + eframe) — placeholder, not yet functional
crates/bench — mmap vs pread benchmark harness
```
## Development
```bash
# Check, test, lint
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# Run benchmarks (requires a pre-generated ~5GB test file)
mkdir -p /tmp/test-logviewer
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
# Test a single crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
```
### CI
The CI gate runs on ubuntu-latest and windows-latest:
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
No custom rustfmt or clippy config — uses toolchain defaults.
## Architecture
The core library (`log-viewer-core`) owns all I/O, parsing, and data types. Key design decisions:
- **Sparse line index** — samples every 256 lines using memchr SIMD acceleration. A 1M-line file produces a ~32KB index instead of ~8MB.
- **Progressive loading** — `ProgressiveFileReader` starts with a quick head+tail sample for instant line estimates, then builds the full index in a background thread via crossbeam-channel.
- **mmap with TOCTOU mitigation** — post-mmap stat check to detect file changes. `read_cache` module exists as a future pread-based alternative to eliminate SIGBUS risk entirely.
- **Persistent cache** — line indexes are serialized to disk with xxh3 content hashing for invalidation. Atomic writes via temp files.
- **Visual height index** — prefix-sum over wrapped-line heights, enabling O(log n) scroll-to-line mapping for long wrapped lines.
Planned but not yet built: filtering, bookmarks, sessions, search engine (stub modules exist).
## Benchmarks
Custom harness comparing mmap vs pread backends across 7 suites (startup, render, jump, memory, growth, rotation, concurrent). Uses wall-clock timing with `/proc/self/` RSS and page fault metrics. Results are written as markdown tables to `benchmark-report.md`.
The harness expects `/tmp/test-logviewer/extreme.log` to already exist. It does not generate the primary 5GB test file automatically; create it first with the `dd` command shown above.
The benchmark binary includes ~75 unit tests for the reader backends and metrics collection.
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "log-viewer-bench"
version = "0.1.0"
edition = "2024"
[lib]
name = "log_viewer_bench"
path = "src/lib.rs"
[[bin]]
name = "log-viewer-bench"
path = "src/main.rs"
[dependencies]
memmap2 = "0.9"
nix = { version = "0.30", features = ["signal", "resource", "mman", "fs"] }
libc = "0.2"
memchr = "2"
serde_json = "1"
clap = { version = "4", features = ["derive"] }
crossbeam-channel = "0.5"
tempfile = "3"
+221
View File
@@ -0,0 +1,221 @@
use std::fs;
use std::io::{BufRead, BufReader, BufWriter, Write};
use std::path::{Path, PathBuf};
pub struct TestFileInfo {
pub path: PathBuf,
pub size_bytes: u64,
pub line_count: u64,
pub avg_line_length: f64,
}
/// Check if test file exists and return its info, or generate it
pub fn ensure_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
if path.exists() {
return get_file_info(path);
}
generate_test_file(path)
}
/// Get info about an existing test file
fn get_file_info(path: &Path) -> std::io::Result<TestFileInfo> {
let metadata = fs::metadata(path)?;
let size_bytes = metadata.len();
let line_count = count_existing_lines(path)?;
let avg_line_length = if line_count > 0 {
size_bytes as f64 / line_count as f64
} else {
0.0
};
Ok(TestFileInfo {
path: path.to_path_buf(),
size_bytes,
line_count,
avg_line_length,
})
}
/// Generate a large test file (~5GB / ~74M lines) if it doesn't exist
fn generate_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(path)?);
let target_lines: u64 = 74_000_000;
for i in 0..target_lines {
writeln!(
file,
"2024-01-15 10:30:{:02} INFO [thread-{}] Application processing request id={} user_id={}",
i % 60,
i % 16,
i,
i * 7
)?;
}
file.flush()?;
drop(file);
get_file_info(path)
}
/// Generate a smaller file (~10MB / ~150K lines) for growth/rotation tests
pub fn generate_growable_file(dir: &Path) -> std::io::Result<PathBuf> {
fs::create_dir_all(dir)?;
let path = dir.join("growable.log");
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(&path)?);
for i in 0..150_000u64 {
writeln!(
file,
"2024-01-15 10:30:{:02} INFO [thread-{}] Appending test line {}",
i % 60,
i % 16,
i
)?;
}
file.flush()?;
Ok(path)
}
/// Append `count` lines to the file
pub fn append_lines(path: &Path, count: usize) -> std::io::Result<()> {
let existing_lines = count_existing_lines(path)?;
let mut file =
BufWriter::with_capacity(64 * 1024, fs::OpenOptions::new().append(true).open(path)?);
for i in 0..count {
writeln!(
file,
"2024-01-15 10:30:00 INFO Appended line {}",
existing_lines + i as u64
)?;
}
file.flush()?;
Ok(())
}
/// Truncate file to specified size
pub fn truncate_file(path: &Path, size: u64) -> std::io::Result<()> {
let file = fs::OpenOptions::new().write(true).open(path)?;
file.set_len(size)
}
/// Rotate file: rename existing file, create new empty file
pub fn rotate_file(path: &Path) -> std::io::Result<PathBuf> {
let rotated = path.with_extension("log.1");
fs::rename(path, &rotated)?;
fs::File::create(path)?;
Ok(rotated)
}
/// Count lines in a file (helper)
fn count_existing_lines(path: &Path) -> std::io::Result<u64> {
let file = fs::File::open(path)?;
let reader = BufReader::new(file);
let mut count = 0u64;
for line in reader.lines() {
line?;
count += 1;
}
Ok(count)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_generate_growable_file_creates_approximately_correct_size() {
let dir = tempfile::tempdir().unwrap();
let path = generate_growable_file(dir.path()).unwrap();
assert!(path.exists());
let metadata = fs::metadata(&path).unwrap();
let size_mb = metadata.len() as f64 / (1024.0 * 1024.0);
// ~150K lines × ~67 bytes ≈ ~10MB; allow 5MB15MB range
assert!(
(5.0..=15.0).contains(&size_mb),
"Expected ~10MB, got {size_mb:.1}MB"
);
}
#[test]
fn test_append_lines_increases_line_count() {
let dir = tempfile::tempdir().unwrap();
let path = {
let mut f = fs::File::create(dir.path().join("test.log")).unwrap();
for i in 0..10u64 {
writeln!(f, "line {i}").unwrap();
}
dir.path().join("test.log")
};
let before = count_existing_lines(&path).unwrap();
append_lines(&path, 5).unwrap();
let after = count_existing_lines(&path).unwrap();
assert_eq!(after, before + 5);
}
#[test]
fn test_truncate_file_reduces_size() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("trunc.log");
{
let mut f = fs::File::create(&path).unwrap();
write!(f, "{}", "A".repeat(1024)).unwrap();
}
let before = fs::metadata(&path).unwrap().len();
assert_eq!(before, 1024);
truncate_file(&path, 512).unwrap();
let after = fs::metadata(&path).unwrap().len();
assert_eq!(after, 512);
}
#[test]
fn test_rotate_file_renames_and_creates_empty() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("rotate.log");
{
let mut f = fs::File::create(&path).unwrap();
write!(f, "original content").unwrap();
}
let rotated = rotate_file(&path).unwrap();
// Rotated file has the old content
assert!(rotated.exists());
assert_eq!(fs::read_to_string(&rotated).unwrap(), "original content");
// New file is empty
assert!(path.exists());
assert_eq!(fs::metadata(&path).unwrap().len(), 0);
}
#[test]
fn test_ensure_test_file_generates_when_missing() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("fresh.log");
assert!(!path.exists());
// Override the generator to use a small file for test speed:
// We'll test ensure_test_file indirectly by checking it calls generate_test_file.
// Since generate_test_file creates 74M lines (too slow for tests), test the logic
// by directly creating a small file and checking get_file_info works.
{
let mut f = fs::File::create(&path).unwrap();
for i in 0..100u64 {
writeln!(f, "2024-01-15 10:30:00 INFO line {i}").unwrap();
}
}
let info = ensure_test_file(&path).unwrap();
assert_eq!(info.line_count, 100);
assert!(info.size_bytes > 0);
assert!(info.avg_line_length > 0.0);
}
}
+24
View File
@@ -0,0 +1,24 @@
pub mod data_gen;
pub mod line_index;
pub mod metrics;
pub mod mmap_reader;
pub mod pread_reader;
pub mod report;
pub mod runner;
pub mod suites;
pub mod types;
use std::path::Path;
/// A single reader backend (mmap or pread)
pub trait FileReaderBackend {
fn name(&self) -> &str;
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized;
fn file_size(&self) -> u64;
fn total_lines(&self) -> usize;
fn get_line(&self, idx: usize) -> Option<String>;
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>>;
fn close(self);
}
+110
View File
@@ -0,0 +1,110 @@
// ─── line_index.rs ───────────────────────────────────────────────────────────
// Vendored from crates/core/src/io/line_index.rs
// Sparse line index: sample every 256 lines to reduce memory usage.
// ──────────────────────────────────────────────────────────────────────────────
const BLOCK_SIZE: usize = 256;
pub struct LineIndex {
pub(crate) sampled_offsets: Vec<u64>,
pub(crate) total_lines: u64,
#[allow(dead_code)]
pub(crate) has_trailing_newline: bool,
}
impl LineIndex {
/// Build sparse line index from a streaming reader.
/// Uses fill_buf()/consume() to avoid loading the entire file into memory.
/// RSS stays at ~64KB (BufReader buffer size), independent of file size.
pub fn from_reader(reader: &mut impl std::io::BufRead) -> std::io::Result<Self> {
let mut sampled_offsets: Vec<u64> = vec![0]; // line 0 starts at offset 0
let mut next_line_idx: usize = 1;
let mut newline_count: usize = 0;
let mut chunk_offset: u64 = 0;
let mut last_byte: Option<u8> = None;
loop {
let buf = reader.fill_buf()?;
if buf.is_empty() {
break;
}
if let Some(&b) = buf.last() {
last_byte = Some(b);
}
for pos in memchr::memchr_iter(b'\n', buf) {
newline_count += 1;
if next_line_idx.is_multiple_of(BLOCK_SIZE) {
sampled_offsets.push(chunk_offset + pos as u64 + 1);
}
next_line_idx += 1;
}
let consumed = buf.len();
chunk_offset += consumed as u64;
reader.consume(consumed);
}
// Empty file: no data at all
if chunk_offset == 0 {
return Ok(LineIndex {
sampled_offsets: vec![],
total_lines: 0,
has_trailing_newline: false,
});
}
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
let total_lines: u64 = if has_trailing_newline && newline_count > 0 {
newline_count as u64
} else {
(1 + newline_count) as u64
};
// Trailing \n pop logic
if has_trailing_newline && newline_count > 0 {
let trailing_line_idx = newline_count;
if trailing_line_idx.is_multiple_of(BLOCK_SIZE) {
sampled_offsets.pop();
}
}
Ok(LineIndex {
sampled_offsets,
total_lines,
has_trailing_newline,
})
}
/// Return total line count.
pub fn line_count(&self) -> usize {
self.total_lines as usize
}
/// Retrieve the content of line `idx` from the given data slice.
/// Uses sparse index to locate the block start, then scans forward
/// a small number of newlines to find the target line.
pub fn get_line<'a>(&self, data: &'a [u8], idx: usize) -> Option<&'a str> {
if idx >= self.total_lines as usize || data.is_empty() {
return None;
}
let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE;
let mut pos = self.sampled_offsets[block] as usize;
for _ in 0..offset_in_block {
match memchr::memchr(b'\n', &data[pos..]) {
Some(rel) => pos = pos + rel + 1,
None => return None,
}
}
let end = memchr::memchr(b'\n', &data[pos..])
.map(|rel| pos + rel)
.unwrap_or(data.len());
let line_bytes = &data[pos..end];
std::str::from_utf8(line_bytes)
.map(|s| s.trim_end_matches(['\r', '\n']))
.ok()
}
}
+83
View File
@@ -0,0 +1,83 @@
use clap::Parser;
use std::path::PathBuf;
/// Benchmark: mmap vs pread for large file reading
#[derive(Parser, Debug)]
#[command(name = "log-viewer-bench", version, about = "Benchmark mmap vs pread")]
struct Args {
/// Path to the test file (default: /tmp/test-logviewer/extreme.log)
#[arg(default_value = "/tmp/test-logviewer/extreme.log")]
test_file: PathBuf,
/// Quick mode: use smaller iterations and skip cold cache tests
#[arg(long)]
quick: bool,
/// Output report path (default: benchmark-report.md)
#[arg(long, default_value = "benchmark-report.md")]
output: PathBuf,
/// Only run specified suites (comma-separated: startup,render,jump,memory,growth,rotation,concurrent)
#[arg(long, value_delimiter = ',')]
suites: Option<Vec<String>>,
}
fn main() {
let args = Args::parse();
let suites = match args.suites {
Some(names) => {
let parsed: Result<Vec<_>, _> = names
.iter()
.map(|s| s.parse::<log_viewer_bench::runner::Suite>())
.collect();
match parsed {
Ok(s) => Some(s),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
}
None => None,
};
println!("=== Benchmark: mmap vs pread ===");
println!("Test file: {}", args.test_file.display());
println!("Quick mode: {}", args.quick);
println!();
let config = log_viewer_bench::runner::BenchConfig {
test_file: args.test_file.clone(),
quick_mode: args.quick,
suites,
};
if !config.test_file.exists() {
eprintln!("ERROR: Test file not found: {}", config.test_file.display());
eprintln!(
"Generate one with: dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000"
);
std::process::exit(1);
}
log_viewer_bench::runner::warn_reset_hwm();
println!("Running benchmarks...");
let results = log_viewer_bench::runner::run_all(&config);
println!("Completed {} benchmarks.\n", results.len());
let report = log_viewer_bench::report::format_report(&results);
println!("{}", report);
if let Err(e) = std::fs::write(&args.output, &report) {
eprintln!(
"WARNING: Failed to save report to {}: {}",
args.output.display(),
e
);
} else {
eprintln!("Report saved to {}", args.output.display());
}
}
+275
View File
@@ -0,0 +1,275 @@
use std::fs::{self, File};
use std::os::unix::fs::MetadataExt;
use std::os::unix::io::AsRawFd;
use std::path::Path;
pub struct RssMetrics {
pub vm_rss_kb: u64,
pub vm_hwm_kb: u64,
}
pub struct PageFaultMetrics {
pub minor_faults: u64,
pub major_faults: u64,
}
pub struct MetricsCollector;
impl MetricsCollector {
/// Read VmRSS and VmHWM from /proc/self/status
pub fn read_rss() -> RssMetrics {
let status = fs::read_to_string("/proc/self/status").unwrap_or_default();
let mut vm_rss_kb: u64 = 0;
let mut vm_hwm_kb: u64 = 0;
for line in status.lines() {
if line.starts_with("VmRSS:") {
vm_rss_kb = parse_kb_value(line);
} else if line.starts_with("VmHWM:") {
vm_hwm_kb = parse_kb_value(line);
}
}
RssMetrics {
vm_rss_kb,
vm_hwm_kb,
}
}
/// Read page fault counts from getrusage
pub fn read_page_faults() -> PageFaultMetrics {
let usage =
nix::sys::resource::getrusage(nix::sys::resource::UsageWho::RUSAGE_SELF).unwrap();
PageFaultMetrics {
// getrusage() returns c_long (i64 on 64-bit Linux) — explicit as u64 conversion
minor_faults: usage.minor_page_faults() as u64,
major_faults: usage.major_page_faults() as u64,
}
}
/// Clear page cache (requires root: sync + drop_caches)
/// Falls back to doing nothing if no permission
pub fn clear_page_cache() -> std::io::Result<()> {
let _ = std::process::Command::new("sync").status();
fs::write("/proc/sys/vm/drop_caches", "1")
}
/// Clear file cache using posix_fadvise(DONTNEED) — no root required
pub fn clear_file_cache(path: &Path) -> std::io::Result<()> {
let file = File::open(path)?;
let len = file.metadata()?.len();
let ret = unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, len as i64, libc::POSIX_FADV_DONTNEED)
};
// posix_fadvise returns error code directly (not errno), 0 = success
if ret != 0 {
return Err(std::io::Error::from_raw_os_error(ret));
}
Ok(())
}
/// Reset VmHWM by writing to /proc/self/clear_refs (requires root)
pub fn reset_vm_hwm() -> std::io::Result<()> {
fs::write("/proc/self/clear_refs", "5").map_err(|e| {
if e.kind() == std::io::ErrorKind::PermissionDenied {
std::io::Error::new(
std::io::ErrorKind::PermissionDenied,
"VmHWM reset requires root (can't write /proc/self/clear_refs)",
)
} else {
e
}
})
}
/// Get file inode number
pub fn get_inode(path: &Path) -> std::io::Result<u64> {
let meta = fs::metadata(path)?;
Ok(meta.ino())
}
/// Check if file was rotated (inode changed)
pub fn detect_rotation(original_inode: u64, path: &Path) -> bool {
Self::get_inode(path)
.map(|ino| ino != original_inode)
.unwrap_or(true)
}
}
fn parse_kb_value(line: &str) -> u64 {
// Format: "VmRSS: 12345 kB"
line.split_whitespace()
.nth(1)
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(0)
}
pub fn mean(data: &[u64]) -> f64 {
if data.is_empty() {
return 0.0;
}
data.iter().sum::<u64>() as f64 / data.len() as f64
}
/// Percentile of data at given fraction (0.01.0). Returns from a sorted copy.
pub fn percentile(data: &[u64], p: f64) -> u64 {
if data.is_empty() {
return 0;
}
let mut sorted: Vec<u64> = data.to_vec();
sorted.sort_unstable();
let idx = ((p * (sorted.len() - 1) as f64).round()) as usize;
sorted[idx.min(sorted.len() - 1)]
}
pub fn stdev(data: &[u64]) -> f64 {
if data.len() < 2 {
return 0.0;
}
let m = mean(data);
let variance: f64 = data
.iter()
.map(|&v| {
let d = v as f64 - m;
d * d
})
.sum::<f64>()
/ (data.len() - 1) as f64;
variance.sqrt()
}
pub fn p50(data: &[u64]) -> u64 {
percentile(data, 0.50)
}
pub fn p95(data: &[u64]) -> u64 {
percentile(data, 0.95)
}
pub fn p99(data: &[u64]) -> u64 {
percentile(data, 0.99)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_rss_returns_values() {
let rss = MetricsCollector::read_rss();
assert!(
rss.vm_rss_kb > 0,
"VmRSS should be non-zero for a running process"
);
assert!(
rss.vm_hwm_kb > 0,
"VmHWM should be non-zero for a running process"
);
}
#[test]
fn test_page_faults_returns_values() {
let faults = MetricsCollector::read_page_faults();
assert!(
faults.minor_faults > 0,
"Should have some minor page faults"
);
}
#[test]
fn test_mean() {
let data = vec![100, 200, 300, 400, 500];
let result = mean(&data);
assert!(
(result - 300.0).abs() < f64::EPSILON,
"mean should be 300.0, got {result}"
);
}
#[test]
fn test_mean_empty() {
assert_eq!(mean(&[]), 0.0);
}
#[test]
fn test_percentile_p50() {
let data = vec![100, 200, 300, 400, 500];
assert_eq!(percentile(&data, 0.50), 300);
}
#[test]
fn test_percentile_p99() {
let data = vec![10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
let p99_result = percentile(&data, 0.99);
assert!(p99_result >= 90, "P99 should be near max, got {p99_result}");
}
#[test]
fn test_percentile_empty() {
assert_eq!(percentile(&[], 0.5), 0);
}
#[test]
fn test_stdev() {
let data = vec![100, 200, 300, 400, 500];
let s = stdev(&data);
assert!(s > 100.0, "stdev should be significant, got {s}");
assert!(s < 200.0, "stdev should be < 200, got {s}");
}
#[test]
fn test_stdev_single() {
assert_eq!(stdev(&[42]), 0.0);
assert_eq!(stdev(&[]), 0.0);
}
#[test]
fn test_parse_kb_value() {
assert_eq!(parse_kb_value("VmRSS: 12345 kB"), 12345);
assert_eq!(parse_kb_value("VmHWM:\t2048 kB"), 2048);
assert_eq!(parse_kb_value("VmRSS: 0 kB"), 0);
}
#[test]
fn test_parse_kb_value_malformed() {
assert_eq!(parse_kb_value("VmRSS: NaN kB"), 0);
assert_eq!(parse_kb_value("garbage"), 0);
}
#[test]
fn test_convenience_percentiles() {
let data = vec![10, 20, 30, 40, 50];
assert_eq!(p50(&data), 30);
assert_eq!(p95(&data), 50);
assert_eq!(p99(&data), 50);
}
#[test]
fn test_inode_for_existing_file() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
assert!(inode > 0, "inode should be non-zero");
}
#[test]
fn test_detect_rotation_no_rotation() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
assert!(!MetricsCollector::detect_rotation(inode, tmp.path()));
}
#[test]
fn test_detect_rotation_file_removed() {
let inode: u64 = 99999;
let result = MetricsCollector::detect_rotation(inode, Path::new("/no/such/file"));
assert!(result, "missing file should indicate rotation");
}
#[test]
fn test_clear_file_cache() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let result = MetricsCollector::clear_file_cache(tmp.path());
assert!(
result.is_ok(),
"clear_file_cache should succeed on temp file: {result:?}"
);
}
}
+752
View File
@@ -0,0 +1,752 @@
// ─── mmap_reader.rs ──────────────────────────────────────────────────────────
// mmap-based FileReaderBackend implementations with 5 variants for benchmarking.
// Includes SIGBUS handler for file-truncation resilience and remap support
// for growing-file scenarios.
// ──────────────────────────────────────────────────────────────────────────────
use std::fs::File;
use std::io::BufReader;
use std::path::Path;
use std::sync::Once;
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU8, Ordering};
use memmap2::{Advice, Mmap, MmapOptions, RemapOptions};
use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction};
use crate::FileReaderBackend;
use crate::line_index::LineIndex;
// ─── SIGBUS Handler ──────────────────────────────────────────────────────────
//
// Signal-safety architecture:
// - Old handler state is stored in raw atomics (AtomicU8 + AtomicPtr) that are
// async-signal-safe to read — no OnceLock, Mutex, or other non-trivial abstractions
// in the signal handler path.
// - Installation uses `Once` for idempotency and follows a strict publish-then-install
// sequence to close the "handler-active-but-state-unpublished" race window.
/// Global flag set by the SIGBUS handler when a bus error is intercepted.
/// Process-global: concurrent benchmarks that reset/check this flag may interfere.
/// Currently acceptable because only `rotation` suite (sequential) uses it.
static SIGBUS_OCCURRED: AtomicBool = AtomicBool::new(false);
/// Discriminant values for old SIGBUS handler type.
const HANDLER_NONE: u8 = 0;
const HANDLER_DEFAULT: u8 = 1;
const HANDLER_IGNORE: u8 = 2;
const HANDLER_PLAIN: u8 = 3; // extern "C" fn(c_int)
#[allow(
clippy::unseparated_literal_suffix,
reason = "clarity: this is the SA_SIGACTION variant"
)]
const HANDLER_SIGACTION: u8 = 4; // extern "C" fn(c_int, *mut siginfo_t, *mut c_void)
/// Old SIGBUS handler type — raw atomic, async-signal-safe to read.
static OLD_HANDLER_KIND: AtomicU8 = AtomicU8::new(HANDLER_NONE);
/// Old SIGBUS handler function pointer — raw atomic, async-signal-safe to read.
static OLD_HANDLER_PTR: AtomicPtr<std::ffi::c_void> = AtomicPtr::new(std::ptr::null_mut());
/// Ensures `install_sigbus_handler` runs exactly once across all threads.
static INSTALL_ONCE: Once = Once::new();
/// Returns `true` if a SIGBUS was intercepted since the last reset.
pub fn sigbus_flag() -> bool {
SIGBUS_OCCURRED.load(Ordering::SeqCst)
}
/// Resets the SIGBUS flag. Call before operations where you want to detect
/// a fresh SIGBUS.
pub fn reset_sigbus_flag() {
SIGBUS_OCCURRED.store(false, Ordering::SeqCst)
}
/// SIGBUS signal handler.
///
/// # Safety Constraints (signal handler context)
/// - NO TLS access
/// - NO memory allocation
/// - NO lock acquisition
/// - Only: AtomicBool store → mmap → raw atomic loads → chain
extern "C" fn sigbus_handler(
sig: libc::c_int,
info: *mut libc::siginfo_t,
ctx: *mut std::ffi::c_void,
) {
SIGBUS_OCCURRED.store(true, Ordering::SeqCst);
// Align fault address down to page boundary.
// si_addr is NOT guaranteed to be page-aligned; without this, mmap(MAP_FIXED)
// could replace the wrong page.
let addr = unsafe { (*info).si_addr() };
let aligned = (addr as usize & !0xFFF) as *mut libc::c_void;
// Map anonymous zero page at fault address to prevent crash on file truncation.
let result = unsafe {
libc::mmap(
aligned,
4096,
libc::PROT_READ,
libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_FIXED,
-1,
0,
)
};
if result == libc::MAP_FAILED {
// Chain to old handler via raw atomics (async-signal-safe).
let kind = OLD_HANDLER_KIND.load(Ordering::Acquire);
match kind {
HANDLER_PLAIN => {
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
if !ptr.is_null() {
// SAFETY: ptr was derived from a valid function pointer
// published by install_sigbus_handler before this handler was installed.
let f: extern "C" fn(libc::c_int) = unsafe { std::mem::transmute(ptr) };
f(sig);
} else {
unsafe { libc::_exit(128 + sig) };
}
}
HANDLER_SIGACTION => {
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
if !ptr.is_null() {
let f: extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut std::ffi::c_void) =
unsafe { std::mem::transmute(ptr) };
f(sig, info, ctx);
} else {
unsafe { libc::_exit(128 + sig) };
}
}
_ => {
// HANDLER_NONE / HANDLER_DEFAULT / HANDLER_IGNORE — no safe chaining.
unsafe { libc::_exit(128 + sig) };
}
}
}
}
/// Install the SIGBUS handler via sigaction(). Chains to any previous handler.
/// Uses `Once` to guarantee exactly-once installation — safe to call from
/// multiple threads concurrently.
///
/// The install sequence closes the "install-then-publish" race by:
/// 1. Querying the current handler via raw `libc::sigaction` (no modification).
/// 2. Publishing old handler state to raw atomics (`Release` stores).
/// 3. Installing our handler via `nix::sigaction`.
///
/// This ensures the atomics are readable *before* our handler can fire.
fn install_sigbus_handler() {
INSTALL_ONCE.call_once(|| {
// Step 1: Query current SIGBUS disposition without changing it.
let mut old_act: libc::sigaction = unsafe { std::mem::zeroed() };
let ret = unsafe { libc::sigaction(libc::SIGBUS, std::ptr::null(), &mut old_act) };
if ret != 0 {
// Failed to query — skip installation entirely.
return;
}
// Step 2: Publish old handler to atomics BEFORE installing ours.
let is_siginfo = (old_act.sa_flags & libc::SA_SIGINFO) != 0;
let raw_usize = old_act.sa_sigaction as usize;
if raw_usize == 0 {
// SIG_DFL (null function pointer → default disposition)
OLD_HANDLER_KIND.store(HANDLER_DEFAULT, Ordering::Release);
} else if raw_usize == 1 {
// SIG_IGN (sentinel value 1 → ignore disposition)
OLD_HANDLER_KIND.store(HANDLER_IGNORE, Ordering::Release);
} else {
// Real handler function pointer
OLD_HANDLER_PTR.store(raw_usize as *mut std::ffi::c_void, Ordering::Release);
if is_siginfo {
OLD_HANDLER_KIND.store(HANDLER_SIGACTION, Ordering::Release);
} else {
OLD_HANDLER_KIND.store(HANDLER_PLAIN, Ordering::Release);
}
}
// Step 3: Install our handler. Atomics are already published,
// so the handler can safely read them from the moment it's installed.
let new_action = SigAction::new(
SigHandler::SigAction(sigbus_handler),
SaFlags::SA_SIGINFO,
SigSet::empty(),
);
let _ = unsafe { sigaction(Signal::SIGBUS, &new_action) };
});
}
// ─── Core MmapReader ─────────────────────────────────────────────────────────
/// Core mmap-based reader. Holds the memory mapping, file handle, and
/// line index. Used as the engine inside each variant wrapper.
pub struct MmapReader {
mmap: Mmap,
#[allow(dead_code)]
file: File,
line_index: LineIndex,
file_size: u64,
}
impl MmapReader {
/// Open a file and create an mmap. Does NOT apply madvise.
/// The caller is responsible for applying the desired advice.
fn open_raw(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let mmap = unsafe { Mmap::map(&file)? };
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
mmap,
file,
line_index,
file_size,
})
}
/// Open with MmapOptions (for populate, etc.).
fn open_with_options(path: &Path, opts: &MmapOptions) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let mmap = if file_size == 0 {
unsafe { Mmap::map(&file)? }
} else {
unsafe { opts.map(&file)? }
};
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
mmap,
file,
line_index,
file_size,
})
}
/// Apply madvise to the entire mapping.
fn advise(&self, advice: Advice) {
let _ = self.mmap.advise(advice);
}
/// Grow the mmap to `new_size` bytes.
///
/// # Safety
/// Caller must ensure no `&[u8]` references to the old mmap data exist.
/// After remap, any pointers derived from the old mapping may be invalid
/// (if the kernel moved the mapping).
pub unsafe fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
unsafe {
self.mmap
.remap(new_size, RemapOptions::new().may_move(true))?;
}
self.file_size = new_size as u64;
Ok(())
}
#[inline]
pub fn file_size(&self) -> u64 {
self.file_size
}
#[inline]
pub fn total_lines(&self) -> usize {
self.line_index.line_count()
}
#[inline]
pub fn get_line(&self, idx: usize) -> Option<String> {
self.line_index
.get_line(&self.mmap, idx)
.map(|s| s.to_owned())
}
#[inline]
pub fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
let start = offset as usize;
let end = start.checked_add(len)?;
if end > self.mmap.len() {
return None;
}
Some(self.mmap[start..end].to_vec())
}
}
// ─── 5 Variants ──────────────────────────────────────────────────────────────
/// Variant 1: Plain mmap, no madvise.
pub struct MmapReaderPlain {
inner: MmapReader,
}
impl MmapReaderPlain {
pub fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
unsafe { self.inner.remap(new_size) }
}
}
impl FileReaderBackend for MmapReaderPlain {
fn name(&self) -> &str {
"mmap_plain"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 2: mmap with MADV_SEQUENTIAL — optimal for sequential scan (index build).
pub struct MmapReaderSequential {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderSequential {
fn name(&self) -> &str {
"mmap_sequential"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
inner.advise(Advice::Sequential);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 3: mmap with MADV_RANDOM — optimal for random line access after index is built.
pub struct MmapReaderRandom {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderRandom {
fn name(&self) -> &str {
"mmap_random"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
inner.advise(Advice::Random);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 4: mmap with MAP_POPULATE — pre-fault all pages at mmap time.
/// Trades higher upfront cost for smoother subsequent access.
pub struct MmapReaderPopulate {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderPopulate {
fn name(&self) -> &str {
"mmap_populate"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let mut opts = MmapOptions::new();
opts.populate();
let inner = MmapReader::open_with_options(path, &opts)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 5: Phase-aware — MADV_SEQUENTIAL during index build, then MADV_RANDOM
/// for line access. Best of both worlds for the read-index-then-query pattern.
pub struct MmapReaderPhaseAware {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderPhaseAware {
fn name(&self) -> &str {
"mmap_phase_aware"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
// Index build used sequential streaming via BufReader. Now switch to random.
inner.advise(Advice::Random);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
// ─── Tests ───────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write as _;
use tempfile::NamedTempFile;
fn create_temp_file(content: &[u8]) -> NamedTempFile {
let mut f = NamedTempFile::new().unwrap();
f.write_all(content).unwrap();
f.flush().unwrap();
f
}
#[test]
fn test_plain_open_and_read_lines() {
let f = create_temp_file(b"hello\nworld\nfoo\n");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_plain");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
assert_eq!(reader.get_line(1), Some("world".to_owned()));
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
assert_eq!(reader.get_line(3), None);
reader.close();
}
#[test]
fn test_plain_out_of_bounds_returns_none() {
let f = create_temp_file(b"line1\nline2\n");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(100), None);
assert_eq!(reader.get_line(usize::MAX), None);
reader.close();
}
#[test]
fn test_plain_read_range() {
let content = b"hello world\nfoo bar\n";
let f = create_temp_file(content);
let reader = MmapReaderPlain::open(f.path()).unwrap();
let range = reader.read_range(0, 5).unwrap();
assert_eq!(&range, b"hello");
let range = reader.read_range(6, 5).unwrap();
assert_eq!(&range, b"world");
assert_eq!(reader.read_range(0, 1000), None);
reader.close();
}
#[test]
fn test_plain_empty_file() {
let f = create_temp_file(b"");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 0);
assert_eq!(reader.file_size(), 0);
assert_eq!(reader.get_line(0), None);
reader.close();
}
#[test]
fn test_sequential_variant() {
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
let reader = MmapReaderSequential::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_sequential");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
reader.close();
}
#[test]
fn test_random_variant() {
let f = create_temp_file(b"one\ntwo\nthree\n");
let reader = MmapReaderRandom::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_random");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(2), Some("three".to_owned()));
reader.close();
}
#[test]
fn test_populate_variant() {
let f = create_temp_file(b"x\ny\nz\n");
let reader = MmapReaderPopulate::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_populate");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("x".to_owned()));
reader.close();
}
#[test]
fn test_phase_aware_variant() {
let f = create_temp_file(b"a\nb\nc\n");
let reader = MmapReaderPhaseAware::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_phase_aware");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("b".to_owned()));
reader.close();
}
#[test]
fn test_file_size() {
let content = b"hello world";
let f = create_temp_file(content);
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.file_size(), content.len() as u64);
reader.close();
}
#[test]
fn test_sigbus_flag_api() {
reset_sigbus_flag();
assert!(!sigbus_flag());
}
#[test]
fn test_concurrent_open_installs_handler_once() {
let f = create_temp_file(b"line1\nline2\nline3\n");
let path = f.path().to_owned();
let num_threads = 8;
let handles: Vec<_> = (0..num_threads)
.map(|thread_id| {
let path = path.clone();
std::thread::spawn(move || {
let reader = match thread_id % 5 {
0 => {
let r = MmapReaderPlain::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"plain"
}
1 => {
let r = MmapReaderSequential::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"sequential"
}
2 => {
let r = MmapReaderRandom::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"random"
}
3 => {
let r = MmapReaderPopulate::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"populate"
}
_ => {
let r = MmapReaderPhaseAware::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"phase_aware"
}
};
reader.to_owned()
})
})
.collect();
let results: Vec<String> = handles
.into_iter()
.map(|h| h.join().expect("thread panicked"))
.collect();
assert_eq!(results.len(), num_threads);
}
#[test]
fn test_no_trailing_newline() {
let f = create_temp_file(b"line1\nline2");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
reader.close();
}
/// Diagnostic test: measure how long each stage of `open_raw()` takes
/// on a large file (e.g. the 5GB extreme.log).
/// Run with: cargo test -p log-viewer-bench --release -- --nocapture diag_open_stages
#[test]
fn diag_open_stages() {
let path = std::path::Path::new("/tmp/test-logviewer/extreme.log");
if !path.exists() {
eprintln!("SKIP: test file not found");
return;
}
// Stage 1: File::open + metadata
let t0 = std::time::Instant::now();
let file = File::open(path).unwrap();
let file_size = file.metadata().unwrap().len();
let stage1 = t0.elapsed();
eprintln!(
"Stage 1 - File::open + metadata: {:.2}ms ({:.1}GB)",
stage1.as_secs_f64() * 1000.0,
file_size as f64 / 1073741824.0
);
// Stage 2: mmap::map
let t1 = std::time::Instant::now();
let _mmap = unsafe { Mmap::map(&file) }.unwrap();
let stage2 = t1.elapsed();
eprintln!(
"Stage 2 - mmap::map: {:.2}ms",
stage2.as_secs_f64() * 1000.0
);
drop(_mmap);
// Stage 3: LineIndex::from_reader via BufReader (default 8KB buffer)
let t2 = std::time::Instant::now();
let mut reader = BufReader::new(&file);
let line_index = crate::line_index::LineIndex::from_reader(&mut reader).unwrap();
let stage3 = t2.elapsed();
eprintln!(
"Stage 3 - LineIndex::from_reader: {:.2}ms ({} lines, {} sampled_offsets)",
stage3.as_secs_f64() * 1000.0,
line_index.line_count(),
line_index.sampled_offsets.len()
);
// Stage 3b: Try with 1MB buffer
let file2 = File::open(path).unwrap();
let t2b = std::time::Instant::now();
let mut reader2 = BufReader::with_capacity(1024 * 1024, &file2);
let line_index2 = crate::line_index::LineIndex::from_reader(&mut reader2).unwrap();
let stage3b = t2b.elapsed();
eprintln!(
"Stage 3b - LineIndex (1MB buffer): {:.2}ms",
stage3b.as_secs_f64() * 1000.0
);
eprintln!("\n=== 瓶颈分析 ===");
let total_ms = stage1.as_secs_f64() * 1000.0
+ stage2.as_secs_f64() * 1000.0
+ stage3.as_secs_f64() * 1000.0;
let total_dur = stage1 + stage2 + stage3;
eprintln!("Total ~{:.0}ms", total_ms);
eprintln!(
" File::open: {:.1}% ({:.0}ms)",
stage1.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage1.as_secs_f64() * 1000.0
);
eprintln!(
" mmap::map: {:.1}% ({:.0}ms)",
stage2.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage2.as_secs_f64() * 1000.0
);
eprintln!(
" from_reader: {:.1}% ({:.0}ms)",
stage3.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage3.as_secs_f64() * 1000.0
);
// Suppress unused warnings
let _ = line_index2;
}
}
+533
View File
@@ -0,0 +1,533 @@
// ─── pread_reader.rs ──────────────────────────────────────────────────────────
// pread-based FileReaderBackend implementations with 3 variants for benchmarking.
// Uses ReadCache for 4KB block caching to reduce syscalls, and posix_fadvise
// for kernel readahead control.
//
// CRITICAL: get_line() is custom — it does NOT use LineIndex::get_line() which
// requires a full &[u8] data slice. Instead it uses sampled_offsets directly
// and reads on-demand via pread.
// ──────────────────────────────────────────────────────────────────────────────
use std::cell::RefCell;
use std::fs::File;
use std::io::{BufReader, Seek as _, SeekFrom};
use std::os::unix::fs::FileExt;
use std::os::unix::io::AsRawFd;
use std::path::Path;
use crate::FileReaderBackend;
use crate::line_index::LineIndex;
const BLOCK_SIZE: usize = 256;
const CACHE_CHUNK: usize = 4096;
// ─── ReadCache ────────────────────────────────────────────────────────────────
/// Single-block read cache. Reduces syscalls by caching the last read.
/// Typical cache hit: sequential get_line() calls within the same 4KB block.
struct ReadCache {
buf: Vec<u8>,
buf_offset: u64,
buf_len: usize,
}
impl ReadCache {
fn new() -> Self {
Self {
buf: vec![0u8; CACHE_CHUNK],
buf_offset: 0,
buf_len: 0,
}
}
/// Read `len` bytes starting at `offset`. Returns a slice into the cache.
/// On cache hit (range fully within cached block), no syscall needed.
/// On miss, performs a `read_exact_at` syscall.
fn invalidate(&mut self) {
self.buf_len = 0;
}
fn get(&mut self, file: &File, offset: u64, len: usize) -> std::io::Result<&[u8]> {
let end = offset + len as u64;
if offset >= self.buf_offset && end <= self.buf_offset + self.buf_len as u64 {
let start = (offset - self.buf_offset) as usize;
return Ok(&self.buf[start..start + len]);
}
let alloc_len = CACHE_CHUNK.max(len);
self.buf.resize(alloc_len, 0);
file.read_exact_at(&mut self.buf[..len], offset)?;
self.buf_offset = offset;
self.buf_len = len;
Ok(&self.buf[..len])
}
}
// ─── PreadReaderCore ──────────────────────────────────────────────────────────
/// Core pread-based reader. Uses ReadCache and sparse LineIndex for on-demand
/// line retrieval without mmap.
pub struct PreadReaderCore {
file: File,
line_index: LineIndex,
file_size: u64,
cache: RefCell<ReadCache>,
}
impl PreadReaderCore {
/// Open file, build line index via streaming BufReader, prepare pread reader.
fn open_raw(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
file,
line_index,
file_size,
cache: RefCell::new(ReadCache::new()),
})
}
/// Apply posix_fadvise to the entire file.
fn advise(&self, advice: libc::c_int) -> std::io::Result<()> {
let fd = self.file.as_raw_fd();
let ret = unsafe { libc::posix_fadvise(fd, 0, self.file_size as i64, advice) };
if ret != 0 {
return Err(std::io::Error::from_raw_os_error(ret));
}
Ok(())
}
/// Custom get_line using sparse index + pread.
/// Does NOT use LineIndex::get_line() (which needs full &[u8] slice).
///
/// Algorithm:
/// 1. Look up sampled_offsets[block] to get approximate byte position
/// 2. Scan forward through offset_in_block newlines via memchr
/// 3. Collect bytes from that position until next newline (may span 4KB blocks)
fn get_line_impl(&self, idx: usize) -> Option<String> {
let total = self.line_index.total_lines as usize;
if idx >= total || total == 0 {
return None;
}
let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE;
let mut cache = self.cache.borrow_mut();
// Phase 1: Find byte position of the start of line `idx`.
// Start at sampled_offsets[block], scan forward through `offset_in_block` newlines.
let start_offset = self.line_index.sampled_offsets[block];
let mut pos = start_offset;
let mut newlines_found = 0;
while newlines_found < offset_in_block {
let remaining = self.file_size.saturating_sub(pos) as usize;
if remaining == 0 {
return None;
}
let to_read = CACHE_CHUNK.min(remaining);
let data = cache.get(&self.file, pos, to_read).ok()?;
for byte_pos in memchr::memchr_iter(b'\n', data) {
newlines_found += 1;
if newlines_found == offset_in_block {
pos = pos + byte_pos as u64 + 1;
break;
}
}
if newlines_found < offset_in_block {
pos += to_read as u64;
}
}
// Phase 2: Collect bytes from `pos` until next newline or EOF.
// Line data may span multiple 4KB cache blocks.
let mut result = Vec::new();
while pos < self.file_size {
let remaining = (self.file_size - pos) as usize;
let to_read = CACHE_CHUNK.min(remaining);
let data = cache.get(&self.file, pos, to_read).ok()?;
match memchr::memchr(b'\n', data) {
Some(rel) => {
result.extend_from_slice(&data[..rel]);
break;
}
None => {
result.extend_from_slice(data);
pos += to_read as u64;
}
}
}
String::from_utf8(result)
.ok()
.map(|s| s.trim_end_matches(['\r', '\n']).to_owned())
}
/// Read a raw byte range from the file using pread.
fn read_range_impl(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
let end = offset.checked_add(len as u64)?;
if end > self.file_size {
return None;
}
let mut buf = vec![0u8; len];
self.file.read_exact_at(&mut buf, offset).ok()?;
Some(buf)
}
#[inline]
pub fn file_size(&self) -> u64 {
self.file_size
}
#[inline]
pub fn total_lines(&self) -> usize {
self.line_index.line_count()
}
pub fn refresh_index(&mut self) -> std::io::Result<()> {
let new_size = self.file.metadata()?.len();
self.file.seek(SeekFrom::Start(0))?;
let new_index = {
let mut reader = BufReader::new(&self.file);
LineIndex::from_reader(&mut reader)?
};
self.file_size = new_size;
self.line_index = new_index;
self.cache.get_mut().invalidate();
Ok(())
}
}
// ─── 3 Variants ───────────────────────────────────────────────────────────────
/// Variant 1: Plain pread, no fadvise.
pub struct PreadReaderPlain {
inner: PreadReaderCore,
}
impl PreadReaderPlain {
pub fn refresh_index(&mut self) -> std::io::Result<()> {
self.inner.refresh_index()
}
}
impl FileReaderBackend for PreadReaderPlain {
fn name(&self) -> &str {
"pread_plain"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
/// Variant 2: pread with POSIX_FADV_RANDOM — disable kernel readahead.
/// Best for random-access line lookup patterns.
pub struct PreadReaderRandom {
inner: PreadReaderCore,
}
impl FileReaderBackend for PreadReaderRandom {
fn name(&self) -> &str {
"pread_random"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
inner.advise(libc::POSIX_FADV_RANDOM)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
/// Variant 3: pread with POSIX_FADV_SEQUENTIAL — aggressive kernel readahead.
/// Best for sequential scan patterns.
pub struct PreadReaderSequential {
inner: PreadReaderCore,
}
impl FileReaderBackend for PreadReaderSequential {
fn name(&self) -> &str {
"pread_sequential"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
inner.advise(libc::POSIX_FADV_SEQUENTIAL)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
// ─── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write as _;
use tempfile::NamedTempFile;
fn create_temp_file(content: &[u8]) -> NamedTempFile {
let mut f = NamedTempFile::new().unwrap();
f.write_all(content).unwrap();
f.flush().unwrap();
f
}
#[test]
fn test_plain_open_and_read_lines() {
let f = create_temp_file(b"hello\nworld\nfoo\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_plain");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
assert_eq!(reader.get_line(1), Some("world".to_owned()));
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
assert_eq!(reader.get_line(3), None);
reader.close();
}
#[test]
fn test_plain_out_of_bounds_returns_none() {
let f = create_temp_file(b"line1\nline2\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(100), None);
assert_eq!(reader.get_line(usize::MAX), None);
reader.close();
}
#[test]
fn test_plain_read_range() {
let content = b"hello world\nfoo bar\n";
let f = create_temp_file(content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
let range = reader.read_range(0, 5).unwrap();
assert_eq!(&range, b"hello");
let range = reader.read_range(6, 5).unwrap();
assert_eq!(&range, b"world");
assert_eq!(reader.read_range(0, 1000), None);
reader.close();
}
#[test]
fn test_plain_empty_file() {
let f = create_temp_file(b"");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 0);
assert_eq!(reader.file_size(), 0);
assert_eq!(reader.get_line(0), None);
reader.close();
}
#[test]
fn test_random_variant() {
let f = create_temp_file(b"one\ntwo\nthree\n");
let reader = PreadReaderRandom::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_random");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(2), Some("three".to_owned()));
reader.close();
}
#[test]
fn test_sequential_variant() {
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
let reader = PreadReaderSequential::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_sequential");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
reader.close();
}
#[test]
fn test_file_size() {
let content = b"hello world";
let f = create_temp_file(content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.file_size(), content.len() as u64);
reader.close();
}
#[test]
fn test_no_trailing_newline() {
let f = create_temp_file(b"line1\nline2");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
reader.close();
}
#[test]
fn test_line_spanning_cache_boundary() {
// Create a line that spans a 4KB boundary
let mut content = vec![b'a'; 4090];
content.push(b'\n');
content.extend_from_slice(b"target\n");
let f = create_temp_file(&content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.get_line(0).unwrap().len(), 4090);
assert_eq!(reader.get_line(1), Some("target".to_owned()));
assert_eq!(reader.get_line(2), None);
reader.close();
}
#[test]
fn test_many_lines_random_access() {
// Create 300 lines to test block boundary (256 lines per block)
let mut content = String::new();
for i in 0..300 {
content.push_str(&format!("line_{}\n", i));
}
let f = create_temp_file(content.as_bytes());
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 300);
// Test lines across the 256-line block boundary
assert_eq!(reader.get_line(0), Some("line_0".to_owned()));
assert_eq!(reader.get_line(255), Some("line_255".to_owned()));
assert_eq!(reader.get_line(256), Some("line_256".to_owned()));
assert_eq!(reader.get_line(299), Some("line_299".to_owned()));
assert_eq!(reader.get_line(300), None);
reader.close();
}
#[test]
fn test_refresh_index_sees_appended_lines() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("refresh_test.log");
// Phase 1: write 3 initial lines
{
let mut f = std::fs::File::create(&path).unwrap();
std::io::Write::write_all(&mut f, b"alpha\nbeta\ngamma\n").unwrap();
}
let mut reader = PreadReaderPlain::open(&path).unwrap();
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("alpha".to_owned()));
assert_eq!(
reader.get_line(3),
None,
"should be out of bounds before append"
);
// Phase 2: append 2 more lines
{
use std::io::Write as _;
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
f.write_all(b"delta\nepsilon\n").unwrap();
}
// Still stale — refresh needed
assert_eq!(reader.get_line(3), None, "stale index before refresh");
reader.refresh_index().unwrap();
assert_eq!(reader.total_lines(), 5);
assert_eq!(reader.get_line(3), Some("delta".to_owned()));
assert_eq!(reader.get_line(4), Some("epsilon".to_owned()));
assert_eq!(reader.get_line(5), None);
reader.close();
}
#[test]
fn test_refresh_index_no_change_is_noop() {
let f = create_temp_file(b"one\ntwo\n");
let mut reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
reader.refresh_index().unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("one".to_owned()));
assert_eq!(reader.get_line(1), Some("two".to_owned()));
reader.close();
}
#[test]
fn test_truncation_graceful_error() {
// Verify that truncated file access returns None/Err instead of panicking
let f = create_temp_file(b"hello\nworld\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
// Truncate behind the reader — pread may return zeros or error
let _ = reader.get_line(1); // must not panic
reader.close();
}
}
+301
View File
@@ -0,0 +1,301 @@
use crate::metrics;
use crate::types::BenchmarkResult;
fn format_rss_mb(kb: u64) -> String {
format!("{:.1}MB", kb as f64 / 1024.0)
}
fn format_faults(count: u64) -> String {
if count < 1000 {
count.to_string()
} else {
let s = count.to_string();
let mut result = String::new();
for (i, c) in s.chars().rev().enumerate() {
if i > 0 && i % 3 == 0 {
result.push(',');
}
result.push(c);
}
result.chars().rev().collect()
}
}
// Format benchmark results as Markdown, grouped by category.
pub fn format_report(results: &[BenchmarkResult]) -> String {
let mut report = String::new();
report.push_str("# Benchmark: mmap vs pread\n\n");
let mut categories: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
std::collections::BTreeMap::new();
for r in results {
categories.entry(&r.category).or_default().push(r);
}
for (category, category_results) in &categories {
report.push_str(&format!("## {}\n\n", capitalize(category)));
let mut tests: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
std::collections::BTreeMap::new();
for r in category_results {
tests.entry(&r.test_name).or_default().push(r);
}
let mut variants: Vec<(String, String)> = Vec::new();
for r in category_results {
if !variants
.iter()
.any(|(b, v)| b == &r.backend && v == &r.variant)
{
variants.push((r.backend.clone(), r.variant.clone()));
}
}
variants.sort();
report.push_str("### Latency\n\n");
report.push_str("| Test |");
for (backend, variant) in &variants {
report.push_str(&format!(" {} ({}) |", backend, variant));
}
report.push_str(" Winner |\n");
report.push_str("|------|");
for _ in &variants {
report.push_str("------|");
}
report.push_str("--------|\n");
for (test_name, test_results) in &tests {
report.push_str(&format!("| {} |", test_name));
let mut all_means: Vec<f64> = Vec::new();
for (backend, variant) in &variants {
if let Some(r) = test_results
.iter()
.find(|r| r.backend == *backend && r.variant == *variant)
{
let avg = if r.latency_us.is_empty() {
0.0
} else {
metrics::mean(&r.latency_us)
};
all_means.push(avg);
}
}
let use_ms = all_means.iter().any(|&v| v >= 1000.0);
let mut best_backend = String::new();
let mut best_latency = f64::MAX;
for (backend, variant) in &variants {
let matching = test_results
.iter()
.find(|r| r.backend == *backend && r.variant == *variant);
if let Some(r) = matching {
let avg = if r.latency_us.is_empty() {
0.0
} else {
metrics::mean(&r.latency_us)
};
let sd = metrics::stdev(&r.latency_us);
let p95_val = metrics::p95(&r.latency_us) as f64;
let cell = if use_ms {
format!(
"{:.2}\u{00b1}{:.2}ms (p95:{:.2}ms)",
avg / 1000.0,
sd / 1000.0,
p95_val / 1000.0
)
} else {
format!(
"{:.1}\u{00b1}{:.1}\u{b5}s (p95:{:.1}\u{b5}s)",
avg, sd, p95_val
)
};
report.push_str(&format!(" {} |", cell));
if avg > 0.0 && avg < best_latency {
best_latency = avg;
best_backend = format!("{} ({})", backend, variant);
}
} else {
report.push_str(" - |");
}
}
report.push_str(&format!(
" {} |\n",
if best_backend.is_empty() {
"-"
} else {
&best_backend
}
));
}
report.push('\n');
let has_memory = category_results
.iter()
.any(|r| r.rss_kb > 0 || r.rss_peak_kb > 0);
if has_memory {
report.push_str("### Memory\n\n");
report.push_str("| Test | Variant | RSS | Peak RSS | Page Faults |\n");
report.push_str("|------|---------|-----|----------|-------------|\n");
let mut mem_rows: Vec<&BenchmarkResult> = category_results.to_vec();
mem_rows.sort_by(|a, b| {
(&a.test_name, &a.backend, &a.variant).cmp(&(&b.test_name, &b.backend, &b.variant))
});
for r in mem_rows {
let variant_label = format!("{} ({})", r.backend, r.variant);
report.push_str(&format!(
"| {} | {} | {} | {} | {} |\n",
r.test_name,
variant_label,
format_rss_mb(r.rss_kb),
format_rss_mb(r.rss_peak_kb),
format_faults(r.page_faults),
));
}
report.push('\n');
}
type ExtraEntry = (String, f64);
type ExtraGroup = (String, String, Vec<ExtraEntry>);
let mut extras: Vec<ExtraGroup> = category_results
.iter()
.filter(|r| !r.extra.is_empty())
.map(|r| {
let mut pairs: Vec<(String, f64)> =
r.extra.iter().map(|(k, &v)| (k.clone(), v)).collect();
pairs.sort_by(|a, b| a.0.cmp(&b.0));
(
r.test_name.clone(),
format!("{} ({})", r.backend, r.variant),
pairs,
)
})
.collect();
extras.sort_by(|a, b| (&a.0, &a.1).cmp(&(&b.0, &b.1)));
if !extras.is_empty() {
report.push_str("### Extra Metrics\n\n");
report.push_str("| Test | Variant | Metric | Value |\n");
report.push_str("|------|---------|--------|-------|\n");
for (test_name, variant_label, pairs) in &extras {
for (key, val) in pairs {
report.push_str(&format!(
"| {} | {} | {} | {:.3} |\n",
test_name, variant_label, key, val
));
}
}
report.push('\n');
}
}
report.push_str("## Summary\n\n");
report.push_str(&format!("- Total benchmarks: {}\n", results.len()));
report.push_str("- Categories: ");
report.push_str(&categories.keys().cloned().collect::<Vec<_>>().join(", "));
report.push('\n');
report
}
fn capitalize(s: &str) -> String {
let mut c = s.chars();
match c.next() {
None => String::new(),
Some(f) => f.to_uppercase().chain(c).collect(),
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
fn make_result(
category: &str,
test_name: &str,
backend: &str,
variant: &str,
latency_us: Vec<u64>,
) -> BenchmarkResult {
BenchmarkResult {
category: category.to_string(),
test_name: test_name.to_string(),
backend: backend.to_string(),
variant: variant.to_string(),
latency_us,
rss_kb: 0,
rss_peak_kb: 0,
page_faults: 0,
extra: HashMap::new(),
}
}
#[test]
fn report_ordering_independent_of_input_order() {
let set_a = vec![
make_result(
"sequential",
"read_1mb",
"pread",
"default",
vec![100, 110, 105],
),
make_result(
"sequential",
"read_1mb",
"mmap",
"default",
vec![80, 85, 90],
),
make_result(
"sequential",
"read_4kb",
"pread",
"default",
vec![10, 12, 11],
),
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
];
let set_b = vec![
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
make_result(
"sequential",
"read_1mb",
"mmap",
"default",
vec![80, 85, 90],
),
make_result(
"sequential",
"read_4kb",
"pread",
"default",
vec![10, 12, 11],
),
make_result(
"sequential",
"read_1mb",
"pread",
"default",
vec![100, 110, 105],
),
];
let report_a = format_report(&set_a);
let report_b = format_report(&set_b);
assert_eq!(
report_a, report_b,
"Reports must be identical regardless of input order"
);
}
}
+225
View File
@@ -0,0 +1,225 @@
use crate::metrics::MetricsCollector;
use crate::types::BenchmarkResult;
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
/// All recognized benchmark suite identifiers.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Suite {
Startup,
Render,
Jump,
Memory,
Growth,
Rotation,
Concurrent,
}
impl Suite {
/// All valid suite identifiers, in execution order.
pub const ALL: &[Suite] = &[
Suite::Startup,
Suite::Render,
Suite::Jump,
Suite::Memory,
Suite::Growth,
Suite::Rotation,
Suite::Concurrent,
];
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Suite::Startup => "startup",
Suite::Render => "render",
Suite::Jump => "jump",
Suite::Memory => "memory",
Suite::Growth => "growth",
Suite::Rotation => "rotation",
Suite::Concurrent => "concurrent",
}
}
}
impl std::str::FromStr for Suite {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
Suite::ALL
.iter()
.find(|suite| suite.as_str() == s)
.copied()
.ok_or_else(|| {
let valid = Suite::ALL
.iter()
.map(|s| s.as_str())
.collect::<Vec<_>>()
.join(", ");
format!("invalid value '{s}' for '--suites': valid values are {valid}")
})
}
}
pub struct BenchConfig {
pub test_file: PathBuf,
pub quick_mode: bool,
pub suites: Option<Vec<Suite>>,
}
/// Track whether we have already warned about VmHWM reset failure.
/// Prevents duplicate warnings across multiple suite runs.
static HWM_WARNED: AtomicBool = AtomicBool::new(false);
pub fn warn_reset_hwm() {
if let Err(e) = MetricsCollector::reset_vm_hwm() {
if HWM_WARNED.swap(true, Ordering::Relaxed) {
return;
}
if e.kind() == std::io::ErrorKind::PermissionDenied {
eprintln!(
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
Memory peak values may be contaminated across benchmark suites. \
Try running as root."
);
} else {
eprintln!(
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
Memory peak values may be contaminated across benchmark suites."
);
}
}
}
#[cfg(test)]
fn reset_hwm_warned() {
HWM_WARNED.store(false, Ordering::Relaxed);
}
pub fn run_all(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let should_run = |suite: Suite| -> bool {
match &config.suites {
Some(suites) => suites.contains(&suite),
None => true,
}
};
if should_run(Suite::Startup) {
warn_reset_hwm();
results.extend(crate::suites::startup::run(config));
}
if should_run(Suite::Render) {
warn_reset_hwm();
results.extend(crate::suites::render::run(config));
}
if should_run(Suite::Jump) {
warn_reset_hwm();
results.extend(crate::suites::jump::run(config));
}
if should_run(Suite::Memory) {
warn_reset_hwm();
results.extend(crate::suites::memory::run(config));
}
if should_run(Suite::Growth) {
warn_reset_hwm();
results.extend(crate::suites::growth::run(config));
}
if should_run(Suite::Rotation) {
warn_reset_hwm();
results.extend(crate::suites::rotation::run(config));
}
if should_run(Suite::Concurrent) {
warn_reset_hwm();
results.extend(crate::suites::concurrent::run(config));
}
results
}
#[cfg(test)]
mod tests {
use super::Suite;
use std::str::FromStr;
#[test]
fn parse_all_valid_suites() {
let expected = [
("startup", Suite::Startup),
("render", Suite::Render),
("jump", Suite::Jump),
("memory", Suite::Memory),
("growth", Suite::Growth),
("rotation", Suite::Rotation),
("concurrent", Suite::Concurrent),
];
for (s, expected_suite) in expected {
assert_eq!(
Suite::from_str(s).unwrap(),
expected_suite,
"failed to parse '{s}'"
);
}
}
#[test]
fn misspelled_suite_returns_error() {
let err = Suite::from_str("startp").unwrap_err();
assert!(
err.contains("invalid value 'startp'"),
"error should mention the bad value: {err}"
);
}
#[test]
fn error_message_lists_all_valid_values() {
let err = Suite::from_str("bogus").unwrap_err();
for name in Suite::ALL.iter().map(|s| s.as_str()) {
assert!(
err.contains(name),
"error should list valid suite '{name}': {err}"
);
}
}
#[test]
fn mixed_valid_invalid_stops_at_first_error() {
// "startup" is valid, "zzz" is not — collect hits the first Err
let names = ["startup".to_string(), "zzz".to_string()];
let result: Result<Vec<Suite>, _> = names.iter().map(|s| s.parse()).collect();
assert!(result.is_err());
}
#[test]
fn hwm_warned_flag_prevents_reentry() {
use super::reset_hwm_warned;
use std::sync::atomic::Ordering;
reset_hwm_warned();
assert!(
!super::HWM_WARNED.load(Ordering::Relaxed),
"flag should be false after reset"
);
// Simulate the flag being set (as warn_reset_hwm would do on error)
super::HWM_WARNED.store(true, Ordering::Relaxed);
// swap should now return true (old value), indicating already warned
assert!(
super::HWM_WARNED.swap(true, Ordering::Relaxed),
"swap should return the previous value (true)"
);
}
#[test]
fn warn_reset_hwm_does_not_panic() {
use super::reset_hwm_warned;
reset_hwm_warned();
// Whether reset_vm_hwm succeeds or fails, warn_reset_hwm must not panic.
// Multiple calls must also be safe.
super::warn_reset_hwm();
super::warn_reset_hwm();
super::warn_reset_hwm();
}
}
+117
View File
@@ -0,0 +1,117 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_parallel_reads::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_parallel_reads::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_parallel_reads::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_parallel_reads::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_parallel_reads::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_parallel_reads::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_parallel_reads::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_parallel_reads::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results
}
fn bench_parallel_reads<B: FileReaderBackend + Send + 'static>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = config.test_file.clone();
let iterations = if config.quick_mode { 250 } else { 1000 };
let total_lines = {
let reader = B::open(&path).expect("Failed to open file for line count");
let count = reader.total_lines();
reader.close();
count
};
let overall_start = std::time::Instant::now();
let num_threads = 4usize;
let handles: Vec<_> = (0..num_threads)
.map(|thread_id| {
let path = path.clone();
std::thread::spawn(move || {
let reader = B::open(&path).expect("Failed to open file in thread");
let mut latencies = Vec::with_capacity(iterations);
for i in 0..iterations {
let line_idx = (thread_id * iterations + i) % total_lines.max(1);
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
reader.close();
latencies
})
})
.collect();
let thread_latencies: Vec<Vec<u64>> = handles
.into_iter()
.map(|h| h.join().expect("Thread panicked"))
.collect();
let total_elapsed = overall_start.elapsed();
let all_latencies: Vec<u64> = thread_latencies.into_iter().flatten().collect();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("num_threads".into(), num_threads as f64);
extra.insert("iterations_per_thread".into(), iterations as f64);
extra.insert("total_time_us".into(), total_elapsed.as_micros() as f64);
extra.insert("total_lines".into(), total_lines as f64);
vec![BenchmarkResult {
category: "concurrent".into(),
test_name: "parallel_reads".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: all_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+295
View File
@@ -0,0 +1,295 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::MmapReaderPlain;
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let dir = tempfile::tempdir().expect("Failed to create temp dir");
results.extend(bench_append_visibility_mmap(config, dir.path()));
results.extend(bench_append_visibility_pread(config, dir.path()));
results.extend(bench_remap_cost(config, dir.path()));
results.extend(bench_scroll_during_append(config, dir.path()));
results.extend(bench_high_frequency_append(config, dir.path()));
results
}
fn bench_append_visibility_mmap(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let path = data_gen::generate_growable_file(dir).expect("Failed to create growable file");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open growable file");
let original_lines = reader.total_lines();
let original_size = reader.file_size();
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
let new_metadata = std::fs::metadata(&path).expect("Failed to read metadata");
let new_size = new_metadata.len() as usize;
let remap_start = std::time::Instant::now();
reader.remap(new_size).expect("Failed to remap");
let remap_elapsed = remap_start.elapsed();
let new_line_bytes =
reader.read_range(original_size, (new_size - original_size as usize).min(256));
let visible = new_line_bytes.is_some();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("original_lines".into(), original_lines as f64);
extra.insert("appended_lines".into(), append_count as f64);
extra.insert("new_bytes_visible".into(), visible as u64 as f64);
extra.insert("original_size".into(), original_size as f64);
extra.insert("new_size".into(), new_size as f64);
extra.insert("remap_us".into(), remap_elapsed.as_micros() as f64);
reader.close();
vec![BenchmarkResult {
category: "growth".into(),
test_name: "append_visibility_mmap".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: vec![remap_elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_append_visibility_pread(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("pread_growth");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let reader = PreadReaderPlain::open(&path).expect("Failed to open growable file");
let original_lines = reader.total_lines();
reader.close();
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
let reopen_start = std::time::Instant::now();
let new_reader = PreadReaderPlain::open(&path).expect("Failed to reopen file");
let reopen_elapsed = reopen_start.elapsed();
let new_lines = new_reader.total_lines();
let can_read_new = new_lines > original_lines;
if can_read_new {
let last_line = new_lines.saturating_sub(1);
let _ = new_reader.get_line(last_line);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("original_lines".into(), original_lines as f64);
extra.insert("appended_lines".into(), append_count as f64);
extra.insert("new_total_lines".into(), new_lines as f64);
extra.insert("reopen_us".into(), reopen_elapsed.as_micros() as f64);
new_reader.close();
vec![BenchmarkResult {
category: "growth".into(),
test_name: "append_visibility_pread".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: vec![reopen_elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_remap_cost(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("remap_cost");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let iterations: usize = if config.quick_mode { 5 } else { 20 };
let mut latencies = Vec::with_capacity(iterations);
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open file");
data_gen::append_lines(&path, append_count).expect("Failed to append");
let new_size = std::fs::metadata(&path).expect("metadata").len() as usize;
let t = std::time::Instant::now();
reader.remap(new_size).expect("remap failed");
latencies.push(t.elapsed().as_micros() as u64);
reader.close();
let _ = std::fs::remove_file(&path);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("appended_per_iter".into(), append_count as f64);
extra.insert("iterations".into(), iterations as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "remap_cost".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_scroll_during_append(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("scroll_append");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let duration_secs: u64 = if config.quick_mode { 2 } else { 10 };
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
let bg_path = path.clone();
let bg_handle = std::thread::spawn(move || {
let batch_size = 100;
let batch_interval =
std::time::Duration::from_micros(1_000_000 / (append_rate / batch_size).max(1) as u64);
let start = std::time::Instant::now();
while start.elapsed().as_secs() < duration_secs {
data_gen::append_lines(&bg_path, batch_size).ok();
std::thread::sleep(batch_interval);
}
});
let mut reader = PreadReaderPlain::open(&path).expect("Failed to open file");
let initial_lines = reader.total_lines();
let mut frame_latencies = Vec::new();
let mut current_line = 0usize;
let mut refresh_count: u64 = 0;
let mut none_count: u64 = 0;
let scroll_start = std::time::Instant::now();
let mut last_refresh = std::time::Instant::now();
let refresh_interval = std::time::Duration::from_millis(250);
while scroll_start.elapsed().as_secs() < duration_secs {
if last_refresh.elapsed() >= refresh_interval {
reader.refresh_index().ok();
refresh_count += 1;
last_refresh = std::time::Instant::now();
continue;
}
if let Some(_line) = reader.get_line(current_line) {
let t = std::time::Instant::now();
frame_latencies.push(t.elapsed().as_micros() as u64);
current_line += 1;
} else {
none_count += 1;
std::thread::sleep(std::time::Duration::from_millis(1));
}
}
let max_line = current_line;
assert!(
max_line > initial_lines,
"benchmark never read past initial {initial_lines} lines (max={max_line})"
);
reader.close();
bg_handle.join().ok();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("duration_secs".into(), duration_secs as f64);
extra.insert("append_rate_per_sec".into(), append_rate as f64);
extra.insert("frames_rendered".into(), frame_latencies.len() as f64);
extra.insert("refresh_count".into(), refresh_count as f64);
extra.insert("none_count".into(), none_count as f64);
extra.insert("initial_lines".into(), initial_lines as f64);
extra.insert("max_line_seen".into(), max_line as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "scroll_during_append".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: frame_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_high_frequency_append(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("high_freq_append");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let duration_secs: u64 = if config.quick_mode { 3 } else { 30 };
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
let batch_size: usize = 100;
let batches_per_sec = append_rate / batch_size;
let total_batches = (duration_secs as usize * batches_per_sec).max(1);
let mut detect_latencies = Vec::with_capacity(total_batches);
for _ in 0..total_batches {
data_gen::append_lines(&path, batch_size).expect("Failed to append");
let t = std::time::Instant::now();
if let Ok(reader) = PreadReaderPlain::open(&path) {
let total = reader.total_lines();
let _ = reader.get_line(total.saturating_sub(1));
reader.close();
}
detect_latencies.push(t.elapsed().as_micros() as u64);
std::thread::sleep(std::time::Duration::from_micros(
1_000_000 / batches_per_sec as u64,
));
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("duration_secs".into(), duration_secs as f64);
extra.insert("append_rate_per_sec".into(), append_rate as f64);
extra.insert("batch_size".into(), batch_size as f64);
extra.insert("total_batches".into(), total_batches as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "high_frequency_append".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: detect_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+251
View File
@@ -0,0 +1,251 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_near_jump::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_near_jump::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_near_jump::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_near_jump::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_near_jump::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_near_jump::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_near_jump::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_near_jump::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_far_jump::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_far_jump::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_far_jump::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_far_jump::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_far_jump::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_far_jump::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_far_jump::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_far_jump::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_jump_end::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_jump_end::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_reverse_scan::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_reverse_scan::<PreadReaderPlain>(
"pread", "plain", config,
));
results
}
fn bench_near_jump<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations: usize = if config.quick_mode { 10 } else { 100 };
let mut latencies = Vec::with_capacity(iterations);
let mut current = 0usize;
for _ in 0..iterations {
let target = (current + 15).min(total.saturating_sub(1));
let t = std::time::Instant::now();
let _ = reader.get_line(target);
latencies.push(t.elapsed().as_micros() as u64);
current = target;
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "near_jump".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: HashMap::new(),
}]
}
fn bench_far_jump<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let repetitions: usize = if config.quick_mode { 3 } else { 10 };
let fractions = [0.25, 0.50, 0.75];
let mut latencies = Vec::new();
let mut extra = HashMap::new();
for &frac in &fractions {
let target = ((total as f64 * frac) as usize).min(total.saturating_sub(1));
for _ in 0..repetitions {
let t = std::time::Instant::now();
let _ = reader.get_line(target);
latencies.push(t.elapsed().as_micros() as u64);
}
}
extra.insert("jump_positions".into(), 3.0);
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "far_jump".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_jump_end<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations: usize = if config.quick_mode { 5 } else { 10 };
let last_line = total.saturating_sub(1);
let mut latencies = Vec::with_capacity(iterations);
for _ in 0..iterations {
let t = std::time::Instant::now();
let _ = reader.get_line(last_line);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("last_line_idx".into(), last_line as f64);
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "jump_end".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_reverse_scan<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
if total <= FRAME_LINES {
reader.close();
return vec![];
}
let iterations: usize = if config.quick_mode { 5 } else { 10 };
let mut latencies = Vec::with_capacity(iterations * FRAME_LINES);
for _ in 0..iterations {
let start = total - FRAME_LINES;
for i in (start..total).rev() {
let t = std::time::Instant::now();
let _ = reader.get_line(i);
latencies.push(t.elapsed().as_micros() as u64);
}
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
let mut extra = HashMap::new();
extra.insert("lines_per_scan".into(), FRAME_LINES as f64);
extra.insert("iterations".into(), iterations as f64);
vec![BenchmarkResult {
category: "jump".into(),
test_name: "reverse_scan".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+237
View File
@@ -0,0 +1,237 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_idle_rss::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_idle_rss::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_idle_rss::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_idle_rss::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_idle_rss::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_idle_rss::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_idle_rss::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_idle_rss::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_scroll_rss::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_scroll_rss::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_jump_end_rss::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_jump_end_rss::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_rss_reclaim::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_rss_reclaim::<PreadReaderPlain>(
"pread", "plain", config,
));
results
}
fn bench_idle_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("total_lines".into(), reader.total_lines() as f64);
extra.insert(
"file_size_mb".into(),
reader.file_size() as f64 / (1024.0 * 1024.0),
);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "idle_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_scroll_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let sample_interval = 100_000;
let max_lines = if config.quick_mode { 100_000 } else { total };
let upper = max_lines.min(total);
let mut rss_samples = Vec::new();
let mut hwm_samples = Vec::new();
let mut lines_read = 0usize;
for i in (0..upper).step_by(sample_interval) {
if reader.get_line(i).is_some() {
lines_read += 1;
}
let rss = MetricsCollector::read_rss();
rss_samples.push(rss.vm_rss_kb);
hwm_samples.push(rss.vm_hwm_kb);
}
let final_rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("rss_samples_count".into(), rss_samples.len() as f64);
extra.insert(
"max_rss_kb".into(),
rss_samples.iter().copied().fold(0u64, u64::max) as f64,
);
extra.insert(
"max_hwm_kb".into(),
hwm_samples.iter().copied().fold(0u64, u64::max) as f64,
);
extra.insert("lines_read".into(), lines_read as f64);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "scroll_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: final_rss.vm_rss_kb,
rss_peak_kb: final_rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_jump_end_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let last_line = total.saturating_sub(1);
let _ = reader.get_line(last_line);
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("last_line_idx".into(), last_line as f64);
extra.insert("total_lines".into(), total as f64);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "jump_end_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_rss_reclaim<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let last_line = total.saturating_sub(1);
let _ = reader.get_line(last_line);
let wait_secs: u64 = if config.quick_mode { 5 } else { 30 };
let sample_interval: u64 = 5;
let num_samples = (wait_secs / sample_interval) as usize;
let mut rss_samples = Vec::with_capacity(num_samples);
let mut hwm_samples = Vec::with_capacity(num_samples);
for _ in 0..num_samples {
std::thread::sleep(std::time::Duration::from_secs(sample_interval));
let rss = MetricsCollector::read_rss();
rss_samples.push(rss.vm_rss_kb);
hwm_samples.push(rss.vm_hwm_kb);
}
let final_rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
let mut extra = HashMap::new();
extra.insert("wait_total_secs".into(), wait_secs as f64);
extra.insert("rss_samples".into(), rss_samples.len() as f64);
if let (Some(&first), Some(&last)) = (rss_samples.first(), rss_samples.last()) {
extra.insert("rss_first_kb".into(), first as f64);
extra.insert("rss_last_kb".into(), last as f64);
extra.insert(
"rss_change_pct".into(),
if first > 0 {
((last as f64 - first as f64) / first as f64) * 100.0
} else {
0.0
},
);
}
vec![BenchmarkResult {
category: "memory".into(),
test_name: "rss_reclaim".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: final_rss.vm_rss_kb,
rss_peak_kb: final_rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+12
View File
@@ -0,0 +1,12 @@
pub mod concurrent;
pub mod growth;
pub mod jump;
pub mod memory;
pub mod render;
pub mod rotation;
pub mod startup;
/// Number of lines read per single-frame render benchmark.
/// Also used as the scan window size for reverse-scan benchmarks.
/// Shared across suites to ensure consistent workload sizing.
pub(crate) const FRAME_LINES: usize = 35;
+325
View File
@@ -0,0 +1,325 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_single_frame::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_single_frame::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_single_frame::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_single_frame::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_single_frame::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_single_frame::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_single_frame::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_single_frame::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_continuous_scroll::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_continuous_scroll::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_continuous_scroll::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_continuous_scroll::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_continuous_scroll::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_continuous_scroll::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_continuous_scroll::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_continuous_scroll::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results
}
fn bench_single_frame<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let mut results = Vec::new();
for (pos_name, start_line) in select_frame_positions(total) {
let mut latencies = Vec::with_capacity(FRAME_LINES);
for i in 0..FRAME_LINES {
let line_idx = start_line + i;
if line_idx >= total {
break;
}
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
results.push(BenchmarkResult {
category: "render".into(),
test_name: format!("single_frame_{pos_name}"),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: HashMap::new(),
});
}
reader.close();
results
}
/// Select non-overlapping benchmark positions for single-frame tests.
///
/// Returns `(label, start_line)` pairs. Positions are chosen so that no
/// two frames overlap, skipping any position that cannot fit without
/// colliding with a previously selected range.
pub(crate) fn select_frame_positions(total: usize) -> Vec<(&'static str, usize)> {
let frame = FRAME_LINES;
if total == 0 {
return vec![];
}
let mut positions: Vec<(&'static str, usize)> = vec![("head", 0)];
// Need at least 3 full frames to place head, middle, and tail without overlap.
if total >= 3 * frame {
let head_end = frame;
let tail_start = total - frame;
let gap = tail_start - head_end;
// Center the middle frame within the gap, ensuring it fits entirely
let middle_start = head_end + gap.saturating_sub(frame) / 2;
positions.push(("middle", middle_start));
positions.push(("tail", tail_start));
}
positions
}
fn bench_continuous_scroll<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations = if config.quick_mode { 100 } else { 1000 };
let timeout = if config.quick_mode {
std::time::Duration::from_secs(1)
} else {
std::time::Duration::from_secs(10)
};
let mut latencies = Vec::with_capacity(iterations);
let start = std::time::Instant::now();
for i in 0..iterations {
if start.elapsed() > timeout {
break;
}
let line_idx = i % total.max(1);
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("total_lines_scrolled".into(), latencies.len() as f64);
reader.close();
vec![BenchmarkResult {
category: "render".into(),
test_name: "continuous_scroll".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
#[cfg(test)]
mod tests {
use super::*;
fn starts(pos: &[(&str, usize)]) -> Vec<usize> {
pos.iter().map(|&(_, s)| s).collect()
}
fn labels<'a>(pos: &[(&'a str, usize)]) -> Vec<&'a str> {
pos.iter().map(|&(l, _)| l).collect()
}
fn ranges_overlap(a_start: usize, b_start: usize) -> bool {
let a_end = a_start + FRAME_LINES;
let b_end = b_start + FRAME_LINES;
a_start < b_end && b_start < a_end
}
#[test]
fn zero_lines_no_positions() {
assert!(select_frame_positions(0).is_empty());
}
#[test]
fn one_line_head_only() {
let pos = select_frame_positions(1);
assert_eq!(labels(&pos), vec!["head"]);
assert_eq!(starts(&pos), vec![0]);
}
#[test]
fn at_frame_lines_head_only() {
let pos = select_frame_positions(FRAME_LINES);
assert_eq!(labels(&pos), vec!["head"]);
assert_eq!(starts(&pos), vec![0]);
}
#[test]
fn just_above_threshold_head_middle_tail() {
let total = 3 * FRAME_LINES;
let pos = select_frame_positions(total);
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
assert_eq!(starts(&pos)[0], 0);
assert_eq!(*starts(&pos).last().unwrap(), total - FRAME_LINES);
// Verify no overlap
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap: {:?} @ {} vs {:?} @ {} (total={})",
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1,
total
);
}
}
}
#[test]
fn no_overlap_at_total_70() {
// total=70 < 3*35=105, so only head is returned
let pos = select_frame_positions(70);
assert_eq!(labels(&pos), vec!["head"]);
}
#[test]
fn no_overlap_at_total_104() {
let pos = select_frame_positions(104);
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=104: {:?} @ {} vs {:?} @ {}",
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1
);
}
}
}
#[test]
fn no_overlap_at_total_105() {
let pos = select_frame_positions(105);
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=105: {:?} @ {} vs {:?} @ {}",
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1
);
}
}
}
#[test]
fn middle_is_centered_between_head_and_tail() {
let total = 1000;
let pos = select_frame_positions(total);
assert_eq!(pos.len(), 3);
let (_, head_start) = pos[0];
let (_, middle_start) = pos[1];
let (_, tail_start) = pos[2];
assert_eq!(head_start, 0);
assert_eq!(tail_start, total - FRAME_LINES);
let head_end = head_start + FRAME_LINES;
let gap = tail_start - head_end;
assert_eq!(middle_start, head_end + gap.saturating_sub(FRAME_LINES) / 2);
}
#[test]
fn large_file_all_three_positions() {
let pos = select_frame_positions(1_000_000);
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
}
}
+170
View File
@@ -0,0 +1,170 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{self, MmapReaderPlain};
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let dir = tempfile::tempdir().expect("Failed to create temp dir");
results.extend(bench_truncate_safety_mmap(config, dir.path()));
results.extend(bench_truncate_safety_pread(config, dir.path()));
results.extend(bench_rotation_detection(config, dir.path()));
results
}
fn bench_truncate_safety_mmap(
_config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_mmap");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut sigbus_detected = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
mmap_reader::reset_sigbus_flag();
let reader = MmapReaderPlain::open(&path).expect("Failed to open file");
let original_size = reader.file_size();
let truncate_size = original_size / 2;
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
let t = std::time::Instant::now();
let mid_offset = original_size as u64 / 2;
let _ = reader.read_range(mid_offset, 64);
latencies.push(t.elapsed().as_micros() as u64);
if mmap_reader::sigbus_flag() {
sigbus_detected += 1;
}
reader.close();
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("sigbus_detected".into(), sigbus_detected as f64);
extra.insert("crashed".into(), 0.0);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "truncate_safety_mmap".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_truncate_safety_pread(
_config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_pread");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut error_count = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let reader = PreadReaderPlain::open(&path).expect("Failed to open file");
let original_size = reader.file_size();
let truncate_size = original_size / 2;
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
let t = std::time::Instant::now();
let mid_offset = original_size as u64 / 2;
let result = reader.read_range(mid_offset, 64);
latencies.push(t.elapsed().as_micros() as u64);
if result.is_none() {
error_count += 1;
}
reader.close();
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("errors_returned".into(), error_count as f64);
extra.insert("crashed".into(), 0.0);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "truncate_safety_pread".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_rotation_detection(_config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("rotation_detect");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut detected_count = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let original_inode = MetricsCollector::get_inode(&path).expect("Failed to get inode");
let _rotated = data_gen::rotate_file(&path).expect("Failed to rotate file");
let t = std::time::Instant::now();
let detected = MetricsCollector::detect_rotation(original_inode, &path);
latencies.push(t.elapsed().as_micros() as u64);
if detected {
detected_count += 1;
}
let _ = std::fs::remove_file(&path);
let rotated = sub_dir.join("growable.log.1");
let _ = std::fs::remove_file(&rotated);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("rotations_detected".into(), detected_count as f64);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "rotation_detection".into(),
backend: "both".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+143
View File
@@ -0,0 +1,143 @@
use std::collections::HashMap;
use std::path::Path;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_hot_open::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_hot_open::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_hot_open::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_hot_open::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_hot_open::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_hot_open::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_hot_open::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_hot_open::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
if !config.quick_mode {
match MetricsCollector::clear_file_cache(&config.test_file) {
Ok(()) => {
results.extend(bench_cold_open::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_cold_open::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_cold_open::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_cold_open::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_cold_open::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_cold_open::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_cold_open::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_cold_open::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
}
Err(e) => {
eprintln!(
"WARNING: Failed to clear file cache; skipping cold startup benchmarks because results would not be cold: {e}"
);
}
}
}
results
}
fn bench_hot_open<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
open_and_measure::<B>(backend, variant, &config.test_file, "hot_open")
}
fn bench_cold_open<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
match MetricsCollector::clear_file_cache(&config.test_file) {
Ok(()) => open_and_measure::<B>(backend, variant, &config.test_file, "cold_open"),
Err(e) => {
eprintln!(
"WARNING: Failed to clear file cache for {backend}/{variant}; skipping cold benchmark: {e}"
);
Vec::new()
}
}
}
fn open_and_measure<B: FileReaderBackend>(
backend: &str,
variant: &str,
path: &Path,
test_name: &str,
) -> Vec<BenchmarkResult> {
let start = std::time::Instant::now();
let reader = B::open(path).expect("Failed to open file");
let elapsed = start.elapsed();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let result = BenchmarkResult {
category: "startup".into(),
test_name: test_name.into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: {
let mut m = HashMap::new();
m.insert("total_lines".into(), reader.total_lines() as f64);
m.insert(
"file_size_mb".into(),
reader.file_size() as f64 / (1024.0 * 1024.0),
);
m
},
};
reader.close();
vec![result]
}
+13
View File
@@ -0,0 +1,13 @@
use std::collections::HashMap;
pub struct BenchmarkResult {
pub category: String,
pub test_name: String,
pub backend: String,
pub variant: String,
pub latency_us: Vec<u64>,
pub rss_kb: u64,
pub rss_peak_kb: u64,
pub page_faults: u64,
pub extra: HashMap<String, f64>,
}
+2 -1
View File
@@ -17,7 +17,8 @@ memmap2.workspace = true
directories.workspace = true directories.workspace = true
xxhash-rust.workspace = true xxhash-rust.workspace = true
bincode.workspace = true bincode.workspace = true
unicode-width.workspace = true
tempfile.workspace = true
[dev-dependencies] [dev-dependencies]
insta.workspace = true insta.workspace = true
tempfile.workspace = true
+221 -32
View File
@@ -9,6 +9,14 @@ use crate::io::index_cache::IndexCache;
use crate::io::line_index::LineIndex; use crate::io::line_index::LineIndex;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AppendStatus {
Unchanged,
Appended(u64),
/// File shrank; mmap and index rebuilt via reload().
Reloaded,
}
pub struct FileReader { pub struct FileReader {
path: PathBuf, path: PathBuf,
mmap: Option<memmap2::Mmap>, mmap: Option<memmap2::Mmap>,
@@ -25,20 +33,23 @@ impl FileReader {
} else { } else {
// SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。 // SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。
// memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。 // memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。
// let m =
// ⚠️ Known limitation (Phase 5): 如果文件在 mmap 期间被外部进程截断, unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
// 访问截断区域的内存会触发 SIGBUS(致命信号,无法恢复)。
// FileWatcher Phase 将添加文件修改检测和 re-mmap 机制来处理此情况。 // Layer 3: mmap 后立即 stat 同一 fd,检测截断(TOCTOU 缓解,非安全证明)
// 在 Phase 5 中,假设打开的文件不会被外部修改。 let current_size = file.metadata()?.len();
Some(unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?) if current_size < m.len() as u64 {
None
} else {
Some(m)
}
}; };
let line_index = { // 直接从 mmap 快照构建行索引,确保索引与数据来自同一内存映射,
let mut reader = std::io::BufReader::new(&file); // 消除 mmap + BufReader 双读之间的 TOCTOU 竞态窗口。
LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io { let line_index = match &mmap {
source: e, Some(m) => LineIndex::from_bytes(m.as_ref()),
context: "building line index".into(), None => LineIndex::from_bytes(&[]),
})?
}; };
Ok(Self { Ok(Self {
@@ -76,9 +87,8 @@ impl FileReader {
} }
} }
/// Save the line index cache to disk.
pub fn save_cache(&self) -> std::io::Result<()> { pub fn save_cache(&self) -> std::io::Result<()> {
IndexCache::save(&self.path, &self.line_index)?; IndexCache::save_with_hash(&self.path, &self.line_index, self.data())?;
Ok(()) Ok(())
} }
@@ -94,27 +104,35 @@ impl FileReader {
self.mmap = None; self.mmap = None;
if file_size > 0 { if file_size > 0 {
self.mmap = Some( let m =
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?, unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
); let current_size = file.metadata()?.len();
self.mmap = if current_size < m.len() as u64 {
None
} else {
Some(m)
};
} }
let mut reader = std::io::BufReader::new(&file); self.line_index = match &self.mmap {
self.line_index = LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io { Some(m) => LineIndex::from_bytes(m.as_ref()),
source: e, None => LineIndex::from_bytes(&[]),
context: "rebuilding line index on reload".into(), };
})?;
Ok(()) Ok(())
} }
pub fn update_for_append(&mut self) -> Result<u64> { pub fn update_for_append(&mut self) -> Result<AppendStatus> {
let file = std::fs::File::open(&self.path)?; let file = std::fs::File::open(&self.path)?;
let new_size = file.metadata()?.len(); let new_size = file.metadata()?.len();
let old_size = self.mmap.as_ref().map_or(0u64, |m| m.len() as u64); let old_size = self.mmap.as_ref().map_or(0u64, |m| m.len() as u64);
if new_size <= old_size { if new_size < old_size {
return Ok(0); self.reload()?;
return Ok(AppendStatus::Reloaded);
}
if new_size == old_size {
return Ok(AppendStatus::Unchanged);
} }
let old_lines = self.line_index.line_count() as u64; let old_lines = self.line_index.line_count() as u64;
@@ -123,11 +141,19 @@ impl FileReader {
let mmap = let mmap =
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?; unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
let current_size = file.metadata()?.len();
if current_size < mmap.len() as u64 {
self.line_index = LineIndex::from_bytes(&[]);
return Ok(AppendStatus::Reloaded);
}
self.line_index self.line_index
.extend_from_bytes(&mmap[old_size as usize..], old_size); .extend_from_bytes(&mmap[old_size as usize..], old_size);
self.mmap = Some(mmap); self.mmap = Some(mmap);
Ok(self.line_index.line_count() as u64 - old_lines) Ok(AppendStatus::Appended(
self.line_index.line_count() as u64 - old_lines,
))
} }
} }
@@ -305,8 +331,8 @@ mod tests {
file.write_all(b"ccc\nddd\n").unwrap(); file.write_all(b"ccc\nddd\n").unwrap();
} }
let new_lines = reader.update_for_append().unwrap(); let status = reader.update_for_append().unwrap();
assert_eq!(new_lines, 2); assert_eq!(status, AppendStatus::Appended(2));
assert_eq!(reader.line_count(), 4); assert_eq!(reader.line_count(), 4);
assert_eq!(reader.get_line(0), Some("aaa")); assert_eq!(reader.get_line(0), Some("aaa"));
assert_eq!(reader.get_line(1), Some("bbb")); assert_eq!(reader.get_line(1), Some("bbb"));
@@ -320,8 +346,8 @@ mod tests {
let mut reader = FileReader::open(f.path()).unwrap(); let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 1); assert_eq!(reader.line_count(), 1);
let new_lines = reader.update_for_append().unwrap(); let status = reader.update_for_append().unwrap();
assert_eq!(new_lines, 0); assert_eq!(status, AppendStatus::Unchanged);
assert_eq!(reader.line_count(), 1); assert_eq!(reader.line_count(), 1);
} }
@@ -341,8 +367,11 @@ mod tests {
file.write_all(b"x\n").unwrap(); file.write_all(b"x\n").unwrap();
} }
let new_lines = reader.update_for_append().unwrap(); let status = reader.update_for_append().unwrap();
assert_eq!(new_lines, 0); assert_eq!(status, AppendStatus::Reloaded);
assert_eq!(reader.line_count(), 1);
assert_eq!(reader.file_size(), 2);
assert_eq!(reader.get_line(0), Some("x"));
} }
#[test] #[test]
@@ -368,4 +397,164 @@ mod tests {
let idx = reader.line_index(); let idx = reader.line_index();
assert_eq!(idx.line_count(), 2); assert_eq!(idx.line_count(), 2);
} }
// ─── TOCTOU 修复验证 ─────────────────────────────────────────────────
// open() 和 reload() 现在从 mmap 快照构建索引(from_bytes),
// 而非从独立 BufReader 读取(from_reader)。以下测试验证:
// 1. from_bytes 与 from_reader 产出完全一致的索引
// 2. 修改文件后 reload 仍返回正确内容(无残留旧状态)
// 3. 极端情况下(空文件、单行、跨块)open+reload 一致性
#[test]
fn test_open_from_bytes_matches_from_reader() {
let cases: Vec<&[u8]> = vec![
b"",
b"hello",
b"hello\n",
b"aaa\nbbb\nccc",
b"\n",
b"a\n\nb",
];
let mut large_cases = Vec::new();
for &n in &[256usize, 300, 512] {
let mut buf = Vec::new();
for i in 0..n {
buf.extend_from_slice(format!("line{}\n", i).as_bytes());
}
large_cases.push(buf);
}
let all_contents: Vec<&[u8]> = cases
.iter()
.copied()
.chain(large_cases.iter().map(Vec::as_slice))
.collect();
for content in all_contents {
let f = create_temp_file(content);
let reader = FileReader::open(f.path()).unwrap();
let from_bytes_idx = LineIndex::from_bytes(content);
let mut buf_reader = std::io::BufReader::new(content);
let from_reader_idx = LineIndex::from_reader(&mut buf_reader).unwrap();
assert_eq!(
reader.line_index().total_lines(),
from_bytes_idx.total_lines(),
"open() line_count should match from_bytes for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.total_lines(),
from_reader_idx.total_lines(),
"from_bytes should match from_reader total_lines for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.sampled_offsets(),
from_reader_idx.sampled_offsets(),
"from_bytes should match from_reader sampled_offsets for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.has_trailing_newline(),
from_reader_idx.has_trailing_newline(),
"from_bytes should match from_reader has_trailing_newline for {:?}B file",
content.len()
);
}
}
#[test]
fn test_reload_after_external_modify_returns_correct_content() {
let f = create_temp_file(b"aaa\nbbb\n");
let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.get_line(0), Some("aaa"));
assert_eq!(reader.get_line(1), Some("bbb"));
assert_eq!(reader.line_count(), 2);
// 外部修改:覆盖写入新内容
{
use std::io::Write;
std::fs::write(f.path(), b"xxx\nyyy\nzzz\n").unwrap();
}
reader.reload().unwrap();
assert_eq!(reader.line_count(), 3);
assert_eq!(reader.get_line(0), Some("xxx"));
assert_eq!(reader.get_line(1), Some("yyy"));
assert_eq!(reader.get_line(2), Some("zzz"));
}
#[test]
fn test_reload_after_truncate_then_rewrite_no_stale_data() {
let f = create_temp_file(b"line0\nline1\nline2\nline3\n");
let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 4);
// 截断后写入更短内容
{
use std::io::Write;
let mut file = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
file.write_all(b"new\n").unwrap();
}
reader.reload().unwrap();
assert_eq!(reader.line_count(), 1);
assert_eq!(reader.get_line(0), Some("new"));
assert_eq!(
reader.get_line(1),
None,
"old line should not be accessible"
);
}
#[test]
fn test_open_reload_idempotent_cross_block() {
let mut content = Vec::new();
for i in 0..600u32 {
content.extend_from_slice(format!("line{}\n", i).as_bytes());
}
let f = create_temp_file(&content);
let reader = FileReader::open(f.path()).unwrap();
let mut reloaded = FileReader::open(f.path()).unwrap();
reloaded.reload().unwrap();
assert_eq!(reader.line_count(), reloaded.line_count());
for i in 0..600 {
assert_eq!(
reader.get_line(i),
reloaded.get_line(i),
"line {i} mismatch between open and reload"
);
}
}
#[test]
fn test_open_stat_after_mmap_detects_truncation() {
let content = b"line0\nline1\nline2\nline3\n";
let f = create_temp_file(content);
let reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 4);
{
use std::io::Write;
let _ = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
}
let reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 0);
assert_eq!(reader.file_size(), 0);
}
} }
+161 -20
View File
@@ -1,36 +1,74 @@
use std::io::{Read as _, Write as _}; use std::io::{Read as _, Write as _};
use std::path::Path; use std::path::Path;
use crate::io::cache_util::{cache_path, CACHE_VERSION}; use crate::io::cache_util::{CACHE_VERSION, cache_path};
use crate::io::line_index::LineIndex; use crate::io::line_index::LineIndex;
pub struct IndexCache; pub struct IndexCache;
/// Write `buf` to `dest` atomically using a unique temporary file in the same directory.
///
/// Each call creates its own temp file via `tempfile::Builder`, eliminating collisions
/// when multiple threads (or processes) save to the same cache path concurrently.
/// The temp file is created in `dest.parent()` so the final `rename` stays on the
/// same filesystem and remains atomic.
fn write_cache_atomically(dest: &Path, buf: &[u8]) -> std::io::Result<()> {
let dir = dest.parent().ok_or_else(|| {
std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"cache destination has no parent directory",
)
})?;
let mut tmp = tempfile::Builder::new()
.prefix("index-cache-")
.suffix(".tmp")
.tempfile_in(dir)?;
tmp.as_file_mut().write_all(buf)?;
tmp.as_file_mut().sync_all()?;
tmp.persist(dest).map(|_| ()).map_err(|e| e.error)
}
fn encode_cache(file_hash: u64, index: &LineIndex) -> std::io::Result<Vec<u8>> {
let index_bytes = bincode::serialize(index)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
let mut buf = Vec::with_capacity(1 + 8 + index_bytes.len());
buf.push(CACHE_VERSION);
buf.extend_from_slice(&file_hash.to_le_bytes());
buf.extend_from_slice(&index_bytes);
Ok(buf)
}
impl IndexCache { impl IndexCache {
/// Save a `LineIndex` to disk using atomic write (write to .tmp, then rename). /// Save a `LineIndex` to disk using atomic write (unique temp file, then rename).
///
/// The file hash is derived from `data` (the same byte slice used to build the index),
/// avoiding TOCTOU issues from re-reading the file from disk.
pub fn save_with_hash(file_path: &Path, index: &LineIndex, data: &[u8]) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
})?;
let file_hash = compute_data_hash(data);
let buf = encode_cache(file_hash, index)?;
write_cache_atomically(&dest, &buf)
}
/// Save a `LineIndex` to disk, computing the hash by re-reading the file.
///
/// Prefer [`save_with_hash`] when the source data is already in memory
/// to avoid a TOCTOU race between indexing and hash computation.
pub fn save(file_path: &Path, index: &LineIndex) -> std::io::Result<()> { pub fn save(file_path: &Path, index: &LineIndex) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| { let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path") std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
})?; })?;
let file_hash = compute_file_hash(file_path)?; let file_hash = compute_file_hash(file_path)?;
let index_bytes = bincode::serialize(index) let buf = encode_cache(file_hash, index)?;
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?; write_cache_atomically(&dest, &buf)
let mut buf = Vec::with_capacity(1 + 8 + index_bytes.len());
buf.push(CACHE_VERSION);
buf.extend_from_slice(&file_hash.to_le_bytes());
buf.extend_from_slice(&index_bytes);
let tmp_path = dest.with_extension("index.tmp");
{
let mut f = std::fs::File::create(&tmp_path)?;
f.write_all(&buf)?;
f.sync_all()?;
}
std::fs::rename(&tmp_path, &dest)?;
Ok(())
} }
/// Load a cached `LineIndex` from disk. /// Load a cached `LineIndex` from disk.
@@ -60,6 +98,32 @@ impl IndexCache {
} }
} }
/// Compute a fingerprint from in-memory data, using the same algorithm as `compute_file_hash`.
/// Returns 0 for empty data.
pub fn compute_data_hash(data: &[u8]) -> u64 {
let len = data.len();
if len == 0 {
return 0;
}
let head_size = 4096.min(len);
let tail_size = 4096.min(len);
let mut hasher_state = xxhash_rust::xxh3::Xxh3::new();
hasher_state.update(&data[..head_size]);
if len > head_size + tail_size {
hasher_state.update(&data[len - tail_size..]);
} else {
// For small files the tail overlaps with the head; hash from the real tail start.
let tail_start = len.saturating_sub(tail_size);
hasher_state.update(&data[tail_start..]);
}
hasher_state.update(&(len as u64).to_le_bytes());
hasher_state.digest()
}
/// Compute a fast fingerprint of the file: xxhash of (head 4KB + tail 4KB + file size). /// Compute a fast fingerprint of the file: xxhash of (head 4KB + tail 4KB + file size).
/// Returns 0 for empty files. /// Returns 0 for empty files.
fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> { fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> {
@@ -243,4 +307,81 @@ mod tests {
assert_ne!(h1, h2, "hash should change when content changes"); assert_ne!(h1, h2, "hash should change when content changes");
} }
}
#[test]
fn test_concurrent_save_with_hash_no_corruption() {
use std::sync::{Arc, Barrier};
let file = make_test_file(300);
let data = std::fs::read(file.path()).unwrap();
let num_threads = 8;
let iterations = 50;
let barrier = Arc::new(Barrier::new(num_threads));
let path = file.path().to_path_buf();
let handles: Vec<_> = (0..num_threads)
.map(|_| {
let barrier = Arc::clone(&barrier);
let path = path.clone();
let data = data.clone();
std::thread::spawn(move || {
let index = LineIndex::from_bytes(&data);
barrier.wait();
for _ in 0..iterations {
IndexCache::save_with_hash(&path, &index, &data)
.expect("concurrent save_with_hash should succeed");
}
})
})
.collect();
for h in handles {
h.join().expect("thread should not panic");
}
let loaded = IndexCache::load(file.path()).expect("final cache should load successfully");
let expected = LineIndex::from_bytes(&data);
assert_eq!(loaded.line_count(), expected.line_count());
assert_eq!(
loaded.sampled_offsets(),
expected.sampled_offsets(),
"concurrent writes must not produce interleaved data"
);
}
#[test]
fn test_concurrent_save_same_dest_all_succeed() {
use std::sync::{Arc, Barrier};
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("test.index");
let payloads: Vec<Vec<u8>> = (0..8).map(|i| vec![i; 64 * 1024]).collect();
let barrier = Arc::new(Barrier::new(8));
let handles: Vec<_> = payloads
.into_iter()
.map(|payload| {
let barrier = Arc::clone(&barrier);
let dest = dest.clone();
std::thread::spawn(move || {
barrier.wait();
write_cache_atomically(&dest, &payload)
.expect("concurrent atomic write should succeed");
})
})
.collect();
for h in handles {
h.join().expect("thread should not panic");
}
let final_data = std::fs::read(&dest).expect("dest file should exist");
assert_eq!(
final_data.len(),
64 * 1024,
"final file must be exactly one payload"
);
}
}
+8 -6
View File
@@ -20,14 +20,14 @@ const BLOCK_SIZE: usize = 256;
pub struct LineIndex { pub struct LineIndex {
// 采样偏移量:每 BLOCK_SIZE 行记录一个起始字节偏移。 // 采样偏移量:每 BLOCK_SIZE 行记录一个起始字节偏移。
// sampled_offsets[i] 存储第 (i * BLOCK_SIZE) 行的字节起始位置。 // sampled_offsets[i] 存储第 (i * BLOCK_SIZE) 行的字节起始位置。
sampled_offsets: Vec<u64>, pub(crate) sampled_offsets: Vec<u64>,
// 文件总行数。 // 文件总行数。
total_lines: u64, pub(crate) total_lines: u64,
// 文件最后一个字节是否是换行符 \n。 // 文件最后一个字节是否是换行符 \n。
#[allow(dead_code)] #[allow(dead_code)]
has_trailing_newline: bool, pub(crate) has_trailing_newline: bool,
} }
impl LineIndex { impl LineIndex {
@@ -158,7 +158,7 @@ impl LineIndex {
// If the junction falls on a block boundary, record the start offset // If the junction falls on a block boundary, record the start offset
// (analogous to from_bytes always pushing offset 0 for line 0). // (analogous to from_bytes always pushing offset 0 for line 0).
if starts_new_line && (old_total as usize) % BLOCK_SIZE == 0 { if starts_new_line && (old_total as usize).is_multiple_of(BLOCK_SIZE) {
self.sampled_offsets.push(start_offset); self.sampled_offsets.push(start_offset);
} }
@@ -212,15 +212,17 @@ impl LineIndex {
self.total_lines as usize self.total_lines as usize
} }
// ─── getter 方法 ──────────────────────────────────────────────────── #[cfg(test)]
pub(crate) fn sampled_offsets(&self) -> &[u64] { pub(crate) fn sampled_offsets(&self) -> &[u64] {
&self.sampled_offsets &self.sampled_offsets
} }
#[cfg(test)]
pub(crate) fn total_lines(&self) -> u64 { pub(crate) fn total_lines(&self) -> u64 {
self.total_lines self.total_lines
} }
#[cfg(test)]
pub(crate) fn has_trailing_newline(&self) -> bool { pub(crate) fn has_trailing_newline(&self) -> bool {
self.has_trailing_newline self.has_trailing_newline
} }
@@ -233,7 +235,7 @@ impl LineIndex {
} }
let block = idx / BLOCK_SIZE; let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE; let offset_in_block = idx % BLOCK_SIZE;
let mut pos = self.sampled_offsets[block] as usize; let mut pos = (*self.sampled_offsets.get(block)?) as usize;
for _ in 0..offset_in_block { for _ in 0..offset_in_block {
match memchr::memchr(b'\n', &data[pos..]) { match memchr::memchr(b'\n', &data[pos..]) {
Some(rel) => pos = pos + rel + 1, Some(rel) => pos = pos + rel + 1,
+1
View File
@@ -18,4 +18,5 @@ pub mod cache_util;
pub mod index_cache; pub mod index_cache;
pub mod line_sampler; pub mod line_sampler;
pub mod progressive_reader; pub mod progressive_reader;
pub mod read_cache;
pub mod wrap; pub mod wrap;
+511 -119
View File
@@ -1,13 +1,32 @@
use std::cell::RefCell; use std::cell::RefCell;
use std::fmt; use std::fmt;
use std::io::BufRead;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::error::{CoreError, Result}; use crate::error::{CoreError, Result};
use crate::io::file_reader::FileReader; use crate::io::file_reader::{AppendStatus, FileReader};
use crate::io::index_cache::IndexCache; use crate::io::index_cache::IndexCache;
use crate::io::line_index::LineIndex; use crate::io::line_index::LineIndex;
use crate::io::line_sampler::sample_line_count; use crate::io::line_sampler::sample_line_count;
use crate::io::wrap::{format_json_line, wrap_line_chars, MAX_WRAP_INPUT_LEN}; use crate::io::wrap::{MAX_WRAP_INPUT_LEN, format_json_line};
// ─── Cancel-aware channel helpers ────────────────────────────────────────────
/// Send a message on `tx`, but abort if `cancel_rx` fires first.
///
/// Uses `crossbeam_channel::select!` so the thread sleeps efficiently instead
/// of busy-looping. Used for terminal messages (Complete / Error) that must
/// not be silently dropped while the receiver is still alive.
fn send_cancelable<T>(
tx: &crossbeam_channel::Sender<T>,
msg: T,
cancel_rx: &crossbeam_channel::Receiver<()>,
) {
crossbeam_channel::select! {
send(tx, msg) -> _ => {}
recv(cancel_rx) -> _ => {}
}
}
// ─── IndexerMessage ────────────────────────────────────────────────────────── // ─── IndexerMessage ──────────────────────────────────────────────────────────
@@ -101,7 +120,7 @@ impl VisualHeightIndex {
} }
} }
fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self { pub fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self {
self.json_format = json_format; self.json_format = json_format;
self.terminal_width = terminal_width; self.terminal_width = terminal_width;
self self
@@ -158,6 +177,31 @@ impl VisualHeightIndex {
self.total_visual_rows += h as u64; self.total_visual_rows += h as u64;
} }
} }
/// Replace the visual height of the last logical line. O(1).
///
/// Must be called **before** `extend_from_heights` so that the last line
/// index still refers to the pre-extension line.
pub fn replace_last_line_height(&mut self, new_height: usize) {
let n = self.prefix_sums.len();
if n < 2 {
return;
}
let last_line = n - 2;
let old_height = self.prefix_sums[last_line + 1] - self.prefix_sums[last_line];
let new_height = new_height as u64;
if new_height == old_height {
return;
}
let delta = new_height.abs_diff(old_height);
if new_height > old_height {
self.prefix_sums[last_line + 1] += delta;
self.total_visual_rows += delta;
} else {
self.prefix_sums[last_line + 1] -= delta;
self.total_visual_rows -= delta;
}
}
} }
// ─── VisualHeightRebuildResult ──────────────────────────────────────────────── // ─── VisualHeightRebuildResult ────────────────────────────────────────────────
@@ -179,6 +223,9 @@ pub fn compute_line_visual_height(
} }
if json_format { if json_format {
let formatted = format_json_line(line_text); let formatted = format_json_line(line_text);
if formatted.len() > MAX_WRAP_INPUT_LEN {
return 1;
}
compute_text_visual_height(&formatted, terminal_width) compute_text_visual_height(&formatted, terminal_width)
} else { } else {
compute_text_visual_height(line_text, terminal_width) compute_text_visual_height(line_text, terminal_width)
@@ -188,29 +235,11 @@ pub fn compute_line_visual_height(
fn compute_text_visual_height(text: &str, width: usize) -> usize { fn compute_text_visual_height(text: &str, width: usize) -> usize {
let mut height = 0; let mut height = 0;
for sub_line in text.split('\n') { for sub_line in text.split('\n') {
height += wrap_line_chars(sub_line, width).len(); height += crate::io::wrap::wrap_line_count(sub_line, width);
} }
height.max(1) height.max(1)
} }
fn compute_visual_heights(
reader: &FileReader,
terminal_width: usize,
json_format: bool,
) -> Vec<usize> {
let line_count = reader.line_count();
let mut visual_heights = Vec::with_capacity(line_count);
for i in 0..line_count {
let line_text = reader.get_line(i).unwrap_or("");
visual_heights.push(compute_line_visual_height(
line_text,
terminal_width,
json_format,
));
}
visual_heights
}
// ─── ReaderState ───────────────────────────────────────────────────────────── // ─── ReaderState ─────────────────────────────────────────────────────────────
pub enum ReaderState { pub enum ReaderState {
@@ -234,8 +263,8 @@ pub enum ReaderState {
pub fn spawn_indexer( pub fn spawn_indexer(
path: PathBuf, path: PathBuf,
generation: u64, generation: u64,
terminal_width: usize, _terminal_width: usize,
json_format: bool, _json_format: bool,
cancel_rx: crossbeam_channel::Receiver<()>, cancel_rx: crossbeam_channel::Receiver<()>,
) -> crossbeam_channel::Receiver<IndexerMessage> { ) -> crossbeam_channel::Receiver<IndexerMessage> {
let (tx, rx) = crossbeam_channel::bounded(10); let (tx, rx) = crossbeam_channel::bounded(10);
@@ -244,62 +273,84 @@ pub fn spawn_indexer(
let file = match std::fs::File::open(&path) { let file = match std::fs::File::open(&path) {
Ok(f) => f, Ok(f) => f,
Err(e) => { Err(e) => {
let _ = tx.send(IndexerMessage::Error { send_cancelable(
generation, &tx,
message: e.to_string(), IndexerMessage::Error {
});
return;
}
};
let file_size = match file.metadata() {
Ok(m) => m.len(),
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
generation,
message: e.to_string(),
});
return;
}
};
let mmap = if file_size == 0 {
None
} else {
match unsafe { memmap2::Mmap::map(&file) } {
Ok(m) => Some(m),
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
generation, generation,
message: e.to_string(), message: e.to_string(),
}); },
return; &cancel_rx,
} );
return;
}
};
let target_len = match file.metadata() {
Ok(m) => m.len(),
Err(e) => {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
},
&cancel_rx,
);
return;
} }
}; };
let data = mmap.as_deref().unwrap_or(&[]); let mut buf_reader = std::io::BufReader::with_capacity(64 * 1024, file);
let mut sampled_offsets: Vec<u64> = vec![0];
let mut next_line_idx: usize = 1;
let mut newline_count: usize = 0;
let mut chunk_offset: u64 = 0;
let mut last_byte: Option<u8> = None;
let mut bytes_since_check: usize = 0;
if !data.is_empty() { loop {
let mut newline_count: usize = 0; let buf = match buf_reader.fill_buf() {
let mut chars_since_check: usize = 0; Ok(b) => b,
let mut prev_pos: usize = 0; Err(e) => {
send_cancelable(
for pos in memchr::memchr_iter(b'\n', data) { &tx,
chars_since_check += pos - prev_pos; IndexerMessage::Error {
prev_pos = pos; generation,
message: e.to_string(),
if chars_since_check >= 1_000_000 { },
chars_since_check = 0; &cancel_rx,
if cancel_rx.try_recv().is_ok() { );
return; return;
}
} }
};
if buf.is_empty() {
break;
}
if let Some(&b) = buf.last() {
last_byte = Some(b);
}
for pos in memchr::memchr_iter(b'\n', buf) {
newline_count += 1; newline_count += 1;
if next_line_idx.is_multiple_of(256) {
sampled_offsets.push(chunk_offset + pos as u64 + 1);
}
next_line_idx += 1;
}
if newline_count % 256_000 == 0 { let consumed = buf.len();
let percent = (pos as f64 / file_size as f64) * 100.0; bytes_since_check += consumed;
let _ = tx.send(IndexerMessage::Progress { chunk_offset += consumed as u64;
buf_reader.consume(consumed);
if bytes_since_check >= 1_000_000 {
bytes_since_check = 0;
if cancel_rx.try_recv().is_ok() {
return;
}
if target_len > 0 {
let percent = (chunk_offset as f64 / target_len as f64) * 100.0;
let _ = tx.try_send(IndexerMessage::Progress {
generation, generation,
percent, percent,
lines_scanned: newline_count as u64, lines_scanned: newline_count as u64,
@@ -312,24 +363,84 @@ pub fn spawn_indexer(
return; return;
} }
let line_index = LineIndex::from_bytes(data); let line_index = if chunk_offset == 0 {
LineIndex {
sampled_offsets: vec![],
total_lines: 0,
has_trailing_newline: false,
}
} else {
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
let total_lines = if has_trailing_newline && newline_count > 0 {
newline_count as u64
} else {
(1 + newline_count) as u64
};
let _ = IndexCache::save(&path, &line_index); if has_trailing_newline && newline_count > 0 {
let trailing_line_idx = newline_count;
if trailing_line_idx.is_multiple_of(256) {
sampled_offsets.pop();
}
}
LineIndex {
sampled_offsets,
total_lines,
has_trailing_newline,
}
};
let mmap = if target_len == 0 {
None
} else {
match std::fs::File::open(&path) {
Ok(mmap_file) => match unsafe { memmap2::Mmap::map(&mmap_file) } {
Ok(m) => match mmap_file.metadata() {
Ok(metadata) if metadata.len() >= m.len() as u64 => Some(m),
Ok(_) | Err(_) => None,
},
Err(e) => {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
},
&cancel_rx,
);
return;
}
},
Err(e) => {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
},
&cancel_rx,
);
return;
}
}
};
if let Some(data) = mmap.as_deref() {
let _ = IndexCache::save_with_hash(&path, &line_index, data);
}
let reader = FileReader::from_parts(path, mmap, line_index); let reader = FileReader::from_parts(path, mmap, line_index);
let visual_height_index = if terminal_width > 0 { send_cancelable(
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format); &tx,
Some(VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width)) IndexerMessage::Complete {
} else { generation,
None reader,
}; visual_height_index: None,
},
let _ = tx.send(IndexerMessage::Complete { &cancel_rx,
generation, );
reader,
visual_height_index,
});
}); });
rx rx
@@ -355,30 +466,67 @@ pub fn spawn_visual_height_rebuild(
Err(_) => return, Err(_) => return,
}; };
let file_size = match file.metadata() { let mmap = match unsafe { memmap2::Mmap::map(&file) } {
Ok(m) => m.len(), Ok(m) => match file.metadata() {
Ok(meta) if meta.len() >= m.len() as u64 => m,
_ => return,
},
Err(_) => return, Err(_) => return,
}; };
let data = mmap.as_ref();
let total_lines = line_index.line_count();
let mut visual_heights: Vec<usize> = Vec::with_capacity(total_lines);
let mmap = if file_size == 0 { let mut line_start = 0usize;
None let mut bytes_since_cancel_check = 0usize;
} else {
match unsafe { memmap2::Mmap::map(&file) } { loop {
Ok(m) => Some(m), if bytes_since_cancel_check >= 1_000_000 {
Err(_) => return, bytes_since_cancel_check = 0;
if cancel_rx.try_recv().is_ok() {
return;
}
} }
};
if cancel_rx.try_recv().is_ok() { let newline_rel = memchr::memchr(b'\n', &data[line_start..]);
let line_end = match newline_rel {
Some(p) => line_start + p,
None => data.len(),
};
let mut slice_end = line_end;
if slice_end > line_start && data[slice_end - 1] == b'\r' {
slice_end -= 1;
}
let line_text = std::str::from_utf8(&data[line_start..slice_end]).unwrap_or("");
visual_heights.push(compute_line_visual_height(
line_text,
terminal_width,
json_format,
));
bytes_since_cancel_check =
bytes_since_cancel_check.saturating_add(line_end - line_start + 1);
match newline_rel {
Some(_) => line_start = line_end + 1,
None => break,
}
}
if visual_heights.len() != total_lines {
return; return;
} }
let reader = FileReader::from_parts(path, mmap, line_index);
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format);
let index = let index =
VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width); VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width);
let _ = tx.send(VisualHeightRebuildResult { generation, index }); send_cancelable(
&tx,
VisualHeightRebuildResult { generation, index },
&cancel_rx,
);
}); });
rx rx
@@ -656,10 +804,10 @@ impl ProgressiveFileReader {
} }
} }
pub fn update_for_append(&mut self) -> Result<u64> { pub fn update_for_append(&mut self) -> Result<AppendStatus> {
match &mut self.state { match &mut self.state {
ReaderState::Ready { reader, .. } => reader.update_for_append(), ReaderState::Ready { reader, .. } => reader.update_for_append(),
_ => Ok(0), _ => Ok(AppendStatus::Unchanged),
} }
} }
@@ -675,7 +823,7 @@ impl ProgressiveFileReader {
pub fn start_visual_height_rebuild(&mut self, terminal_width: usize, json_format: bool) { pub fn start_visual_height_rebuild(&mut self, terminal_width: usize, json_format: bool) {
if let Some(tx) = self.vh_rebuild_cancel_tx.take() { if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
let _ = tx.send(()); let _ = tx.try_send(());
} }
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1); let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
@@ -730,10 +878,10 @@ impl ProgressiveFileReader {
impl Drop for ProgressiveFileReader { impl Drop for ProgressiveFileReader {
fn drop(&mut self) { fn drop(&mut self) {
if let Some(tx) = &self.cancel_tx { if let Some(tx) = &self.cancel_tx {
let _ = tx.send(()); let _ = tx.try_send(());
} }
if let Some(tx) = self.vh_rebuild_cancel_tx.take() { if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
let _ = tx.send(()); let _ = tx.try_send(());
} }
} }
} }
@@ -991,12 +1139,10 @@ mod tests {
assert_eq!(reader.get_line(0), Some("line1")); assert_eq!(reader.get_line(0), Some("line1"));
assert_eq!(reader.get_line(1), Some("line2")); assert_eq!(reader.get_line(1), Some("line2"));
assert_eq!(reader.get_line(2), Some("line3")); assert_eq!(reader.get_line(2), Some("line3"));
assert!(visual_height_index.is_some()); assert!(
let idx = visual_height_index.unwrap(); visual_height_index.is_none(),
assert_eq!(idx.total_visual_rows(), 3); "spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
assert_eq!(idx.visual_row_to_logical_row(0), 0); );
assert_eq!(idx.visual_row_to_logical_row(1), 1);
assert_eq!(idx.visual_row_to_logical_row(2), 2);
} }
other => panic!("expected Complete, got {:?}", other), other => panic!("expected Complete, got {:?}", other),
} }
@@ -1193,18 +1339,23 @@ mod tests {
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1); let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(temp.path().to_path_buf(), 1, 80, false, cancel_rx); let rx = spawn_indexer(temp.path().to_path_buf(), 1, 80, false, cancel_rx);
let msg = rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap(); loop {
match rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap() {
match msg { IndexerMessage::Progress { .. } => continue,
IndexerMessage::Complete { IndexerMessage::Complete {
visual_height_index, reader,
.. visual_height_index,
} => { ..
let idx = visual_height_index.expect("should have visual height index"); } => {
assert_eq!(idx.visual_height_of_line(0), 1); assert_eq!(reader.line_count(), 2);
assert_eq!(idx.visual_height_of_line(1), 1); assert!(
visual_height_index.is_none(),
"spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
);
break;
}
other => panic!("expected Complete, got {:?}", other),
} }
other => panic!("expected Complete, got {:?}", other),
} }
} }
@@ -1258,6 +1409,247 @@ mod tests {
idx.extend_from_heights(&[1, 2, 3]); idx.extend_from_heights(&[1, 2, 3]);
assert_eq!(idx.total_visual_rows(), 6); assert_eq!(idx.total_visual_rows(), 6);
}
#[test]
fn test_replace_last_line_height_increase() {
let heights = [2, 3];
let mut idx = VisualHeightIndex::build(&heights);
assert_eq!(idx.total_visual_rows(), 5);
assert_eq!(idx.visual_height_of_line(1), 3);
idx.replace_last_line_height(7);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 7);
assert_eq!(idx.total_visual_rows(), 9);
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
}
#[test]
fn test_replace_last_line_height_decrease() {
let heights = [2, 5];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(1);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 1);
assert_eq!(idx.total_visual_rows(), 3);
}
#[test]
fn test_replace_last_line_height_same_is_noop() {
let heights = [2, 3];
let mut idx = VisualHeightIndex::build(&heights);
let total_before = idx.total_visual_rows();
idx.replace_last_line_height(3);
assert_eq!(idx.total_visual_rows(), total_before);
}
#[test]
fn test_replace_last_line_height_then_extend() {
let heights = [2, 1];
let mut idx = VisualHeightIndex::build(&heights);
assert_eq!(idx.total_visual_rows(), 3);
idx.replace_last_line_height(4);
idx.extend_from_heights(&[3]);
assert_eq!(idx.line_count(), 3); assert_eq!(idx.line_count(), 3);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 4);
assert_eq!(idx.visual_height_of_line(2), 3);
assert_eq!(idx.total_visual_rows(), 9);
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
assert_eq!(idx.cursor_to_first_visual_row(2), 6);
}
#[test]
fn test_replace_last_line_height_single_line() {
let heights = [5];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(2);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.total_visual_rows(), 2);
}
#[test]
fn test_replace_last_line_height_empty_index() {
let heights: [usize; 0] = [];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(5);
assert_eq!(idx.total_visual_rows(), 0);
}
#[test]
fn test_spawn_indexer_file_truncated_during_scan() {
let mut content = Vec::new();
for i in 0..100_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
{
use std::io::Write;
let _ = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
}
let result = rx.recv_timeout(std::time::Duration::from_secs(10));
match result {
Ok(IndexerMessage::Complete { reader, .. }) => {
assert!(reader.line_count() <= 100_000);
}
Ok(IndexerMessage::Error { .. }) => {}
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
other => panic!("unexpected: {:?}", other),
}
}
#[test]
fn test_spawn_visual_height_rebuild_line_count_mismatch_discards() {
let content = b"line0\nline1\nline2\n";
let f = create_temp_file(content);
let data = std::fs::read(f.path()).unwrap();
let index = LineIndex::from_bytes(&data);
IndexCache::save(f.path(), &index).unwrap();
{
use std::io::Write;
let mut file = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
file.write_all(b"only_one_line\n").unwrap();
}
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_visual_height_rebuild(f.path().to_path_buf(), 1, 80, false, cancel_rx);
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
match result {
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
Ok(_) => panic!("should have been discarded due to line count mismatch"),
}
}
#[test]
fn test_send_cancelable_delivers_on_empty_channel() {
let (tx, rx) = crossbeam_channel::bounded(2);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
send_cancelable(&tx, 42, &cancel_rx);
assert_eq!(rx.try_recv(), Ok(42));
}
#[test]
fn test_send_cancelable_aborts_on_cancel() {
let (tx, rx) = crossbeam_channel::bounded(1);
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
tx.send("filler").unwrap();
let handle = std::thread::spawn(move || {
send_cancelable(&tx, "important", &cancel_rx);
});
cancel_tx.send(()).unwrap();
handle.join().unwrap();
assert_eq!(rx.try_recv(), Ok("filler"));
}
#[test]
fn test_send_cancelable_drains_when_room_available() {
let (tx, rx) = crossbeam_channel::bounded(1);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
tx.send("first").unwrap();
let rx_clone = rx.clone();
let handle = std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_millis(50));
let _ = rx_clone.try_recv();
});
send_cancelable(&tx, "second", &cancel_rx);
handle.join().unwrap();
assert_eq!(rx.try_recv(), Ok("second"));
}
#[test]
fn test_progress_try_send_does_not_block_full_channel() {
let mut content = Vec::new();
for i in 0..50_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
let mut got_complete = false;
let timeout = std::time::Duration::from_secs(15);
let start = std::time::Instant::now();
while start.elapsed() < timeout {
match rx.recv_timeout(std::time::Duration::from_secs(1)) {
Ok(IndexerMessage::Progress { .. }) => {}
Ok(IndexerMessage::Complete { .. }) => {
got_complete = true;
break;
}
Ok(IndexerMessage::Error { message, .. }) => {
panic!("unexpected error: {}", message);
}
Err(e) => panic!("recv error: {:?}", e),
}
}
assert!(
got_complete,
"indexer should complete even when Progress fills channel"
);
}
#[test]
fn test_indexer_cancel_with_full_channel() {
let mut content = Vec::new();
for i in 0..500_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
cancel_tx.send(()).unwrap();
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
match result {
Err(crossbeam_channel::RecvTimeoutError::Timeout)
| Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Ok(IndexerMessage::Complete { .. }) => {}
Ok(IndexerMessage::Error { .. }) => {}
Ok(IndexerMessage::Progress { .. }) => {}
}
} }
} }
+68 -8
View File
@@ -56,9 +56,15 @@ impl LruReadCache {
/// on a hit, or fills a cache slot on a miss. Cross-block reads go through /// on a hit, or fills a cache slot on a miss. Cross-block reads go through
/// the spill buffer and are not cached. /// the spill buffer and are not cached.
pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> { pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> {
if len == 0 {
return Ok(&[]);
}
let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1); let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1);
let request_end = offset.saturating_add(len as u64); let request_end = offset.checked_add(len as u64).ok_or_else(|| {
let block_end = aligned_key + BLOCK_ALIGN as u64; io::Error::new(io::ErrorKind::InvalidInput, "read range overflows u64")
})?;
let block_end = aligned_key.saturating_add(BLOCK_ALIGN as u64);
if request_end > block_end { if request_end > block_end {
self.spill_buf.resize(len, 0); self.spill_buf.resize(len, 0);
@@ -74,7 +80,8 @@ impl LruReadCache {
} }
let hit_idx = self.slots.iter().position(|slot| { let hit_idx = self.slots.iter().position(|slot| {
slot.block_offset == aligned_key && request_end <= slot.block_offset + slot.len as u64 let slot_end = slot.block_offset.saturating_add(slot.len as u64);
slot.len > 0 && slot.block_offset == aligned_key && request_end <= slot_end
}); });
if let Some(idx) = hit_idx { if let Some(idx) = hit_idx {
@@ -96,8 +103,8 @@ impl LruReadCache {
let slot = &mut self.slots[evict_idx]; let slot = &mut self.slots[evict_idx];
let bytes_read = file.read_at(&mut slot.buf, aligned_key)?; let bytes_read = file.read_at(&mut slot.buf, aligned_key)?;
// Note: get(file, 0, 0) on an empty file now returns Err (old code returned Ok(&[])). // Non-empty reads that return 0 are EOF. Zero-length reads are handled above
// No callers pass len == 0, so this is a safe semantic change. // as a successful no-op.
if bytes_read == 0 { if bytes_read == 0 {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes")); return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
} }
@@ -107,7 +114,8 @@ impl LruReadCache {
slot.last_access = self.tick; slot.last_access = self.tick;
self.tick += 1; self.tick += 1;
if request_end > aligned_key + bytes_read as u64 { let bytes_end = aligned_key.saturating_add(bytes_read as u64);
if request_end > bytes_end {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read")); return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
} }
@@ -118,7 +126,9 @@ impl LruReadCache {
/// Invalidate all cache slots and the spill buffer. /// Invalidate all cache slots and the spill buffer.
pub fn clear(&mut self) { pub fn clear(&mut self) {
for slot in &mut self.slots { for slot in &mut self.slots {
slot.block_offset = 0;
slot.len = 0; slot.len = 0;
slot.last_access = 0;
} }
self.spill_len = 0; self.spill_len = 0;
} }
@@ -314,9 +324,11 @@ mod tests {
cache.clear(); cache.clear();
// All slots should have len == 0. // All slots should be fully reset.
for slot in &cache.slots { for slot in &cache.slots {
assert_eq!(slot.block_offset, 0);
assert_eq!(slot.len, 0); assert_eq!(slot.len, 0);
assert_eq!(slot.last_access, 0);
} }
assert_eq!(cache.spill_len, 0); assert_eq!(cache.spill_len, 0);
@@ -432,4 +444,52 @@ mod tests {
assert_eq!(&line2[..4090], &[b'B'; 4090]); assert_eq!(&line2[..4090], &[b'B'; 4090]);
assert_eq!(line2[4090], b'\n'); assert_eq!(line2[4090], b'\n');
} }
}
#[test]
fn zero_len_read_is_noop_on_fresh_cache() {
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, 0, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, 0);
assert!(cache.slots.iter().all(|s| s.len == 0));
}
#[test]
fn zero_len_read_is_noop_on_populated_cache() {
let f = make_file(b"abc");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
cache.get(&file, 0, 1).unwrap();
let tick_before = cache.tick;
let result = cache.get(&file, 0, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, tick_before);
}
#[test]
fn zero_len_read_at_max_offset_is_ok() {
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, u64::MAX, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, 0);
assert!(cache.slots.iter().all(|s| s.len == 0));
}
#[test]
fn nonzero_read_range_overflow_returns_invalid_input() {
let f = make_file(b"abc");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let err = cache.get(&file, u64::MAX, 1).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
}
}
+307 -26
View File
@@ -1,10 +1,21 @@
use std::borrow::Cow;
/// Maximum input length for wrap/format operations (10 MB). /// Maximum input length for wrap/format operations (10 MB).
/// Lines exceeding this are returned as-is to avoid pathological cases. /// Callers should check against this constant before invoking `wrap_line_chars`
/// to avoid pathological cases on oversized lines.
pub const MAX_WRAP_INPUT_LEN: usize = 10 * 1024 * 1024; pub const MAX_WRAP_INPUT_LEN: usize = 10 * 1024 * 1024;
/// Split a line into chunks of exactly `width` characters (display columns). /// Column spacing for tab stop alignment.
const TAB_WIDTH: usize = 4;
/// Split a line into chunks of exactly `width` display columns.
/// For a log viewer, we want character-level wrapping, not word-level. /// For a log viewer, we want character-level wrapping, not word-level.
/// Uses `unicode-width` for correct CJK/emoji/zero-width handling.
/// Tab characters expand to the next tab-stop boundary and split across
/// rows when the expansion exceeds the remaining width.
pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> { pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
use unicode_width::UnicodeWidthChar;
if width == 0 { if width == 0 {
return vec![String::new()]; return vec![String::new()];
} }
@@ -15,21 +26,40 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
let mut row = String::new(); let mut row = String::new();
let mut col = 0; let mut col = 0;
for ch in line.chars() { for ch in line.chars() {
let w = if ch == '\t' { 4 } else { 1 };
if col + w > width && !row.is_empty() {
result.push(std::mem::take(&mut row));
col = 0;
}
if ch == '\t' { if ch == '\t' {
row.push_str(" "); let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
col += 4; let mut remaining = tab_stop;
while remaining > 0 {
let avail = width.saturating_sub(col);
let fill = remaining.min(avail);
for _ in 0..fill {
row.push(' ');
}
col += fill;
remaining -= fill;
if col >= width {
result.push(std::mem::take(&mut row));
col = 0;
}
}
} else { } else {
let w = if ch.is_control() {
// Control characters (except tab): width 0, still pushed to preserve content.
// Visible rendering is the caller's responsibility.
0
} else {
ch.width().unwrap_or(0)
};
if col + w > width && !row.is_empty() {
result.push(std::mem::take(&mut row));
col = 0;
}
row.push(ch); row.push(ch);
col += w; col += w;
} if col >= width {
if col >= width { result.push(std::mem::take(&mut row));
result.push(std::mem::take(&mut row)); col = 0;
col = 0; }
} }
} }
if !row.is_empty() { if !row.is_empty() {
@@ -41,21 +71,81 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
result result
} }
/// Format a line as pretty-printed JSON if it's a JSON Object. /// Count wrapped rows for a line without allocating the wrapped strings.
/// Returns the original line unchanged for non-JSON or non-Object content. /// MUST produce the same count as `wrap_line_chars(line, width).len()`.
pub fn format_json_line(line: &str) -> String { pub fn wrap_line_count(line: &str, width: usize) -> usize {
if line.trim().is_empty() { use unicode_width::UnicodeWidthChar;
return String::new();
if width == 0 || line.is_empty() {
return 1;
} }
// Quick pre-check: only try parsing if it starts with '{' let mut count = 0usize;
if !line.trim_start().starts_with('{') { let mut col = 0usize;
return line.to_string(); let mut row_has_content = false;
for ch in line.chars() {
if ch == '\t' {
let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
let mut remaining = tab_stop;
while remaining > 0 {
let avail = width.saturating_sub(col);
let fill = remaining.min(avail);
col += fill;
remaining -= fill;
if col >= width {
count += 1;
col = 0;
row_has_content = false;
} else {
row_has_content = true;
}
}
} else {
let w = if ch.is_control() {
0
} else {
ch.width().unwrap_or(0)
};
if col + w > width && row_has_content {
count += 1;
col = 0;
}
col += w;
if col >= width {
count += 1;
col = 0;
row_has_content = false;
} else {
row_has_content = true;
}
}
}
if row_has_content {
count += 1;
}
if count == 0 {
count = 1;
}
count
}
/// Format a line as pretty-printed JSON if it's a JSON Object.
/// Returns the original line borrowed for non-JSON or non-Object content,
/// only allocating when pretty-printing actually happens.
pub fn format_json_line(line: &str) -> Cow<'_, str> {
let trimmed = line.trim();
if trimmed.is_empty() {
return Cow::Borrowed("");
}
if !trimmed.starts_with('{') {
return Cow::Borrowed(line);
} }
match serde_json::from_str::<serde_json::Value>(line) { match serde_json::from_str::<serde_json::Value>(line) {
Ok(value) if value.is_object() => { Ok(value) if value.is_object() => match serde_json::to_string_pretty(&value) {
serde_json::to_string_pretty(&value).unwrap_or_else(|_| line.to_string()) Ok(s) => Cow::Owned(s),
} Err(_) => Cow::Borrowed(line),
_ => line.to_string(), },
_ => Cow::Borrowed(line),
} }
} }
@@ -96,7 +186,7 @@ mod tests {
#[test] #[test]
fn test_wrap_with_tab() { fn test_wrap_with_tab() {
let result = wrap_line_chars("a\tb", 4); let result = wrap_line_chars("a\tb", 4);
assert_eq!(result, vec!["a", " ", "b"]); assert_eq!(result, vec!["a ", "b"]);
} }
#[test] #[test]
@@ -132,4 +222,195 @@ mod tests {
fn test_max_wrap_input_len_constant() { fn test_max_wrap_input_len_constant() {
assert_eq!(MAX_WRAP_INPUT_LEN, 10 * 1024 * 1024); assert_eq!(MAX_WRAP_INPUT_LEN, 10 * 1024 * 1024);
} }
#[test]
fn test_wrap_cjk_chars() {
let result = wrap_line_chars("你好", 3);
assert_eq!(result, vec!["", ""]);
}
#[test]
fn test_wrap_cjk_ascii_mixed() {
let result = wrap_line_chars("a你好", 4);
assert_eq!(result, vec!["a你", ""]);
}
#[test]
fn test_wrap_zero_width_char() {
let result = wrap_line_chars("a\u{200B}b", 2);
assert_eq!(result, vec!["a\u{200B}b"]);
}
#[test]
fn test_wrap_emoji() {
let result = wrap_line_chars("😀a", 3);
assert_eq!(result, vec!["😀a"]);
}
#[test]
fn test_wrap_emoji_exact_wrap() {
let result = wrap_line_chars("😀a", 2);
assert_eq!(result, vec!["😀", "a"]);
}
#[test]
fn test_wrap_combining_mark() {
// Scalar-width wrapping: combining mark (width 0) stays with next base char,
// not the preceding one, because the base char already triggered a flush.
let result = wrap_line_chars("a\u{0301}b", 1);
assert_eq!(result, vec!["a", "\u{0301}b"]);
}
#[test]
fn test_wrap_cjk_width_one() {
let result = wrap_line_chars("你好", 1);
assert_eq!(result, vec!["", ""]);
}
#[test]
fn test_tab_narrow_width() {
let result = wrap_line_chars("\t", 2);
assert_eq!(result, vec![" ", " "]);
let result = wrap_line_chars("\t", 1);
assert_eq!(result, vec![" ", " ", " ", " "]);
}
#[test]
fn test_tab_stop_alignment() {
assert_eq!(wrap_line_chars("a\tb", 8), vec!["a b"]);
assert_eq!(wrap_line_chars("ab\t", 4), vec!["ab "]);
assert_eq!(wrap_line_chars("abc\tb", 8), vec!["abc b"]);
}
#[test]
fn test_tab_at_line_boundary() {
let result = wrap_line_chars("a\tb", 4);
assert_eq!(result, vec!["a ", "b"]);
}
#[test]
fn test_tab_regression_ab_tab() {
let result = wrap_line_chars("ab\t", 4);
assert_eq!(result, vec!["ab "]);
}
#[test]
fn test_wrap_count_matches_chars_empty() {
assert_eq!(wrap_line_count("", 80), wrap_line_chars("", 80).len());
}
#[test]
fn test_wrap_count_matches_chars_zero_width() {
assert_eq!(
wrap_line_count("hello", 0),
wrap_line_chars("hello", 0).len()
);
}
#[test]
fn test_wrap_count_matches_chars_ascii_short() {
for width in 1..20 {
assert_eq!(
wrap_line_count("hello world", width),
wrap_line_chars("hello world", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_cjk() {
for width in 1..10 {
assert_eq!(
wrap_line_count("你好世界", width),
wrap_line_chars("你好世界", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_emoji() {
for width in 1..6 {
assert_eq!(
wrap_line_count("😀👨‍👩‍👧", width),
wrap_line_chars("😀👨‍👩‍👧", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_combining_mark() {
for width in 1..6 {
assert_eq!(
wrap_line_count("a\u{0301}b\u{0301}c", width),
wrap_line_chars("a\u{0301}b\u{0301}c", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_tab_variants() {
let inputs = ["\t", "a\tb", "ab\t", "\t\t", "abc\tb", "a\t\t\tb"];
for input in inputs {
for width in 1..9 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"input={input:?} width={width}"
);
}
}
}
#[test]
fn test_wrap_count_matches_chars_mixed() {
let input = "log: 你好 😀\tvalue=true [ERROR]";
for width in 1..40 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_zero_width_chars() {
let input = "a\u{200B}b\u{200C}c\u{200D}d";
for width in 1..8 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_control_chars() {
let input = "a\x07b\x1Fc";
for width in 1..8 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_long_log_line() {
let input = "2025-01-14T10:23:45.123Z [INFO] [auth] request handled status=200 \
latency=0.234s endpoint=/api/v1/users user=user_42 request_id=req_abc123def456";
for width in [10, 20, 40, 60, 80, 100, 120] {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
} }
+306 -90
View File
@@ -16,7 +16,14 @@
// 类似于 Python 的 dict、JavaScript 的 Object/Map、Java 的 HashMap。 // 类似于 Python 的 dict、JavaScript 的 Object/Map、Java 的 HashMap。
// 它存储键值对(key-value pairs),可以通过键快速查找对应的值。 // 它存储键值对(key-value pairs),可以通过键快速查找对应的值。
// 这里用 HashMap<String, Value> 来存储 JSON 中除 timestamp/level 之外的其他字段。 // 这里用 HashMap<String, Value> 来存储 JSON 中除 timestamp/level 之外的其他字段。
use std::collections::HashMap; use std::collections::{HashMap, HashSet};
// serde::de 中的 Visitor / MapAccess 允许我们自定义 JSON 对象的反序列化过程。
// 默认的 serde_json::from_str::<HashMap<_, _>>() 遇到重复键时会采用"后者覆盖前者"last-wins),
// 前面的值被静默丢弃。这里我们通过自定义 Visitor 在反序列化过程中逐个观察 key-value 对,
// 在保持 last-wins 行为的同时,将重复 key 的所有值记录到 DuplicateKey 中。
use serde::Deserializer;
use serde::de::{MapAccess, Visitor};
// serde_json::Value — 来自 serde_json 库(Rust 中最流行的 JSON 处理库)。 // serde_json::Value — 来自 serde_json 库(Rust 中最流行的 JSON 处理库)。
// Value 是一个枚举类型,可以表示任意 JSON 值: // Value 是一个枚举类型,可以表示任意 JSON 值:
@@ -32,7 +39,20 @@ use serde_json::Value;
// ─── 引入项目内部类型 ────────────────────────────────────────────────────── // ─── 引入项目内部类型 ──────────────────────────────────────────────────────
// crate 表示"当前项目(crate"。 // crate 表示"当前项目(crate"。
// types 模块中定义了 LogEntry(一条日志记录)和 LogLevel(日志级别,如 INFO/ERROR)。 // types 模块中定义了 LogEntry(一条日志记录)和 LogLevel(日志级别,如 INFO/ERROR)。
use crate::types::{LogEntry, LogLevel}; use crate::types::{DuplicateKey, LogEntry, LogLevel};
// ─── strip_bom 辅助函数 ──────────────────────────────────────────────────
// 剥离行首的 UTF-8 BOMByte Order Mark, U+FEFF)。
//
// Windows 环境和某些导出工具生成的文件会在行首插入 BOM,
// 而 serde_json 不接受 BOM 前缀的 JSON 文本(会报 "expected value" 错误)。
// 只剥离一个前导 BOM,不处理多个 BOM 或行内 BOM(那些是畸形输入)。
//
// 参数: line: &str — 输入字符串切片。
// 返回: &str — 去掉 BOM 后的字符串切片(借用原始字符串,零分配)。
fn strip_bom(line: &str) -> &str {
line.strip_prefix('\u{FEFF}').unwrap_or(line)
}
// ─── detect_json_log 函数 ────────────────────────────────────────────────── // ─── detect_json_log 函数 ──────────────────────────────────────────────────
// 检测一行文本是否是一个 JSON 对象。 // 检测一行文本是否是一个 JSON 对象。
@@ -63,7 +83,92 @@ pub fn detect_json_log(line: &str) -> bool {
// 则匹配成功。_ 是通配符,表示"不关心对象里面的具体内容"。 // 则匹配成功。_ 是通配符,表示"不关心对象里面的具体内容"。
// //
// 如果匹配到 Ok(Value::Object(_)) 返回 true,否则返回 false。 // 如果匹配到 Ok(Value::Object(_)) 返回 true,否则返回 false。
matches!(serde_json::from_str::<Value>(line), Ok(Value::Object(_))) matches!(
serde_json::from_str::<Value>(strip_bom(line)),
Ok(Value::Object(_))
)
}
// ─── DuplicateKeyVisitor ──────────────────────────────────────────────────
// 自定义 serde Visitor,在反序列化 JSON 对象时检测重复 key。
//
// 工作原理:
// serde 的 MapAccess trait 允许我们逐个遍历 JSON 对象的 key-value 对。
// 每读到一个 (key, value),我们:
// 1. 检查这个 key 是否已经见过(通过 HashSet)
// 2. 如果是重复 key,记录到 Vec<DuplicateKey> 中(包含所有出现过的值)
// 3. 将 key-value 插入 Maplast-wins,与 serde_json 默认行为一致)
//
// 这样既保持了兼容性(last-wins),又不丢失信息(所有值都记录在 DuplicateKey 中)。
struct DuplicateKeyVisitor;
impl<'de> Visitor<'de> for DuplicateKeyVisitor {
// 返回类型:(serde_json::Map, 重复 key 列表)
type Value = (serde_json::Map<String, Value>, Vec<DuplicateKey>);
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("a JSON object")
}
fn visit_map<A>(self, mut access: A) -> Result<Self::Value, A::Error>
where
A: MapAccess<'de>,
{
let mut map = serde_json::Map::new();
let mut seen = HashSet::new();
let mut duplicates: Vec<DuplicateKey> = Vec::new();
while let Some((key, value)) = access.next_entry::<String, Value>()? {
if !seen.insert(key.clone()) {
// 重复 key:将之前 map 中的值和当前值都记录下来
if let Some(existing) = duplicates.iter_mut().find(|d| d.key == key) {
// 同一个 key 第三次及以上出现:追加当前值
existing.values.push(value.clone());
} else {
// 同一个 key 第二次出现:记录第一次的值 + 当前值
let prev_value = map.get(&key).cloned().unwrap_or(Value::Null);
duplicates.push(DuplicateKey {
key: key.clone(),
values: vec![prev_value, value.clone()],
});
}
}
// last-wins:后出现的值覆盖前面的值,与 serde_json 默认行为一致
map.insert(key, value);
}
Ok((map, duplicates))
}
}
/// 使用自定义 Visitor 解析 JSON 对象,同时检测重复 key。
///
/// 返回 (serde_json::Map, Vec<DuplicateKey>)
/// - Map 中存储所有 key-value(重复 key 取 last-wins
/// - Vec 中记录所有重复 key 及其全部值
fn parse_json_object_with_duplicates(
json: &str,
) -> Option<(serde_json::Map<String, Value>, Vec<DuplicateKey>)> {
let mut deserializer = serde_json::Deserializer::from_str(json);
deserializer.deserialize_map(DuplicateKeyVisitor).ok()
}
// ─── take_string_field_from_map 辅助函数 ──────────────────────────────────
// 从 serde_json::Map 中安全提取字符串字段。
// 功能与原 take_string_field 相同,但操作 serde_json::Map 而非 HashMap。
fn take_string_field_from_map(
obj: &mut serde_json::Map<String, Value>,
keys: &[&str],
) -> Option<String> {
for key in keys {
if obj.get(*key).is_some_and(Value::is_string) {
let Some(Value::String(v)) = obj.remove(*key) else {
unreachable!("value was checked as string");
};
return Some(v);
}
}
None
} }
// ─── parse_line 函数 ────────────────────────────────────────────────────── // ─── parse_line 函数 ──────────────────────────────────────────────────────
@@ -73,107 +178,35 @@ pub fn detect_json_log(line: &str) -> bool {
// 返回: Option<LogEntry> — 解析成功返回 Some(LogEntry),失败或不合法返回 None。 // 返回: Option<LogEntry> — 解析成功返回 Some(LogEntry),失败或不合法返回 None。
// Option 是 Rust 的可选类型:Some(值) 表示有值,None 表示没有值。 // Option 是 Rust 的可选类型:Some(值) 表示有值,None 表示没有值。
pub fn parse_line(line: &str) -> Option<LogEntry> { pub fn parse_line(line: &str) -> Option<LogEntry> {
// ─── 跳过空行 ────────────────────────────────────────────────────────── let line = strip_bom(line);
// line.trim() 去除首尾空白字符(空格、制表符、换行符等)。
// .is_empty() 检查是否为空字符串。
// 如果去除空白后是空的,说明是空行,不需要解析,直接返回 None。
if line.trim().is_empty() { if line.trim().is_empty() {
return None; return None;
} }
// ─── 解析 JSON 为 HashMap ────────────────────────────────────────────── // ─── 使用自定义 Visitor 解析 JSON ──────────────────────────────────
// serde_json::from_str(line) 尝试将字符串解析为 JSON // 通过 DuplicateKeyVisitor 反序列化,在保持 last-wins 的同时检测重复 key
// 由于我们声明了 HashMap<String, Value> 类型,Rust 会自动将 JSON 对象 // 返回的 (serde_json::Map, Vec<DuplicateKey>) 中:
// 转换为 HashMap,其中每个键是 String,每个值是 serde_json::Value。 // - Map 包含所有 key-value(重复 key 取最后一个值)
// // - Vec 记录了所有重复 key 及其出现过的全部值
// .ok() 将 Result 转换为 Option let (mut obj, duplicate_keys) = parse_json_object_with_duplicates(line)?;
// Ok(值) → Some(值)
// Err(_) → None
//
// 末尾的 ? 是"问号操作符"try operator),在这里的作用是:
// 如果 .ok() 返回 None(即 JSON 解析失败),则整个函数直接返回 None。
// 如果返回 Some(hashmap),则将 hashmap 取出并绑定到 fields 变量。
//
// let mut 表示这是一个"可变变量"mutable variable),
// 后续代码会修改这个 HashMap(从中删除已识别的字段)。
let mut fields: HashMap<String, Value> = serde_json::from_str(line).ok()?;
// ─── 保存原始行内容 ──────────────────────────────────────────────────
// line.to_string() 将 &str(字符串切片引用)转换为 String(拥有所有权的字符串)。
// 保存原始行是为了在 UI 中显示未经修改的原始日志内容。
let raw_line = line.to_string(); let raw_line = line.to_string();
let timestamp =
// ─── 提取时间戳字段 ────────────────────────────────────────────────── take_string_field_from_map(&mut obj, &["timestamp", "time", "ts", "@timestamp"]);
// 这段代码尝试从 JSON 中提取时间戳,逻辑如下: let level = take_string_field_from_map(&mut obj, &["level", "lvl", "severity"])
//
// 1. ["timestamp", "time", "ts", "@timestamp"] — 候选键名数组。
// 不同日志系统使用不同的时间戳字段名,这里列出常见的几种。
//
// 2. .iter() — 创建数组的迭代器,可以逐个遍历元素。
//
// 3. .find_map(|key| fields.remove(*key)) — 对每个候选键名:
// - fields.remove(*key): 尝试从 HashMap 中移除该键并返回对应的值。
// 如果键不存在,remove 返回 None。
// *key 是解引用(deref),将 &str(引用)转换为 str,因为 remove 接受 &str 类型。
// - find_map: 遍历所有候选键,返回第一个 Some(值) 的结果。
// 即找到第一个存在的键就停止。
//
// 4. .and_then(|v| v.as_str().map(String::from)) — 如果找到了时间戳值:
// - v.as_str(): 尝试将 serde_json::Value 转换为 &str(字符串切片)。
// 如果 Value 不是字符串类型(比如是数字),返回 None。
// - .map(String::from): 如果是字符串,将其转换为 String(拥有所有权的字符串)。
// - and_then: 类似于 map,但用于"扁平化"嵌套的 Option。
// 如果 as_str() 返回 None,整个链返回 None。
let timestamp = ["timestamp", "time", "ts", "@timestamp"]
.iter()
.find_map(|key| fields.remove(*key))
.and_then(|v| v.as_str().map(String::from));
// ─── 提取日志级别字段 ──────────────────────────────────────────────
// 与时间戳提取类似,但多了一步:将字符串解析为 LogLevel 枚举。
let level = ["level", "lvl", "severity"]
.iter()
.find_map(|key| fields.remove(*key))
.and_then(|v| v.as_str().map(String::from))
// .map(|s| s.parse::<LogLevel>(...)) — 尝试将字符串解析为 LogLevel 枚举。
// parse::<LogLevel> 中的 ::<LogLevel> 是泛型参数(turbofish 语法),
// 指定我们要将字符串解析为 LogLevel 类型。
//
// .unwrap_or_else(|e| match e {}) — 错误处理:
// - 如果解析成功,直接返回 LogLevel 值。
// - 如果解析失败(字符串不匹配任何已知的日志级别),执行闭包。
// - |e| match e {}: 这个闭包接收解析错误 e,用 match e {} 进行"穷尽匹配"。
// 由于 LogLevel 的 parse 错误类型是一个空枚举(没有任何变体),
// match e {} 意味着"这个分支永远不会执行"unreachable)。
// 但实际上,如果 parse 失败,unwrap_or_else 不会执行这个闭包——
// 等等,这里有个细微之处:
// unwrap_or_else 只在 Err 时执行闭包,但 match e {} 对空枚举是合法的
// (因为空枚举没有任何可能的值,所以 match 是穷尽的)。
// 不过这里的实际效果是:如果 parse 失败,整个 .map() 返回 None
// (因为 unwrap_or_else 返回的类型是 LogLevel,而空 match 不会有返回值)。
//
// 实际上更准确的解释:parse() 的错误类型是 Infallible(不可失败的),
// 即解析总是成功。所以 unwrap_or_else 永远不会被执行。
// 但即使如此,unwrap_or_else 的闭包也需要类型正确,match e {} 满足这一点。
.map(|s| s.parse::<LogLevel>().unwrap_or_else(|e| match e {})); .map(|s| s.parse::<LogLevel>().unwrap_or_else(|e| match e {}));
// ─── 构建 LogEntry 并返回 ────────────────────────────────────────── // serde_json::Map → HashMap:剩余字段转为 HashMap 存入 fields
// 此时 fields HashMap 中还剩下未被提取的字段(如 message、自定义字段等)。 let fields: HashMap<String, Value> = obj.into_iter().collect();
// timestamp 和 level 已经从 fields 中移除了(通过 remove)。
//
// Some(LogEntry { ... }) — 使用结构体字面量创建 LogEntry 实例,
// 并用 Some() 包裹表示"有值"。
Some(LogEntry { Some(LogEntry {
// line_number 设为 0,由调用者(如 parse_line_with_number)设置正确的值。
line_number: 0, line_number: 0,
// 原始行内容。
raw_line, raw_line,
// 时间戳(可能为 None,如果 JSON 中没有时间戳字段)。
timestamp, timestamp,
// 日志级别(可能为 None,如果 JSON 中没有级别字段)。
level, level,
// 剩余的 JSON 字段(已移除 timestamp 和 level)。
fields, fields,
duplicate_keys,
}) })
} }
@@ -317,6 +350,13 @@ mod tests {
assert_eq!(parse_line(warn_line).unwrap().level, Some(LogLevel::Warn)); assert_eq!(parse_line(warn_line).unwrap().level, Some(LogLevel::Warn));
} }
#[test]
// Regression: level field with surrounding whitespace should still be recognized.
fn test_level_whitespace_in_json() {
let line = r#"{"level":" WARN ","message":"test"}"#;
assert_eq!(parse_line(line).unwrap().level, Some(LogLevel::Warn));
}
#[test] #[test]
// 测试:所有候选时间戳键名(timestamp, time, ts, @timestamp)都能被识别。 // 测试:所有候选时间戳键名(timestamp, time, ts, @timestamp)都能被识别。
fn test_timestamp_key_names() { fn test_timestamp_key_names() {
@@ -357,4 +397,180 @@ mod tests {
assert_eq!(entry.level, Some(LogLevel::Info), "failed for key: {key}"); assert_eq!(entry.level, Some(LogLevel::Info), "failed for key: {key}");
} }
} }
#[test]
// 测试:数字类型的 level 值不应被提取,应保留在 fields 中。
fn test_numeric_level_preserved_in_fields() {
let line = r#"{"level":30,"message":"hello"}"#;
let entry = parse_line(line).unwrap();
// level 不是字符串,应返回 None。
assert!(entry.level.is_none());
// 数字 level 应保留在 fields 中,不被静默丢弃。
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
// message 仍正常存在。
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("hello".to_string()))
);
}
#[test]
// 测试:数字类型的 timestamp 值不应被提取,应保留在 fields 中。
fn test_numeric_timestamp_preserved_in_fields() {
let line = r#"{"timestamp":1718000000,"message":"hello"}"#;
let entry = parse_line(line).unwrap();
// timestamp 不是字符串,应返回 None。
assert!(entry.timestamp.is_none());
// 数字 timestamp 应保留在 fields 中。
assert_eq!(
entry.fields.get("timestamp"),
Some(&Value::Number(1718000000.into()))
);
}
#[test]
// 测试:当第一个候选键是数字时,应回退到下一个字符串类型的候选键。
fn test_fallback_to_string_key() {
let line = r#"{"level":30,"lvl":"INFO","message":"hello"}"#;
let entry = parse_line(line).unwrap();
// "level" 是数字,应跳过;"lvl" 是字符串,应成功提取。
assert_eq!(entry.level, Some(LogLevel::Info));
// 数字 "level" 保留在 fields 中。
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
// "lvl" 已被成功提取并从 fields 中移除。
assert!(entry.fields.get("lvl").is_none());
}
#[test]
// 测试:timestamp 的 fallback 行为 — 数字 timestamp 被保留,字符串 time 被提取。
fn test_timestamp_fallback_preserves_numeric() {
let line = r#"{"timestamp":1718000000,"time":"2024-01-01T00:00:00Z"}"#;
let entry = parse_line(line).unwrap();
// "timestamp" 是数字,跳过;"time" 是字符串,成功提取。
assert_eq!(entry.timestamp, Some("2024-01-01T00:00:00Z".to_string()));
// 数字 "timestamp" 保留在 fields 中。
assert_eq!(
entry.fields.get("timestamp"),
Some(&Value::Number(1718000000.into()))
);
// "time" 已被提取并从 fields 中移除。
assert!(entry.fields.get("time").is_none());
}
#[test]
fn test_bom_prefixed_json() {
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(entry.level, Some(LogLevel::Info));
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("hello".into()))
);
}
#[test]
fn test_bom_prefixed_detect() {
assert!(detect_json_log("\u{FEFF}{\"level\":\"INFO\"}"));
}
#[test]
fn test_bom_only_whitespace() {
assert!(parse_line("\u{FEFF} ").is_none());
}
#[test]
fn test_bom_stripped_from_raw_line() {
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(entry.raw_line, "{\"level\":\"INFO\",\"message\":\"hello\"}");
}
#[test]
fn test_internal_bom_not_stripped() {
let line = "{\"message\":\"\u{FEFF}hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("\u{FEFF}hello".into()))
);
}
// ─── 重复 key 检测测试 ──────────────────────────────────────────────
#[test]
fn test_no_duplicate_keys_normal_json() {
let line = r#"{"level":"INFO","message":"hello"}"#;
let entry = parse_line(line).unwrap();
assert!(entry.duplicate_keys.is_empty());
}
#[test]
fn test_duplicate_message_key_detected() {
let line = r#"{"level":"INFO","message":"first","message":"second"}"#;
let entry = parse_line(line).unwrap();
// last-wins: fields 中保留第二个值
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("second".into()))
);
// 重复 key 记录中包含所有值
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "message");
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
assert_eq!(
entry.duplicate_keys[0].values[0],
Value::String("first".into())
);
assert_eq!(
entry.duplicate_keys[0].values[1],
Value::String("second".into())
);
}
#[test]
fn test_duplicate_key_last_wins() {
let line = r#"{"msg":"a","msg":"b","msg":"c"}"#;
let entry = parse_line(line).unwrap();
// last-wins: 最终值是 "c"
assert_eq!(entry.fields.get("msg"), Some(&Value::String("c".into())));
// 三个值都被记录
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].values.len(), 3);
assert_eq!(entry.duplicate_keys[0].values[0], Value::String("a".into()));
assert_eq!(entry.duplicate_keys[0].values[1], Value::String("b".into()));
assert_eq!(entry.duplicate_keys[0].values[2], Value::String("c".into()));
}
#[test]
fn test_multiple_different_duplicate_keys() {
let line = r#"{"a":"1","b":"2","a":"3","b":"4"}"#;
let entry = parse_line(line).unwrap();
assert_eq!(entry.duplicate_keys.len(), 2);
let dup_a = entry.duplicate_keys.iter().find(|d| d.key == "a").unwrap();
let dup_b = entry.duplicate_keys.iter().find(|d| d.key == "b").unwrap();
assert_eq!(dup_a.values.len(), 2);
assert_eq!(dup_b.values.len(), 2);
}
#[test]
fn test_duplicate_level_key_detected() {
let line = r#"{"level":"INFO","level":"ERROR","message":"hello"}"#;
let entry = parse_line(line).unwrap();
// last-wins: level 被提取为 ERROR
assert_eq!(entry.level, Some(LogLevel::Error));
// 重复 key 被记录
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "level");
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
}
#[test]
fn test_duplicate_timestamp_key_detected() {
let line = r#"{"timestamp":"2024-01-01","timestamp":"2024-06-01"}"#;
let entry = parse_line(line).unwrap();
// last-wins: timestamp 提取为后者
assert_eq!(entry.timestamp, Some("2024-06-01".to_string()));
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "timestamp");
}
} }
+43 -3
View File
@@ -100,13 +100,21 @@ fn detect_level_from_text(line: &str) -> Option<LogLevel> {
) )
} }
// ─── is_ident_char ─────────────────────────────────────────────────────────
/// Whether a byte looks like an ASCII identifier continuation character
/// (letter / digit / underscore). Log-level keywords must NOT be adjacent to
/// such characters to count as a valid word boundary.
fn is_ident_char(b: u8) -> bool {
b.is_ascii_alphanumeric() || b == b'_'
}
// ─── is_word_boundary ─────────────────────────────────────────────────────── // ─── is_word_boundary ───────────────────────────────────────────────────────
/// Check that the match at `start..start+len` is surrounded by non-alphabetic /// Check that the match at `start..start+len` is surrounded by non-identifier
/// characters (or the string edge). /// characters (or the string edge).
fn is_word_boundary(text: &str, start: usize, len: usize) -> bool { fn is_word_boundary(text: &str, start: usize, len: usize) -> bool {
let before_ok = start == 0 || !text.as_bytes()[start - 1].is_ascii_alphabetic(); let before_ok = start == 0 || !is_ident_char(text.as_bytes()[start - 1]);
let after_idx = start + len; let after_idx = start + len;
let after_ok = after_idx >= text.len() || !text.as_bytes()[after_idx].is_ascii_alphabetic(); let after_ok = after_idx >= text.len() || !is_ident_char(text.as_bytes()[after_idx]);
before_ok && after_ok before_ok && after_ok
} }
@@ -209,4 +217,36 @@ mod tests {
let line = format!("{prefix} ERROR something"); let line = format!("{prefix} ERROR something");
assert_eq!(detect_level(&line), Some(LogLevel::Error)); assert_eq!(detect_level(&line), Some(LogLevel::Error));
} }
#[test]
fn test_boundary_rejects_trailing_digits() {
assert_eq!(detect_level("ERROR123"), None);
assert_eq!(detect_level("WARN2: bad"), None);
assert_eq!(detect_level("ERR2"), None);
}
#[test]
fn test_boundary_rejects_underscore() {
assert_eq!(detect_level("INFO_foo"), None);
assert_eq!(detect_level("DBG_value=5"), None);
}
#[test]
fn test_boundary_rejects_leading_digits_and_underscore() {
assert_eq!(detect_level("123ERROR: fail"), None);
assert_eq!(detect_level("foo_ERROR: fail"), None);
assert_eq!(detect_level("1WRN"), None);
}
#[test]
fn test_boundary_accepts_valid_suffixes() {
assert_eq!(detect_level("ERROR: fail"), Some(LogLevel::Error));
assert_eq!(detect_level("[ERROR] fail"), Some(LogLevel::Error));
assert_eq!(detect_level("ERROR fail"), Some(LogLevel::Error));
}
#[test]
fn test_boundary_camel_case_regression() {
assert_eq!(detect_level("errorLevel"), None);
}
} }
+50 -5
View File
@@ -71,13 +71,17 @@ impl FromStr for LogLevel {
// 接收一个字符串切片 &str,返回 Result<LogLevel, Infallible>。 // 接收一个字符串切片 &str,返回 Result<LogLevel, Infallible>。
// 由于 Err 类型是 Infallible,实际上返回值总是 Ok(LogLevel)。 // 由于 Err 类型是 Infallible,实际上返回值总是 Ok(LogLevel)。
fn from_str(s: &str) -> Result<Self, Self::Err> { fn from_str(s: &str) -> Result<Self, Self::Err> {
// `s.to_uppercase()` — 字符串转换为大写,实现不区分大小写的匹配 // `s.trim()` — 去除字符串前后的 Unicode 空白字符
// 例如 " WARN " → "WARN""\tINFO\n" → "INFO"。
let trimmed = s.trim();
// `trimmed.to_uppercase()` — 将 trimmed 后的字符串转换为大写,实现不区分大小写的匹配。
// 例如 "info"、"Info"、"INFO" 都会被转换为 "INFO"。 // 例如 "info"、"Info"、"INFO" 都会被转换为 "INFO"。
// 返回一个新的 String(堆分配)。 // 返回一个新的 String(堆分配)。
// //
// `.as_str()` — 将 String 转换回 &str(字符串切片引用)。 // `.as_str()` — 将 String 转换回 &str(字符串切片引用)。
// 因为 match 需要匹配 &str 而不是 String。 // 因为 match 需要匹配 &str 而不是 String。
match s.to_uppercase().as_str() { match trimmed.to_uppercase().as_str() {
// `|` 在 match 分支中表示"或"multiple patterns)。 // `|` 在 match 分支中表示"或"multiple patterns)。
// "ERROR" | "ERR" | "SEVERE" | "FATAL" 都匹配到 LogLevel::Error。 // "ERROR" | "ERR" | "SEVERE" | "FATAL" 都匹配到 LogLevel::Error。
"ERROR" | "ERR" | "SEVERE" | "FATAL" => Ok(LogLevel::Error), "ERROR" | "ERR" | "SEVERE" | "FATAL" => Ok(LogLevel::Error),
@@ -86,9 +90,9 @@ impl FromStr for LogLevel {
"DEBUG" | "DBG" => Ok(LogLevel::Debug), "DEBUG" | "DBG" => Ok(LogLevel::Debug),
"TRACE" | "TRC" => Ok(LogLevel::Trace), "TRACE" | "TRC" => Ok(LogLevel::Trace),
// `_` 是通配符,匹配所有未被上面分支捕获的值。 // `_` 是通配符,匹配所有未被上面分支捕获的值。
// 对于未知级别,包装为 Unknown 并保存原始字符串。 // 对于未知级别,包装为 Unknown 并保存 trimmed 后的字符串。
// s.to_string() 将 &str 转换为 String(注意这里用原始 s,不是大写后的)。 // s.to_string() 将 &str 转换为 String(注意这里用 trimmed,不是原始 s)。
_ => Ok(LogLevel::Unknown(s.to_string())), _ => Ok(LogLevel::Unknown(trimmed.to_string())),
} }
} }
} }
@@ -119,6 +123,18 @@ impl fmt::Display for LogLevel {
} }
// ─── LogEntry 结构体 ──────────────────────────────────────────────────────── // ─── LogEntry 结构体 ────────────────────────────────────────────────────────
/// 记录 JSON 日志中出现的重复 key 信息
///
/// 当 JSON 对象中同一个 key 出现多次时,serde_json 默认 last-wins(后值覆盖前值),
/// 前面的值会静默丢失。此结构记录所有重复出现的 key 及其全部值,供 UI 展示警告。
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct DuplicateKey {
/// 重复的 key 名称
pub key: String,
/// 该 key 出现的所有值(按出现顺序排列,最后一个值是 fields 中的最终值)
pub values: Vec<Value>,
}
/// 一行解析后的日志 /// 一行解析后的日志
/// ///
/// 表示日志文件中经过解析器处理后的一行内容。 /// 表示日志文件中经过解析器处理后的一行内容。
@@ -144,6 +160,9 @@ pub struct LogEntry {
/// HashMap<String, Value> 是一个字典,键是字段名,值是 JSON 值。 /// HashMap<String, Value> 是一个字典,键是字段名,值是 JSON 值。
/// 例如 {"message": "hello", "request_id": "abc123"}。 /// 例如 {"message": "hello", "request_id": "abc123"}。
pub fields: HashMap<String, Value>, pub fields: HashMap<String, Value>,
/// JSON 中重复出现的 key 记录(正常日志为空 Vec)
pub duplicate_keys: Vec<DuplicateKey>,
} }
// ─── SearchResult 结构体 ──────────────────────────────────────────────────── // ─── SearchResult 结构体 ────────────────────────────────────────────────────
@@ -288,6 +307,31 @@ mod tests {
); );
} }
#[test]
fn test_from_str_whitespace_trimmed() {
assert_eq!(" WARN ".parse::<LogLevel>(), Ok(LogLevel::Warn));
assert_eq!("\tINFO".parse::<LogLevel>(), Ok(LogLevel::Info));
assert_eq!("ERROR\n".parse::<LogLevel>(), Ok(LogLevel::Error));
assert_eq!(" debug ".parse::<LogLevel>(), Ok(LogLevel::Debug));
assert_eq!("\tTRACE\t".parse::<LogLevel>(), Ok(LogLevel::Trace));
}
#[test]
fn test_from_str_whitespace_unknown_trimmed() {
// Unknown stores the trimmed value, not the original.
assert_eq!(
" CUSTOM ".parse::<LogLevel>(),
Ok(LogLevel::Unknown("CUSTOM".into()))
);
// Pure whitespace becomes Unknown("").
assert_eq!(" ".parse::<LogLevel>(), Ok(LogLevel::Unknown("".into())));
// Internal whitespace is NOT collapsed.
assert_eq!(
"W ARN".parse::<LogLevel>(),
Ok(LogLevel::Unknown("W ARN".into()))
);
}
#[test] #[test]
// 测试:LogLevel 的 Display 输出格式是否正确。 // 测试:LogLevel 的 Display 输出格式是否正确。
fn test_display_output() { fn test_display_output() {
@@ -319,6 +363,7 @@ mod tests {
timestamp: Some("2024-01-01T00:00:00".to_string()), timestamp: Some("2024-01-01T00:00:00".to_string()),
level: Some(LogLevel::Info), level: Some(LogLevel::Info),
fields, fields,
duplicate_keys: vec![],
}; };
// 逐字段验证。 // 逐字段验证。
+111 -37
View File
@@ -1,7 +1,7 @@
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex}; use std::sync::{Arc, Mutex};
use crossbeam_channel::{bounded, Receiver, Sender}; use crossbeam_channel::{Receiver, Sender, bounded};
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher}; use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use crate::error::Result; use crate::error::Result;
@@ -13,6 +13,7 @@ pub enum FileEvent {
Truncated { new_size: u64 }, Truncated { new_size: u64 },
Rotated { new_inode: u64 }, Rotated { new_inode: u64 },
Removed, Removed,
WatcherError { message: String },
} }
// ─── get_inode ────────────────────────────────────────────────────────────── // ─── get_inode ──────────────────────────────────────────────────────────────
@@ -34,6 +35,43 @@ struct WatchState {
last_inode: u64, last_inode: u64,
} }
fn process_event(event: Event, watch_path: &Path, state: &mut WatchState) -> Option<FileEvent> {
if !event.paths.iter().any(|p| p == watch_path) {
return None;
}
match event.kind {
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
EventKind::Remove(_) => return Some(FileEvent::Removed),
_ => return None,
}
let current_inode = get_inode(watch_path).unwrap_or(0);
let current_size = std::fs::metadata(watch_path).map(|m| m.len()).unwrap_or(0);
if current_inode != 0 && state.last_inode != 0 && current_inode != state.last_inode {
state.last_inode = current_inode;
state.last_size = current_size;
Some(FileEvent::Rotated {
new_inode: current_inode,
})
} else if current_size > state.last_size {
state.last_size = current_size;
state.last_inode = current_inode;
Some(FileEvent::Appended {
new_size: current_size,
})
} else if current_size < state.last_size {
state.last_size = current_size;
state.last_inode = current_inode;
Some(FileEvent::Truncated {
new_size: current_size,
})
} else {
None
}
}
// ─── FileWatcher ──────────────────────────────────────────────────────────── // ─── FileWatcher ────────────────────────────────────────────────────────────
pub struct FileWatcher { pub struct FileWatcher {
rx: Receiver<FileEvent>, rx: Receiver<FileEvent>,
@@ -56,24 +94,13 @@ impl FileWatcher {
notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| { notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| {
let event = match res { let event = match res {
Ok(e) => e, Ok(e) => e,
Err(_) => return, Err(error) => {
}; let _ = tx.try_send(FileEvent::WatcherError {
message: error.to_string(),
match event.kind { });
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
EventKind::Remove(_) => {
let _ = tx.send(FileEvent::Removed);
return; return;
} }
_ => return, };
}
if !event.paths.iter().any(|p| p == &watch_path) {
return;
}
let current_inode = get_inode(&watch_path).unwrap_or(0);
let current_size = std::fs::metadata(&watch_path).map(|m| m.len()).unwrap_or(0);
let mut st = state.lock().unwrap_or_else(|poison| { let mut st = state.lock().unwrap_or_else(|poison| {
// Recover from poisoned mutex — state only tracks last_size // Recover from poisoned mutex — state only tracks last_size
@@ -81,25 +108,8 @@ impl FileWatcher {
// cause a duplicate event, which is harmless. // cause a duplicate event, which is harmless.
poison.into_inner() poison.into_inner()
}); });
if let Some(fe) = process_event(event, &watch_path, &mut st) {
if current_inode != 0 && st.last_inode != 0 && current_inode != st.last_inode { let _ = tx.try_send(fe);
let _ = tx.send(FileEvent::Rotated {
new_inode: current_inode,
});
st.last_inode = current_inode;
st.last_size = current_size;
} else if current_size > st.last_size {
let _ = tx.send(FileEvent::Appended {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
} else if current_size < st.last_size {
let _ = tx.send(FileEvent::Truncated {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
} }
})?; })?;
@@ -144,6 +154,56 @@ mod tests {
events events
} }
#[test]
fn test_remove_wrong_path_ignored() {
let dir = tempfile::tempdir().expect("create temp dir");
let watched = dir.path().join("watched.log");
let other = dir.path().join("other.log");
std::fs::write(&watched, b"hello\n").expect("write watched");
std::fs::write(&other, b"other\n").expect("write other");
let mut state = WatchState {
last_size: 6,
last_inode: get_inode(&watched).unwrap_or(0),
};
let event = Event {
kind: EventKind::Remove(notify::event::RemoveKind::File),
paths: vec![other.clone()],
attrs: Default::default(),
};
let result = process_event(event, &watched, &mut state);
assert_eq!(
result, None,
"Remove for non-watched path should be ignored"
);
}
#[test]
fn test_remove_correct_path_emits_removed() {
let dir = tempfile::tempdir().expect("create temp dir");
let watched = dir.path().join("watched.log");
std::fs::write(&watched, b"hello\n").expect("write watched");
let mut state = WatchState {
last_size: 6,
last_inode: get_inode(&watched).unwrap_or(0),
};
let event = Event {
kind: EventKind::Remove(notify::event::RemoveKind::File),
paths: vec![watched.clone()],
attrs: Default::default(),
};
let result = process_event(event, &watched, &mut state);
assert!(
matches!(result, Some(FileEvent::Removed)),
"Remove for watched path should emit Removed"
);
}
#[test] #[test]
fn test_watcher_append() { fn test_watcher_append() {
let dir = tempfile::tempdir().expect("create temp dir"); let dir = tempfile::tempdir().expect("create temp dir");
@@ -232,5 +292,19 @@ mod tests {
let d = FileEvent::Rotated { new_inode: 42 }; let d = FileEvent::Rotated { new_inode: 42 };
assert_ne!(a, d); assert_ne!(a, d);
let e1 = FileEvent::WatcherError {
message: "io error".into(),
};
let e2 = FileEvent::WatcherError {
message: "io error".into(),
};
assert_eq!(e1, e2);
assert_ne!(
e1,
FileEvent::WatcherError {
message: "other".into()
}
);
} }
} }
+1
View File
@@ -11,6 +11,7 @@ anyhow.workspace = true
log-viewer-core.workspace = true log-viewer-core.workspace = true
serde_json.workspace = true serde_json.workspace = true
crossbeam-channel.workspace = true crossbeam-channel.workspace = true
unicode-width.workspace = true
[dev-dependencies] [dev-dependencies]
tempfile = { workspace = true } tempfile = { workspace = true }
+138
View File
@@ -0,0 +1,138 @@
use std::time::Instant;
use crossterm::event::{KeyCode, KeyEvent, KeyEventKind, KeyModifiers};
use super::{App, AppMode};
impl App {
pub fn handle_key(&mut self, key: KeyEvent) {
let should_handle = match key.kind {
KeyEventKind::Press => true,
KeyEventKind::Repeat => self.is_repeatable_key(&key),
KeyEventKind::Release => false,
};
if !should_handle {
return;
}
match self.mode {
AppMode::Normal => self.handle_normal_key(key),
AppMode::Settings => self.handle_settings_key(key),
}
}
/// Keys that should auto-repeat when held (scroll/navigation only).
fn is_repeatable_key(&self, key: &KeyEvent) -> bool {
let plain = key.modifiers.is_empty();
let ctrl = key.modifiers == KeyModifiers::CONTROL;
match self.mode {
AppMode::Normal => {
(plain
&& matches!(
key.code,
KeyCode::Char('j')
| KeyCode::Down
| KeyCode::Char('k')
| KeyCode::Up
| KeyCode::PageDown
| KeyCode::PageUp
))
|| (ctrl
&& matches!(
key.code,
KeyCode::Char('d')
| KeyCode::Char('u')
| KeyCode::Char('f')
| KeyCode::Char('b')
))
}
AppMode::Settings => {
plain
&& matches!(
key.code,
KeyCode::Char('j')
| KeyCode::Down
| KeyCode::Char('k')
| KeyCode::Up
| KeyCode::Left
| KeyCode::Right
)
}
}
}
fn handle_normal_key(&mut self, key: KeyEvent) {
match key.code {
KeyCode::Char('q') | KeyCode::Esc => {
self.should_quit = true;
self.last_g_press = None;
}
KeyCode::Char('j') | KeyCode::Down => {
self.scroll_down_line();
self.last_g_press = None;
}
KeyCode::Char('k') | KeyCode::Up => {
self.scroll_up_line();
self.last_g_press = None;
}
KeyCode::Char('d') if key.modifiers.contains(KeyModifiers::CONTROL) => {
self.scroll_down_half_page();
self.last_g_press = None;
}
KeyCode::Char('u') if key.modifiers.contains(KeyModifiers::CONTROL) => {
self.scroll_up_half_page();
self.last_g_press = None;
}
KeyCode::Char('f') if key.modifiers.contains(KeyModifiers::CONTROL) => {
self.scroll_down_page();
self.last_g_press = None;
}
KeyCode::PageDown => {
self.scroll_down_page();
self.last_g_press = None;
}
KeyCode::Char('b') if key.modifiers.contains(KeyModifiers::CONTROL) => {
self.scroll_up_page();
self.last_g_press = None;
}
KeyCode::PageUp => {
self.scroll_up_page();
self.last_g_press = None;
}
KeyCode::Char('G') | KeyCode::End => {
self.scroll_to_bottom();
self.last_g_press = None;
}
KeyCode::Char('g') => {
if let Some(instant) = self.last_g_press
&& instant.elapsed().as_millis() < 500
{
self.scroll_to_top();
self.last_g_press = None;
return;
}
self.last_g_press = Some(Instant::now());
}
KeyCode::Home => {
self.scroll_to_top();
self.last_g_press = None;
}
KeyCode::Tab => {
self.toggle_json_format();
self.last_g_press = None;
}
KeyCode::Char('s') | KeyCode::Char('S')
if !key.modifiers.contains(KeyModifiers::CONTROL) =>
{
self.settings_draft = self.color_config.clone();
self.settings_error = None;
self.mode = AppMode::Settings;
}
_ => {
self.last_g_press = None;
}
}
}
}
+169
View File
@@ -0,0 +1,169 @@
use std::path::Path;
use log_viewer_core::io::progressive_reader::{
IndexerMessage, ProgressiveFileReader, spawn_indexer,
};
use log_viewer_core::watcher::file_watcher::FileWatcher;
use super::{App, AppLoadingState};
impl App {
pub fn load_file(&mut self, path: &str) -> anyhow::Result<()> {
// ── Phase 1: Pure computation, no mutation of self ──────────
// If any step fails and returns ?, self remains completely untouched,
// preserving the old file's watcher, loading_state, and file_path.
let mut pfr =
ProgressiveFileReader::open(Path::new(path)).map_err(|e| anyhow::anyhow!("{e}"))?;
let new_loading_state = if pfr.is_sampling() {
// Cache miss: spawn background indexer
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let generation = pfr.generation();
let indexer_rx =
spawn_indexer(pfr.path().to_path_buf(), generation, 80, false, cancel_rx);
pfr = ProgressiveFileReader::with_channels(
Path::new(path),
cancel_tx,
indexer_rx,
generation,
)
.map_err(|e| anyhow::anyhow!("{e}"))?;
let estimated = pfr.line_count() as u64;
AppLoadingState::Loading {
reader: pfr,
estimated_lines: estimated,
progress_percent: 0.0,
}
} else {
// Cache hit: Ready state
AppLoadingState::Ready { reader: pfr }
};
let new_watcher = FileWatcher::watch(Path::new(path)).ok();
// ── Phase 2: Commit — swap self to new state ───────────────
// SAFETY: Do NOT add any fallible operations (with ?) below this point.
// The old file_watcher and loading_state are dropped here, cancelling
// any background indexer for the previous file.
self.file_watcher = new_watcher;
self.loading_state = new_loading_state;
self.file_path = Some(path.to_string());
// Reset UI state for the new file
self.cursor_line = 0;
self.v_offset = 0;
self.v_sub_offset = 0;
self.cursor_sub_offset = 0;
self.viewport_cache.invalidate();
self.last_g_press = None;
self.json_format = false;
self.mode = super::AppMode::Normal;
self.reload_after_loading = false;
Ok(())
}
/// Poll the background indexer for progress/completion.
/// Transitions Loading → Ready when indexing completes.
/// Must be called every frame in the event loop.
pub fn poll_background_indexer(&mut self) {
// Poll visual height rebuild (Ready state only).
// Recalibrate v_offset only when VHI actually changed.
if let AppLoadingState::Ready { reader } = &mut self.loading_state
&& let Some(index) = reader.poll_visual_height_rebuild()
{
if let log_viewer_core::io::progressive_reader::ReaderState::Ready {
visual_height_index,
..
} = &mut reader.state
{
*visual_height_index = Some(index);
}
let logical_top = self.v_offset.min(self.total_lines().saturating_sub(1));
let sub = self.v_sub_offset;
let first_visual = self.cursor_to_first_visual_row(logical_top);
let line_height = self
.get_visual_height_index()
.map_or(1, |idx| idx.visual_height_of_line(logical_top));
self.v_offset = first_visual.saturating_add(sub.min(line_height.saturating_sub(1)));
self.v_sub_offset = 0;
self.clamp_cursor_sub_offset_no_vhi();
self.clamp_v_offset();
self.viewport_cache.invalidate();
}
// Poll main indexer (Loading state)
let old_state = std::mem::replace(&mut self.loading_state, AppLoadingState::Empty);
if let AppLoadingState::Loading {
mut reader,
estimated_lines,
mut progress_percent,
} = old_state
{
if let Some(msg) = reader.poll_indexer() {
match msg {
IndexerMessage::Progress { percent, .. } => {
progress_percent = percent;
self.loading_state = AppLoadingState::Loading {
reader,
estimated_lines,
progress_percent,
};
}
IndexerMessage::Complete {
reader: fr,
visual_height_index,
..
} => {
let saved_cursor = self.cursor_line;
reader.set_ready(fr, visual_height_index);
self.loading_state = AppLoadingState::Ready { reader };
self.viewport_cache.invalidate();
// Clamp cursor if exact count < estimated
self.cursor_line = saved_cursor.min(self.total_lines().saturating_sub(1));
// Loading uses 1:1 logical-line offsets; Ready uses visual-row
// offsets derived from the prefix-sum index. Recompute v_offset
// so the same logical line stays visible (falls back to 1:1 when
// the index is absent, which is the case right after invalidate).
self.v_offset = self.cursor_to_first_visual_row(self.cursor_line);
self.clamp_v_offset();
self.v_sub_offset = 0;
self.cursor_sub_offset = 0;
// Gutter width changes (~N → N) shift content_width, so any
// VisualHeightIndex built with the old width is stale.
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
reader.invalidate_visual_height_index();
}
self.v_offset = new_offset;
self.v_sub_offset = new_sub;
self.cursor_sub_offset = 0;
if self.reload_after_loading {
self.reload_after_loading = false;
self.reload_ready_reader();
}
}
IndexerMessage::Error { message, .. } => {
self.loading_state = AppLoadingState::Error(message);
self.reload_after_loading = false;
}
}
} else {
self.loading_state = AppLoadingState::Loading {
reader,
estimated_lines,
progress_percent,
};
}
} else {
self.loading_state = old_state;
}
}
}
File diff suppressed because it is too large Load Diff
+209
View File
@@ -0,0 +1,209 @@
use log_viewer_core::config::ColorConfig;
use log_viewer_core::io::progressive_reader::VisualHeightIndex;
use super::viewport_cache::gutter_width_for;
use super::{App, AppLoadingState, AppMode, ViewportRenderRow};
impl App {
#[allow(dead_code)]
pub fn get_line(&self, idx: usize) -> Option<String> {
match &self.loading_state {
AppLoadingState::Ready { reader } => reader.get_line(idx),
AppLoadingState::Loading { reader, .. } => reader.get_line(idx),
_ => None,
}
}
#[allow(dead_code)]
pub fn file_name(&self) -> Option<&str> {
self.file_path
.as_ref()
.and_then(|p| std::path::Path::new(p).file_name().and_then(|n| n.to_str()))
}
pub fn total_lines(&self) -> usize {
match &self.loading_state {
AppLoadingState::Ready { reader } => reader.line_count(),
AppLoadingState::Loading {
reader,
estimated_lines,
..
} => {
// Use estimated total lines (not sampled_line_count) so the user can
// scroll freely during indexing. get_line() incrementally scans
// forward on demand, so lines beyond the initial 64KB are still
// accessible. The .max() guards against under-estimates.
(*estimated_lines as usize).max(reader.sampled_line_count())
}
_ => 0,
}
}
pub fn is_loaded(&self) -> bool {
matches!(
self.loading_state,
AppLoadingState::Ready { .. } | AppLoadingState::Loading { .. }
)
}
pub fn is_loading(&self) -> bool {
matches!(self.loading_state, AppLoadingState::Loading { .. })
}
pub fn is_error(&self) -> bool {
matches!(self.loading_state, AppLoadingState::Error(_))
}
pub(super) fn get_visual_height_index(&self) -> Option<&VisualHeightIndex> {
match &self.loading_state {
AppLoadingState::Ready { reader } => match &reader.state {
log_viewer_core::io::progressive_reader::ReaderState::Ready {
visual_height_index,
..
} => visual_height_index.as_ref(),
_ => None,
},
_ => None,
}
}
pub fn error_message(&self) -> Option<&str> {
match &self.loading_state {
AppLoadingState::Error(msg) => Some(msg),
_ => None,
}
}
pub fn loading_progress(&self) -> Option<f64> {
match &self.loading_state {
AppLoadingState::Loading {
progress_percent, ..
} => Some(*progress_percent),
_ => None,
}
}
pub fn estimated_lines(&self) -> Option<u64> {
match &self.loading_state {
AppLoadingState::Loading {
estimated_lines, ..
} => Some(*estimated_lines),
_ => None,
}
}
pub(crate) fn mode(&self) -> AppMode {
self.mode
}
pub(crate) fn settings_error(&self) -> Option<&str> {
self.settings_error.as_deref()
}
pub(crate) fn settings_draft(&self) -> &ColorConfig {
&self.settings_draft
}
pub(crate) fn settings_cursor(&self) -> usize {
self.settings_cursor
}
pub(crate) fn color_config(&self) -> &ColorConfig {
&self.color_config
}
pub(crate) fn cursor_line(&self) -> usize {
self.cursor_line
}
pub(crate) fn cursor_sub_offset(&self) -> usize {
self.cursor_sub_offset
}
#[allow(dead_code)]
pub(crate) fn content_width(&self) -> u16 {
self.content_width
}
#[allow(dead_code)]
pub(crate) fn content_height(&self) -> u16 {
self.content_height
}
pub(crate) fn set_content_area(&mut self, width: u16, height: u16) {
self.content_width = width;
self.content_height = height;
}
pub(crate) fn viewport_rows(
&self,
start_logical: usize,
offset_in_line: usize,
available_rows: usize,
) -> Vec<ViewportRenderRow<'_>> {
let mut rows = Vec::new();
for (entry_idx, entry) in self.viewport_cache.entries.iter().enumerate() {
let logical_line = self.viewport_cache.logical_start + entry_idx;
let start_row = if logical_line == start_logical {
offset_in_line
} else {
0
};
for (visual_row, text) in entry.wrapped_rows.iter().enumerate().skip(start_row) {
if rows.len() >= available_rows {
return rows;
}
rows.push(ViewportRenderRow {
logical_line,
visual_row,
text,
level: entry.level.as_ref(),
});
}
}
rows
}
#[cfg(test)]
pub(crate) fn enter_settings_mode_for_test(&mut self) {
self.mode = AppMode::Settings;
self.settings_draft = self.color_config.clone();
}
#[cfg(test)]
pub(crate) fn set_cursor_for_test(&mut self, line: usize, sub_offset: usize) {
self.cursor_line = line;
self.cursor_sub_offset = sub_offset;
}
#[cfg(test)]
pub fn set_error_state(&mut self, msg: impl Into<String>) {
self.loading_state = AppLoadingState::Error(msg.into());
}
#[allow(dead_code)]
pub fn file_size(&self) -> u64 {
match &self.loading_state {
AppLoadingState::Ready { reader } => reader.reader().map_or(0, |r| r.file_size()),
AppLoadingState::Loading { reader, .. } => reader.reader().map_or(0, |r| r.file_size()),
_ => 0,
}
}
/// MUST match the renderer's `gutter_width` formula in `ui.rs::render_content`.
pub(crate) fn gutter_width(&self) -> usize {
gutter_width_for(self.total_lines(), self.is_loading())
}
pub(super) fn get_content_width(&self) -> usize {
let total = if self.content_width > 0 {
self.content_width as usize
} else {
80
};
total.saturating_sub(gutter_width_for(self.total_lines(), self.is_loading()))
}
}
+359
View File
@@ -0,0 +1,359 @@
use super::App;
impl App {
fn line_visual_height(&self, line: usize, width: usize) -> usize {
if line >= self.total_lines() {
return 1;
}
self.compute_visual_height(line, width).max(1)
}
fn advance_visual_pos(
&self,
mut line: usize,
mut sub: usize,
mut n: usize,
width: usize,
) -> (usize, usize) {
let last = self.total_lines().saturating_sub(1);
while n > 0 && line < last {
let h = self.line_visual_height(line, width);
let remaining_in_line = h.saturating_sub(sub);
if n < remaining_in_line {
sub += n;
return (line, sub);
}
n -= remaining_in_line;
line += 1;
sub = 0;
}
if line >= last {
let h = self.line_visual_height(last, width);
sub = (sub + n).min(h.saturating_sub(1));
line = last;
}
(line, sub)
}
fn retreat_visual_pos(
&self,
mut line: usize,
mut sub: usize,
mut n: usize,
width: usize,
) -> (usize, usize) {
while n > 0 {
if sub >= n {
sub -= n;
return (line, sub);
}
n -= sub + 1;
if line == 0 {
return (0, 0);
}
line -= 1;
sub = self.line_visual_height(line, width).saturating_sub(1);
}
(line, sub)
}
pub(super) fn clamp_cursor_sub_offset_no_vhi(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
let width = self.get_content_width();
if width == 0 {
return;
}
let h = self.line_visual_height(self.cursor_line, width);
if self.cursor_sub_offset >= h {
self.cursor_sub_offset = h.saturating_sub(1);
}
}
pub(super) fn ensure_cursor_visible_no_vhi(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 || self.content_height == 0 {
return;
}
let width = self.get_content_width();
if width == 0 {
return;
}
let content_h = self.content_height as usize;
let cur_line = self.cursor_line;
let cur_sub = self.cursor_sub_offset;
let v_top_line = self.v_offset;
let v_top_sub = self.v_sub_offset;
if cur_line < v_top_line || (cur_line == v_top_line && cur_sub < v_top_sub) {
self.v_offset = cur_line;
self.v_sub_offset = cur_sub;
return;
}
let mut line = v_top_line;
let mut sub = v_top_sub;
let mut walked = 0usize;
let total = self.total_lines();
while walked < content_h {
if line == cur_line && sub == cur_sub {
return;
}
let h = self.line_visual_height(line, width);
if sub + 1 < h {
sub += 1;
} else if line + 1 < total {
line += 1;
sub = 0;
} else {
return;
}
walked += 1;
}
let target_distance = content_h.saturating_sub(1);
let (new_line, new_sub) =
self.retreat_visual_pos(cur_line, cur_sub, target_distance, width);
self.v_offset = new_line;
self.v_sub_offset = new_sub;
}
pub fn scroll_down_line(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
if let Some(index) = self.get_visual_height_index() {
let last = self.total_lines().saturating_sub(1);
let current_height = index.visual_height_of_line(self.cursor_line);
if self.cursor_sub_offset + 1 < current_height {
self.cursor_sub_offset += 1;
} else if self.cursor_line < last {
self.cursor_line += 1;
self.cursor_sub_offset = 0;
}
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 && self.total_lines() > 0 {
let (new_line, new_sub) =
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, 1, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_up_line(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
if let Some(index) = self.get_visual_height_index() {
if self.cursor_sub_offset > 0 {
self.cursor_sub_offset -= 1;
} else if self.cursor_line > 0 {
let previous_line = self.cursor_line - 1;
let previous_line_last_sub =
index.visual_height_of_line(previous_line).saturating_sub(1);
self.cursor_line = previous_line;
self.cursor_sub_offset = previous_line_last_sub;
}
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 {
let (new_line, new_sub) =
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, 1, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_down_half_page(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
let half = self.content_height as usize / 2;
if let Some(index) = self.get_visual_height_index() {
let total_vr = index.total_visual_rows();
if total_vr == 0 {
return;
}
let cur_vr = self.cursor_visual_row();
let target_vr = cur_vr.saturating_add(half as u64).min(total_vr - 1);
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 && half > 0 {
let (new_line, new_sub) =
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, half, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_up_half_page(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
let half = self.content_height as usize / 2;
if let Some(index) = self.get_visual_height_index() {
let total_vr = index.total_visual_rows();
if total_vr == 0 {
return;
}
let cur_vr = self.cursor_visual_row();
let target_vr = cur_vr.saturating_sub(half as u64);
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 && half > 0 {
let (new_line, new_sub) =
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, half, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_down_page(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
let page = self.content_height as usize;
if let Some(index) = self.get_visual_height_index() {
let total_vr = index.total_visual_rows();
if total_vr == 0 {
return;
}
let cur_vr = self.cursor_visual_row();
let target_vr = cur_vr.saturating_add(page as u64).min(total_vr - 1);
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 && page > 0 {
let (new_line, new_sub) =
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, page, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_up_page(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
let page = self.content_height as usize;
if let Some(index) = self.get_visual_height_index() {
let total_vr = index.total_visual_rows();
if total_vr == 0 {
return;
}
let cur_vr = self.cursor_visual_row();
let target_vr = cur_vr.saturating_sub(page as u64);
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible();
} else {
let width = self.get_content_width();
if width > 0 && page > 0 {
let (new_line, new_sub) =
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, page, width);
self.cursor_line = new_line;
self.cursor_sub_offset = new_sub;
self.ensure_cursor_visible_no_vhi();
}
}
}
pub fn scroll_to_top(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
self.cursor_line = 0;
self.v_offset = 0;
self.v_sub_offset = 0;
self.cursor_sub_offset = 0;
}
pub fn scroll_to_bottom(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 {
return;
}
self.cursor_line = self.total_lines().saturating_sub(1);
if let Some(idx) = self.get_visual_height_index() {
self.cursor_sub_offset = idx
.visual_height_of_line(self.cursor_line)
.saturating_sub(1);
self.v_sub_offset = 0;
self.ensure_cursor_visible();
self.clamp_v_offset();
} else {
let width = self.get_content_width();
self.cursor_sub_offset = if width > 0 {
self.line_visual_height(self.cursor_line, width)
.saturating_sub(1)
} else {
0
};
self.ensure_cursor_visible_no_vhi();
}
}
pub(super) fn ensure_cursor_visible(&mut self) {
if !self.is_loaded() || self.total_lines() == 0 || self.content_height == 0 {
return;
}
if self.is_loading() {
return;
}
let cursor_visual = self.cursor_visual_row() as usize;
let content_h = self.content_height as usize;
if cursor_visual < self.v_offset {
self.v_offset = cursor_visual;
} else if cursor_visual >= self.v_offset.saturating_add(content_h) {
self.v_offset = cursor_visual.saturating_sub(content_h).saturating_add(1);
}
self.clamp_v_offset();
}
pub(super) fn clamp_v_offset(&mut self) {
let max_offset = self
.total_visual_rows()
.saturating_sub(self.content_height as usize);
self.v_offset = self.v_offset.min(max_offset);
}
fn cursor_visual_row(&self) -> u64 {
if let Some(index) = self.get_visual_height_index() {
let first = index.cursor_to_first_visual_row(self.cursor_line);
let height = index.visual_height_of_line(self.cursor_line);
let max_sub = height.saturating_sub(1);
let sub = self.cursor_sub_offset.min(max_sub) as u64;
first + sub
} else {
self.cursor_line as u64
}
}
}
+98
View File
@@ -0,0 +1,98 @@
use crossterm::event::{KeyCode, KeyEvent};
use crate::color::AVAILABLE_COLORS;
use super::{App, AppMode};
impl App {
pub(super) fn handle_settings_key(&mut self, key: KeyEvent) {
match key.code {
KeyCode::Esc | KeyCode::Char('q') => {
self.settings_error = None;
self.mode = AppMode::Normal;
}
KeyCode::Enter => {
let draft = self.settings_draft.clone();
match draft.save() {
Ok(()) => {
self.color_config = draft;
self.settings_error = None;
self.mode = AppMode::Normal;
}
Err(e) => {
self.settings_error = Some(format!("Failed to save settings: {e}"));
}
}
}
KeyCode::Char('j') | KeyCode::Down => {
if self.settings_cursor < 5 {
self.settings_cursor += 1;
}
}
KeyCode::Char('k') | KeyCode::Up => {
self.settings_cursor = self.settings_cursor.saturating_sub(1);
}
KeyCode::Left => {
self.cycle_color(self.settings_cursor, false);
}
KeyCode::Right => {
self.cycle_color(self.settings_cursor, true);
}
KeyCode::Char(c) if ('1'..='8').contains(&c) => {
let idx = (c as usize) - ('1' as usize);
self.set_color(self.settings_cursor, AVAILABLE_COLORS[idx]);
}
_ => {}
}
}
fn cycle_color(&mut self, level_idx: usize, forward: bool) {
let current = self.get_settings_color(level_idx).to_string();
let colors = AVAILABLE_COLORS;
let pos = colors.iter().position(|&c| c == current);
let new_pos = match pos {
Some(p) => {
if forward {
(p + 1) % colors.len()
} else if p == 0 {
colors.len() - 1
} else {
p - 1
}
}
None => {
if forward {
0
} else {
colors.len() - 1
}
}
};
self.set_color(level_idx, colors[new_pos]);
}
fn get_settings_color(&self, level_idx: usize) -> &str {
match level_idx {
0 => &self.settings_draft.error,
1 => &self.settings_draft.warn,
2 => &self.settings_draft.info,
3 => &self.settings_draft.debug,
4 => &self.settings_draft.trace,
5 => &self.settings_draft.unknown,
_ => "white",
}
}
fn set_color(&mut self, level_idx: usize, color_name: &str) {
self.settings_error = None;
match level_idx {
0 => self.settings_draft.error = color_name.to_string(),
1 => self.settings_draft.warn = color_name.to_string(),
2 => self.settings_draft.info = color_name.to_string(),
3 => self.settings_draft.debug = color_name.to_string(),
4 => self.settings_draft.trace = color_name.to_string(),
5 => self.settings_draft.unknown = color_name.to_string(),
_ => {}
}
}
}
+222
View File
@@ -0,0 +1,222 @@
use std::borrow::Cow;
use log_viewer_core::io::wrap::{MAX_WRAP_INPUT_LEN, format_json_line, wrap_line_chars};
use super::viewport_cache::{ViewportEntry, truncate_to_columns};
use super::{App, AppLoadingState};
impl App {
/// Compute a single line's viewport entry (wrapped rows + level + height).
pub(super) fn compute_line_entry(&self, line: usize, width: usize) -> ViewportEntry {
let raw = self.get_line(line).unwrap_or_default();
// Guard 1: oversized raw input — skip detect_level and JSON formatting
// to avoid O(n) parsing overhead on huge lines.
if raw.len() > MAX_WRAP_INPUT_LEN {
return ViewportEntry {
wrapped_rows: vec![truncate_to_columns(&raw, width)],
level: None,
visual_height: 1,
};
}
let level = log_viewer_core::parser::level::detect_level(&raw);
let display_text: Cow<'_, str> = if self.json_format {
format_json_line(&raw)
} else {
Cow::Borrowed(raw.as_str())
};
// Guard 2: JSON pretty-printing may expand a line beyond the limit.
if display_text.len() > MAX_WRAP_INPUT_LEN {
return ViewportEntry {
wrapped_rows: vec![truncate_to_columns(&display_text, width)],
level,
visual_height: 1,
};
}
let mut wrapped = Vec::new();
for sub_line in display_text.split('\n') {
wrapped.extend(wrap_line_chars(sub_line, width));
}
let visual_height = wrapped.len().max(1);
ViewportEntry {
wrapped_rows: wrapped,
level,
visual_height,
}
}
/// Compute visual height for a single line without storing it.
pub(super) fn compute_visual_height(&self, line: usize, width: usize) -> usize {
let raw = self.get_line(line).unwrap_or_default();
// Guard 1: oversized raw input.
if raw.len() > MAX_WRAP_INPUT_LEN {
return 1;
}
let display_text: Cow<'_, str> = if self.json_format {
format_json_line(&raw)
} else {
Cow::Borrowed(raw.as_str())
};
if display_text.len() > MAX_WRAP_INPUT_LEN {
return 1;
}
let mut height = 0;
for sub_line in display_text.split('\n') {
height += wrap_line_chars(sub_line, width).len();
}
height.max(1)
}
/// Find (logical_line, offset_in_line) for a given visual row offset.
fn find_logical_line_at_visual_row(&self, visual_row: usize, _width: usize) -> (usize, usize) {
if let Some(index) = self.get_visual_height_index() {
return index.visual_row_to_logical_row_with_offset(visual_row as u64);
}
(visual_row.min(self.total_lines().saturating_sub(1)), 0)
}
/// Ensure the viewport cache covers the visible range.
/// Returns (start_logical, offset_in_line) for rendering.
pub(crate) fn ensure_viewport_cache(&mut self, width: usize) -> (usize, usize) {
let viewport_height = self.content_height as usize;
if !self.is_loaded() || width == 0 || viewport_height == 0 {
return (0, 0);
}
let params_changed = self.viewport_cache.needs_recompute(width, self.json_format);
if params_changed {
self.viewport_cache.invalidate();
self.viewport_cache.width = width;
self.viewport_cache.set_json_format(self.json_format);
self.ensure_visual_height_index(width);
if self.get_visual_height_index().is_some() {
self.ensure_cursor_visible();
} else {
self.ensure_cursor_visible_no_vhi();
}
}
// Find start logical line from v_offset
let (start_logical, offset_in_line) = if self.is_loading() {
(
self.v_offset.min(self.total_lines().saturating_sub(1)),
self.v_sub_offset,
)
} else {
self.find_logical_line_at_visual_row(self.v_offset, width)
};
// Compute viewport entries
self.viewport_cache.entries.clear();
self.viewport_cache.logical_start = start_logical;
let total = self.total_lines();
let mut rows_remaining = viewport_height + offset_in_line;
for line_idx in start_logical..total {
if rows_remaining == 0 {
break;
}
let entry = self.compute_line_entry(line_idx, width);
rows_remaining = rows_remaining.saturating_sub(entry.visual_height);
self.viewport_cache.entries.push(entry);
}
(start_logical, offset_in_line)
}
/// Compute total visual rows (cached, lazily evaluated).
/// Returns `total_lines` for sampling mode (1:1 mapping).
pub(super) fn total_visual_rows(&mut self) -> usize {
if self.is_loading() {
return self.total_lines();
}
if let Some(index) = self.get_visual_height_index() {
return index.total_visual_rows() as usize;
}
self.total_lines()
}
pub(super) fn cursor_to_first_visual_row(&self, line: usize) -> usize {
if self.is_loading() {
return line;
}
if let Some(index) = self.get_visual_height_index() {
return index.cursor_to_first_visual_row(line) as usize;
}
line
}
pub(super) fn visual_row_to_logical_row(&self, visual_row: usize) -> usize {
if self.is_loading() {
return visual_row.min(self.total_lines().saturating_sub(1));
}
if let Some(index) = self.get_visual_height_index() {
return index.visual_row_to_logical_row(visual_row as u64);
}
visual_row.min(self.total_lines().saturating_sub(1))
}
/// Compute the rebased offset pair (logical_line, sub_row) from the
/// current visual-row `v_offset`. Returns `(v_offset, v_sub_offset)`
/// suitable for the no-VHI fallback scrolling path.
///
/// MUST be called before borrowing `&mut self.loading_state` for
/// `invalidate_visual_height_index`, because it reads VHI through
/// `&self`.
pub(super) fn rebase_offset_for_invalidate(&self) -> (usize, usize) {
if self.get_visual_height_index().is_some() {
let top_visual = self.v_offset;
let top_line = self.visual_row_to_logical_row(top_visual);
let line_first_visual = self.cursor_to_first_visual_row(top_line);
let sub = top_visual.saturating_sub(line_first_visual);
(top_line, sub)
} else {
(self.v_offset, self.v_sub_offset)
}
}
pub(super) fn ensure_visual_height_index(&mut self, width: usize) {
let needs_rebuild = match self.get_visual_height_index() {
Some(idx) => !idx.is_valid_for(self.json_format, width),
None => true,
};
if needs_rebuild {
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
reader.invalidate_visual_height_index();
reader.start_visual_height_rebuild(width, self.json_format);
}
self.v_offset = new_offset;
self.v_sub_offset = new_sub;
self.clamp_cursor_sub_offset_no_vhi();
}
}
pub(super) fn toggle_json_format(&mut self) {
self.json_format = !self.json_format;
self.viewport_cache.invalidate();
let width = self.viewport_cache.width;
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
reader.invalidate_visual_height_index();
if width > 0 {
reader.start_visual_height_rebuild(width, self.json_format);
}
}
self.v_offset = new_offset;
self.v_sub_offset = new_sub;
self.clamp_cursor_sub_offset_no_vhi();
}
}
+99
View File
@@ -0,0 +1,99 @@
use log_viewer_core::types::LogLevel;
use unicode_width::UnicodeWidthChar;
pub(super) struct ViewportEntry {
pub(super) wrapped_rows: Vec<String>,
pub(super) level: Option<LogLevel>,
pub(super) visual_height: usize,
}
pub(super) struct ViewportCache {
pub(super) entries: Vec<ViewportEntry>,
pub(super) logical_start: usize,
pub(super) width: usize,
json_format: bool,
pub(super) cached_total_visual_rows: Option<usize>,
}
impl ViewportCache {
pub(super) fn new() -> Self {
Self {
entries: Vec::new(),
logical_start: 0,
width: 0,
json_format: false,
cached_total_visual_rows: None,
}
}
pub(super) fn invalidate(&mut self) {
self.entries.clear();
self.logical_start = 0;
self.width = 0;
self.cached_total_visual_rows = None;
}
pub(super) fn needs_recompute(&self, width: usize, json_format: bool) -> bool {
self.width != width || self.json_format != json_format
}
pub(super) fn set_json_format(&mut self, json_format: bool) {
self.json_format = json_format;
}
#[allow(dead_code)]
pub(super) fn get_entry(&self, logical_line: usize) -> Option<&ViewportEntry> {
if logical_line >= self.logical_start {
let idx = logical_line - self.logical_start;
self.entries.get(idx)
} else {
None
}
}
}
pub(super) const TRUNCATE_TAB_WIDTH: usize = 4;
pub(super) fn gutter_width_for(total_lines: usize, is_loading: bool) -> usize {
if total_lines == 0 {
return 0;
}
let line_num_width = total_lines.to_string().len();
let loading_extra = if is_loading { 1 } else { 0 };
line_num_width + loading_extra + 1 + 1
}
pub(super) fn truncate_to_columns(s: &str, max_cols: usize) -> String {
if max_cols == 0 || s.is_empty() {
return String::new();
}
let mut out = String::new();
let mut col = 0;
for ch in s.chars() {
if ch == '\t' {
let tab_stop = TRUNCATE_TAB_WIDTH - (col % TRUNCATE_TAB_WIDTH);
if col + tab_stop > max_cols {
break;
}
for _ in 0..tab_stop {
out.push(' ');
}
col += tab_stop;
} else {
let w = if ch.is_control() {
0
} else {
ch.width().unwrap_or(0)
};
if col + w > max_cols {
break;
}
out.push(ch);
col += w;
}
}
out
}
+162
View File
@@ -0,0 +1,162 @@
use log_viewer_core::io::file_reader::AppendStatus;
use log_viewer_core::io::progressive_reader::{ReaderState, compute_line_visual_height};
use log_viewer_core::watcher::file_watcher::FileEvent;
use super::viewport_cache::gutter_width_for;
use super::{App, AppLoadingState};
impl App {
pub fn poll_file_watcher(&mut self) {
let events: Vec<FileEvent> = match &mut self.file_watcher {
Some(w) => std::iter::from_fn(|| w.try_recv()).collect(),
None => return,
};
for event in events {
match event {
FileEvent::Appended { new_size: _ } => {
self.handle_file_appended();
}
FileEvent::Truncated { new_size: _ } => {
self.handle_file_truncated();
}
FileEvent::Rotated { new_inode: _ } => {
// Don't auto-switch; old content preserved.
// User can reload manually if desired.
}
FileEvent::Removed => {
self.loading_state = AppLoadingState::Error("File has been deleted".into());
}
FileEvent::WatcherError { message: _ } => {}
}
}
}
pub(super) fn handle_file_appended(&mut self) {
let rebased = self.rebase_offset_for_invalidate();
match &mut self.loading_state {
AppLoadingState::Ready { reader } => {
let old_reader_line_count = reader.line_count();
let status = reader.update_for_append();
let width = {
let total = if self.content_width > 0 {
self.content_width as usize
} else {
80
};
total.saturating_sub(gutter_width_for(reader.line_count(), false))
};
match status {
Ok(AppendStatus::Appended(_new_lines)) => {
let _ = reader.save_cache();
let (old_line_count, can_extend) = {
match &reader.state {
ReaderState::Ready {
visual_height_index: Some(idx),
..
} => (idx.line_count(), idx.is_valid_for(self.json_format, width)),
_ => (0, false),
}
};
let new_line_count = reader.line_count();
if can_extend && old_line_count == old_reader_line_count {
if let ReaderState::Ready {
visual_height_index: Some(index),
reader: fr,
} = &mut reader.state
{
if old_line_count > 0 {
let last_old_line_text =
fr.get_line(old_line_count - 1).unwrap_or("");
let new_h = compute_line_visual_height(
last_old_line_text,
width,
self.json_format,
);
index.replace_last_line_height(new_h);
}
let mut new_heights = Vec::with_capacity(
new_line_count.saturating_sub(old_line_count),
);
for i in old_line_count..new_line_count {
let line_text = fr.get_line(i).unwrap_or("");
new_heights.push(compute_line_visual_height(
line_text,
width,
self.json_format,
));
}
index.extend_from_heights(&new_heights);
}
} else {
let (new_offset, new_sub) = rebased;
self.v_offset = new_offset;
self.v_sub_offset = new_sub;
self.cursor_sub_offset = 0;
reader.invalidate_visual_height_index();
reader.start_visual_height_rebuild(width, self.json_format);
}
self.viewport_cache.invalidate();
}
Ok(AppendStatus::Reloaded) => {
let _ = reader.save_cache();
let (new_offset, _new_sub) = rebased;
reader.invalidate_visual_height_index();
reader.start_visual_height_rebuild(width, self.json_format);
self.cursor_line =
self.cursor_line.min(self.total_lines().saturating_sub(1));
self.v_offset = new_offset;
self.v_sub_offset = 0;
self.cursor_sub_offset = 0;
self.viewport_cache.invalidate();
self.clamp_v_offset();
}
Ok(AppendStatus::Unchanged) | Err(_) => {}
}
}
AppLoadingState::Loading { .. } => {
self.reload_after_loading = true;
}
_ => {}
}
}
pub(super) fn reload_ready_reader(&mut self) {
let (new_offset, _new_sub) = self.rebase_offset_for_invalidate();
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
let _ = reader.reload();
let width = {
let total = if self.content_width > 0 {
self.content_width as usize
} else {
80
};
total.saturating_sub(gutter_width_for(reader.line_count(), false))
};
let _ = reader.save_cache();
reader.invalidate_visual_height_index();
reader.start_visual_height_rebuild(width, self.json_format);
self.cursor_line = self.cursor_line.min(self.total_lines().saturating_sub(1));
self.v_offset = new_offset;
self.v_sub_offset = 0;
self.clamp_cursor_sub_offset_no_vhi();
self.viewport_cache.invalidate();
self.clamp_v_offset();
}
}
pub(super) fn handle_file_truncated(&mut self) {
match &mut self.loading_state {
AppLoadingState::Ready { .. } => {
self.reload_ready_reader();
}
AppLoadingState::Loading { .. } => {
self.reload_after_loading = true;
}
_ => {}
}
}
}
+73 -21
View File
@@ -1,3 +1,6 @@
use std::time::Duration;
use anyhow::Context;
use clap::Parser; use clap::Parser;
mod app; mod app;
@@ -11,44 +14,93 @@ struct Cli {
files: Vec<String>, files: Vec<String>,
} }
// ── RAII terminal guard ────────────────────────────────────────────
//
// Holds the ratatui Terminal and guarantees terminal state restoration
// (disable raw mode, leave alternate screen, show cursor) on **every**
// exit path: normal return, `?` error propagation, and panic unwind.
//
// IMPORTANT: `Drop` does **not** run on `std::process::exit` or
// `panic = "abort"`. Therefore we must never call `process::exit`
// inside the guarded scope — use `?` to return `Err` instead.
type Backend = ratatui::backend::CrosstermBackend<std::io::Stdout>;
struct TerminalGuard {
terminal: ratatui::Terminal<Backend>,
}
impl TerminalGuard {
/// Enable raw mode, enter alternate screen, and create the terminal.
/// On partial failure, rolls back any steps that already succeeded.
fn enter() -> anyhow::Result<Self> {
crossterm::terminal::enable_raw_mode()
.map_err(|e| anyhow::anyhow!("enable_raw_mode: {e}"))?;
let mut stdout = std::io::stdout();
if let Err(e) = crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen) {
let _ = crossterm::terminal::disable_raw_mode();
return Err(anyhow::anyhow!("EnterAlternateScreen: {e}"));
}
let backend = ratatui::backend::CrosstermBackend::new(stdout);
match ratatui::Terminal::new(backend) {
Ok(terminal) => Ok(Self { terminal }),
Err(e) => {
// Roll back: leave alternate screen + disable raw mode
let _ = crossterm::execute!(
std::io::stdout(),
crossterm::terminal::LeaveAlternateScreen
);
let _ = crossterm::terminal::disable_raw_mode();
Err(anyhow::anyhow!("Terminal::new: {e}"))
}
}
}
fn terminal(&mut self) -> &mut ratatui::Terminal<Backend> {
&mut self.terminal
}
}
impl Drop for TerminalGuard {
fn drop(&mut self) {
// Best-effort cleanup; suppress errors (Drop must not panic).
let _ = crossterm::terminal::disable_raw_mode();
let _ = crossterm::execute!(
self.terminal.backend_mut(),
crossterm::terminal::LeaveAlternateScreen
);
let _ = self.terminal.show_cursor();
}
}
// ── main ───────────────────────────────────────────────────────────
fn main() -> anyhow::Result<()> { fn main() -> anyhow::Result<()> {
let cli = Cli::parse(); let cli = Cli::parse();
crossterm::terminal::enable_raw_mode()?; let mut guard = TerminalGuard::enter()?;
let mut stdout = std::io::stdout();
crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen)?;
let backend = ratatui::backend::CrosstermBackend::new(stdout);
let mut terminal = ratatui::Terminal::new(backend)?;
let mut app = app::App::new(); let mut app = app::App::new();
app.color_config = log_viewer_core::config::ColorConfig::load(); app.color_config = log_viewer_core::config::ColorConfig::load();
if let Some(file) = cli.files.first() if let Some(file) = cli.files.first() {
&& let Err(e) = app.load_file(file) app.load_file(file)
{ .with_context(|| format!("loading file {file}"))?;
eprintln!("Error loading file: {e}");
std::process::exit(1);
} }
while !app.should_quit { while !app.should_quit {
app.poll_background_indexer(); app.poll_background_indexer();
app.poll_file_watcher(); app.poll_file_watcher();
terminal.draw(|frame| ui::render(frame, &mut app))?; guard.terminal().draw(|frame| ui::render(frame, &mut app))?;
if crossterm::event::poll(std::time::Duration::from_millis(100))? { if crossterm::event::poll(Duration::from_millis(100))? {
match crossterm::event::read()? { match crossterm::event::read()? {
crossterm::event::Event::Key(key) => app.handle_key(key), crossterm::event::Event::Key(key) => app.handle_key(key),
crossterm::event::Event::Resize(_w, _h) => {} crossterm::event::Event::Resize(_, _) => {}
_ => {} _ => {}
} }
} }
} }
crossterm::terminal::disable_raw_mode()?;
crossterm::execute!(
terminal.backend_mut(),
crossterm::terminal::LeaveAlternateScreen
)?;
terminal.show_cursor()?;
Ok(()) Ok(())
} }
+176 -84
View File
@@ -6,6 +6,7 @@ use crate::color::level_fg;
use log_viewer_core::config::ColorConfig; use log_viewer_core::config::ColorConfig;
use log_viewer_core::types::LogLevel; use log_viewer_core::types::LogLevel;
#[allow(dead_code)]
pub(crate) fn build_line_spans( pub(crate) fn build_line_spans(
gutter_text: String, gutter_text: String,
content_text: String, content_text: String,
@@ -28,6 +29,10 @@ pub(crate) fn build_line_spans(
]) ])
} }
fn is_cursor_visual_row(app: &App, logical_line: usize, visual_row: usize) -> bool {
logical_line == app.cursor_line() && visual_row == app.cursor_sub_offset()
}
pub fn render(frame: &mut ratatui::Frame, app: &mut App) { pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
use ratatui::layout::{Constraint, Layout}; use ratatui::layout::{Constraint, Layout};
use ratatui::widgets::Paragraph; use ratatui::widgets::Paragraph;
@@ -40,7 +45,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
.split(frame.area()); .split(frame.area());
// ── Title bar ────────────────────────────────────────────────── // ── Title bar ──────────────────────────────────────────────────
let title_text = if app.mode == AppMode::Settings { let title_text = if app.mode() == AppMode::Settings {
" Color Settings".to_string() " Color Settings".to_string()
} else if app.is_loading() { } else if app.is_loading() {
let name = app.file_name().unwrap_or("unknown"); let name = app.file_name().unwrap_or("unknown");
@@ -51,7 +56,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
let cursor_display = if app.total_lines() == 0 { let cursor_display = if app.total_lines() == 0 {
0 0
} else { } else {
app.cursor_line + 1 app.cursor_line() + 1
}; };
format!(" {} [{}/{}]", name, cursor_display, app.total_lines()) format!(" {} [{}/{}]", name, cursor_display, app.total_lines())
} else { } else {
@@ -63,7 +68,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
); );
// ── Content area ─────────────────────────────────────────────── // ── Content area ───────────────────────────────────────────────
if app.mode == AppMode::Settings { if app.mode() == AppMode::Settings {
render_settings(frame, app, outer[1]); render_settings(frame, app, outer[1]);
} else if app.is_error() { } else if app.is_error() {
let msg = app.error_message().unwrap_or_default(); let msg = app.error_message().unwrap_or_default();
@@ -88,9 +93,21 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
} }
// ── Status bar ───────────────────────────────────────────────── // ── Status bar ─────────────────────────────────────────────────
let status_text = if app.mode == AppMode::Settings { if app.mode() == AppMode::Settings {
" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel" if let Some(err) = app.settings_error() {
} else if app.is_error() { frame.render_widget(
Paragraph::new(err).style(Style::default().fg(Color::Red)),
outer[2],
);
} else {
frame.render_widget(
Paragraph::new(" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"),
outer[2],
);
}
return;
}
let status_text = if app.is_error() {
" Press q to quit" " Press q to quit"
} else if app.is_loading() { } else if app.is_loading() {
let pct = app.loading_progress().map_or(0, |p| p as usize); let pct = app.loading_progress().map_or(0, |p| p as usize);
@@ -110,8 +127,11 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
} else if app.is_loaded() { } else if app.is_loaded() {
let name = app.file_name().unwrap_or("unknown"); let name = app.file_name().unwrap_or("unknown");
let total = app.total_lines(); let total = app.total_lines();
let cursor_display = if total == 0 { 0 } else { app.cursor_line + 1 }; let cursor_display = if total == 0 { 0 } else { app.cursor_line() + 1 };
let status = format!(" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit", name, cursor_display, total); let status = format!(
" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit",
name, cursor_display, total
);
frame.render_widget(Paragraph::new(status), outer[2]); frame.render_widget(Paragraph::new(status), outer[2]);
return; return;
} else { } else {
@@ -130,26 +150,31 @@ pub fn render_settings(frame: &mut ratatui::Frame, app: &mut App, area: ratatui:
let popup_w = ((area.width as u32 * 4 / 5).max(40)).min(area.width as u32) as u16; let popup_w = ((area.width as u32 * 4 / 5).max(40)).min(area.width as u32) as u16;
let popup_h = ((area.height as u32 * 4 / 5).max(14)).min(area.height as u32) as u16; let popup_h = ((area.height as u32 * 4 / 5).max(14)).min(area.height as u32) as u16;
let popup_x = area.width.saturating_sub(popup_w) / 2; let popup_x = area
let popup_y = area.height.saturating_sub(popup_h) / 2; .x
.saturating_add(area.width.saturating_sub(popup_w) / 2);
let popup_y = area
.y
.saturating_add(area.height.saturating_sub(popup_h) / 2);
let popup = ratatui::layout::Rect::new(popup_x, popup_y, popup_w, popup_h); let popup = ratatui::layout::Rect::new(popup_x, popup_y, popup_w, popup_h);
let block = Block::new().borders(Borders::ALL).title(" Color Settings "); let block = Block::new().borders(Borders::ALL).title(" Color Settings ");
let inner = block.inner(popup); let inner = block.inner(popup);
frame.render_widget(block, popup); frame.render_widget(block, popup);
let settings_draft = app.settings_draft();
let levels = [ let levels = [
("ERROR", &app.settings_draft.error), ("ERROR", &settings_draft.error),
("WARN", &app.settings_draft.warn), ("WARN", &settings_draft.warn),
("INFO", &app.settings_draft.info), ("INFO", &settings_draft.info),
("DEBUG", &app.settings_draft.debug), ("DEBUG", &settings_draft.debug),
("TRACE", &app.settings_draft.trace), ("TRACE", &settings_draft.trace),
("UNKNOWN", &app.settings_draft.unknown), ("UNKNOWN", &settings_draft.unknown),
]; ];
let mut lines = Vec::new(); let mut lines = Vec::new();
for (i, (level_name, color_name)) in levels.iter().enumerate() { for (i, (level_name, color_name)) in levels.iter().enumerate() {
let is_selected = i == app.settings_cursor; let is_selected = i == app.settings_cursor();
let cursor_marker = if is_selected { "" } else { " " }; let cursor_marker = if is_selected { "" } else { " " };
let preview_color = color_name.parse::<Color>().unwrap_or(Color::White); let preview_color = color_name.parse::<Color>().unwrap_or(Color::White);
@@ -200,8 +225,7 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
let content_width = area.width as usize; let content_width = area.width as usize;
let content_height = area.height as usize; let content_height = area.height as usize;
app.content_height = area.height; app.set_content_area(area.width, area.height);
app.content_width = area.width;
let total_lines = app.total_lines(); let total_lines = app.total_lines();
let line_num_width = if total_lines > 0 { let line_num_width = if total_lines > 0 {
@@ -209,14 +233,8 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
} else { } else {
0 0
}; };
let is_loading = app.is_loading(); let is_loading = app.is_loading();
let gutter_prefix_extra = if is_loading { 1 } else { 0 }; let gutter_width = app.gutter_width();
let gutter_width = if total_lines > 0 {
line_num_width + gutter_prefix_extra + 1 + 1
} else {
0
};
let actual_content_width = content_width.saturating_sub(gutter_width); let actual_content_width = content_width.saturating_sub(gutter_width);
@@ -227,7 +245,6 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
let (start_logical, offset_in_line) = app.ensure_viewport_cache(actual_content_width); let (start_logical, offset_in_line) = app.ensure_viewport_cache(actual_content_width);
let mut lines: Vec<Line> = Vec::new(); let mut lines: Vec<Line> = Vec::new();
let mut current_visual_offset: usize = 0;
let available_rows = content_height; let available_rows = content_height;
let gutter_style = if is_loading { let gutter_style = if is_loading {
@@ -238,66 +255,50 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
Style::default().fg(Color::DarkGray) Style::default().fg(Color::DarkGray)
}; };
for (entry_idx, entry) in app.viewport_cache.entries.iter().enumerate() { for row in app.viewport_rows(start_logical, offset_in_line, available_rows) {
let logical_line = app.viewport_cache.logical_start + entry_idx; let is_cursor = is_cursor_visual_row(app, row.logical_line, row.visual_row);
let start_row = if logical_line == start_logical { let level = row.level;
offset_in_line
let bg_color = if is_cursor {
Color::DarkGray
} else { } else {
0 Color::Reset
};
let level_fg = level_fg(level, app.color_config()).unwrap_or(Color::White);
let gutter_text = if row.visual_row == 0 {
if is_loading {
format!(
"~{:>width$} \u{2502}",
row.logical_line + 1,
width = line_num_width
)
} else {
format!(
"{:>width$} \u{2502}",
row.logical_line + 1,
width = line_num_width
)
}
} else if is_loading {
format!(" {:width$} \u{2502}", "", width = line_num_width)
} else {
format!("{:width$} \u{2502}", "", width = line_num_width)
}; };
for (visual_row, text) in entry.wrapped_rows.iter().enumerate().skip(start_row) { let effective_gutter_style = if is_cursor {
if current_visual_offset >= available_rows { gutter_style.bg(bg_color)
break; } else {
} gutter_style
};
let is_cursor = logical_line == app.cursor_line; lines.push(Line::from(vec![
let level = entry.level.as_ref(); Span::styled(gutter_text, effective_gutter_style),
Span::styled(
let bg_color = if is_cursor { row.text.to_string(),
Color::DarkGray Style::default().fg(level_fg).bg(bg_color),
} else { ),
Color::Reset ]));
};
let level_fg = level_fg(level, &app.color_config).unwrap_or(Color::White);
let gutter_text = if visual_row == 0 {
if is_loading {
format!(
"~{:>width$} \u{2502}",
logical_line + 1,
width = line_num_width
)
} else {
format!(
"{:>width$} \u{2502}",
logical_line + 1,
width = line_num_width
)
}
} else if is_loading {
format!(" {:width$} \u{2502}", "", width = line_num_width)
} else {
format!("{:width$} \u{2502}", "", width = line_num_width)
};
let effective_gutter_style = if is_cursor {
gutter_style.bg(bg_color)
} else {
gutter_style
};
lines.push(Line::from(vec![
Span::styled(gutter_text, effective_gutter_style),
Span::styled(text.clone(), Style::default().fg(level_fg).bg(bg_color)),
]));
current_visual_offset += 1;
}
if current_visual_offset >= available_rows {
break;
}
} }
while lines.len() < available_rows { while lines.len() < available_rows {
@@ -363,6 +364,16 @@ mod tests {
assert_eq!(line.spans[1].style.bg, Some(Color::Reset)); assert_eq!(line.spans[1].style.bg, Some(Color::Reset));
} }
#[test]
fn test_cursor_highlight_predicate_uses_cursor_sub_offset() {
let mut app = App::new();
app.set_cursor_for_test(4, 2);
assert!(!is_cursor_visual_row(&app, 4, 0));
assert!(!is_cursor_visual_row(&app, 3, 2));
assert!(is_cursor_visual_row(&app, 4, 2));
}
fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer { fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer {
let backend = ratatui::backend::TestBackend::new(width, height); let backend = ratatui::backend::TestBackend::new(width, height);
let mut terminal = ratatui::Terminal::new(backend).unwrap(); let mut terminal = ratatui::Terminal::new(backend).unwrap();
@@ -469,7 +480,8 @@ mod tests {
let result = std::panic::catch_unwind(|| { let result = std::panic::catch_unwind(|| {
let data = std::fs::read(&path).unwrap(); let data = std::fs::read(&path).unwrap();
let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data); let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data);
let _ = log_viewer_core::io::index_cache::IndexCache::save(&path, &index); let _ =
log_viewer_core::io::index_cache::IndexCache::save_with_hash(&path, &index, &data);
let mut app = App::new(); let mut app = App::new();
app.load_file(path.to_str().unwrap()).unwrap(); app.load_file(path.to_str().unwrap()).unwrap();
@@ -493,4 +505,84 @@ mod tests {
let _ = std::fs::remove_file(&path); let _ = std::fs::remove_file(&path);
assert!(result.is_ok()); assert!(result.is_ok());
} }
// ── Issue #31: Settings popup area offset tests ────────────────
/// Helper: enter settings mode and render to buffer.
fn render_settings_to_buffer(
app: &mut App,
width: u16,
height: u16,
) -> ratatui::buffer::Buffer {
app.enter_settings_mode_for_test();
render_to_buffer(app, width, height)
}
/// Find the top-left corner of the popup border by scanning for '┌'.
fn find_popup_top_left(
buf: &ratatui::buffer::Buffer,
width: u16,
height: u16,
) -> Option<(u16, u16)> {
for row in 0..height {
for col in 0..width {
if buf.cell((col, row)).unwrap().symbol() == "" {
return Some((col, row));
}
}
}
None
}
#[test]
fn test_settings_popup_includes_area_offset() {
// In an 80x24 frame with Layout [Length(1), Min(1), Length(1)]:
// outer[0] = title bar -> y=0
// outer[1] = content -> y=1, height=22
// outer[2] = status bar -> y=23
// The popup is centered within outer[1], so its y must be >= outer[1].y (which is 1).
let mut app = App::new();
let buf = render_settings_to_buffer(&mut app, 80, 24);
let (_px, py) =
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
// outer[1].y == 1; the popup is centered inside a 22-row area,
// so popup_y must be at least 1 (not 0).
assert!(
py >= 1,
"popup top row should account for area.y offset, got y={py} (expected >= 1)"
);
}
#[test]
fn test_settings_popup_horizontal_centering_uses_area_x() {
// outer[1].x is 0 for this layout, so this mainly verifies the popup
// is centered and not shifted left. A non-zero area.x layout would
// need a different layout to trigger, but the formula is the same.
let mut app = App::new();
let buf = render_settings_to_buffer(&mut app, 80, 24);
let (px, _py) =
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
// popup_w = 80*4/5 = 64, centered: (80-64)/2 = 8
assert_eq!(
px, 8,
"popup should start at x=8 (centered 64-wide popup in 80-col area)"
);
}
#[test]
fn test_settings_popup_small_frame_no_panic() {
// Frame smaller than the min popup size (40x14) should not panic.
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let mut app = App::new();
let _buf = render_settings_to_buffer(&mut app, 30, 10);
}));
assert!(
result.is_ok(),
"rendering settings in a small frame should not panic"
);
}
} }