Author SHA1 Message Date
dailz ef26481976 style: apply rustfmt across bench and core crates
CI / ci (ubuntu-latest) (push) Has been cancelled
CI / ci (windows-latest) (push) Has been cancelled
Pure formatting pass - no semantic changes. Mostly long-line reflow,
import statement reordering (alphabetical), and trailing-newline fixes
that had been accumulating in the working tree.

Affected files:
* crates/bench/src/{data_gen,main,mmap_reader,pread_reader,report,runner}.rs
* crates/bench/src/suites/{concurrent,growth,jump,memory,render,rotation,startup}.rs
* crates/core/src/io/{file_reader,index_cache,read_cache}.rs
* crates/core/src/{types.rs, watcher/file_watcher.rs}
2026-06-22 16:38:28 +08:00
dailz c4ba196016 chore: fix CI branch trigger and add project READMEs
CI / ci (ubuntu-latest) (push) Has been cancelled
CI / ci (windows-latest) (push) Has been cancelled
* ci.yml: trigger on pushes/PRs to master instead of main. The repo's
  default branch is master, so the previous config never fired and CI
  silently passed on every commit.

* Add README.md (English) and README-zh.md (Chinese), the bilingual
  project documentation referenced by AGENTS.md.
2026-06-22 16:32:52 +08:00
dailz ec5a163f05 chore: expand .gitignore for per-crate target dirs and local artifacts
* crates/*/target/ - per-crate build output (root /target/ was already
  covered, but workspace members' own target/ dirs were leaking through)
* .omo/ .sisyphus/ - local agent tooling state, not project content
* benchmark-report.md - output of cargo run -p log-viewer-bench,
  regenerated per run so should not be tracked
2026-06-22 15:14:52 +08:00
dailz 95f259a2bb fix(core): silence three clippy warnings blocking CI gate
* line_index.rs: replace manual `x % BLOCK_SIZE == 0` with
  `.is_multiple_of(BLOCK_SIZE)` (Rust 1.81+).

* line_index.rs: gate three pub(crate) accessor methods
  (`sampled_offsets`, `total_lines`, `has_trailing_newline`) behind
  `#[cfg(test)]`. They are only consumed by tests in file_reader.rs;
  marking them test-only removes them from production builds entirely,
  eliminating the dead_code warning without suppressing it.

* json.rs: drop redundant `Some(... .ok()?)` wrapper in
  parse_json_object_with_duplicates. `.ok()` already returns Option,
  so wrapping it in Some and unwrapping with ? was a no-op.

No behavior change. Unblocks `cargo clippy --workspace -- -D warnings`.
2026-06-22 15:02:33 +08:00
dailz e69b7af32a fix(tui): decouple cursor from viewport + correct width calc in no-VHI paths
Three intertwined fixes for the no-VHI scroll path that together eliminate
the 'switching to a new entry scrolls it to viewport top' behavior observed
during Loading state, Loading->Ready transition window, and Tab-toggle
VHI rebuild window.

Cursor/viewport decoupling (the user-visible bug):
* scroll_down_line / scroll_up_line no-VHI branches now advance only the
  cursor (cursor_line, cursor_sub_offset), then call ensure_cursor_visible_no_vhi
  to adjust the viewport minimally if needed. Previously they locked
  v_offset = cursor_line, forcing the cursor's entry to the top of the
  viewport on every keypress.
* New helpers: advance_visual_pos, retreat_visual_pos (bounded visual-row
  walks using compute_visual_height), ensure_cursor_visible_no_vhi (matches
  the VHI-path semantics - top-align when cursor is above viewport,
  bottom-align with minimal scroll when below, no movement when within).
* Same pattern applied to scroll_down/up_half_page and scroll_down/up_page.
* scroll_to_bottom sets cursor_sub_offset to the last visual row of the
  last line in no-VHI state (was 0).
* ensure_viewport_cache hidden anchors removed: the params-change recenter
  and the Loading post-check that did v_offset = cursor_line are replaced
  with the visibility helper.
* 4 transition sites (Tab handler, ensure_visual_height_index, reload,
  poll VHI complete) now clamp cursor_sub_offset against the current
  line height instead of blindly zeroing.

Width calculation correctness (root cause of the residual j/k bug):
* gutter_width() pub(crate) method extracted from ui.rs's inline formula,
  shared between renderer and App to prevent drift.
* get_content_width() now returns effective text-wrap width (area minus
  gutter), matching the renderer's actual_content_width. Previously it
  returned raw area width, causing compute_visual_height to under-count
  wrap rows for lines that wrapped only after gutter subtraction.
* Free function gutter_width_for(total_lines, is_loading) added so
  handle_file_appended and reload_ready_reader can recompute width
  after line-count changes without conflicting borrows on self.
* handle_file_appended: width now captured AFTER update_for_append
  (was before), so digit-boundary crossings (9->10, 99->100) don't
  leave stale gutter widths.

ui.rs render_content now calls app.gutter_width() instead of inlining
the formula (DRY).

Tests: 7 existing tests that locked in the old buggy v_offset = cursor_line
coupling have been rewritten to assert the new correct behavior. New
property tests verify wrap_line_count matches wrap_line_chars across
CJK/emoji/tab/combining-mark/control-char inputs.
2026-06-22 14:52:32 +08:00
dailz 4421da35f4 perf(core): linear mmap VHI rebuild + zero-alloc wrap helpers
Four changes that compound to a 2-5x speedup on VHI rebuild for large
files, plus eliminating one full wasted VHI pass on initial open.

* spawn_indexer no longer computes VHI inline (progressive_reader.rs).
  The inline VHI used a hardcoded width=80, was invalidated immediately
  on Loading->Ready transition (gutter digit-boundary), and triggered
  a second full rebuild. Now returns visual_height_index: None and lets
  the UI trigger one rebuild with the correct post-layout width.

* spawn_visual_height_rebuild rewritten as a single linear pass over
  mmap bytes using memchr::memchr (SIMD) to find newlines, replacing
  the byte-by-byte BufRead::read_until scan. Cancel check throttled
  to every 1MB instead of per-line.

* wrap_line_count(&str, width) -> usize added as a zero-allocation
  counterpart to wrap_line_chars. Property tests cover ASCII, CJK,
  emoji, combining marks, tabs, zero-width chars, control chars, and
  real log lines across many widths - all match wrap_line_chars().len()
  exactly. compute_text_visual_height now uses wrap_line_count, removing
  10M Vec allocations on a 10M-line file.

* format_json_line returns Cow<'_, str> instead of String. Non-JSON
  lines borrow the input (zero allocation); only pretty-printed JSON
  objects allocate. Removes another 10M allocations when json_format=true.

Removes now-unused compute_visual_heights (only caller was the deleted
inline VHI path).
2026-06-22 14:52:01 +08:00
dailz e765f8967f docs: add AGENTS.md with repo-specific agent guidance
Compact instruction file covering workspace layout, the non-obvious
default-members behavior (bare cargo run builds core, not tui), the
12 CoreError variants, the CRITICAL TerminalGuard Drop caveat, per-module
temp-file test helpers, and the benchmark test-file generation requirement.
2026-06-22 14:51:27 +08:00
dailz 967c11fea9 fix(tui): preserve old state on load_file failure (issue #32)
load_file previously cleared file_watcher before attempting to open the
new file. If open failed, the watcher was lost while loading_state and
file_path still pointed to the old file, breaking change detection.

Refactor to build-then-swap pattern: all fallible work runs on local
variables first, then self is updated only on success. On failure, the
old file's watcher, loading_state and file_path remain intact.

Update test to verify all three fields are preserved on failure.

Closes #32
2026-06-11 17:23:16 +08:00
dailz 10323ce814 fix(tui): add area offset to settings popup positioning (closes #31) 2026-06-11 16:49:37 +08:00
dailz c1a931551b fix(tui): handle settings save errors and rewrite truncate_to_columns (closes #30)
- Report save failures in settings instead of silently discarding errors
- Save draft first, commit color_config only on success (prevents split-brain)
- Show error message in red on status bar, stay in Settings mode on failure
- Clear error on Esc, entering settings, or any settings edit
- Rewrite truncate_to_columns as dedicated O(prefix) truncator
- Tab: stop before tab if expansion would exceed width
- Fixes pre-existing test_truncate_to_columns_tab failure
2026-06-11 16:08:57 +08:00
dailz dfc016c348 fix(tui): wrap color backward cycle at index 0 instead of clamping (closes #29) 2026-06-11 15:05:42 +08:00
dailz 19a3b877f9 fix(core): tighten word boundary to reject digits and underscores in level detection (closes #28) 2026-06-11 14:28:13 +08:00
dailz 5cb56dafd8 fix(core): correct tab-stop alignment and width overflow in wrap_line_chars
- Extract TAB_WIDTH constant (4) replacing magic numbers
- Calculate tab stop as TAB_WIDTH - (col % TAB_WIDTH) for proper alignment
- Split tab expansion across rows when width < TAB_WIDTH
- Update test_wrap_with_tab expected value for new behavior
- Add tests: narrow width, stop alignment, line boundary, regression

Fixes: #27
2026-06-11 13:37:14 +08:00
dailz e99861c76d fix(tui): add MAX_WRAP_INPUT_LEN guard to prevent UI freeze on oversized lines (closes #26)
- compute_line_entry/compute_visual_height: double guard (raw + post-format)
- Skip detect_level on oversized raw input to avoid O(n) JSON parsing
- Add post-format guard for JSON lines that expand beyond 10MB
- progressive_reader: add post-format guard to compute_line_visual_height
- Add truncate_to_columns helper using existing wrap_line_chars
- Fix misleading docstring on MAX_WRAP_INPUT_LEN constant
- Add 6 regression tests covering all guard paths
2026-06-11 13:15:11 +08:00
dailz a43ef673b0 fix(tui): rebase v_offset before VHI invalidation to prevent viewport jump (closes #25)
Ready state can have VHI=None during async rebuild (Tab toggle, resize,
file append, Loading→Ready transition). Without rebase, v_offset retains
a visual-row value that gets treated as a logical-line offset, causing
viewport jumps and cursor drift.

Changes:
- Add rebase_offset_for_invalidate() to convert v_offset from visual-row
  to logical-line + sub-row before VHI invalidation
- Call it at all 6 invalidation sites in Ready state
- Recalibrate v_offset from viewport top when VHI rebuild completes
- Clamp preserved sub_row to new line height after JSON/width changes

Regression tests: 6 new tests covering rebase conversion, Tab toggle,
VHI rebuild recalibration, sub-row clamping, and Loading→Ready transition.
2026-06-11 09:49:10 +08:00
dailz 70f930eef7 fix(tui): use updated v_offset after params_changed in ensure_viewport_cache (#24)
The loading branch of ensure_viewport_cache captured v_offset before the
params_changed block, which could reassign self.v_offset. This caused the
viewport to use a stale offset when loading + width/format changed together.

Remove the stale local variable and read self.v_offset directly, consistent
with the non-loading branch. Add regression test.
2026-06-11 08:58:10 +08:00
dailz 463c53148b fix(tui): filter KeyEventKind to prevent Release/Repeat from triggering commands (closes #23) 2026-06-10 17:34:17 +08:00
dailz e9f75ce3b1 fix(parser): detect duplicate JSON keys via custom Visitor instead of silent last-wins (closes #22) 2026-06-10 15:22:55 +08:00
dailz ef1889767a fix(parser): trim whitespace in LogLevel::from_str to prevent misclassification (closes #21)
Before this fix, level strings with surrounding whitespace (e.g. " WARN ")
were incorrectly parsed as Unknown instead of the matching variant.
This primarily affected the JSON log path where serde preserves the raw
field value including whitespace.

Changes:
- Add s.trim() before case-insensitive matching in FromStr impl
- Store trimmed value in Unknown variant to avoid whitespace noise
- Add unit tests for whitespace-padded known levels
- Add unit tests for Unknown trimming semantics (empty, internal ws)
- Add JSON regression test for level field with surrounding whitespace
2026-06-10 13:37:11 +08:00
dailz eedab3ac96 fix(parser): strip UTF-8 BOM before JSON parsing
serde_json rejects BOM (U+FEFF) prefixed input with 'expected value'
error, causing BOM-prefixed JSON log lines to be silently dropped.

Add strip_bom() helper that strips exactly one leading BOM character,
apply it in detect_json_log() and parse_line().

Closes #20
2026-06-10 11:37:00 +08:00
dailzandSisyphus 8e9600dda2 fix(parser): preserve non-string timestamp/level fields instead of silently dropping them (closes #19)
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-06-10 10:47:04 +08:00
dailz 2cebbd94c4 fix: concurrent cache save uses unique temp files (#17)
Replace deterministic .index.tmp path with per-save unique temp files
via tempfile::Builder. Eliminates race condition where background
indexer thread and TUI main thread could collide on the same temp path,
causing data truncation or corrupt cache writes.

Changes:
- Add write_cache_atomically() helper using tempfile::Builder
- Refactor save_with_hash() and save() to use the helper
- Extract encode_cache() to deduplicate serialization logic
- Move tempfile from [dev-dependencies] to [dependencies]
- Add 2 concurrent tests validating no corruption under parallel writes

Fixes #17
2026-06-09 16:13:39 +08:00
dailz 0d88e933e6 fix(io): replace blocking channel sends with cancel-aware alternatives (closes #16)
Background worker threads used blocking tx.send() on bounded channels.
If the consumer stopped draining, threads hung forever with no way to
reach the cancel check. Drop-issued cancellation was ineffective.

Changes:
- Progress messages: tx.try_send() (discard if full, never blocks loop)
- Terminal messages (Complete/Error): new send_cancelable<T>() helper
  using crossbeam select! — sleeps efficiently until send succeeds or
  cancel arrives
- Drop cancellation: tx.try_send() — Drop must never block
- spawn_visual_height_rebuild: same fix for its bounded(1) channel
- 5 new tests covering full-channel + cancel scenarios
2026-06-09 15:14:37 +08:00
dailz 420b853cb9 fix(watcher): filter Remove events by path to prevent false removed reports (closes #15) 2026-06-09 13:18:23 +08:00
dailz 7852e92ecc fix(watcher): forward notify backend errors instead of silently discarding
Previously Err(_) => return in the notify callback silently dropped all
backend errors (inotify exhaustion, fs unmount, permission loss), leaving
the application unaware that file monitoring had stopped working.

Add FileEvent::WatcherError { message: String } variant to propagate
backend errors through the existing bounded channel. The TUI consumer
receives the event without disrupting the UI for transient errors.

Closes #14
2026-06-09 11:26:54 +08:00
dailz d37ed6df68 fix(io): harden read_cache against zero-length false hits and overflow (closes #13)
- Add early return for len==0 (Ok(&[])) matching std::io semantics
- Add slot.len > 0 guard to cache hit predicate to prevent empty-slot
  false matches
- Replace unchecked arithmetic with checked_add/saturating_add for
  request_end, block_end, and post-read coverage check
- Fix misleading comment about get(file,0,0) behavior on miss path
- Strengthen clear() to fully reset block_offset and last_access
- Register read_cache module in io/mod.rs
- Add 4 regression tests: zero-len on fresh/populated cache,
  zero-len at u64::MAX, overflow error on nonzero read at u64::MAX
2026-06-09 10:48:34 +08:00
dailz b58d66f2aa fix(io): use unicode-width for correct CJK/emoji/zero-width display width (closes #12) 2026-06-07 12:50:17 +08:00
dailz d4679a7543 fix(io): update visual height of last line on append without trailing newline (closes #11)
When a file does not end with a newline, appending content extends the
last logical line's text and thus its visual height. The incremental
extend path in handle_file_appended only computed heights for newly
created logical lines, missing the old last line whose content changed.

Add VisualHeightIndex::replace_last_line_height() — an O(1) method that
rewrites the final prefix sum entry and total. Called before
extend_from_heights so the correct line is targeted.

Changes:
- progressive_reader.rs: add replace_last_line_height, pub with_params,
  7 VHI unit tests
- app.rs: save old_reader_line_count before update, recompute last old
  line height in extend path, 2 integration regression tests
2026-06-07 09:46:24 +08:00
dailz 8844e58cb4 Merge fix/m20-append-lines-error-handling: fix append_lines I/O error swallowing (closes #45) + clippy cleanup 2026-06-07 09:17:41 +08:00
dailz 6a2f8ecb66 fix(bench): resolve pre-existing clippy warnings in report.rs and mmap_reader.rs 2026-06-07 09:15:34 +08:00
dailz f6081b9fe9 fix(bench): propagate I/O errors in append_lines instead of silently defaulting to 0 (closes #45) 2026-06-07 09:13:37 +08:00
dailz 97a2c6a925 fix(bench): regenerate growable file each iteration in truncate safety benchmarks (closes #44) 2026-06-07 09:02:19 +08:00
dailz e6e0e2cc90 fix(bench): correct lines_read to actual successful reads in bench_scroll_rss
lines_read was incorrectly set to max_lines.min(total) (the loop upper bound)
instead of the actual number of successfully read lines. Now tracks lines_read
via get_line(i).is_some() counter and uses max_lines.min(total) as the correct
loop upper bound to handle empty file edge case.

Fixes #43
2026-06-07 08:50:20 +08:00
dailz ffaf462bae Merge fix/m23-single-frame-tail-overlap: [M23] small file single_frame_tail overlap fix 2026-06-07 08:32:58 +08:00
35 changed files with 3788 additions and 682 deletions
+2 -2
View File
@@ -2,9 +2,9 @@ name: CI
on:
push:
branches: [main]
branches: [master]
pull_request:
branches: [main]
branches: [master]
env:
CARGO_TERM_COLOR: always
+8
View File
@@ -1,5 +1,13 @@
/target/
crates/*/target/
*.swp
*.swo
.idea/
.vscode/
# Agent tooling state
.omo/
.sisyphus/
# Benchmark output
benchmark-report.md
+114
View File
@@ -0,0 +1,114 @@
# logViewer
High-performance log file viewer. Rust workspace with 4 crates, edition 2024, MSRV 1.92 (pinned in `rust-toolchain.toml`).
## Workspace Layout
```
crates/core — log-viewer-core (library) — I/O, parsing, types, config, file watching
crates/tui — log-viewer-tui (binary) — terminal UI (ratatui + crossterm)
crates/gui — log-viewer-gui (binary) — GUI (egui + eframe) — STUB, not yet functional
crates/bench — log-viewer-bench (binary) — mmap vs pread benchmark harness
```
`core` owns all I/O, parsing, data types, and file watching. TUI/GUI consume core's public API. GUI is a placeholder with no core integration yet.
**`default-members = ["crates/core"]`** — bare `cargo run` / `cargo test` target *core*, not the TUI. Always pass `-p log-viewer-tui` (or `--workspace`) explicitly.
## Commands
```bash
# Run the TUI (primary interface)
cargo run -p log-viewer-tui -- <log-file>
# Build/check/test/lint the whole workspace
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# Single crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
# Benchmarks (test file is NOT auto-generated — see "Benchmarks" section below)
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
```
## CI Gate (must pass before merge)
Runs on ubuntu-latest + windows-latest (`.github/workflows/ci.yml`):
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
Toolchain installed via `dtolnay/rust-toolchain@stable` reading `rust-toolchain.toml` (channel `1.92`). No rustfmt.toml or clippy.toml — uses toolchain defaults.
## Core Architecture
All ten modules are declared `pub` in `lib.rs` and re-exported as `log_viewer_core::<module>`.
### Key Modules
- **`types`** — `LogLevel`, `LogEntry`, `SearchQuery`, `SearchFilter`, `Bookmark`, `FileSession`, `DuplicateKey`, `SearchResult`
- **`error`** — `CoreError` (**12 variants** via thiserror: Io, Parse, Search, Index, Config, TomlSerialize, Encoding, Watch, Mmap, Cache, FileNotFound, Other) + `Result<T>` alias. `From` impls for `io::Error` / `serde_json::Error` / `toml::de::Error` / `toml::ser::Error` / `notify::Error` mean `?` works freely.
- **`io::file_reader`** — mmap-backed `FileReader` with TOCTOU mitigation (post-mmap stat check), append/reload detection
- **`io::progressive_reader`** — `ProgressiveFileReader` state machine (`ReaderState::Sampling { .. } | Ready { .. }`), background indexer via crossbeam-channel, `VisualHeightIndex` for wrapped-line scroll
- **`io::line_index`** — Sparse index (every `BLOCK_SIZE = 256` lines, memchr SIMD). Serializable to disk.
- **`io::index_cache`** — Persistent cache with xxh3 content hash, atomic writes via temp files
- **`io::line_sampler`** / **`io::cache_util`** — shared helpers used by the index path
- **`io::wrap`** — Unicode-aware line wrapping (CJK/emoji/tab), JSON pretty-printing; enforces `MAX_WRAP_INPUT_LEN`
- **`io::read_cache`** — LRU read cache (not yet integrated into the live reader; future pread backend)
- **`parser::json`** — NDJSON parser with BOM handling, duplicate key detection
- **`parser::level`** — JSON-first level detection (`detect_level`); falls back to bounded keyword scan with word-boundary check on non-JSON lines
- **`watcher::file_watcher`** — File event watcher (notify + crossbeam) with append/truncate/rotation detection
- **`config`** — `ColorConfig` TOML load/save with per-field serde defaults
### Stubs (single-line TODOs — planned, not yet built)
- `filter::Filter`
- `bookmark::BookmarkManager`
- `session::SessionManager`
- `search::engine::SearchEngine`
## TUI Architecture
- **`main.rs`** — clap CLI (`files: Vec<String>`), `TerminalGuard` RAII for raw mode + alternate screen. **CRITICAL**: `Drop` does not run on `std::process::exit` or `panic = "abort"` — never call `process::exit` inside the guarded scope; return `Err` and use `?` instead. Event loop: poll indexer → poll watcher → draw → poll keys, 100ms timeout.
- **`app.rs`** — `App` struct (~4000 lines including ~135 inline tests, test mod starts at `#[cfg(test)] mod tests`). `AppLoadingState: Empty | Loading { reader, estimated_lines, progress_percent } | Ready { reader } | Error(String)`. `AppMode: Normal | Settings`. Viewport cache for scroll. All key handling.
- **`ui.rs`** — ratatui rendering: title bar, content area, status bar, settings popup
- **`color.rs`** — `LogLevel` → ratatui `Color` via `ColorConfig`
### TUI Keybindings
j/k scroll, Ctrl+d/u half-page, Ctrl+f/b full-page, G/gg jump end/top, Tab toggle JSON formatting, S settings panel, q/Esc quit. Settings: j/k select level, ←/→ cycle colors, Enter save, Esc cancel.
## Testing
All tests are inline (`#[cfg(test)] mod tests` blocks). No `tests/` directories, no integration tests, no async tests.
- **~450 tests total**: core 251, tui 136, bench 63, gui 0
- Temp-file helpers vary per module — check the local `tests` mod before assuming an API:
- `make_temp_file(content) -> PathBuf``tui/app.rs`, `tui/ui.rs`
- `make_test_file(lines) -> NamedTempFile``core/io/index_cache.rs`
- `make_file(data) -> NamedTempFile``core/io/read_cache.rs`
- `struct TempFile { ... }``core/io/line_sampler.rs`
- `insta` is a declared dev-dependency but currently **unused** (no `insta::` calls anywhere in the workspace)
- TUI render tests use ratatui `TestBackend` (see `tui/ui.rs`)
- Bench crate has its own unit tests for mmap/pread backends, metrics, and data_gen helpers
## Benchmarks
Custom harness (not criterion). Wall-clock + `/proc/self/` RSS/page fault metrics. 7 suites: startup, render, jump, memory, growth, rotation, concurrent. Tests mmap (plain/sequential/random/populate/phase_aware) vs pread (plain/random/sequential) backends. Output: markdown tables in `benchmark-report.md`.
**The harness does NOT auto-generate the test file.** `main.rs` requires `/tmp/test-logviewer/extreme.log` to already exist; if missing it prints the `dd` command and exits. Generate it manually first:
```bash
mkdir -p /tmp/test-logviewer
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000 # ~5GB
```
(`crates/bench/src/data_gen.rs` has `generate_test_file` / `ensure_test_file` helpers and is used by the `growth` and `rotation` suites for per-run growable files, but it is **not** wired into `main.rs` for the primary test file.)
Note: `log-viewer-bench` depends directly on `memmap2` / `nix` / `libc` / `memchr` — it does **not** depend on `log-viewer-core`, so it can be built standalone with `cargo build -p log-viewer-bench`.
Generated
+2
View File
@@ -2333,6 +2333,7 @@ dependencies = [
"tempfile",
"thiserror 2.0.18",
"toml",
"unicode-width",
"xxhash-rust",
]
@@ -2357,6 +2358,7 @@ dependencies = [
"ratatui",
"serde_json",
"tempfile",
"unicode-width",
]
[[package]]
+1
View File
@@ -27,3 +27,4 @@ textwrap = "0.16"
tempfile = "3"
xxhash-rust = { version = "0.8", features = ["xxh3"] }
bincode = "1"
unicode-width = "0.2"
+105
View File
@@ -0,0 +1,105 @@
# logViewer
**[English](README.md)**
用 Rust 构建的高性能终端日志查看器,专为处理超大日志文件(GB 级别)而设计,内存占用极低。
基于 mmap 内存映射、稀疏行索引和后台渐进式加载,打开文件几乎瞬时完成——即使是 5GB 以上的日志文件,也能在索引完成前就开始滚动浏览。
## 特性
- **瞬时打开** — mmap 读取 + 后台渐进式索引,索引未完成即可开始浏览
- **超大文件支持** — 稀疏行索引(每 256 行采样一次),5GB 文件仅占用约 8MB 内存
- **实时文件追踪** — 通过 `notify` 监控文件追加、截断和日志轮转
- **JSON 日志解析** — 支持 NDJSON 格式,自动处理 BOM、检测重复键,可切换格式化显示
- **Unicode 感知的换行** — 正确处理中文、emoji 和制表符的行宽计算
- **持久化索引缓存** — 行索引保存到磁盘,使用 xxh3 哈希校验内容变更,再次打开几乎无需等待
- **Vim 风格快捷键** — 终端用户无学习成本
- **可自定义配色** — 每个日志级别独立配色,通过 TOML 配置文件管理,TUI 内直接调整
## 快速开始
```bash
# 直接编译运行
cargo run -p log-viewer-tui -- path/to/logfile.log
# 或先编译再运行
cargo build --release -p log-viewer-tui
./target/release/log-viewer-tui path/to/logfile.log
```
需要 Rust 1.92 及以上版本(见 `rust-toolchain.toml`)。
## 快捷键
| 按键 | 操作 |
|------|------|
| `j` / `↓` | 向下滚动一行 |
| `k` / `↑` | 向上滚动一行 |
| `Ctrl+d` | 向下滚动半页 |
| `Ctrl+u` | 向上滚动半页 |
| `Ctrl+f` / `PgDn` | 向下滚动一页 |
| `Ctrl+b` / `PgUp` | 向上滚动一页 |
| `G` / `End` | 跳转到文件末尾 |
| `gg` / `Home` | 跳转到文件开头 |
| `Tab` | 切换 JSON 格式化显示 |
| `S` | 打开配色设置 |
| `q` / `Esc` | 退出 |
设置面板:`j`/`k` 选择日志级别,`←`/`→` 切换颜色,`Enter` 保存,`Esc` 取消。
## 项目结构
```
crates/core — 核心库:I/O、解析、数据类型、配置、文件监控
crates/tui — 终端界面(ratatui + crossterm
crates/gui — 图形界面(egui + eframe)— 占位模块,尚未实现
crates/bench — mmap 与 pread 性能对比基准测试
```
## 开发
```bash
# 检查、测试、代码规范
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# 运行基准测试(首次运行会生成约 5GB 测试文件)
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
# 单独测试某个 crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
```
### CI
CI 在 ubuntu-latest 和 windows-latest 上运行以下检查:
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
无自定义 rustfmt 或 clippy 配置——使用工具链默认值。
## 架构
核心库 `log-viewer-core` 负责所有 I/O、解析和数据类型。关键设计决策:
- **稀疏行索引** — 每 256 行采样一次,使用 memchr SIMD 加速。100 万行的文件仅产生约 32KB 的索引,而非 8MB。
- **渐进式加载** — `ProgressiveFileReader` 先读取文件头尾进行快速行数估算,再通过 crossbeam-channel 在后台线程构建完整索引。
- **mmap 与 TOCTOU 防护** — mmap 映射后进行 stat 校验以检测文件变更。`read_cache` 模块为未来基于 pread 的实现预留,可彻底消除 SIGBUS 风险。
- **持久化缓存** — 行索引序列化到磁盘,使用 xxh3 内容哈希进行失效校验,通过临时文件实现原子写入。
- **视觉高度索引** — 基于换行后行高的前缀和数组,支持 O(log n) 的滚动定位,适用于长行换行场景。
计划中但尚未实现的功能:过滤、书签、会话管理、搜索引擎(stub 模块已创建)。
## 基准测试
自研基准测试框架,对比 mmap 与 pread 两种后端在 7 个测试场景(启动、渲染、跳转、内存、增长、轮转、并发)下的表现。使用挂钟计时和 `/proc/self/` 的 RSS 及页错误指标。结果以 Markdown 表格形式输出到 `benchmark-report.md`
基准测试二进制包含约 75 个单元测试,覆盖读取器后端和指标采集逻辑。
+104
View File
@@ -0,0 +1,104 @@
# logViewer
**[中文文档](README-zh.md)**
A high-performance terminal log file viewer built in Rust, designed to handle multi-gigabyte files with minimal memory overhead.
Uses memory-mapped I/O with a sparse line index and progressive background loading to open files instantly — even 5GB+ logs scroll smoothly from the first keystroke.
## Features
- **Instant file open** — mmap-backed reader with background progressive indexing; start scrolling before indexing finishes
- **Handles huge files** — sparse line index (1 entry per 256 lines) keeps memory usage at ~8MB even for 5GB files
- **Live file tracking** — watches for appends, truncations, and log rotation via `notify`
- **JSON log support** — NDJSON parsing with BOM handling, duplicate key detection, and toggleable pretty-printing
- **Unicode-aware wrapping** — correct line wrapping for CJK, emoji, and tabs
- **Persistent index cache** — line indexes saved to disk with xxh3 content hashing; re-opens are near-instant
- **Vim-like keybindings** — familiar navigation for terminal users
- **Customizable colors** — per-log-level colors via TOML config, adjustable from within the TUI
## Quick Start
```bash
# Build and run
cargo run -p log-viewer-tui -- path/to/logfile.log
# Or build first
cargo build --release -p log-viewer-tui
./target/release/log-viewer-tui path/to/logfile.log
```
Requires Rust 1.92+ (see `rust-toolchain.toml`).
## Keybindings
| Key | Action |
|-----|--------|
| `j` / `↓` | Scroll down one line |
| `k` / `↑` | Scroll up one line |
| `Ctrl+d` | Scroll down half page |
| `Ctrl+u` | Scroll up half page |
| `Ctrl+f` / `PgDn` | Scroll down full page |
| `Ctrl+b` / `PgUp` | Scroll up full page |
| `G` / `End` | Jump to end of file |
| `gg` / `Home` | Jump to start of file |
| `Tab` | Toggle JSON pretty-printing |
| `S` | Open color settings |
| `q` / `Esc` | Quit |
Settings panel: use `j`/`k` to select a log level, `←`/`→` to cycle colors, `Enter` to save, `Esc` to cancel.
## Workspace Structure
```
crates/core — Shared library: I/O, parsing, types, config, file watching
crates/tui — Terminal UI (ratatui + crossterm)
crates/gui — GUI (egui + eframe) — placeholder, not yet functional
crates/bench — mmap vs pread benchmark harness
```
## Development
```bash
# Check, test, lint
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings
# Run benchmarks (generates ~5GB test file on first run)
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
# Test a single crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui
```
### CI
The CI gate runs on ubuntu-latest and windows-latest:
1. `cargo fmt --check --all`
2. `cargo check --workspace`
3. `cargo test --workspace`
4. `cargo clippy --workspace -- -D warnings`
No custom rustfmt or clippy config — uses toolchain defaults.
## Architecture
The core library (`log-viewer-core`) owns all I/O, parsing, and data types. Key design decisions:
- **Sparse line index** — samples every 256 lines using memchr SIMD acceleration. A 1M-line file produces a ~32KB index instead of ~8MB.
- **Progressive loading** — `ProgressiveFileReader` starts with a quick head+tail sample for instant line estimates, then builds the full index in a background thread via crossbeam-channel.
- **mmap with TOCTOU mitigation** — post-mmap stat check to detect file changes. `read_cache` module exists as a future pread-based alternative to eliminate SIGBUS risk entirely.
- **Persistent cache** — line indexes are serialized to disk with xxh3 content hashing for invalidation. Atomic writes via temp files.
- **Visual height index** — prefix-sum over wrapped-line heights, enabling O(log n) scroll-to-line mapping for long wrapped lines.
Planned but not yet built: filtering, bookmarks, sessions, search engine (stub modules exist).
## Benchmarks
Custom harness comparing mmap vs pread backends across 7 suites (startup, render, jump, memory, growth, rotation, concurrent). Uses wall-clock timing with `/proc/self/` RSS and page fault metrics. Results are written as markdown tables to `benchmark-report.md`.
The benchmark binary includes ~75 unit tests for the reader backends and metrics collection.
+9 -6
View File
@@ -83,11 +83,9 @@ pub fn generate_growable_file(dir: &Path) -> std::io::Result<PathBuf> {
/// Append `count` lines to the file
pub fn append_lines(path: &Path, count: usize) -> std::io::Result<()> {
let mut file = BufWriter::with_capacity(
64 * 1024,
fs::OpenOptions::new().append(true).open(path)?,
);
let existing_lines = count_existing_lines(path).unwrap_or(0);
let existing_lines = count_existing_lines(path)?;
let mut file =
BufWriter::with_capacity(64 * 1024, fs::OpenOptions::new().append(true).open(path)?);
for i in 0..count {
writeln!(
file,
@@ -117,7 +115,12 @@ pub fn rotate_file(path: &Path) -> std::io::Result<PathBuf> {
fn count_existing_lines(path: &Path) -> std::io::Result<u64> {
let file = fs::File::open(path)?;
let reader = BufReader::new(file);
Ok(reader.lines().count() as u64)
let mut count = 0u64;
for line in reader.lines() {
line?;
count += 1;
}
Ok(count)
}
#[cfg(test)]
+4 -2
View File
@@ -27,8 +27,10 @@ fn main() {
let suites = match args.suites {
Some(names) => {
let parsed: Result<Vec<_>, _> =
names.iter().map(|s| s.parse::<log_viewer_bench::runner::Suite>()).collect();
let parsed: Result<Vec<_>, _> = names
.iter()
.map(|s| s.parse::<log_viewer_bench::runner::Suite>())
.collect();
match parsed {
Ok(s) => Some(s),
Err(e) => {
+9 -9
View File
@@ -7,14 +7,14 @@
use std::fs::File;
use std::io::BufReader;
use std::path::Path;
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU8, Ordering};
use std::sync::Once;
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU8, Ordering};
use memmap2::{Advice, Mmap, MmapOptions, RemapOptions};
use nix::sys::signal::{sigaction, SaFlags, SigAction, SigHandler, SigSet, Signal};
use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction};
use crate::line_index::LineIndex;
use crate::FileReaderBackend;
use crate::line_index::LineIndex;
// ─── SIGBUS Handler ──────────────────────────────────────────────────────────
//
@@ -35,7 +35,10 @@ const HANDLER_NONE: u8 = 0;
const HANDLER_DEFAULT: u8 = 1;
const HANDLER_IGNORE: u8 = 2;
const HANDLER_PLAIN: u8 = 3; // extern "C" fn(c_int)
#[expect(clippy::unseparated_literal_suffix, reason = "clarity: this is the SA_SIGACTION variant")]
#[allow(
clippy::unseparated_literal_suffix,
reason = "clarity: this is the SA_SIGACTION variant"
)]
const HANDLER_SIGACTION: u8 = 4; // extern "C" fn(c_int, *mut siginfo_t, *mut c_void)
/// Old SIGBUS handler type — raw atomic, async-signal-safe to read.
@@ -107,11 +110,8 @@ extern "C" fn sigbus_handler(
HANDLER_SIGACTION => {
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
if !ptr.is_null() {
let f: extern "C" fn(
libc::c_int,
*mut libc::siginfo_t,
*mut std::ffi::c_void,
) = unsafe { std::mem::transmute(ptr) };
let f: extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut std::ffi::c_void) =
unsafe { std::mem::transmute(ptr) };
f(sig, info, ctx);
} else {
unsafe { libc::_exit(128 + sig) };
+10 -3
View File
@@ -15,8 +15,8 @@ use std::os::unix::fs::FileExt;
use std::os::unix::io::AsRawFd;
use std::path::Path;
use crate::line_index::LineIndex;
use crate::FileReaderBackend;
use crate::line_index::LineIndex;
const BLOCK_SIZE: usize = 256;
const CACHE_CHUNK: usize = 4096;
@@ -478,12 +478,19 @@ mod tests {
let mut reader = PreadReaderPlain::open(&path).unwrap();
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("alpha".to_owned()));
assert_eq!(reader.get_line(3), None, "should be out of bounds before append");
assert_eq!(
reader.get_line(3),
None,
"should be out of bounds before append"
);
// Phase 2: append 2 more lines
{
use std::io::Write as _;
let mut f = std::fs::OpenOptions::new().append(true).open(&path).unwrap();
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.unwrap();
f.write_all(b"delta\nepsilon\n").unwrap();
}
+52 -12
View File
@@ -143,11 +143,9 @@ pub fn format_report(results: &[BenchmarkResult]) -> String {
report.push_str("| Test | Variant | RSS | Peak RSS | Page Faults |\n");
report.push_str("|------|---------|-----|----------|-------------|\n");
let mut mem_rows: Vec<&BenchmarkResult> =
category_results.iter().copied().collect();
let mut mem_rows: Vec<&BenchmarkResult> = category_results.to_vec();
mem_rows.sort_by(|a, b| {
(&a.test_name, &a.backend, &a.variant)
.cmp(&(&b.test_name, &b.backend, &b.variant))
(&a.test_name, &a.backend, &a.variant).cmp(&(&b.test_name, &b.backend, &b.variant))
});
for r in mem_rows {
let variant_label = format!("{} ({})", r.backend, r.variant);
@@ -163,7 +161,10 @@ pub fn format_report(results: &[BenchmarkResult]) -> String {
report.push('\n');
}
let mut extras: Vec<(String, String, Vec<(String, f64)>)> = category_results
type ExtraEntry = (String, f64);
type ExtraGroup = (String, String, Vec<ExtraEntry>);
let mut extras: Vec<ExtraGroup> = category_results
.iter()
.filter(|r| !r.extra.is_empty())
.map(|r| {
@@ -241,21 +242,60 @@ mod tests {
#[test]
fn report_ordering_independent_of_input_order() {
let set_a = vec![
make_result("sequential", "read_1mb", "pread", "default", vec![100, 110, 105]),
make_result("sequential", "read_1mb", "mmap", "default", vec![80, 85, 90]),
make_result("sequential", "read_4kb", "pread", "default", vec![10, 12, 11]),
make_result(
"sequential",
"read_1mb",
"pread",
"default",
vec![100, 110, 105],
),
make_result(
"sequential",
"read_1mb",
"mmap",
"default",
vec![80, 85, 90],
),
make_result(
"sequential",
"read_4kb",
"pread",
"default",
vec![10, 12, 11],
),
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
];
let set_b = vec![
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
make_result("sequential", "read_1mb", "mmap", "default", vec![80, 85, 90]),
make_result("sequential", "read_4kb", "pread", "default", vec![10, 12, 11]),
make_result("sequential", "read_1mb", "pread", "default", vec![100, 110, 105]),
make_result(
"sequential",
"read_1mb",
"mmap",
"default",
vec![80, 85, 90],
),
make_result(
"sequential",
"read_4kb",
"pread",
"default",
vec![10, 12, 11],
),
make_result(
"sequential",
"read_1mb",
"pread",
"default",
vec![100, 110, 105],
),
];
let report_a = format_report(&set_a);
let report_b = format_report(&set_b);
assert_eq!(report_a, report_b, "Reports must be identical regardless of input order");
assert_eq!(
report_a, report_b,
"Reports must be identical regardless of input order"
);
}
}
+5 -1
View File
@@ -154,7 +154,11 @@ mod tests {
("concurrent", Suite::Concurrent),
];
for (s, expected_suite) in expected {
assert_eq!(Suite::from_str(s).unwrap(), expected_suite, "failed to parse '{s}'");
assert_eq!(
Suite::from_str(s).unwrap(),
expected_suite,
"failed to parse '{s}'"
);
}
}
+1 -1
View File
@@ -1,5 +1,6 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
@@ -8,7 +9,6 @@ use crate::mmap_reader::{
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
+1 -1
View File
@@ -1,12 +1,12 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::MmapReaderPlain;
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
+1 -1
View File
@@ -1,6 +1,7 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
@@ -9,7 +10,6 @@ use crate::mmap_reader::{
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
+8 -4
View File
@@ -1,5 +1,6 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
@@ -8,7 +9,6 @@ use crate::mmap_reader::{
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
@@ -101,11 +101,15 @@ fn bench_scroll_rss<B: FileReaderBackend>(
let sample_interval = 100_000;
let max_lines = if config.quick_mode { 100_000 } else { total };
let upper = max_lines.min(total);
let mut rss_samples = Vec::new();
let mut hwm_samples = Vec::new();
let mut lines_read = 0usize;
for i in (0..max_lines).step_by(sample_interval) {
let _ = reader.get_line(i);
for i in (0..upper).step_by(sample_interval) {
if reader.get_line(i).is_some() {
lines_read += 1;
}
let rss = MetricsCollector::read_rss();
rss_samples.push(rss.vm_rss_kb);
hwm_samples.push(rss.vm_hwm_kb);
@@ -124,7 +128,7 @@ fn bench_scroll_rss<B: FileReaderBackend>(
"max_hwm_kb".into(),
hwm_samples.iter().copied().fold(0u64, u64::max) as f64,
);
extra.insert("lines_read".into(), max_lines.min(total) as f64);
extra.insert("lines_read".into(), lines_read as f64);
reader.close();
+14 -4
View File
@@ -1,6 +1,7 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
@@ -9,7 +10,6 @@ use crate::mmap_reader::{
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
@@ -248,7 +248,11 @@ mod tests {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap: {:?} @ {} vs {:?} @ {} (total={})",
pos[i].0, pos[i].1, pos[j].0, pos[j].1, total
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1,
total
);
}
}
@@ -269,7 +273,10 @@ mod tests {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=104: {:?} @ {} vs {:?} @ {}",
pos[i].0, pos[i].1, pos[j].0, pos[j].1
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1
);
}
}
@@ -283,7 +290,10 @@ mod tests {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=105: {:?} @ {} vs {:?} @ {}",
pos[i].0, pos[i].1, pos[j].0, pos[j].1
pos[i].0,
pos[i].1,
pos[j].0,
pos[j].1
);
}
}
+5 -15
View File
@@ -1,12 +1,12 @@
use std::collections::HashMap;
use crate::FileReaderBackend;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{self, MmapReaderPlain};
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
@@ -24,13 +24,14 @@ fn bench_truncate_safety_mmap(
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_mmap");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut sigbus_detected = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
mmap_reader::reset_sigbus_flag();
let reader = MmapReaderPlain::open(&path).expect("Failed to open file");
@@ -48,12 +49,6 @@ fn bench_truncate_safety_mmap(
sigbus_detected += 1;
}
reader.close();
let mut f = std::fs::File::create(&path).expect("Failed to recreate file");
use std::io::Write;
for i in 0..1000u64 {
writeln!(f, "restored line {i}").unwrap();
}
}
let rss = MetricsCollector::read_rss();
@@ -82,13 +77,14 @@ fn bench_truncate_safety_pread(
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_pread");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut error_count = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let reader = PreadReaderPlain::open(&path).expect("Failed to open file");
let original_size = reader.file_size();
@@ -104,12 +100,6 @@ fn bench_truncate_safety_pread(
error_count += 1;
}
reader.close();
let mut f = std::fs::File::create(&path).expect("Failed to recreate file");
use std::io::Write;
for i in 0..1000u64 {
writeln!(f, "restored line {i}").unwrap();
}
}
let rss = MetricsCollector::read_rss();
+1 -1
View File
@@ -1,6 +1,7 @@
use std::collections::HashMap;
use std::path::Path;
use crate::FileReaderBackend;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
@@ -9,7 +10,6 @@ use crate::mmap_reader::{
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
+2 -1
View File
@@ -17,7 +17,8 @@ memmap2.workspace = true
directories.workspace = true
xxhash-rust.workspace = true
bincode.workspace = true
unicode-width.workspace = true
tempfile.workspace = true
[dev-dependencies]
insta.workspace = true
tempfile.workspace = true
+7 -3
View File
@@ -33,8 +33,8 @@ impl FileReader {
} else {
// SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。
// memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。
let m = unsafe { memmap2::Mmap::map(&file) }
.map_err(|e| CoreError::Mmap(e.to_string()))?;
let m =
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
// Layer 3: mmap 后立即 stat 同一 fd,检测截断(TOCTOU 缓解,非安全证明)
let current_size = file.metadata()?.len();
@@ -507,7 +507,11 @@ mod tests {
reader.reload().unwrap();
assert_eq!(reader.line_count(), 1);
assert_eq!(reader.get_line(0), Some("new"));
assert_eq!(reader.get_line(1), None, "old line should not be accessible");
assert_eq!(
reader.get_line(1),
None,
"old line should not be accessible"
);
}
#[test]
+117 -38
View File
@@ -1,26 +1,37 @@
use std::io::{Read as _, Write as _};
use std::path::Path;
use crate::io::cache_util::{cache_path, CACHE_VERSION};
use crate::io::cache_util::{CACHE_VERSION, cache_path};
use crate::io::line_index::LineIndex;
pub struct IndexCache;
impl IndexCache {
/// Save a `LineIndex` to disk using atomic write (write to .tmp, then rename).
/// Write `buf` to `dest` atomically using a unique temporary file in the same directory.
///
/// The file hash is derived from `data` (the same byte slice used to build the index),
/// avoiding TOCTOU issues from re-reading the file from disk.
pub fn save_with_hash(
file_path: &Path,
index: &LineIndex,
data: &[u8],
) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
/// Each call creates its own temp file via `tempfile::Builder`, eliminating collisions
/// when multiple threads (or processes) save to the same cache path concurrently.
/// The temp file is created in `dest.parent()` so the final `rename` stays on the
/// same filesystem and remains atomic.
fn write_cache_atomically(dest: &Path, buf: &[u8]) -> std::io::Result<()> {
let dir = dest.parent().ok_or_else(|| {
std::io::Error::new(
std::io::ErrorKind::InvalidInput,
"cache destination has no parent directory",
)
})?;
let file_hash = compute_data_hash(data);
let mut tmp = tempfile::Builder::new()
.prefix("index-cache-")
.suffix(".tmp")
.tempfile_in(dir)?;
tmp.as_file_mut().write_all(buf)?;
tmp.as_file_mut().sync_all()?;
tmp.persist(dest).map(|_| ()).map_err(|e| e.error)
}
fn encode_cache(file_hash: u64, index: &LineIndex) -> std::io::Result<Vec<u8>> {
let index_bytes = bincode::serialize(index)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
@@ -28,16 +39,22 @@ impl IndexCache {
buf.push(CACHE_VERSION);
buf.extend_from_slice(&file_hash.to_le_bytes());
buf.extend_from_slice(&index_bytes);
let tmp_path = dest.with_extension("index.tmp");
{
let mut f = std::fs::File::create(&tmp_path)?;
f.write_all(&buf)?;
f.sync_all()?;
Ok(buf)
}
std::fs::rename(&tmp_path, &dest)?;
Ok(())
impl IndexCache {
/// Save a `LineIndex` to disk using atomic write (unique temp file, then rename).
///
/// The file hash is derived from `data` (the same byte slice used to build the index),
/// avoiding TOCTOU issues from re-reading the file from disk.
pub fn save_with_hash(file_path: &Path, index: &LineIndex, data: &[u8]) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
})?;
let file_hash = compute_data_hash(data);
let buf = encode_cache(file_hash, index)?;
write_cache_atomically(&dest, &buf)
}
/// Save a `LineIndex` to disk, computing the hash by re-reading the file.
@@ -50,23 +67,8 @@ impl IndexCache {
})?;
let file_hash = compute_file_hash(file_path)?;
let index_bytes = bincode::serialize(index)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
let mut buf = Vec::with_capacity(1 + 8 + index_bytes.len());
buf.push(CACHE_VERSION);
buf.extend_from_slice(&file_hash.to_le_bytes());
buf.extend_from_slice(&index_bytes);
let tmp_path = dest.with_extension("index.tmp");
{
let mut f = std::fs::File::create(&tmp_path)?;
f.write_all(&buf)?;
f.sync_all()?;
}
std::fs::rename(&tmp_path, &dest)?;
Ok(())
let buf = encode_cache(file_hash, index)?;
write_cache_atomically(&dest, &buf)
}
/// Load a cached `LineIndex` from disk.
@@ -305,4 +307,81 @@ mod tests {
assert_ne!(h1, h2, "hash should change when content changes");
}
#[test]
fn test_concurrent_save_with_hash_no_corruption() {
use std::sync::{Arc, Barrier};
let file = make_test_file(300);
let data = std::fs::read(file.path()).unwrap();
let num_threads = 8;
let iterations = 50;
let barrier = Arc::new(Barrier::new(num_threads));
let path = file.path().to_path_buf();
let handles: Vec<_> = (0..num_threads)
.map(|_| {
let barrier = Arc::clone(&barrier);
let path = path.clone();
let data = data.clone();
std::thread::spawn(move || {
let index = LineIndex::from_bytes(&data);
barrier.wait();
for _ in 0..iterations {
IndexCache::save_with_hash(&path, &index, &data)
.expect("concurrent save_with_hash should succeed");
}
})
})
.collect();
for h in handles {
h.join().expect("thread should not panic");
}
let loaded = IndexCache::load(file.path()).expect("final cache should load successfully");
let expected = LineIndex::from_bytes(&data);
assert_eq!(loaded.line_count(), expected.line_count());
assert_eq!(
loaded.sampled_offsets(),
expected.sampled_offsets(),
"concurrent writes must not produce interleaved data"
);
}
#[test]
fn test_concurrent_save_same_dest_all_succeed() {
use std::sync::{Arc, Barrier};
let dir = tempfile::tempdir().unwrap();
let dest = dir.path().join("test.index");
let payloads: Vec<Vec<u8>> = (0..8).map(|i| vec![i; 64 * 1024]).collect();
let barrier = Arc::new(Barrier::new(8));
let handles: Vec<_> = payloads
.into_iter()
.map(|payload| {
let barrier = Arc::clone(&barrier);
let dest = dest.clone();
std::thread::spawn(move || {
barrier.wait();
write_cache_atomically(&dest, &payload)
.expect("concurrent atomic write should succeed");
})
})
.collect();
for h in handles {
h.join().expect("thread should not panic");
}
let final_data = std::fs::read(&dest).expect("dest file should exist");
assert_eq!(
final_data.len(),
64 * 1024,
"final file must be exactly one payload"
);
}
}
+4 -2
View File
@@ -158,7 +158,7 @@ impl LineIndex {
// If the junction falls on a block boundary, record the start offset
// (analogous to from_bytes always pushing offset 0 for line 0).
if starts_new_line && (old_total as usize) % BLOCK_SIZE == 0 {
if starts_new_line && (old_total as usize).is_multiple_of(BLOCK_SIZE) {
self.sampled_offsets.push(start_offset);
}
@@ -212,15 +212,17 @@ impl LineIndex {
self.total_lines as usize
}
// ─── getter 方法 ────────────────────────────────────────────────────
#[cfg(test)]
pub(crate) fn sampled_offsets(&self) -> &[u64] {
&self.sampled_offsets
}
#[cfg(test)]
pub(crate) fn total_lines(&self) -> u64 {
self.total_lines
}
#[cfg(test)]
pub(crate) fn has_trailing_newline(&self) -> bool {
self.has_trailing_newline
}
+1
View File
@@ -18,4 +18,5 @@ pub mod cache_util;
pub mod index_cache;
pub mod line_sampler;
pub mod progressive_reader;
pub mod read_cache;
pub mod wrap;
+325 -78
View File
@@ -8,7 +8,25 @@ use crate::io::file_reader::{AppendStatus, FileReader};
use crate::io::index_cache::IndexCache;
use crate::io::line_index::LineIndex;
use crate::io::line_sampler::sample_line_count;
use crate::io::wrap::{format_json_line, wrap_line_chars, MAX_WRAP_INPUT_LEN};
use crate::io::wrap::{MAX_WRAP_INPUT_LEN, format_json_line};
// ─── Cancel-aware channel helpers ────────────────────────────────────────────
/// Send a message on `tx`, but abort if `cancel_rx` fires first.
///
/// Uses `crossbeam_channel::select!` so the thread sleeps efficiently instead
/// of busy-looping. Used for terminal messages (Complete / Error) that must
/// not be silently dropped while the receiver is still alive.
fn send_cancelable<T>(
tx: &crossbeam_channel::Sender<T>,
msg: T,
cancel_rx: &crossbeam_channel::Receiver<()>,
) {
crossbeam_channel::select! {
send(tx, msg) -> _ => {}
recv(cancel_rx) -> _ => {}
}
}
// ─── IndexerMessage ──────────────────────────────────────────────────────────
@@ -102,7 +120,7 @@ impl VisualHeightIndex {
}
}
fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self {
pub fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self {
self.json_format = json_format;
self.terminal_width = terminal_width;
self
@@ -159,6 +177,31 @@ impl VisualHeightIndex {
self.total_visual_rows += h as u64;
}
}
/// Replace the visual height of the last logical line. O(1).
///
/// Must be called **before** `extend_from_heights` so that the last line
/// index still refers to the pre-extension line.
pub fn replace_last_line_height(&mut self, new_height: usize) {
let n = self.prefix_sums.len();
if n < 2 {
return;
}
let last_line = n - 2;
let old_height = self.prefix_sums[last_line + 1] - self.prefix_sums[last_line];
let new_height = new_height as u64;
if new_height == old_height {
return;
}
let delta = new_height.abs_diff(old_height);
if new_height > old_height {
self.prefix_sums[last_line + 1] += delta;
self.total_visual_rows += delta;
} else {
self.prefix_sums[last_line + 1] -= delta;
self.total_visual_rows -= delta;
}
}
}
// ─── VisualHeightRebuildResult ────────────────────────────────────────────────
@@ -180,6 +223,9 @@ pub fn compute_line_visual_height(
}
if json_format {
let formatted = format_json_line(line_text);
if formatted.len() > MAX_WRAP_INPUT_LEN {
return 1;
}
compute_text_visual_height(&formatted, terminal_width)
} else {
compute_text_visual_height(line_text, terminal_width)
@@ -189,29 +235,11 @@ pub fn compute_line_visual_height(
fn compute_text_visual_height(text: &str, width: usize) -> usize {
let mut height = 0;
for sub_line in text.split('\n') {
height += wrap_line_chars(sub_line, width).len();
height += crate::io::wrap::wrap_line_count(sub_line, width);
}
height.max(1)
}
fn compute_visual_heights(
reader: &FileReader,
terminal_width: usize,
json_format: bool,
) -> Vec<usize> {
let line_count = reader.line_count();
let mut visual_heights = Vec::with_capacity(line_count);
for i in 0..line_count {
let line_text = reader.get_line(i).unwrap_or("");
visual_heights.push(compute_line_visual_height(
line_text,
terminal_width,
json_format,
));
}
visual_heights
}
// ─── ReaderState ─────────────────────────────────────────────────────────────
pub enum ReaderState {
@@ -235,8 +263,8 @@ pub enum ReaderState {
pub fn spawn_indexer(
path: PathBuf,
generation: u64,
terminal_width: usize,
json_format: bool,
_terminal_width: usize,
_json_format: bool,
cancel_rx: crossbeam_channel::Receiver<()>,
) -> crossbeam_channel::Receiver<IndexerMessage> {
let (tx, rx) = crossbeam_channel::bounded(10);
@@ -245,20 +273,28 @@ pub fn spawn_indexer(
let file = match std::fs::File::open(&path) {
Ok(f) => f,
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
});
},
&cancel_rx,
);
return;
}
};
let target_len = match file.metadata() {
Ok(m) => m.len(),
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
});
},
&cancel_rx,
);
return;
}
};
@@ -275,10 +311,14 @@ pub fn spawn_indexer(
let buf = match buf_reader.fill_buf() {
Ok(b) => b,
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
});
},
&cancel_rx,
);
return;
}
};
@@ -310,7 +350,7 @@ pub fn spawn_indexer(
}
if target_len > 0 {
let percent = (chunk_offset as f64 / target_len as f64) * 100.0;
let _ = tx.send(IndexerMessage::Progress {
let _ = tx.try_send(IndexerMessage::Progress {
generation,
percent,
lines_scanned: newline_count as u64,
@@ -361,18 +401,26 @@ pub fn spawn_indexer(
Ok(_) | Err(_) => None,
},
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
});
},
&cancel_rx,
);
return;
}
},
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
send_cancelable(
&tx,
IndexerMessage::Error {
generation,
message: e.to_string(),
});
},
&cancel_rx,
);
return;
}
}
@@ -384,18 +432,15 @@ pub fn spawn_indexer(
let reader = FileReader::from_parts(path, mmap, line_index);
let visual_height_index = if terminal_width > 0 {
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format);
Some(VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width))
} else {
None
};
let _ = tx.send(IndexerMessage::Complete {
send_cancelable(
&tx,
IndexerMessage::Complete {
generation,
reader,
visual_height_index,
});
visual_height_index: None,
},
&cancel_rx,
);
});
rx
@@ -421,41 +466,67 @@ pub fn spawn_visual_height_rebuild(
Err(_) => return,
};
let mut reader = std::io::BufReader::with_capacity(64 * 1024, file);
let mut visual_heights = Vec::with_capacity(line_index.line_count());
let mut line_buf = Vec::new();
let mmap = match unsafe { memmap2::Mmap::map(&file) } {
Ok(m) => match file.metadata() {
Ok(meta) if meta.len() >= m.len() as u64 => m,
_ => return,
},
Err(_) => return,
};
let data = mmap.as_ref();
let total_lines = line_index.line_count();
let mut visual_heights: Vec<usize> = Vec::with_capacity(total_lines);
let mut line_start = 0usize;
let mut bytes_since_cancel_check = 0usize;
loop {
if bytes_since_cancel_check >= 1_000_000 {
bytes_since_cancel_check = 0;
if cancel_rx.try_recv().is_ok() {
return;
}
}
line_buf.clear();
match std::io::BufRead::read_until(&mut reader, b'\n', &mut line_buf) {
Ok(0) => break,
Ok(_) => {
let line_text = std::str::from_utf8(&line_buf)
.ok()
.map(|s| s.trim_end_matches(['\r', '\n']))
.unwrap_or("");
let newline_rel = memchr::memchr(b'\n', &data[line_start..]);
let line_end = match newline_rel {
Some(p) => line_start + p,
None => data.len(),
};
let mut slice_end = line_end;
if slice_end > line_start && data[slice_end - 1] == b'\r' {
slice_end -= 1;
}
let line_text = std::str::from_utf8(&data[line_start..slice_end]).unwrap_or("");
visual_heights.push(compute_line_visual_height(
line_text,
terminal_width,
json_format,
));
}
Err(_) => return,
bytes_since_cancel_check =
bytes_since_cancel_check.saturating_add(line_end - line_start + 1);
match newline_rel {
Some(_) => line_start = line_end + 1,
None => break,
}
}
if visual_heights.len() != line_index.line_count() {
if visual_heights.len() != total_lines {
return;
}
let index =
VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width);
let _ = tx.send(VisualHeightRebuildResult { generation, index });
send_cancelable(
&tx,
VisualHeightRebuildResult { generation, index },
&cancel_rx,
);
});
rx
@@ -752,7 +823,7 @@ impl ProgressiveFileReader {
pub fn start_visual_height_rebuild(&mut self, terminal_width: usize, json_format: bool) {
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
let _ = tx.send(());
let _ = tx.try_send(());
}
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
@@ -807,10 +878,10 @@ impl ProgressiveFileReader {
impl Drop for ProgressiveFileReader {
fn drop(&mut self) {
if let Some(tx) = &self.cancel_tx {
let _ = tx.send(());
let _ = tx.try_send(());
}
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
let _ = tx.send(());
let _ = tx.try_send(());
}
}
}
@@ -1068,12 +1139,10 @@ mod tests {
assert_eq!(reader.get_line(0), Some("line1"));
assert_eq!(reader.get_line(1), Some("line2"));
assert_eq!(reader.get_line(2), Some("line3"));
assert!(visual_height_index.is_some());
let idx = visual_height_index.unwrap();
assert_eq!(idx.total_visual_rows(), 3);
assert_eq!(idx.visual_row_to_logical_row(0), 0);
assert_eq!(idx.visual_row_to_logical_row(1), 1);
assert_eq!(idx.visual_row_to_logical_row(2), 2);
assert!(
visual_height_index.is_none(),
"spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
);
}
other => panic!("expected Complete, got {:?}", other),
}
@@ -1274,12 +1343,15 @@ mod tests {
match rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap() {
IndexerMessage::Progress { .. } => continue,
IndexerMessage::Complete {
reader,
visual_height_index,
..
} => {
let idx = visual_height_index.expect("should have visual height index");
assert_eq!(idx.visual_height_of_line(0), 1);
assert_eq!(idx.visual_height_of_line(1), 1);
assert_eq!(reader.line_count(), 2);
assert!(
visual_height_index.is_none(),
"spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
);
break;
}
other => panic!("expected Complete, got {:?}", other),
@@ -1339,6 +1411,85 @@ mod tests {
assert_eq!(idx.total_visual_rows(), 6);
}
#[test]
fn test_replace_last_line_height_increase() {
let heights = [2, 3];
let mut idx = VisualHeightIndex::build(&heights);
assert_eq!(idx.total_visual_rows(), 5);
assert_eq!(idx.visual_height_of_line(1), 3);
idx.replace_last_line_height(7);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 7);
assert_eq!(idx.total_visual_rows(), 9);
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
}
#[test]
fn test_replace_last_line_height_decrease() {
let heights = [2, 5];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(1);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 1);
assert_eq!(idx.total_visual_rows(), 3);
}
#[test]
fn test_replace_last_line_height_same_is_noop() {
let heights = [2, 3];
let mut idx = VisualHeightIndex::build(&heights);
let total_before = idx.total_visual_rows();
idx.replace_last_line_height(3);
assert_eq!(idx.total_visual_rows(), total_before);
}
#[test]
fn test_replace_last_line_height_then_extend() {
let heights = [2, 1];
let mut idx = VisualHeightIndex::build(&heights);
assert_eq!(idx.total_visual_rows(), 3);
idx.replace_last_line_height(4);
idx.extend_from_heights(&[3]);
assert_eq!(idx.line_count(), 3);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.visual_height_of_line(1), 4);
assert_eq!(idx.visual_height_of_line(2), 3);
assert_eq!(idx.total_visual_rows(), 9);
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
assert_eq!(idx.cursor_to_first_visual_row(2), 6);
}
#[test]
fn test_replace_last_line_height_single_line() {
let heights = [5];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(2);
assert_eq!(idx.visual_height_of_line(0), 2);
assert_eq!(idx.total_visual_rows(), 2);
}
#[test]
fn test_replace_last_line_height_empty_index() {
let heights: [usize; 0] = [];
let mut idx = VisualHeightIndex::build(&heights);
idx.replace_last_line_height(5);
assert_eq!(idx.total_visual_rows(), 0);
}
#[test]
fn test_spawn_indexer_file_truncated_during_scan() {
let mut content = Vec::new();
@@ -1390,13 +1541,7 @@ mod tests {
}
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_visual_height_rebuild(
f.path().to_path_buf(),
1,
80,
false,
cancel_rx,
);
let rx = spawn_visual_height_rebuild(f.path().to_path_buf(), 1, 80, false, cancel_rx);
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
match result {
@@ -1405,4 +1550,106 @@ mod tests {
Ok(_) => panic!("should have been discarded due to line count mismatch"),
}
}
#[test]
fn test_send_cancelable_delivers_on_empty_channel() {
let (tx, rx) = crossbeam_channel::bounded(2);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
send_cancelable(&tx, 42, &cancel_rx);
assert_eq!(rx.try_recv(), Ok(42));
}
#[test]
fn test_send_cancelable_aborts_on_cancel() {
let (tx, rx) = crossbeam_channel::bounded(1);
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
tx.send("filler").unwrap();
let handle = std::thread::spawn(move || {
send_cancelable(&tx, "important", &cancel_rx);
});
cancel_tx.send(()).unwrap();
handle.join().unwrap();
assert_eq!(rx.try_recv(), Ok("filler"));
}
#[test]
fn test_send_cancelable_drains_when_room_available() {
let (tx, rx) = crossbeam_channel::bounded(1);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
tx.send("first").unwrap();
let rx_clone = rx.clone();
let handle = std::thread::spawn(move || {
std::thread::sleep(std::time::Duration::from_millis(50));
let _ = rx_clone.try_recv();
});
send_cancelable(&tx, "second", &cancel_rx);
handle.join().unwrap();
assert_eq!(rx.try_recv(), Ok("second"));
}
#[test]
fn test_progress_try_send_does_not_block_full_channel() {
let mut content = Vec::new();
for i in 0..50_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
let mut got_complete = false;
let timeout = std::time::Duration::from_secs(15);
let start = std::time::Instant::now();
while start.elapsed() < timeout {
match rx.recv_timeout(std::time::Duration::from_secs(1)) {
Ok(IndexerMessage::Progress { .. }) => {}
Ok(IndexerMessage::Complete { .. }) => {
got_complete = true;
break;
}
Ok(IndexerMessage::Error { message, .. }) => {
panic!("unexpected error: {}", message);
}
Err(e) => panic!("recv error: {:?}", e),
}
}
assert!(
got_complete,
"indexer should complete even when Progress fills channel"
);
}
#[test]
fn test_indexer_cancel_with_full_channel() {
let mut content = Vec::new();
for i in 0..500_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
cancel_tx.send(()).unwrap();
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
match result {
Err(crossbeam_channel::RecvTimeoutError::Timeout)
| Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Ok(IndexerMessage::Complete { .. }) => {}
Ok(IndexerMessage::Error { .. }) => {}
Ok(IndexerMessage::Progress { .. }) => {}
}
}
}
+67 -7
View File
@@ -56,9 +56,15 @@ impl LruReadCache {
/// on a hit, or fills a cache slot on a miss. Cross-block reads go through
/// the spill buffer and are not cached.
pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> {
if len == 0 {
return Ok(&[]);
}
let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1);
let request_end = offset.saturating_add(len as u64);
let block_end = aligned_key + BLOCK_ALIGN as u64;
let request_end = offset.checked_add(len as u64).ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidInput, "read range overflows u64")
})?;
let block_end = aligned_key.saturating_add(BLOCK_ALIGN as u64);
if request_end > block_end {
self.spill_buf.resize(len, 0);
@@ -74,7 +80,8 @@ impl LruReadCache {
}
let hit_idx = self.slots.iter().position(|slot| {
slot.block_offset == aligned_key && request_end <= slot.block_offset + slot.len as u64
let slot_end = slot.block_offset.saturating_add(slot.len as u64);
slot.len > 0 && slot.block_offset == aligned_key && request_end <= slot_end
});
if let Some(idx) = hit_idx {
@@ -96,8 +103,8 @@ impl LruReadCache {
let slot = &mut self.slots[evict_idx];
let bytes_read = file.read_at(&mut slot.buf, aligned_key)?;
// Note: get(file, 0, 0) on an empty file now returns Err (old code returned Ok(&[])).
// No callers pass len == 0, so this is a safe semantic change.
// Non-empty reads that return 0 are EOF. Zero-length reads are handled above
// as a successful no-op.
if bytes_read == 0 {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
}
@@ -107,7 +114,8 @@ impl LruReadCache {
slot.last_access = self.tick;
self.tick += 1;
if request_end > aligned_key + bytes_read as u64 {
let bytes_end = aligned_key.saturating_add(bytes_read as u64);
if request_end > bytes_end {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
}
@@ -118,7 +126,9 @@ impl LruReadCache {
/// Invalidate all cache slots and the spill buffer.
pub fn clear(&mut self) {
for slot in &mut self.slots {
slot.block_offset = 0;
slot.len = 0;
slot.last_access = 0;
}
self.spill_len = 0;
}
@@ -314,9 +324,11 @@ mod tests {
cache.clear();
// All slots should have len == 0.
// All slots should be fully reset.
for slot in &cache.slots {
assert_eq!(slot.block_offset, 0);
assert_eq!(slot.len, 0);
assert_eq!(slot.last_access, 0);
}
assert_eq!(cache.spill_len, 0);
@@ -432,4 +444,52 @@ mod tests {
assert_eq!(&line2[..4090], &[b'B'; 4090]);
assert_eq!(line2[4090], b'\n');
}
#[test]
fn zero_len_read_is_noop_on_fresh_cache() {
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, 0, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, 0);
assert!(cache.slots.iter().all(|s| s.len == 0));
}
#[test]
fn zero_len_read_is_noop_on_populated_cache() {
let f = make_file(b"abc");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
cache.get(&file, 0, 1).unwrap();
let tick_before = cache.tick;
let result = cache.get(&file, 0, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, tick_before);
}
#[test]
fn zero_len_read_at_max_offset_is_ok() {
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, u64::MAX, 0).unwrap();
assert!(result.is_empty());
assert_eq!(cache.tick, 0);
assert!(cache.slots.iter().all(|s| s.len == 0));
}
#[test]
fn nonzero_read_range_overflow_returns_invalid_input() {
let f = make_file(b"abc");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let err = cache.get(&file, u64::MAX, 1).unwrap_err();
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
}
}
+302 -21
View File
@@ -1,10 +1,21 @@
use std::borrow::Cow;
/// Maximum input length for wrap/format operations (10 MB).
/// Lines exceeding this are returned as-is to avoid pathological cases.
/// Callers should check against this constant before invoking `wrap_line_chars`
/// to avoid pathological cases on oversized lines.
pub const MAX_WRAP_INPUT_LEN: usize = 10 * 1024 * 1024;
/// Split a line into chunks of exactly `width` characters (display columns).
/// Column spacing for tab stop alignment.
const TAB_WIDTH: usize = 4;
/// Split a line into chunks of exactly `width` display columns.
/// For a log viewer, we want character-level wrapping, not word-level.
/// Uses `unicode-width` for correct CJK/emoji/zero-width handling.
/// Tab characters expand to the next tab-stop boundary and split across
/// rows when the expansion exceeds the remaining width.
pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
use unicode_width::UnicodeWidthChar;
if width == 0 {
return vec![String::new()];
}
@@ -15,23 +26,42 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
let mut row = String::new();
let mut col = 0;
for ch in line.chars() {
let w = if ch == '\t' { 4 } else { 1 };
if ch == '\t' {
let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
let mut remaining = tab_stop;
while remaining > 0 {
let avail = width.saturating_sub(col);
let fill = remaining.min(avail);
for _ in 0..fill {
row.push(' ');
}
col += fill;
remaining -= fill;
if col >= width {
result.push(std::mem::take(&mut row));
col = 0;
}
}
} else {
let w = if ch.is_control() {
// Control characters (except tab): width 0, still pushed to preserve content.
// Visible rendering is the caller's responsibility.
0
} else {
ch.width().unwrap_or(0)
};
if col + w > width && !row.is_empty() {
result.push(std::mem::take(&mut row));
col = 0;
}
if ch == '\t' {
row.push_str(" ");
col += 4;
} else {
row.push(ch);
col += w;
}
if col >= width {
result.push(std::mem::take(&mut row));
col = 0;
}
}
}
if !row.is_empty() {
result.push(row);
}
@@ -41,21 +71,81 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
result
}
/// Format a line as pretty-printed JSON if it's a JSON Object.
/// Returns the original line unchanged for non-JSON or non-Object content.
pub fn format_json_line(line: &str) -> String {
if line.trim().is_empty() {
return String::new();
/// Count wrapped rows for a line without allocating the wrapped strings.
/// MUST produce the same count as `wrap_line_chars(line, width).len()`.
pub fn wrap_line_count(line: &str, width: usize) -> usize {
use unicode_width::UnicodeWidthChar;
if width == 0 || line.is_empty() {
return 1;
}
// Quick pre-check: only try parsing if it starts with '{'
if !line.trim_start().starts_with('{') {
return line.to_string();
let mut count = 0usize;
let mut col = 0usize;
let mut row_has_content = false;
for ch in line.chars() {
if ch == '\t' {
let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
let mut remaining = tab_stop;
while remaining > 0 {
let avail = width.saturating_sub(col);
let fill = remaining.min(avail);
col += fill;
remaining -= fill;
if col >= width {
count += 1;
col = 0;
row_has_content = false;
} else {
row_has_content = true;
}
}
} else {
let w = if ch.is_control() {
0
} else {
ch.width().unwrap_or(0)
};
if col + w > width && row_has_content {
count += 1;
col = 0;
}
col += w;
if col >= width {
count += 1;
col = 0;
row_has_content = false;
} else {
row_has_content = true;
}
}
}
if row_has_content {
count += 1;
}
if count == 0 {
count = 1;
}
count
}
/// Format a line as pretty-printed JSON if it's a JSON Object.
/// Returns the original line borrowed for non-JSON or non-Object content,
/// only allocating when pretty-printing actually happens.
pub fn format_json_line(line: &str) -> Cow<'_, str> {
let trimmed = line.trim();
if trimmed.is_empty() {
return Cow::Borrowed("");
}
if !trimmed.starts_with('{') {
return Cow::Borrowed(line);
}
match serde_json::from_str::<serde_json::Value>(line) {
Ok(value) if value.is_object() => {
serde_json::to_string_pretty(&value).unwrap_or_else(|_| line.to_string())
}
_ => line.to_string(),
Ok(value) if value.is_object() => match serde_json::to_string_pretty(&value) {
Ok(s) => Cow::Owned(s),
Err(_) => Cow::Borrowed(line),
},
_ => Cow::Borrowed(line),
}
}
@@ -96,7 +186,7 @@ mod tests {
#[test]
fn test_wrap_with_tab() {
let result = wrap_line_chars("a\tb", 4);
assert_eq!(result, vec!["a", " ", "b"]);
assert_eq!(result, vec!["a ", "b"]);
}
#[test]
@@ -132,4 +222,195 @@ mod tests {
fn test_max_wrap_input_len_constant() {
assert_eq!(MAX_WRAP_INPUT_LEN, 10 * 1024 * 1024);
}
#[test]
fn test_wrap_cjk_chars() {
let result = wrap_line_chars("你好", 3);
assert_eq!(result, vec!["", ""]);
}
#[test]
fn test_wrap_cjk_ascii_mixed() {
let result = wrap_line_chars("a你好", 4);
assert_eq!(result, vec!["a你", ""]);
}
#[test]
fn test_wrap_zero_width_char() {
let result = wrap_line_chars("a\u{200B}b", 2);
assert_eq!(result, vec!["a\u{200B}b"]);
}
#[test]
fn test_wrap_emoji() {
let result = wrap_line_chars("😀a", 3);
assert_eq!(result, vec!["😀a"]);
}
#[test]
fn test_wrap_emoji_exact_wrap() {
let result = wrap_line_chars("😀a", 2);
assert_eq!(result, vec!["😀", "a"]);
}
#[test]
fn test_wrap_combining_mark() {
// Scalar-width wrapping: combining mark (width 0) stays with next base char,
// not the preceding one, because the base char already triggered a flush.
let result = wrap_line_chars("a\u{0301}b", 1);
assert_eq!(result, vec!["a", "\u{0301}b"]);
}
#[test]
fn test_wrap_cjk_width_one() {
let result = wrap_line_chars("你好", 1);
assert_eq!(result, vec!["", ""]);
}
#[test]
fn test_tab_narrow_width() {
let result = wrap_line_chars("\t", 2);
assert_eq!(result, vec![" ", " "]);
let result = wrap_line_chars("\t", 1);
assert_eq!(result, vec![" ", " ", " ", " "]);
}
#[test]
fn test_tab_stop_alignment() {
assert_eq!(wrap_line_chars("a\tb", 8), vec!["a b"]);
assert_eq!(wrap_line_chars("ab\t", 4), vec!["ab "]);
assert_eq!(wrap_line_chars("abc\tb", 8), vec!["abc b"]);
}
#[test]
fn test_tab_at_line_boundary() {
let result = wrap_line_chars("a\tb", 4);
assert_eq!(result, vec!["a ", "b"]);
}
#[test]
fn test_tab_regression_ab_tab() {
let result = wrap_line_chars("ab\t", 4);
assert_eq!(result, vec!["ab "]);
}
#[test]
fn test_wrap_count_matches_chars_empty() {
assert_eq!(wrap_line_count("", 80), wrap_line_chars("", 80).len());
}
#[test]
fn test_wrap_count_matches_chars_zero_width() {
assert_eq!(
wrap_line_count("hello", 0),
wrap_line_chars("hello", 0).len()
);
}
#[test]
fn test_wrap_count_matches_chars_ascii_short() {
for width in 1..20 {
assert_eq!(
wrap_line_count("hello world", width),
wrap_line_chars("hello world", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_cjk() {
for width in 1..10 {
assert_eq!(
wrap_line_count("你好世界", width),
wrap_line_chars("你好世界", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_emoji() {
for width in 1..6 {
assert_eq!(
wrap_line_count("😀👨‍👩‍👧", width),
wrap_line_chars("😀👨‍👩‍👧", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_combining_mark() {
for width in 1..6 {
assert_eq!(
wrap_line_count("a\u{0301}b\u{0301}c", width),
wrap_line_chars("a\u{0301}b\u{0301}c", width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_tab_variants() {
let inputs = ["\t", "a\tb", "ab\t", "\t\t", "abc\tb", "a\t\t\tb"];
for input in inputs {
for width in 1..9 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"input={input:?} width={width}"
);
}
}
}
#[test]
fn test_wrap_count_matches_chars_mixed() {
let input = "log: 你好 😀\tvalue=true [ERROR]";
for width in 1..40 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_zero_width_chars() {
let input = "a\u{200B}b\u{200C}c\u{200D}d";
for width in 1..8 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_control_chars() {
let input = "a\x07b\x1Fc";
for width in 1..8 {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
#[test]
fn test_wrap_count_matches_chars_long_log_line() {
let input = "2025-01-14T10:23:45.123Z [INFO] [auth] request handled status=200 \
latency=0.234s endpoint=/api/v1/users user=user_42 request_id=req_abc123def456";
for width in [10, 20, 40, 60, 80, 100, 120] {
assert_eq!(
wrap_line_count(input, width),
wrap_line_chars(input, width).len(),
"width={width}"
);
}
}
}
+306 -90
View File
@@ -16,7 +16,14 @@
// 类似于 Python 的 dict、JavaScript 的 Object/Map、Java 的 HashMap。
// 它存储键值对(key-value pairs),可以通过键快速查找对应的值。
// 这里用 HashMap<String, Value> 来存储 JSON 中除 timestamp/level 之外的其他字段。
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
// serde::de 中的 Visitor / MapAccess 允许我们自定义 JSON 对象的反序列化过程。
// 默认的 serde_json::from_str::<HashMap<_, _>>() 遇到重复键时会采用"后者覆盖前者"last-wins),
// 前面的值被静默丢弃。这里我们通过自定义 Visitor 在反序列化过程中逐个观察 key-value 对,
// 在保持 last-wins 行为的同时,将重复 key 的所有值记录到 DuplicateKey 中。
use serde::Deserializer;
use serde::de::{MapAccess, Visitor};
// serde_json::Value — 来自 serde_json 库(Rust 中最流行的 JSON 处理库)。
// Value 是一个枚举类型,可以表示任意 JSON 值:
@@ -32,7 +39,20 @@ use serde_json::Value;
// ─── 引入项目内部类型 ──────────────────────────────────────────────────────
// crate 表示"当前项目(crate"。
// types 模块中定义了 LogEntry(一条日志记录)和 LogLevel(日志级别,如 INFO/ERROR)。
use crate::types::{LogEntry, LogLevel};
use crate::types::{DuplicateKey, LogEntry, LogLevel};
// ─── strip_bom 辅助函数 ──────────────────────────────────────────────────
// 剥离行首的 UTF-8 BOMByte Order Mark, U+FEFF)。
//
// Windows 环境和某些导出工具生成的文件会在行首插入 BOM,
// 而 serde_json 不接受 BOM 前缀的 JSON 文本(会报 "expected value" 错误)。
// 只剥离一个前导 BOM,不处理多个 BOM 或行内 BOM(那些是畸形输入)。
//
// 参数: line: &str — 输入字符串切片。
// 返回: &str — 去掉 BOM 后的字符串切片(借用原始字符串,零分配)。
fn strip_bom(line: &str) -> &str {
line.strip_prefix('\u{FEFF}').unwrap_or(line)
}
// ─── detect_json_log 函数 ──────────────────────────────────────────────────
// 检测一行文本是否是一个 JSON 对象。
@@ -63,7 +83,92 @@ pub fn detect_json_log(line: &str) -> bool {
// 则匹配成功。_ 是通配符,表示"不关心对象里面的具体内容"。
//
// 如果匹配到 Ok(Value::Object(_)) 返回 true,否则返回 false。
matches!(serde_json::from_str::<Value>(line), Ok(Value::Object(_)))
matches!(
serde_json::from_str::<Value>(strip_bom(line)),
Ok(Value::Object(_))
)
}
// ─── DuplicateKeyVisitor ──────────────────────────────────────────────────
// 自定义 serde Visitor,在反序列化 JSON 对象时检测重复 key。
//
// 工作原理:
// serde 的 MapAccess trait 允许我们逐个遍历 JSON 对象的 key-value 对。
// 每读到一个 (key, value),我们:
// 1. 检查这个 key 是否已经见过(通过 HashSet)
// 2. 如果是重复 key,记录到 Vec<DuplicateKey> 中(包含所有出现过的值)
// 3. 将 key-value 插入 Maplast-wins,与 serde_json 默认行为一致)
//
// 这样既保持了兼容性(last-wins),又不丢失信息(所有值都记录在 DuplicateKey 中)。
struct DuplicateKeyVisitor;
impl<'de> Visitor<'de> for DuplicateKeyVisitor {
// 返回类型:(serde_json::Map, 重复 key 列表)
type Value = (serde_json::Map<String, Value>, Vec<DuplicateKey>);
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("a JSON object")
}
fn visit_map<A>(self, mut access: A) -> Result<Self::Value, A::Error>
where
A: MapAccess<'de>,
{
let mut map = serde_json::Map::new();
let mut seen = HashSet::new();
let mut duplicates: Vec<DuplicateKey> = Vec::new();
while let Some((key, value)) = access.next_entry::<String, Value>()? {
if !seen.insert(key.clone()) {
// 重复 key:将之前 map 中的值和当前值都记录下来
if let Some(existing) = duplicates.iter_mut().find(|d| d.key == key) {
// 同一个 key 第三次及以上出现:追加当前值
existing.values.push(value.clone());
} else {
// 同一个 key 第二次出现:记录第一次的值 + 当前值
let prev_value = map.get(&key).cloned().unwrap_or(Value::Null);
duplicates.push(DuplicateKey {
key: key.clone(),
values: vec![prev_value, value.clone()],
});
}
}
// last-wins:后出现的值覆盖前面的值,与 serde_json 默认行为一致
map.insert(key, value);
}
Ok((map, duplicates))
}
}
/// 使用自定义 Visitor 解析 JSON 对象,同时检测重复 key。
///
/// 返回 (serde_json::Map, Vec<DuplicateKey>)
/// - Map 中存储所有 key-value(重复 key 取 last-wins
/// - Vec 中记录所有重复 key 及其全部值
fn parse_json_object_with_duplicates(
json: &str,
) -> Option<(serde_json::Map<String, Value>, Vec<DuplicateKey>)> {
let mut deserializer = serde_json::Deserializer::from_str(json);
deserializer.deserialize_map(DuplicateKeyVisitor).ok()
}
// ─── take_string_field_from_map 辅助函数 ──────────────────────────────────
// 从 serde_json::Map 中安全提取字符串字段。
// 功能与原 take_string_field 相同,但操作 serde_json::Map 而非 HashMap。
fn take_string_field_from_map(
obj: &mut serde_json::Map<String, Value>,
keys: &[&str],
) -> Option<String> {
for key in keys {
if obj.get(*key).is_some_and(Value::is_string) {
let Some(Value::String(v)) = obj.remove(*key) else {
unreachable!("value was checked as string");
};
return Some(v);
}
}
None
}
// ─── parse_line 函数 ──────────────────────────────────────────────────────
@@ -73,107 +178,35 @@ pub fn detect_json_log(line: &str) -> bool {
// 返回: Option<LogEntry> — 解析成功返回 Some(LogEntry),失败或不合法返回 None。
// Option 是 Rust 的可选类型:Some(值) 表示有值,None 表示没有值。
pub fn parse_line(line: &str) -> Option<LogEntry> {
// ─── 跳过空行 ──────────────────────────────────────────────────────────
// line.trim() 去除首尾空白字符(空格、制表符、换行符等)。
// .is_empty() 检查是否为空字符串。
// 如果去除空白后是空的,说明是空行,不需要解析,直接返回 None。
let line = strip_bom(line);
if line.trim().is_empty() {
return None;
}
// ─── 解析 JSON 为 HashMap ──────────────────────────────────────────────
// serde_json::from_str(line) 尝试将字符串解析为 JSON
// 由于我们声明了 HashMap<String, Value> 类型,Rust 会自动将 JSON 对象
// 转换为 HashMap,其中每个键是 String,每个值是 serde_json::Value。
//
// .ok() 将 Result 转换为 Option
// Ok(值) → Some(值)
// Err(_) → None
//
// 末尾的 ? 是"问号操作符"try operator),在这里的作用是:
// 如果 .ok() 返回 None(即 JSON 解析失败),则整个函数直接返回 None。
// 如果返回 Some(hashmap),则将 hashmap 取出并绑定到 fields 变量。
//
// let mut 表示这是一个"可变变量"mutable variable),
// 后续代码会修改这个 HashMap(从中删除已识别的字段)。
let mut fields: HashMap<String, Value> = serde_json::from_str(line).ok()?;
// ─── 使用自定义 Visitor 解析 JSON ──────────────────────────────────
// 通过 DuplicateKeyVisitor 反序列化,在保持 last-wins 的同时检测重复 key
// 返回的 (serde_json::Map, Vec<DuplicateKey>) 中:
// - Map 包含所有 key-value(重复 key 取最后一个值)
// - Vec 记录了所有重复 key 及其出现过的全部值
let (mut obj, duplicate_keys) = parse_json_object_with_duplicates(line)?;
// ─── 保存原始行内容 ──────────────────────────────────────────────────
// line.to_string() 将 &str(字符串切片引用)转换为 String(拥有所有权的字符串)。
// 保存原始行是为了在 UI 中显示未经修改的原始日志内容。
let raw_line = line.to_string();
// ─── 提取时间戳字段 ──────────────────────────────────────────────────
// 这段代码尝试从 JSON 中提取时间戳,逻辑如下:
//
// 1. ["timestamp", "time", "ts", "@timestamp"] — 候选键名数组。
// 不同日志系统使用不同的时间戳字段名,这里列出常见的几种。
//
// 2. .iter() — 创建数组的迭代器,可以逐个遍历元素。
//
// 3. .find_map(|key| fields.remove(*key)) — 对每个候选键名:
// - fields.remove(*key): 尝试从 HashMap 中移除该键并返回对应的值。
// 如果键不存在,remove 返回 None。
// *key 是解引用(deref),将 &str(引用)转换为 str,因为 remove 接受 &str 类型。
// - find_map: 遍历所有候选键,返回第一个 Some(值) 的结果。
// 即找到第一个存在的键就停止。
//
// 4. .and_then(|v| v.as_str().map(String::from)) — 如果找到了时间戳值:
// - v.as_str(): 尝试将 serde_json::Value 转换为 &str(字符串切片)。
// 如果 Value 不是字符串类型(比如是数字),返回 None。
// - .map(String::from): 如果是字符串,将其转换为 String(拥有所有权的字符串)。
// - and_then: 类似于 map,但用于"扁平化"嵌套的 Option。
// 如果 as_str() 返回 None,整个链返回 None。
let timestamp = ["timestamp", "time", "ts", "@timestamp"]
.iter()
.find_map(|key| fields.remove(*key))
.and_then(|v| v.as_str().map(String::from));
// ─── 提取日志级别字段 ──────────────────────────────────────────────
// 与时间戳提取类似,但多了一步:将字符串解析为 LogLevel 枚举。
let level = ["level", "lvl", "severity"]
.iter()
.find_map(|key| fields.remove(*key))
.and_then(|v| v.as_str().map(String::from))
// .map(|s| s.parse::<LogLevel>(...)) — 尝试将字符串解析为 LogLevel 枚举。
// parse::<LogLevel> 中的 ::<LogLevel> 是泛型参数(turbofish 语法),
// 指定我们要将字符串解析为 LogLevel 类型。
//
// .unwrap_or_else(|e| match e {}) — 错误处理:
// - 如果解析成功,直接返回 LogLevel 值。
// - 如果解析失败(字符串不匹配任何已知的日志级别),执行闭包。
// - |e| match e {}: 这个闭包接收解析错误 e,用 match e {} 进行"穷尽匹配"。
// 由于 LogLevel 的 parse 错误类型是一个空枚举(没有任何变体),
// match e {} 意味着"这个分支永远不会执行"unreachable)。
// 但实际上,如果 parse 失败,unwrap_or_else 不会执行这个闭包——
// 等等,这里有个细微之处:
// unwrap_or_else 只在 Err 时执行闭包,但 match e {} 对空枚举是合法的
// (因为空枚举没有任何可能的值,所以 match 是穷尽的)。
// 不过这里的实际效果是:如果 parse 失败,整个 .map() 返回 None
// (因为 unwrap_or_else 返回的类型是 LogLevel,而空 match 不会有返回值)。
//
// 实际上更准确的解释:parse() 的错误类型是 Infallible(不可失败的),
// 即解析总是成功。所以 unwrap_or_else 永远不会被执行。
// 但即使如此,unwrap_or_else 的闭包也需要类型正确,match e {} 满足这一点。
let timestamp =
take_string_field_from_map(&mut obj, &["timestamp", "time", "ts", "@timestamp"]);
let level = take_string_field_from_map(&mut obj, &["level", "lvl", "severity"])
.map(|s| s.parse::<LogLevel>().unwrap_or_else(|e| match e {}));
// ─── 构建 LogEntry 并返回 ──────────────────────────────────────────
// 此时 fields HashMap 中还剩下未被提取的字段(如 message、自定义字段等)。
// timestamp 和 level 已经从 fields 中移除了(通过 remove)。
//
// Some(LogEntry { ... }) — 使用结构体字面量创建 LogEntry 实例,
// 并用 Some() 包裹表示"有值"。
// serde_json::Map → HashMap:剩余字段转为 HashMap 存入 fields
let fields: HashMap<String, Value> = obj.into_iter().collect();
Some(LogEntry {
// line_number 设为 0,由调用者(如 parse_line_with_number)设置正确的值。
line_number: 0,
// 原始行内容。
raw_line,
// 时间戳(可能为 None,如果 JSON 中没有时间戳字段)。
timestamp,
// 日志级别(可能为 None,如果 JSON 中没有级别字段)。
level,
// 剩余的 JSON 字段(已移除 timestamp 和 level)。
fields,
duplicate_keys,
})
}
@@ -317,6 +350,13 @@ mod tests {
assert_eq!(parse_line(warn_line).unwrap().level, Some(LogLevel::Warn));
}
#[test]
// Regression: level field with surrounding whitespace should still be recognized.
fn test_level_whitespace_in_json() {
let line = r#"{"level":" WARN ","message":"test"}"#;
assert_eq!(parse_line(line).unwrap().level, Some(LogLevel::Warn));
}
#[test]
// 测试:所有候选时间戳键名(timestamp, time, ts, @timestamp)都能被识别。
fn test_timestamp_key_names() {
@@ -357,4 +397,180 @@ mod tests {
assert_eq!(entry.level, Some(LogLevel::Info), "failed for key: {key}");
}
}
#[test]
// 测试:数字类型的 level 值不应被提取,应保留在 fields 中。
fn test_numeric_level_preserved_in_fields() {
let line = r#"{"level":30,"message":"hello"}"#;
let entry = parse_line(line).unwrap();
// level 不是字符串,应返回 None。
assert!(entry.level.is_none());
// 数字 level 应保留在 fields 中,不被静默丢弃。
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
// message 仍正常存在。
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("hello".to_string()))
);
}
#[test]
// 测试:数字类型的 timestamp 值不应被提取,应保留在 fields 中。
fn test_numeric_timestamp_preserved_in_fields() {
let line = r#"{"timestamp":1718000000,"message":"hello"}"#;
let entry = parse_line(line).unwrap();
// timestamp 不是字符串,应返回 None。
assert!(entry.timestamp.is_none());
// 数字 timestamp 应保留在 fields 中。
assert_eq!(
entry.fields.get("timestamp"),
Some(&Value::Number(1718000000.into()))
);
}
#[test]
// 测试:当第一个候选键是数字时,应回退到下一个字符串类型的候选键。
fn test_fallback_to_string_key() {
let line = r#"{"level":30,"lvl":"INFO","message":"hello"}"#;
let entry = parse_line(line).unwrap();
// "level" 是数字,应跳过;"lvl" 是字符串,应成功提取。
assert_eq!(entry.level, Some(LogLevel::Info));
// 数字 "level" 保留在 fields 中。
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
// "lvl" 已被成功提取并从 fields 中移除。
assert!(entry.fields.get("lvl").is_none());
}
#[test]
// 测试:timestamp 的 fallback 行为 — 数字 timestamp 被保留,字符串 time 被提取。
fn test_timestamp_fallback_preserves_numeric() {
let line = r#"{"timestamp":1718000000,"time":"2024-01-01T00:00:00Z"}"#;
let entry = parse_line(line).unwrap();
// "timestamp" 是数字,跳过;"time" 是字符串,成功提取。
assert_eq!(entry.timestamp, Some("2024-01-01T00:00:00Z".to_string()));
// 数字 "timestamp" 保留在 fields 中。
assert_eq!(
entry.fields.get("timestamp"),
Some(&Value::Number(1718000000.into()))
);
// "time" 已被提取并从 fields 中移除。
assert!(entry.fields.get("time").is_none());
}
#[test]
fn test_bom_prefixed_json() {
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(entry.level, Some(LogLevel::Info));
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("hello".into()))
);
}
#[test]
fn test_bom_prefixed_detect() {
assert!(detect_json_log("\u{FEFF}{\"level\":\"INFO\"}"));
}
#[test]
fn test_bom_only_whitespace() {
assert!(parse_line("\u{FEFF} ").is_none());
}
#[test]
fn test_bom_stripped_from_raw_line() {
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(entry.raw_line, "{\"level\":\"INFO\",\"message\":\"hello\"}");
}
#[test]
fn test_internal_bom_not_stripped() {
let line = "{\"message\":\"\u{FEFF}hello\"}";
let entry = parse_line(line).unwrap();
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("\u{FEFF}hello".into()))
);
}
// ─── 重复 key 检测测试 ──────────────────────────────────────────────
#[test]
fn test_no_duplicate_keys_normal_json() {
let line = r#"{"level":"INFO","message":"hello"}"#;
let entry = parse_line(line).unwrap();
assert!(entry.duplicate_keys.is_empty());
}
#[test]
fn test_duplicate_message_key_detected() {
let line = r#"{"level":"INFO","message":"first","message":"second"}"#;
let entry = parse_line(line).unwrap();
// last-wins: fields 中保留第二个值
assert_eq!(
entry.fields.get("message"),
Some(&Value::String("second".into()))
);
// 重复 key 记录中包含所有值
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "message");
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
assert_eq!(
entry.duplicate_keys[0].values[0],
Value::String("first".into())
);
assert_eq!(
entry.duplicate_keys[0].values[1],
Value::String("second".into())
);
}
#[test]
fn test_duplicate_key_last_wins() {
let line = r#"{"msg":"a","msg":"b","msg":"c"}"#;
let entry = parse_line(line).unwrap();
// last-wins: 最终值是 "c"
assert_eq!(entry.fields.get("msg"), Some(&Value::String("c".into())));
// 三个值都被记录
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].values.len(), 3);
assert_eq!(entry.duplicate_keys[0].values[0], Value::String("a".into()));
assert_eq!(entry.duplicate_keys[0].values[1], Value::String("b".into()));
assert_eq!(entry.duplicate_keys[0].values[2], Value::String("c".into()));
}
#[test]
fn test_multiple_different_duplicate_keys() {
let line = r#"{"a":"1","b":"2","a":"3","b":"4"}"#;
let entry = parse_line(line).unwrap();
assert_eq!(entry.duplicate_keys.len(), 2);
let dup_a = entry.duplicate_keys.iter().find(|d| d.key == "a").unwrap();
let dup_b = entry.duplicate_keys.iter().find(|d| d.key == "b").unwrap();
assert_eq!(dup_a.values.len(), 2);
assert_eq!(dup_b.values.len(), 2);
}
#[test]
fn test_duplicate_level_key_detected() {
let line = r#"{"level":"INFO","level":"ERROR","message":"hello"}"#;
let entry = parse_line(line).unwrap();
// last-wins: level 被提取为 ERROR
assert_eq!(entry.level, Some(LogLevel::Error));
// 重复 key 被记录
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "level");
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
}
#[test]
fn test_duplicate_timestamp_key_detected() {
let line = r#"{"timestamp":"2024-01-01","timestamp":"2024-06-01"}"#;
let entry = parse_line(line).unwrap();
// last-wins: timestamp 提取为后者
assert_eq!(entry.timestamp, Some("2024-06-01".to_string()));
assert_eq!(entry.duplicate_keys.len(), 1);
assert_eq!(entry.duplicate_keys[0].key, "timestamp");
}
}
+43 -3
View File
@@ -100,13 +100,21 @@ fn detect_level_from_text(line: &str) -> Option<LogLevel> {
)
}
// ─── is_ident_char ─────────────────────────────────────────────────────────
/// Whether a byte looks like an ASCII identifier continuation character
/// (letter / digit / underscore). Log-level keywords must NOT be adjacent to
/// such characters to count as a valid word boundary.
fn is_ident_char(b: u8) -> bool {
b.is_ascii_alphanumeric() || b == b'_'
}
// ─── is_word_boundary ───────────────────────────────────────────────────────
/// Check that the match at `start..start+len` is surrounded by non-alphabetic
/// Check that the match at `start..start+len` is surrounded by non-identifier
/// characters (or the string edge).
fn is_word_boundary(text: &str, start: usize, len: usize) -> bool {
let before_ok = start == 0 || !text.as_bytes()[start - 1].is_ascii_alphabetic();
let before_ok = start == 0 || !is_ident_char(text.as_bytes()[start - 1]);
let after_idx = start + len;
let after_ok = after_idx >= text.len() || !text.as_bytes()[after_idx].is_ascii_alphabetic();
let after_ok = after_idx >= text.len() || !is_ident_char(text.as_bytes()[after_idx]);
before_ok && after_ok
}
@@ -209,4 +217,36 @@ mod tests {
let line = format!("{prefix} ERROR something");
assert_eq!(detect_level(&line), Some(LogLevel::Error));
}
#[test]
fn test_boundary_rejects_trailing_digits() {
assert_eq!(detect_level("ERROR123"), None);
assert_eq!(detect_level("WARN2: bad"), None);
assert_eq!(detect_level("ERR2"), None);
}
#[test]
fn test_boundary_rejects_underscore() {
assert_eq!(detect_level("INFO_foo"), None);
assert_eq!(detect_level("DBG_value=5"), None);
}
#[test]
fn test_boundary_rejects_leading_digits_and_underscore() {
assert_eq!(detect_level("123ERROR: fail"), None);
assert_eq!(detect_level("foo_ERROR: fail"), None);
assert_eq!(detect_level("1WRN"), None);
}
#[test]
fn test_boundary_accepts_valid_suffixes() {
assert_eq!(detect_level("ERROR: fail"), Some(LogLevel::Error));
assert_eq!(detect_level("[ERROR] fail"), Some(LogLevel::Error));
assert_eq!(detect_level("ERROR fail"), Some(LogLevel::Error));
}
#[test]
fn test_boundary_camel_case_regression() {
assert_eq!(detect_level("errorLevel"), None);
}
}
+50 -5
View File
@@ -71,13 +71,17 @@ impl FromStr for LogLevel {
// 接收一个字符串切片 &str,返回 Result<LogLevel, Infallible>。
// 由于 Err 类型是 Infallible,实际上返回值总是 Ok(LogLevel)。
fn from_str(s: &str) -> Result<Self, Self::Err> {
// `s.to_uppercase()` — 字符串转换为大写,实现不区分大小写的匹配
// `s.trim()` — 去除字符串前后的 Unicode 空白字符
// 例如 " WARN " → "WARN""\tINFO\n" → "INFO"。
let trimmed = s.trim();
// `trimmed.to_uppercase()` — 将 trimmed 后的字符串转换为大写,实现不区分大小写的匹配。
// 例如 "info"、"Info"、"INFO" 都会被转换为 "INFO"。
// 返回一个新的 String(堆分配)。
//
// `.as_str()` — 将 String 转换回 &str(字符串切片引用)。
// 因为 match 需要匹配 &str 而不是 String。
match s.to_uppercase().as_str() {
match trimmed.to_uppercase().as_str() {
// `|` 在 match 分支中表示"或"multiple patterns)。
// "ERROR" | "ERR" | "SEVERE" | "FATAL" 都匹配到 LogLevel::Error。
"ERROR" | "ERR" | "SEVERE" | "FATAL" => Ok(LogLevel::Error),
@@ -86,9 +90,9 @@ impl FromStr for LogLevel {
"DEBUG" | "DBG" => Ok(LogLevel::Debug),
"TRACE" | "TRC" => Ok(LogLevel::Trace),
// `_` 是通配符,匹配所有未被上面分支捕获的值。
// 对于未知级别,包装为 Unknown 并保存原始字符串。
// s.to_string() 将 &str 转换为 String(注意这里用原始 s,不是大写后的)。
_ => Ok(LogLevel::Unknown(s.to_string())),
// 对于未知级别,包装为 Unknown 并保存 trimmed 后的字符串。
// s.to_string() 将 &str 转换为 String(注意这里用 trimmed,不是原始 s)。
_ => Ok(LogLevel::Unknown(trimmed.to_string())),
}
}
}
@@ -119,6 +123,18 @@ impl fmt::Display for LogLevel {
}
// ─── LogEntry 结构体 ────────────────────────────────────────────────────────
/// 记录 JSON 日志中出现的重复 key 信息
///
/// 当 JSON 对象中同一个 key 出现多次时,serde_json 默认 last-wins(后值覆盖前值),
/// 前面的值会静默丢失。此结构记录所有重复出现的 key 及其全部值,供 UI 展示警告。
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct DuplicateKey {
/// 重复的 key 名称
pub key: String,
/// 该 key 出现的所有值(按出现顺序排列,最后一个值是 fields 中的最终值)
pub values: Vec<Value>,
}
/// 一行解析后的日志
///
/// 表示日志文件中经过解析器处理后的一行内容。
@@ -144,6 +160,9 @@ pub struct LogEntry {
/// HashMap<String, Value> 是一个字典,键是字段名,值是 JSON 值。
/// 例如 {"message": "hello", "request_id": "abc123"}。
pub fields: HashMap<String, Value>,
/// JSON 中重复出现的 key 记录(正常日志为空 Vec)
pub duplicate_keys: Vec<DuplicateKey>,
}
// ─── SearchResult 结构体 ────────────────────────────────────────────────────
@@ -288,6 +307,31 @@ mod tests {
);
}
#[test]
fn test_from_str_whitespace_trimmed() {
assert_eq!(" WARN ".parse::<LogLevel>(), Ok(LogLevel::Warn));
assert_eq!("\tINFO".parse::<LogLevel>(), Ok(LogLevel::Info));
assert_eq!("ERROR\n".parse::<LogLevel>(), Ok(LogLevel::Error));
assert_eq!(" debug ".parse::<LogLevel>(), Ok(LogLevel::Debug));
assert_eq!("\tTRACE\t".parse::<LogLevel>(), Ok(LogLevel::Trace));
}
#[test]
fn test_from_str_whitespace_unknown_trimmed() {
// Unknown stores the trimmed value, not the original.
assert_eq!(
" CUSTOM ".parse::<LogLevel>(),
Ok(LogLevel::Unknown("CUSTOM".into()))
);
// Pure whitespace becomes Unknown("").
assert_eq!(" ".parse::<LogLevel>(), Ok(LogLevel::Unknown("".into())));
// Internal whitespace is NOT collapsed.
assert_eq!(
"W ARN".parse::<LogLevel>(),
Ok(LogLevel::Unknown("W ARN".into()))
);
}
#[test]
// 测试:LogLevel 的 Display 输出格式是否正确。
fn test_display_output() {
@@ -319,6 +363,7 @@ mod tests {
timestamp: Some("2024-01-01T00:00:00".to_string()),
level: Some(LogLevel::Info),
fields,
duplicate_keys: vec![],
};
// 逐字段验证。
+111 -37
View File
@@ -1,7 +1,7 @@
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use crossbeam_channel::{bounded, Receiver, Sender};
use crossbeam_channel::{Receiver, Sender, bounded};
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use crate::error::Result;
@@ -13,6 +13,7 @@ pub enum FileEvent {
Truncated { new_size: u64 },
Rotated { new_inode: u64 },
Removed,
WatcherError { message: String },
}
// ─── get_inode ──────────────────────────────────────────────────────────────
@@ -34,6 +35,43 @@ struct WatchState {
last_inode: u64,
}
fn process_event(event: Event, watch_path: &Path, state: &mut WatchState) -> Option<FileEvent> {
if !event.paths.iter().any(|p| p == watch_path) {
return None;
}
match event.kind {
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
EventKind::Remove(_) => return Some(FileEvent::Removed),
_ => return None,
}
let current_inode = get_inode(watch_path).unwrap_or(0);
let current_size = std::fs::metadata(watch_path).map(|m| m.len()).unwrap_or(0);
if current_inode != 0 && state.last_inode != 0 && current_inode != state.last_inode {
state.last_inode = current_inode;
state.last_size = current_size;
Some(FileEvent::Rotated {
new_inode: current_inode,
})
} else if current_size > state.last_size {
state.last_size = current_size;
state.last_inode = current_inode;
Some(FileEvent::Appended {
new_size: current_size,
})
} else if current_size < state.last_size {
state.last_size = current_size;
state.last_inode = current_inode;
Some(FileEvent::Truncated {
new_size: current_size,
})
} else {
None
}
}
// ─── FileWatcher ────────────────────────────────────────────────────────────
pub struct FileWatcher {
rx: Receiver<FileEvent>,
@@ -56,50 +94,22 @@ impl FileWatcher {
notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| {
let event = match res {
Ok(e) => e,
Err(_) => return,
Err(error) => {
let _ = tx.try_send(FileEvent::WatcherError {
message: error.to_string(),
});
return;
}
};
match event.kind {
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
EventKind::Remove(_) => {
let _ = tx.try_send(FileEvent::Removed);
return;
}
_ => return,
}
if !event.paths.iter().any(|p| p == &watch_path) {
return;
}
let current_inode = get_inode(&watch_path).unwrap_or(0);
let current_size = std::fs::metadata(&watch_path).map(|m| m.len()).unwrap_or(0);
let mut st = state.lock().unwrap_or_else(|poison| {
// Recover from poisoned mutex — state only tracks last_size
// and last_inode for event dedup. Stale values at worst
// cause a duplicate event, which is harmless.
poison.into_inner()
});
if current_inode != 0 && st.last_inode != 0 && current_inode != st.last_inode {
let _ = tx.try_send(FileEvent::Rotated {
new_inode: current_inode,
});
st.last_inode = current_inode;
st.last_size = current_size;
} else if current_size > st.last_size {
let _ = tx.try_send(FileEvent::Appended {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
} else if current_size < st.last_size {
let _ = tx.try_send(FileEvent::Truncated {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
if let Some(fe) = process_event(event, &watch_path, &mut st) {
let _ = tx.try_send(fe);
}
})?;
@@ -144,6 +154,56 @@ mod tests {
events
}
#[test]
fn test_remove_wrong_path_ignored() {
let dir = tempfile::tempdir().expect("create temp dir");
let watched = dir.path().join("watched.log");
let other = dir.path().join("other.log");
std::fs::write(&watched, b"hello\n").expect("write watched");
std::fs::write(&other, b"other\n").expect("write other");
let mut state = WatchState {
last_size: 6,
last_inode: get_inode(&watched).unwrap_or(0),
};
let event = Event {
kind: EventKind::Remove(notify::event::RemoveKind::File),
paths: vec![other.clone()],
attrs: Default::default(),
};
let result = process_event(event, &watched, &mut state);
assert_eq!(
result, None,
"Remove for non-watched path should be ignored"
);
}
#[test]
fn test_remove_correct_path_emits_removed() {
let dir = tempfile::tempdir().expect("create temp dir");
let watched = dir.path().join("watched.log");
std::fs::write(&watched, b"hello\n").expect("write watched");
let mut state = WatchState {
last_size: 6,
last_inode: get_inode(&watched).unwrap_or(0),
};
let event = Event {
kind: EventKind::Remove(notify::event::RemoveKind::File),
paths: vec![watched.clone()],
attrs: Default::default(),
};
let result = process_event(event, &watched, &mut state);
assert!(
matches!(result, Some(FileEvent::Removed)),
"Remove for watched path should emit Removed"
);
}
#[test]
fn test_watcher_append() {
let dir = tempfile::tempdir().expect("create temp dir");
@@ -232,5 +292,19 @@ mod tests {
let d = FileEvent::Rotated { new_inode: 42 };
assert_ne!(a, d);
let e1 = FileEvent::WatcherError {
message: "io error".into(),
};
let e2 = FileEvent::WatcherError {
message: "io error".into(),
};
assert_eq!(e1, e2);
assert_ne!(
e1,
FileEvent::WatcherError {
message: "other".into()
}
);
}
}
+1
View File
@@ -11,6 +11,7 @@ anyhow.workspace = true
log-viewer-core.workspace = true
serde_json.workspace = true
crossbeam-channel.workspace = true
unicode-width.workspace = true
[dev-dependencies]
tempfile = { workspace = true }
+1828 -278
View File
File diff suppressed because it is too large Load Diff
+126 -15
View File
@@ -6,6 +6,7 @@ use crate::color::level_fg;
use log_viewer_core::config::ColorConfig;
use log_viewer_core::types::LogLevel;
#[allow(dead_code)]
pub(crate) fn build_line_spans(
gutter_text: String,
content_text: String,
@@ -28,6 +29,10 @@ pub(crate) fn build_line_spans(
])
}
fn is_cursor_visual_row(app: &App, logical_line: usize, visual_row: usize) -> bool {
logical_line == app.cursor_line && visual_row == app.cursor_sub_offset
}
pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
use ratatui::layout::{Constraint, Layout};
use ratatui::widgets::Paragraph;
@@ -88,9 +93,21 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
}
// ── Status bar ─────────────────────────────────────────────────
let status_text = if app.mode == AppMode::Settings {
" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"
} else if app.is_error() {
if app.mode == AppMode::Settings {
if let Some(ref err) = app.settings_error {
frame.render_widget(
Paragraph::new(err.as_str()).style(Style::default().fg(Color::Red)),
outer[2],
);
} else {
frame.render_widget(
Paragraph::new(" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"),
outer[2],
);
}
return;
}
let status_text = if app.is_error() {
" Press q to quit"
} else if app.is_loading() {
let pct = app.loading_progress().map_or(0, |p| p as usize);
@@ -111,7 +128,10 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
let name = app.file_name().unwrap_or("unknown");
let total = app.total_lines();
let cursor_display = if total == 0 { 0 } else { app.cursor_line + 1 };
let status = format!(" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit", name, cursor_display, total);
let status = format!(
" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit",
name, cursor_display, total
);
frame.render_widget(Paragraph::new(status), outer[2]);
return;
} else {
@@ -130,8 +150,12 @@ pub fn render_settings(frame: &mut ratatui::Frame, app: &mut App, area: ratatui:
let popup_w = ((area.width as u32 * 4 / 5).max(40)).min(area.width as u32) as u16;
let popup_h = ((area.height as u32 * 4 / 5).max(14)).min(area.height as u32) as u16;
let popup_x = area.width.saturating_sub(popup_w) / 2;
let popup_y = area.height.saturating_sub(popup_h) / 2;
let popup_x = area
.x
.saturating_add(area.width.saturating_sub(popup_w) / 2);
let popup_y = area
.y
.saturating_add(area.height.saturating_sub(popup_h) / 2);
let popup = ratatui::layout::Rect::new(popup_x, popup_y, popup_w, popup_h);
let block = Block::new().borders(Borders::ALL).title(" Color Settings ");
@@ -209,14 +233,8 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
} else {
0
};
let is_loading = app.is_loading();
let gutter_prefix_extra = if is_loading { 1 } else { 0 };
let gutter_width = if total_lines > 0 {
line_num_width + gutter_prefix_extra + 1 + 1
} else {
0
};
let gutter_width = app.gutter_width();
let actual_content_width = content_width.saturating_sub(gutter_width);
@@ -251,7 +269,7 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
break;
}
let is_cursor = logical_line == app.cursor_line;
let is_cursor = is_cursor_visual_row(app, logical_line, visual_row);
let level = entry.level.as_ref();
let bg_color = if is_cursor {
@@ -363,6 +381,17 @@ mod tests {
assert_eq!(line.spans[1].style.bg, Some(Color::Reset));
}
#[test]
fn test_cursor_highlight_predicate_uses_cursor_sub_offset() {
let mut app = App::new();
app.cursor_line = 4;
app.cursor_sub_offset = 2;
assert!(!is_cursor_visual_row(&app, 4, 0));
assert!(!is_cursor_visual_row(&app, 3, 2));
assert!(is_cursor_visual_row(&app, 4, 2));
}
fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer {
let backend = ratatui::backend::TestBackend::new(width, height);
let mut terminal = ratatui::Terminal::new(backend).unwrap();
@@ -469,7 +498,8 @@ mod tests {
let result = std::panic::catch_unwind(|| {
let data = std::fs::read(&path).unwrap();
let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data);
let _ = log_viewer_core::io::index_cache::IndexCache::save_with_hash(&path, &index, &data);
let _ =
log_viewer_core::io::index_cache::IndexCache::save_with_hash(&path, &index, &data);
let mut app = App::new();
app.load_file(path.to_str().unwrap()).unwrap();
@@ -493,4 +523,85 @@ mod tests {
let _ = std::fs::remove_file(&path);
assert!(result.is_ok());
}
// ── Issue #31: Settings popup area offset tests ────────────────
/// Helper: enter settings mode and render to buffer.
fn render_settings_to_buffer(
app: &mut App,
width: u16,
height: u16,
) -> ratatui::buffer::Buffer {
app.mode = crate::app::AppMode::Settings;
app.settings_draft = app.color_config.clone();
render_to_buffer(app, width, height)
}
/// Find the top-left corner of the popup border by scanning for '┌'.
fn find_popup_top_left(
buf: &ratatui::buffer::Buffer,
width: u16,
height: u16,
) -> Option<(u16, u16)> {
for row in 0..height {
for col in 0..width {
if buf.cell((col, row)).unwrap().symbol() == "" {
return Some((col, row));
}
}
}
None
}
#[test]
fn test_settings_popup_includes_area_offset() {
// In an 80x24 frame with Layout [Length(1), Min(1), Length(1)]:
// outer[0] = title bar -> y=0
// outer[1] = content -> y=1, height=22
// outer[2] = status bar -> y=23
// The popup is centered within outer[1], so its y must be >= outer[1].y (which is 1).
let mut app = App::new();
let buf = render_settings_to_buffer(&mut app, 80, 24);
let (_px, py) =
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
// outer[1].y == 1; the popup is centered inside a 22-row area,
// so popup_y must be at least 1 (not 0).
assert!(
py >= 1,
"popup top row should account for area.y offset, got y={py} (expected >= 1)"
);
}
#[test]
fn test_settings_popup_horizontal_centering_uses_area_x() {
// outer[1].x is 0 for this layout, so this mainly verifies the popup
// is centered and not shifted left. A non-zero area.x layout would
// need a different layout to trigger, but the formula is the same.
let mut app = App::new();
let buf = render_settings_to_buffer(&mut app, 80, 24);
let (px, _py) =
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
// popup_w = 80*4/5 = 64, centered: (80-64)/2 = 8
assert_eq!(
px, 8,
"popup should start at x=8 (centered 64-wide popup in 80-col area)"
);
}
#[test]
fn test_settings_popup_small_frame_no_panic() {
// Frame smaller than the min popup size (40x14) should not panic.
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let mut app = App::new();
let _buf = render_settings_to_buffer(&mut app, 30, 10);
}));
assert!(
result.is_ok(),
"rendering settings in a small frame should not panic"
);
}
}