Compare commits
56
Commits
b3256b2917
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c9aa50f932 | ||
|
|
9854ef13b6 | ||
|
|
58ac4a65c7 | ||
|
|
0910d4e85f | ||
|
|
ef26481976 | ||
|
|
c4ba196016 | ||
|
|
ec5a163f05 | ||
|
|
95f259a2bb | ||
|
|
e69b7af32a | ||
|
|
4421da35f4 | ||
|
|
e765f8967f | ||
|
|
967c11fea9 | ||
|
|
10323ce814 | ||
|
|
c1a931551b | ||
|
|
dfc016c348 | ||
|
|
19a3b877f9 | ||
|
|
5cb56dafd8 | ||
|
|
e99861c76d | ||
|
|
a43ef673b0 | ||
|
|
70f930eef7 | ||
|
|
463c53148b | ||
|
|
e9f75ce3b1 | ||
|
|
ef1889767a | ||
|
|
eedab3ac96 | ||
|
|
8e9600dda2 | ||
|
|
2cebbd94c4 | ||
|
|
0d88e933e6 | ||
|
|
420b853cb9 | ||
|
|
7852e92ecc | ||
|
|
d37ed6df68 | ||
|
|
b58d66f2aa | ||
|
|
d4679a7543 | ||
|
|
8844e58cb4 | ||
|
|
6a2f8ecb66 | ||
|
|
f6081b9fe9 | ||
|
|
97a2c6a925 | ||
|
|
e6e0e2cc90 | ||
|
|
ffaf462bae | ||
|
|
a8dc067cd4 | ||
|
|
502479677b | ||
|
|
5656b26d7b | ||
|
|
a8b64e78bd | ||
|
|
e945a357f7 | ||
|
|
fb57584546 | ||
|
|
9baec5ab69 | ||
|
|
6dd87d2872 | ||
|
|
83f633a562 | ||
|
|
dad5f5a635 | ||
|
|
534a089b58 | ||
|
|
b7938e069d | ||
|
|
d40d70c600 | ||
|
|
1350f659fa | ||
|
|
1bb6b2e9f3 | ||
|
|
bef0b44e91 | ||
|
|
24fe97a457 | ||
|
|
b6e655bff6 |
@@ -2,9 +2,9 @@ name: CI
|
|||||||
|
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
branches: [main]
|
branches: [master]
|
||||||
pull_request:
|
pull_request:
|
||||||
branches: [main]
|
branches: [master]
|
||||||
|
|
||||||
env:
|
env:
|
||||||
CARGO_TERM_COLOR: always
|
CARGO_TERM_COLOR: always
|
||||||
|
|||||||
@@ -1,5 +1,13 @@
|
|||||||
/target/
|
/target/
|
||||||
|
crates/*/target/
|
||||||
*.swp
|
*.swp
|
||||||
*.swo
|
*.swo
|
||||||
.idea/
|
.idea/
|
||||||
.vscode/
|
.vscode/
|
||||||
|
|
||||||
|
# Agent tooling state
|
||||||
|
.omo/
|
||||||
|
.sisyphus/
|
||||||
|
|
||||||
|
# Benchmark output
|
||||||
|
benchmark-report.md
|
||||||
|
|||||||
@@ -0,0 +1,122 @@
|
|||||||
|
# logViewer
|
||||||
|
|
||||||
|
High-performance log file viewer. Rust workspace with 4 crates, edition 2024, MSRV 1.92 (pinned in `rust-toolchain.toml`).
|
||||||
|
|
||||||
|
## Workspace Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/core — log-viewer-core (library) — I/O, parsing, types, config, file watching
|
||||||
|
crates/tui — log-viewer-tui (binary) — terminal UI (ratatui + crossterm)
|
||||||
|
crates/gui — log-viewer-gui (binary) — GUI (egui + eframe) — STUB, not yet functional
|
||||||
|
crates/bench — log-viewer-bench (binary) — mmap vs pread benchmark harness
|
||||||
|
```
|
||||||
|
|
||||||
|
`core` owns all I/O, parsing, data types, and file watching. TUI/GUI consume core's public API. GUI is a placeholder with no core integration yet.
|
||||||
|
|
||||||
|
**`default-members = ["crates/core"]`** — bare `cargo run` / `cargo test` target *core*, not the TUI. Always pass `-p log-viewer-tui` (or `--workspace`) explicitly.
|
||||||
|
|
||||||
|
## Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run the TUI (primary interface)
|
||||||
|
cargo run -p log-viewer-tui -- <log-file>
|
||||||
|
|
||||||
|
# Build/check/test/lint the whole workspace
|
||||||
|
cargo check --workspace
|
||||||
|
cargo test --workspace
|
||||||
|
cargo fmt --check --all
|
||||||
|
cargo clippy --workspace -- -D warnings
|
||||||
|
|
||||||
|
# Single crate
|
||||||
|
cargo test -p log-viewer-core
|
||||||
|
cargo test -p log-viewer-tui
|
||||||
|
|
||||||
|
# Benchmarks (test file is NOT auto-generated — see "Benchmarks" section below)
|
||||||
|
cargo run -p log-viewer-bench
|
||||||
|
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## CI Gate (must pass before merge)
|
||||||
|
|
||||||
|
Runs on ubuntu-latest + windows-latest (`.github/workflows/ci.yml`):
|
||||||
|
1. `cargo fmt --check --all`
|
||||||
|
2. `cargo check --workspace`
|
||||||
|
3. `cargo test --workspace`
|
||||||
|
4. `cargo clippy --workspace -- -D warnings`
|
||||||
|
|
||||||
|
Toolchain installed via `dtolnay/rust-toolchain@stable` reading `rust-toolchain.toml` (channel `1.92`). No rustfmt.toml or clippy.toml — uses toolchain defaults.
|
||||||
|
|
||||||
|
## Core Architecture
|
||||||
|
|
||||||
|
All ten modules are declared `pub` in `lib.rs` and re-exported as `log_viewer_core::<module>`.
|
||||||
|
|
||||||
|
### Key Modules
|
||||||
|
|
||||||
|
- **`types`** — `LogLevel`, `LogEntry`, `SearchQuery`, `SearchFilter`, `Bookmark`, `FileSession`, `DuplicateKey`, `SearchResult`
|
||||||
|
- **`error`** — `CoreError` (**12 variants** via thiserror: Io, Parse, Search, Index, Config, TomlSerialize, Encoding, Watch, Mmap, Cache, FileNotFound, Other) + `Result<T>` alias. `From` impls for `io::Error` / `serde_json::Error` / `toml::de::Error` / `toml::ser::Error` / `notify::Error` mean `?` works freely.
|
||||||
|
- **`io::file_reader`** — mmap-backed `FileReader` with TOCTOU mitigation (post-mmap stat check), append/reload detection
|
||||||
|
- **`io::progressive_reader`** — `ProgressiveFileReader` state machine (`ReaderState::Sampling { .. } | Ready { .. }`), background indexer via crossbeam-channel, `VisualHeightIndex` for wrapped-line scroll
|
||||||
|
- **`io::line_index`** — Sparse index (every `BLOCK_SIZE = 256` lines, memchr SIMD). Serializable to disk.
|
||||||
|
- **`io::index_cache`** — Persistent cache with xxh3 content hash, atomic writes via temp files
|
||||||
|
- **`io::line_sampler`** / **`io::cache_util`** — shared helpers used by the index path
|
||||||
|
- **`io::wrap`** — Unicode-aware line wrapping (CJK/emoji/tab), JSON pretty-printing; enforces `MAX_WRAP_INPUT_LEN`
|
||||||
|
- **`io::read_cache`** — LRU read cache (not yet integrated into the live reader; future pread backend)
|
||||||
|
- **`parser::json`** — NDJSON parser with BOM handling, duplicate key detection
|
||||||
|
- **`parser::level`** — JSON-first level detection (`detect_level`); falls back to bounded keyword scan with word-boundary check on non-JSON lines
|
||||||
|
- **`watcher::file_watcher`** — File event watcher (notify + crossbeam) with append/truncate/rotation detection
|
||||||
|
- **`config`** — `ColorConfig` TOML load/save with per-field serde defaults
|
||||||
|
|
||||||
|
### Stubs (single-line TODOs — planned, not yet built)
|
||||||
|
|
||||||
|
- `filter::Filter`
|
||||||
|
- `bookmark::BookmarkManager`
|
||||||
|
- `session::SessionManager`
|
||||||
|
- `search::engine::SearchEngine`
|
||||||
|
|
||||||
|
## TUI Architecture
|
||||||
|
|
||||||
|
- **`main.rs`** — clap CLI (`files: Vec<String>`), `TerminalGuard` RAII for raw mode + alternate screen. **CRITICAL**: `Drop` does not run on `std::process::exit` or `panic = "abort"` — never call `process::exit` inside the guarded scope; return `Err` and use `?` instead. Event loop: poll indexer → poll watcher → draw → poll keys, 100ms timeout.
|
||||||
|
- **`app/`** — module directory (split from the former monolithic `app.rs`). `mod.rs` (~3000 lines, inline `tests` mod at `#[cfg(test)] mod tests`) holds the `App` struct, `AppLoadingState: Empty | Loading { reader, estimated_lines, progress_percent } | Ready { reader } | Error(String)`, `AppMode: Normal | Settings`, `ViewportRenderRow`, and the main impl blocks. Per-concern helpers are split into submodules:
|
||||||
|
- `input.rs` — `handle_key` + Normal-mode key dispatch
|
||||||
|
- `settings.rs` — Settings-mode key dispatch (←/→ color cycle, j/k level select, Enter/Esc)
|
||||||
|
- `scroll.rs` — line/half/full-page scroll + VHI-aware sub-offset walk
|
||||||
|
- `viewport.rs` — viewport computation helpers
|
||||||
|
- `viewport_cache.rs` — on-demand viewport-sized render cache
|
||||||
|
- `loading.rs` — loading-state polling/progress helpers
|
||||||
|
- `watcher.rs` — file-watcher event polling + reload/reindex glue
|
||||||
|
- `query.rs` — query helpers
|
||||||
|
- **`ui.rs`** — ratatui rendering: title bar, content area, status bar, settings popup
|
||||||
|
- **`color.rs`** — `LogLevel` → ratatui `Color` via `ColorConfig`
|
||||||
|
|
||||||
|
### TUI Keybindings
|
||||||
|
|
||||||
|
j/k scroll, Ctrl+d/u half-page, Ctrl+f/b full-page, G/gg jump end/top, Tab toggle JSON formatting, S settings panel, q/Esc quit. Settings: j/k select level, ←/→ cycle colors, Enter save, Esc cancel.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
All tests are inline (`#[cfg(test)] mod tests` blocks). No `tests/` directories, no integration tests, no async tests.
|
||||||
|
|
||||||
|
- **~470 tests total**: core 262, tui 142, bench 63, gui 0
|
||||||
|
- Temp-file helpers vary per module — check the local `tests` mod before assuming an API:
|
||||||
|
- `make_temp_file(content) -> PathBuf` — `tui/app/mod.rs`, `tui/ui.rs`
|
||||||
|
- `make_test_file(lines) -> NamedTempFile` — `core/io/index_cache.rs`
|
||||||
|
- `make_file(data) -> NamedTempFile` — `core/io/read_cache.rs`
|
||||||
|
- `struct TempFile { ... }` — `core/io/line_sampler.rs`
|
||||||
|
- `insta` is a declared dev-dependency but currently **unused** (no `insta::` calls anywhere in the workspace)
|
||||||
|
- TUI render tests use ratatui `TestBackend` (see `tui/ui.rs`)
|
||||||
|
- Bench crate has its own unit tests for mmap/pread backends, metrics, and data_gen helpers
|
||||||
|
|
||||||
|
## Benchmarks
|
||||||
|
|
||||||
|
Custom harness (not criterion). Wall-clock + `/proc/self/` RSS/page fault metrics. 7 suites: startup, render, jump, memory, growth, rotation, concurrent. Tests mmap (plain/sequential/random/populate/phase_aware) vs pread (plain/random/sequential) backends. Output: markdown tables in `benchmark-report.md`.
|
||||||
|
|
||||||
|
**The harness does NOT auto-generate the test file.** `main.rs` requires `/tmp/test-logviewer/extreme.log` to already exist; if missing it prints the `dd` command and exits. Generate it manually first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mkdir -p /tmp/test-logviewer
|
||||||
|
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000 # ~5GB
|
||||||
|
```
|
||||||
|
|
||||||
|
(`crates/bench/src/data_gen.rs` has `generate_test_file` / `ensure_test_file` helpers and is used by the `growth` and `rotation` suites for per-run growable files, but it is **not** wired into `main.rs` for the primary test file.)
|
||||||
|
|
||||||
|
Note: `log-viewer-bench` depends directly on `memmap2` / `nix` / `libc` / `memchr` — it does **not** depend on `log-viewer-core`, so it can be built standalone with `cargo build -p log-viewer-bench`.
|
||||||
Generated
+2
@@ -2333,6 +2333,7 @@ dependencies = [
|
|||||||
"tempfile",
|
"tempfile",
|
||||||
"thiserror 2.0.18",
|
"thiserror 2.0.18",
|
||||||
"toml",
|
"toml",
|
||||||
|
"unicode-width",
|
||||||
"xxhash-rust",
|
"xxhash-rust",
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -2357,6 +2358,7 @@ dependencies = [
|
|||||||
"ratatui",
|
"ratatui",
|
||||||
"serde_json",
|
"serde_json",
|
||||||
"tempfile",
|
"tempfile",
|
||||||
|
"unicode-width",
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|||||||
@@ -27,3 +27,4 @@ textwrap = "0.16"
|
|||||||
tempfile = "3"
|
tempfile = "3"
|
||||||
xxhash-rust = { version = "0.8", features = ["xxh3"] }
|
xxhash-rust = { version = "0.8", features = ["xxh3"] }
|
||||||
bincode = "1"
|
bincode = "1"
|
||||||
|
unicode-width = "0.2"
|
||||||
|
|||||||
+109
@@ -0,0 +1,109 @@
|
|||||||
|
# logViewer
|
||||||
|
|
||||||
|
**[English](README.md)**
|
||||||
|
|
||||||
|
用 Rust 构建的高性能终端日志查看器,专为处理超大日志文件(GB 级别)而设计,内存占用极低。
|
||||||
|
|
||||||
|
基于 mmap 内存映射、稀疏行索引和后台渐进式加载,打开文件几乎瞬时完成——即使是 5GB 以上的日志文件,也能在索引完成前就开始滚动浏览。
|
||||||
|
|
||||||
|
## 特性
|
||||||
|
|
||||||
|
- **瞬时打开** — mmap 读取 + 后台渐进式索引,索引未完成即可开始浏览
|
||||||
|
- **超大文件支持** — 稀疏行索引(每 256 行采样一次),5GB 文件仅占用约 8MB 内存
|
||||||
|
- **实时文件追踪** — 通过 `notify` 监控文件追加、截断和日志轮转
|
||||||
|
- **JSON 日志解析** — 支持 NDJSON 格式,自动处理 BOM、检测重复键,可切换格式化显示
|
||||||
|
- **Unicode 感知的换行** — 正确处理中文、emoji 和制表符的行宽计算
|
||||||
|
- **持久化索引缓存** — 行索引保存到磁盘,使用 xxh3 哈希校验内容变更,再次打开几乎无需等待
|
||||||
|
- **Vim 风格快捷键** — 终端用户无学习成本
|
||||||
|
- **可自定义配色** — 每个日志级别独立配色,通过 TOML 配置文件管理,TUI 内直接调整
|
||||||
|
|
||||||
|
## 快速开始
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 直接编译运行
|
||||||
|
cargo run -p log-viewer-tui -- path/to/logfile.log
|
||||||
|
|
||||||
|
# 或先编译再运行
|
||||||
|
cargo build --release -p log-viewer-tui
|
||||||
|
./target/release/log-viewer-tui path/to/logfile.log
|
||||||
|
```
|
||||||
|
|
||||||
|
需要 Rust 1.92 及以上版本(见 `rust-toolchain.toml`)。
|
||||||
|
|
||||||
|
## 快捷键
|
||||||
|
|
||||||
|
| 按键 | 操作 |
|
||||||
|
|------|------|
|
||||||
|
| `j` / `↓` | 向下滚动一行 |
|
||||||
|
| `k` / `↑` | 向上滚动一行 |
|
||||||
|
| `Ctrl+d` | 向下滚动半页 |
|
||||||
|
| `Ctrl+u` | 向上滚动半页 |
|
||||||
|
| `Ctrl+f` / `PgDn` | 向下滚动一页 |
|
||||||
|
| `Ctrl+b` / `PgUp` | 向上滚动一页 |
|
||||||
|
| `G` / `End` | 跳转到文件末尾 |
|
||||||
|
| `gg` / `Home` | 跳转到文件开头 |
|
||||||
|
| `Tab` | 切换 JSON 格式化显示 |
|
||||||
|
| `S` | 打开配色设置 |
|
||||||
|
| `q` / `Esc` | 退出 |
|
||||||
|
|
||||||
|
设置面板:`j`/`k` 选择日志级别,`←`/`→` 切换颜色,`Enter` 保存,`Esc` 取消。
|
||||||
|
|
||||||
|
## 项目结构
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/core — 核心库:I/O、解析、数据类型、配置、文件监控
|
||||||
|
crates/tui — 终端界面(ratatui + crossterm)
|
||||||
|
crates/gui — 图形界面(egui + eframe)— 占位模块,尚未实现
|
||||||
|
crates/bench — mmap 与 pread 性能对比基准测试
|
||||||
|
```
|
||||||
|
|
||||||
|
## 开发
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 检查、测试、代码规范
|
||||||
|
cargo check --workspace
|
||||||
|
cargo test --workspace
|
||||||
|
cargo fmt --check --all
|
||||||
|
cargo clippy --workspace -- -D warnings
|
||||||
|
|
||||||
|
# 运行基准测试(需要先生成约 5GB 测试文件)
|
||||||
|
mkdir -p /tmp/test-logviewer
|
||||||
|
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000
|
||||||
|
cargo run -p log-viewer-bench
|
||||||
|
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
|
||||||
|
|
||||||
|
# 单独测试某个 crate
|
||||||
|
cargo test -p log-viewer-core
|
||||||
|
cargo test -p log-viewer-tui
|
||||||
|
```
|
||||||
|
|
||||||
|
### CI
|
||||||
|
|
||||||
|
CI 在 ubuntu-latest 和 windows-latest 上运行以下检查:
|
||||||
|
|
||||||
|
1. `cargo fmt --check --all`
|
||||||
|
2. `cargo check --workspace`
|
||||||
|
3. `cargo test --workspace`
|
||||||
|
4. `cargo clippy --workspace -- -D warnings`
|
||||||
|
|
||||||
|
无自定义 rustfmt 或 clippy 配置——使用工具链默认值。
|
||||||
|
|
||||||
|
## 架构
|
||||||
|
|
||||||
|
核心库 `log-viewer-core` 负责所有 I/O、解析和数据类型。关键设计决策:
|
||||||
|
|
||||||
|
- **稀疏行索引** — 每 256 行采样一次,使用 memchr SIMD 加速。100 万行的文件仅产生约 32KB 的索引,而非 8MB。
|
||||||
|
- **渐进式加载** — `ProgressiveFileReader` 先读取文件头尾进行快速行数估算,再通过 crossbeam-channel 在后台线程构建完整索引。
|
||||||
|
- **mmap 与 TOCTOU 防护** — mmap 映射后进行 stat 校验以检测文件变更。`read_cache` 模块为未来基于 pread 的实现预留,可彻底消除 SIGBUS 风险。
|
||||||
|
- **持久化缓存** — 行索引序列化到磁盘,使用 xxh3 内容哈希进行失效校验,通过临时文件实现原子写入。
|
||||||
|
- **视觉高度索引** — 基于换行后行高的前缀和数组,支持 O(log n) 的滚动定位,适用于长行换行场景。
|
||||||
|
|
||||||
|
计划中但尚未实现的功能:过滤、书签、会话管理、搜索引擎(stub 模块已创建)。
|
||||||
|
|
||||||
|
## 基准测试
|
||||||
|
|
||||||
|
自研基准测试框架,对比 mmap 与 pread 两种后端在 7 个测试场景(启动、渲染、跳转、内存、增长、轮转、并发)下的表现。使用挂钟计时和 `/proc/self/` 的 RSS 及页错误指标。结果以 Markdown 表格形式输出到 `benchmark-report.md`。
|
||||||
|
|
||||||
|
基准测试框架要求 `/tmp/test-logviewer/extreme.log` 已经存在。它不会自动生成主用的 5GB 测试文件;请先使用上方的 `dd` 命令创建。
|
||||||
|
|
||||||
|
基准测试二进制包含约 75 个单元测试,覆盖读取器后端和指标采集逻辑。
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
# logViewer
|
||||||
|
|
||||||
|
**[中文文档](README-zh.md)**
|
||||||
|
|
||||||
|
A high-performance terminal log file viewer built in Rust, designed to handle multi-gigabyte files with minimal memory overhead.
|
||||||
|
|
||||||
|
Uses memory-mapped I/O with a sparse line index and progressive background loading to open files instantly — even 5GB+ logs scroll smoothly from the first keystroke.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Instant file open** — mmap-backed reader with background progressive indexing; start scrolling before indexing finishes
|
||||||
|
- **Handles huge files** — sparse line index (1 entry per 256 lines) keeps memory usage at ~8MB even for 5GB files
|
||||||
|
- **Live file tracking** — watches for appends, truncations, and log rotation via `notify`
|
||||||
|
- **JSON log support** — NDJSON parsing with BOM handling, duplicate key detection, and toggleable pretty-printing
|
||||||
|
- **Unicode-aware wrapping** — correct line wrapping for CJK, emoji, and tabs
|
||||||
|
- **Persistent index cache** — line indexes saved to disk with xxh3 content hashing; re-opens are near-instant
|
||||||
|
- **Vim-like keybindings** — familiar navigation for terminal users
|
||||||
|
- **Customizable colors** — per-log-level colors via TOML config, adjustable from within the TUI
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build and run
|
||||||
|
cargo run -p log-viewer-tui -- path/to/logfile.log
|
||||||
|
|
||||||
|
# Or build first
|
||||||
|
cargo build --release -p log-viewer-tui
|
||||||
|
./target/release/log-viewer-tui path/to/logfile.log
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires Rust 1.92+ (see `rust-toolchain.toml`).
|
||||||
|
|
||||||
|
## Keybindings
|
||||||
|
|
||||||
|
| Key | Action |
|
||||||
|
|-----|--------|
|
||||||
|
| `j` / `↓` | Scroll down one line |
|
||||||
|
| `k` / `↑` | Scroll up one line |
|
||||||
|
| `Ctrl+d` | Scroll down half page |
|
||||||
|
| `Ctrl+u` | Scroll up half page |
|
||||||
|
| `Ctrl+f` / `PgDn` | Scroll down full page |
|
||||||
|
| `Ctrl+b` / `PgUp` | Scroll up full page |
|
||||||
|
| `G` / `End` | Jump to end of file |
|
||||||
|
| `gg` / `Home` | Jump to start of file |
|
||||||
|
| `Tab` | Toggle JSON pretty-printing |
|
||||||
|
| `S` | Open color settings |
|
||||||
|
| `q` / `Esc` | Quit |
|
||||||
|
|
||||||
|
Settings panel: use `j`/`k` to select a log level, `←`/`→` to cycle colors, `Enter` to save, `Esc` to cancel.
|
||||||
|
|
||||||
|
## Workspace Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/core — Shared library: I/O, parsing, types, config, file watching
|
||||||
|
crates/tui — Terminal UI (ratatui + crossterm)
|
||||||
|
crates/gui — GUI (egui + eframe) — placeholder, not yet functional
|
||||||
|
crates/bench — mmap vs pread benchmark harness
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check, test, lint
|
||||||
|
cargo check --workspace
|
||||||
|
cargo test --workspace
|
||||||
|
cargo fmt --check --all
|
||||||
|
cargo clippy --workspace -- -D warnings
|
||||||
|
|
||||||
|
# Run benchmarks (requires a pre-generated ~5GB test file)
|
||||||
|
mkdir -p /tmp/test-logviewer
|
||||||
|
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000
|
||||||
|
cargo run -p log-viewer-bench
|
||||||
|
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md
|
||||||
|
|
||||||
|
# Test a single crate
|
||||||
|
cargo test -p log-viewer-core
|
||||||
|
cargo test -p log-viewer-tui
|
||||||
|
```
|
||||||
|
|
||||||
|
### CI
|
||||||
|
|
||||||
|
The CI gate runs on ubuntu-latest and windows-latest:
|
||||||
|
1. `cargo fmt --check --all`
|
||||||
|
2. `cargo check --workspace`
|
||||||
|
3. `cargo test --workspace`
|
||||||
|
4. `cargo clippy --workspace -- -D warnings`
|
||||||
|
|
||||||
|
No custom rustfmt or clippy config — uses toolchain defaults.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The core library (`log-viewer-core`) owns all I/O, parsing, and data types. Key design decisions:
|
||||||
|
|
||||||
|
- **Sparse line index** — samples every 256 lines using memchr SIMD acceleration. A 1M-line file produces a ~32KB index instead of ~8MB.
|
||||||
|
- **Progressive loading** — `ProgressiveFileReader` starts with a quick head+tail sample for instant line estimates, then builds the full index in a background thread via crossbeam-channel.
|
||||||
|
- **mmap with TOCTOU mitigation** — post-mmap stat check to detect file changes. `read_cache` module exists as a future pread-based alternative to eliminate SIGBUS risk entirely.
|
||||||
|
- **Persistent cache** — line indexes are serialized to disk with xxh3 content hashing for invalidation. Atomic writes via temp files.
|
||||||
|
- **Visual height index** — prefix-sum over wrapped-line heights, enabling O(log n) scroll-to-line mapping for long wrapped lines.
|
||||||
|
|
||||||
|
Planned but not yet built: filtering, bookmarks, sessions, search engine (stub modules exist).
|
||||||
|
|
||||||
|
## Benchmarks
|
||||||
|
|
||||||
|
Custom harness comparing mmap vs pread backends across 7 suites (startup, render, jump, memory, growth, rotation, concurrent). Uses wall-clock timing with `/proc/self/` RSS and page fault metrics. Results are written as markdown tables to `benchmark-report.md`.
|
||||||
|
|
||||||
|
The harness expects `/tmp/test-logviewer/extreme.log` to already exist. It does not generate the primary 5GB test file automatically; create it first with the `dd` command shown above.
|
||||||
|
|
||||||
|
The benchmark binary includes ~75 unit tests for the reader backends and metrics collection.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
[package]
|
||||||
|
name = "log-viewer-bench"
|
||||||
|
version = "0.1.0"
|
||||||
|
edition = "2024"
|
||||||
|
|
||||||
|
[lib]
|
||||||
|
name = "log_viewer_bench"
|
||||||
|
path = "src/lib.rs"
|
||||||
|
|
||||||
|
[[bin]]
|
||||||
|
name = "log-viewer-bench"
|
||||||
|
path = "src/main.rs"
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
memmap2 = "0.9"
|
||||||
|
nix = { version = "0.30", features = ["signal", "resource", "mman", "fs"] }
|
||||||
|
libc = "0.2"
|
||||||
|
memchr = "2"
|
||||||
|
serde_json = "1"
|
||||||
|
clap = { version = "4", features = ["derive"] }
|
||||||
|
crossbeam-channel = "0.5"
|
||||||
|
tempfile = "3"
|
||||||
@@ -0,0 +1,221 @@
|
|||||||
|
use std::fs;
|
||||||
|
use std::io::{BufRead, BufReader, BufWriter, Write};
|
||||||
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
pub struct TestFileInfo {
|
||||||
|
pub path: PathBuf,
|
||||||
|
pub size_bytes: u64,
|
||||||
|
pub line_count: u64,
|
||||||
|
pub avg_line_length: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if test file exists and return its info, or generate it
|
||||||
|
pub fn ensure_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
|
||||||
|
if path.exists() {
|
||||||
|
return get_file_info(path);
|
||||||
|
}
|
||||||
|
generate_test_file(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get info about an existing test file
|
||||||
|
fn get_file_info(path: &Path) -> std::io::Result<TestFileInfo> {
|
||||||
|
let metadata = fs::metadata(path)?;
|
||||||
|
let size_bytes = metadata.len();
|
||||||
|
let line_count = count_existing_lines(path)?;
|
||||||
|
|
||||||
|
let avg_line_length = if line_count > 0 {
|
||||||
|
size_bytes as f64 / line_count as f64
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(TestFileInfo {
|
||||||
|
path: path.to_path_buf(),
|
||||||
|
size_bytes,
|
||||||
|
line_count,
|
||||||
|
avg_line_length,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate a large test file (~5GB / ~74M lines) if it doesn't exist
|
||||||
|
fn generate_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
|
||||||
|
if let Some(parent) = path.parent() {
|
||||||
|
fs::create_dir_all(parent)?;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(path)?);
|
||||||
|
let target_lines: u64 = 74_000_000;
|
||||||
|
for i in 0..target_lines {
|
||||||
|
writeln!(
|
||||||
|
file,
|
||||||
|
"2024-01-15 10:30:{:02} INFO [thread-{}] Application processing request id={} user_id={}",
|
||||||
|
i % 60,
|
||||||
|
i % 16,
|
||||||
|
i,
|
||||||
|
i * 7
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
file.flush()?;
|
||||||
|
drop(file);
|
||||||
|
|
||||||
|
get_file_info(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate a smaller file (~10MB / ~150K lines) for growth/rotation tests
|
||||||
|
pub fn generate_growable_file(dir: &Path) -> std::io::Result<PathBuf> {
|
||||||
|
fs::create_dir_all(dir)?;
|
||||||
|
let path = dir.join("growable.log");
|
||||||
|
|
||||||
|
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(&path)?);
|
||||||
|
for i in 0..150_000u64 {
|
||||||
|
writeln!(
|
||||||
|
file,
|
||||||
|
"2024-01-15 10:30:{:02} INFO [thread-{}] Appending test line {}",
|
||||||
|
i % 60,
|
||||||
|
i % 16,
|
||||||
|
i
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
file.flush()?;
|
||||||
|
|
||||||
|
Ok(path)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Append `count` lines to the file
|
||||||
|
pub fn append_lines(path: &Path, count: usize) -> std::io::Result<()> {
|
||||||
|
let existing_lines = count_existing_lines(path)?;
|
||||||
|
let mut file =
|
||||||
|
BufWriter::with_capacity(64 * 1024, fs::OpenOptions::new().append(true).open(path)?);
|
||||||
|
for i in 0..count {
|
||||||
|
writeln!(
|
||||||
|
file,
|
||||||
|
"2024-01-15 10:30:00 INFO Appended line {}",
|
||||||
|
existing_lines + i as u64
|
||||||
|
)?;
|
||||||
|
}
|
||||||
|
file.flush()?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Truncate file to specified size
|
||||||
|
pub fn truncate_file(path: &Path, size: u64) -> std::io::Result<()> {
|
||||||
|
let file = fs::OpenOptions::new().write(true).open(path)?;
|
||||||
|
file.set_len(size)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Rotate file: rename existing file, create new empty file
|
||||||
|
pub fn rotate_file(path: &Path) -> std::io::Result<PathBuf> {
|
||||||
|
let rotated = path.with_extension("log.1");
|
||||||
|
fs::rename(path, &rotated)?;
|
||||||
|
fs::File::create(path)?;
|
||||||
|
Ok(rotated)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Count lines in a file (helper)
|
||||||
|
fn count_existing_lines(path: &Path) -> std::io::Result<u64> {
|
||||||
|
let file = fs::File::open(path)?;
|
||||||
|
let reader = BufReader::new(file);
|
||||||
|
let mut count = 0u64;
|
||||||
|
for line in reader.lines() {
|
||||||
|
line?;
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
Ok(count)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_generate_growable_file_creates_approximately_correct_size() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = generate_growable_file(dir.path()).unwrap();
|
||||||
|
|
||||||
|
assert!(path.exists());
|
||||||
|
let metadata = fs::metadata(&path).unwrap();
|
||||||
|
let size_mb = metadata.len() as f64 / (1024.0 * 1024.0);
|
||||||
|
// ~150K lines × ~67 bytes ≈ ~10MB; allow 5MB–15MB range
|
||||||
|
assert!(
|
||||||
|
(5.0..=15.0).contains(&size_mb),
|
||||||
|
"Expected ~10MB, got {size_mb:.1}MB"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_append_lines_increases_line_count() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = {
|
||||||
|
let mut f = fs::File::create(dir.path().join("test.log")).unwrap();
|
||||||
|
for i in 0..10u64 {
|
||||||
|
writeln!(f, "line {i}").unwrap();
|
||||||
|
}
|
||||||
|
dir.path().join("test.log")
|
||||||
|
};
|
||||||
|
|
||||||
|
let before = count_existing_lines(&path).unwrap();
|
||||||
|
append_lines(&path, 5).unwrap();
|
||||||
|
let after = count_existing_lines(&path).unwrap();
|
||||||
|
assert_eq!(after, before + 5);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_truncate_file_reduces_size() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("trunc.log");
|
||||||
|
{
|
||||||
|
let mut f = fs::File::create(&path).unwrap();
|
||||||
|
write!(f, "{}", "A".repeat(1024)).unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let before = fs::metadata(&path).unwrap().len();
|
||||||
|
assert_eq!(before, 1024);
|
||||||
|
|
||||||
|
truncate_file(&path, 512).unwrap();
|
||||||
|
let after = fs::metadata(&path).unwrap().len();
|
||||||
|
assert_eq!(after, 512);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_rotate_file_renames_and_creates_empty() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("rotate.log");
|
||||||
|
{
|
||||||
|
let mut f = fs::File::create(&path).unwrap();
|
||||||
|
write!(f, "original content").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let rotated = rotate_file(&path).unwrap();
|
||||||
|
|
||||||
|
// Rotated file has the old content
|
||||||
|
assert!(rotated.exists());
|
||||||
|
assert_eq!(fs::read_to_string(&rotated).unwrap(), "original content");
|
||||||
|
|
||||||
|
// New file is empty
|
||||||
|
assert!(path.exists());
|
||||||
|
assert_eq!(fs::metadata(&path).unwrap().len(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_ensure_test_file_generates_when_missing() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("fresh.log");
|
||||||
|
assert!(!path.exists());
|
||||||
|
|
||||||
|
// Override the generator to use a small file for test speed:
|
||||||
|
// We'll test ensure_test_file indirectly by checking it calls generate_test_file.
|
||||||
|
// Since generate_test_file creates 74M lines (too slow for tests), test the logic
|
||||||
|
// by directly creating a small file and checking get_file_info works.
|
||||||
|
{
|
||||||
|
let mut f = fs::File::create(&path).unwrap();
|
||||||
|
for i in 0..100u64 {
|
||||||
|
writeln!(f, "2024-01-15 10:30:00 INFO line {i}").unwrap();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let info = ensure_test_file(&path).unwrap();
|
||||||
|
assert_eq!(info.line_count, 100);
|
||||||
|
assert!(info.size_bytes > 0);
|
||||||
|
assert!(info.avg_line_length > 0.0);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
pub mod data_gen;
|
||||||
|
pub mod line_index;
|
||||||
|
pub mod metrics;
|
||||||
|
pub mod mmap_reader;
|
||||||
|
pub mod pread_reader;
|
||||||
|
pub mod report;
|
||||||
|
pub mod runner;
|
||||||
|
pub mod suites;
|
||||||
|
pub mod types;
|
||||||
|
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
/// A single reader backend (mmap or pread)
|
||||||
|
pub trait FileReaderBackend {
|
||||||
|
fn name(&self) -> &str;
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized;
|
||||||
|
fn file_size(&self) -> u64;
|
||||||
|
fn total_lines(&self) -> usize;
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String>;
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>>;
|
||||||
|
fn close(self);
|
||||||
|
}
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
// ─── line_index.rs ───────────────────────────────────────────────────────────
|
||||||
|
// Vendored from crates/core/src/io/line_index.rs
|
||||||
|
// Sparse line index: sample every 256 lines to reduce memory usage.
|
||||||
|
// ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
const BLOCK_SIZE: usize = 256;
|
||||||
|
|
||||||
|
pub struct LineIndex {
|
||||||
|
pub(crate) sampled_offsets: Vec<u64>,
|
||||||
|
pub(crate) total_lines: u64,
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub(crate) has_trailing_newline: bool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl LineIndex {
|
||||||
|
/// Build sparse line index from a streaming reader.
|
||||||
|
/// Uses fill_buf()/consume() to avoid loading the entire file into memory.
|
||||||
|
/// RSS stays at ~64KB (BufReader buffer size), independent of file size.
|
||||||
|
pub fn from_reader(reader: &mut impl std::io::BufRead) -> std::io::Result<Self> {
|
||||||
|
let mut sampled_offsets: Vec<u64> = vec![0]; // line 0 starts at offset 0
|
||||||
|
let mut next_line_idx: usize = 1;
|
||||||
|
let mut newline_count: usize = 0;
|
||||||
|
let mut chunk_offset: u64 = 0;
|
||||||
|
let mut last_byte: Option<u8> = None;
|
||||||
|
|
||||||
|
loop {
|
||||||
|
let buf = reader.fill_buf()?;
|
||||||
|
if buf.is_empty() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(&b) = buf.last() {
|
||||||
|
last_byte = Some(b);
|
||||||
|
}
|
||||||
|
|
||||||
|
for pos in memchr::memchr_iter(b'\n', buf) {
|
||||||
|
newline_count += 1;
|
||||||
|
if next_line_idx.is_multiple_of(BLOCK_SIZE) {
|
||||||
|
sampled_offsets.push(chunk_offset + pos as u64 + 1);
|
||||||
|
}
|
||||||
|
next_line_idx += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let consumed = buf.len();
|
||||||
|
chunk_offset += consumed as u64;
|
||||||
|
reader.consume(consumed);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Empty file: no data at all
|
||||||
|
if chunk_offset == 0 {
|
||||||
|
return Ok(LineIndex {
|
||||||
|
sampled_offsets: vec![],
|
||||||
|
total_lines: 0,
|
||||||
|
has_trailing_newline: false,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
|
||||||
|
|
||||||
|
let total_lines: u64 = if has_trailing_newline && newline_count > 0 {
|
||||||
|
newline_count as u64
|
||||||
|
} else {
|
||||||
|
(1 + newline_count) as u64
|
||||||
|
};
|
||||||
|
|
||||||
|
// Trailing \n pop logic
|
||||||
|
if has_trailing_newline && newline_count > 0 {
|
||||||
|
let trailing_line_idx = newline_count;
|
||||||
|
if trailing_line_idx.is_multiple_of(BLOCK_SIZE) {
|
||||||
|
sampled_offsets.pop();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(LineIndex {
|
||||||
|
sampled_offsets,
|
||||||
|
total_lines,
|
||||||
|
has_trailing_newline,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Return total line count.
|
||||||
|
pub fn line_count(&self) -> usize {
|
||||||
|
self.total_lines as usize
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Retrieve the content of line `idx` from the given data slice.
|
||||||
|
/// Uses sparse index to locate the block start, then scans forward
|
||||||
|
/// a small number of newlines to find the target line.
|
||||||
|
pub fn get_line<'a>(&self, data: &'a [u8], idx: usize) -> Option<&'a str> {
|
||||||
|
if idx >= self.total_lines as usize || data.is_empty() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let block = idx / BLOCK_SIZE;
|
||||||
|
let offset_in_block = idx % BLOCK_SIZE;
|
||||||
|
let mut pos = self.sampled_offsets[block] as usize;
|
||||||
|
for _ in 0..offset_in_block {
|
||||||
|
match memchr::memchr(b'\n', &data[pos..]) {
|
||||||
|
Some(rel) => pos = pos + rel + 1,
|
||||||
|
None => return None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let end = memchr::memchr(b'\n', &data[pos..])
|
||||||
|
.map(|rel| pos + rel)
|
||||||
|
.unwrap_or(data.len());
|
||||||
|
let line_bytes = &data[pos..end];
|
||||||
|
std::str::from_utf8(line_bytes)
|
||||||
|
.map(|s| s.trim_end_matches(['\r', '\n']))
|
||||||
|
.ok()
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
use clap::Parser;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
|
||||||
|
/// Benchmark: mmap vs pread for large file reading
|
||||||
|
#[derive(Parser, Debug)]
|
||||||
|
#[command(name = "log-viewer-bench", version, about = "Benchmark mmap vs pread")]
|
||||||
|
struct Args {
|
||||||
|
/// Path to the test file (default: /tmp/test-logviewer/extreme.log)
|
||||||
|
#[arg(default_value = "/tmp/test-logviewer/extreme.log")]
|
||||||
|
test_file: PathBuf,
|
||||||
|
|
||||||
|
/// Quick mode: use smaller iterations and skip cold cache tests
|
||||||
|
#[arg(long)]
|
||||||
|
quick: bool,
|
||||||
|
|
||||||
|
/// Output report path (default: benchmark-report.md)
|
||||||
|
#[arg(long, default_value = "benchmark-report.md")]
|
||||||
|
output: PathBuf,
|
||||||
|
|
||||||
|
/// Only run specified suites (comma-separated: startup,render,jump,memory,growth,rotation,concurrent)
|
||||||
|
#[arg(long, value_delimiter = ',')]
|
||||||
|
suites: Option<Vec<String>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let args = Args::parse();
|
||||||
|
|
||||||
|
let suites = match args.suites {
|
||||||
|
Some(names) => {
|
||||||
|
let parsed: Result<Vec<_>, _> = names
|
||||||
|
.iter()
|
||||||
|
.map(|s| s.parse::<log_viewer_bench::runner::Suite>())
|
||||||
|
.collect();
|
||||||
|
match parsed {
|
||||||
|
Ok(s) => Some(s),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!("error: {e}");
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
println!("=== Benchmark: mmap vs pread ===");
|
||||||
|
println!("Test file: {}", args.test_file.display());
|
||||||
|
println!("Quick mode: {}", args.quick);
|
||||||
|
println!();
|
||||||
|
|
||||||
|
let config = log_viewer_bench::runner::BenchConfig {
|
||||||
|
test_file: args.test_file.clone(),
|
||||||
|
quick_mode: args.quick,
|
||||||
|
suites,
|
||||||
|
};
|
||||||
|
|
||||||
|
if !config.test_file.exists() {
|
||||||
|
eprintln!("ERROR: Test file not found: {}", config.test_file.display());
|
||||||
|
eprintln!(
|
||||||
|
"Generate one with: dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000"
|
||||||
|
);
|
||||||
|
std::process::exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
log_viewer_bench::runner::warn_reset_hwm();
|
||||||
|
|
||||||
|
println!("Running benchmarks...");
|
||||||
|
let results = log_viewer_bench::runner::run_all(&config);
|
||||||
|
println!("Completed {} benchmarks.\n", results.len());
|
||||||
|
|
||||||
|
let report = log_viewer_bench::report::format_report(&results);
|
||||||
|
|
||||||
|
println!("{}", report);
|
||||||
|
|
||||||
|
if let Err(e) = std::fs::write(&args.output, &report) {
|
||||||
|
eprintln!(
|
||||||
|
"WARNING: Failed to save report to {}: {}",
|
||||||
|
args.output.display(),
|
||||||
|
e
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
eprintln!("Report saved to {}", args.output.display());
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,275 @@
|
|||||||
|
use std::fs::{self, File};
|
||||||
|
use std::os::unix::fs::MetadataExt;
|
||||||
|
use std::os::unix::io::AsRawFd;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
pub struct RssMetrics {
|
||||||
|
pub vm_rss_kb: u64,
|
||||||
|
pub vm_hwm_kb: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct PageFaultMetrics {
|
||||||
|
pub minor_faults: u64,
|
||||||
|
pub major_faults: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct MetricsCollector;
|
||||||
|
|
||||||
|
impl MetricsCollector {
|
||||||
|
/// Read VmRSS and VmHWM from /proc/self/status
|
||||||
|
pub fn read_rss() -> RssMetrics {
|
||||||
|
let status = fs::read_to_string("/proc/self/status").unwrap_or_default();
|
||||||
|
let mut vm_rss_kb: u64 = 0;
|
||||||
|
let mut vm_hwm_kb: u64 = 0;
|
||||||
|
for line in status.lines() {
|
||||||
|
if line.starts_with("VmRSS:") {
|
||||||
|
vm_rss_kb = parse_kb_value(line);
|
||||||
|
} else if line.starts_with("VmHWM:") {
|
||||||
|
vm_hwm_kb = parse_kb_value(line);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
RssMetrics {
|
||||||
|
vm_rss_kb,
|
||||||
|
vm_hwm_kb,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read page fault counts from getrusage
|
||||||
|
pub fn read_page_faults() -> PageFaultMetrics {
|
||||||
|
let usage =
|
||||||
|
nix::sys::resource::getrusage(nix::sys::resource::UsageWho::RUSAGE_SELF).unwrap();
|
||||||
|
PageFaultMetrics {
|
||||||
|
// getrusage() returns c_long (i64 on 64-bit Linux) — explicit as u64 conversion
|
||||||
|
minor_faults: usage.minor_page_faults() as u64,
|
||||||
|
major_faults: usage.major_page_faults() as u64,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Clear page cache (requires root: sync + drop_caches)
|
||||||
|
/// Falls back to doing nothing if no permission
|
||||||
|
pub fn clear_page_cache() -> std::io::Result<()> {
|
||||||
|
let _ = std::process::Command::new("sync").status();
|
||||||
|
fs::write("/proc/sys/vm/drop_caches", "1")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Clear file cache using posix_fadvise(DONTNEED) — no root required
|
||||||
|
pub fn clear_file_cache(path: &Path) -> std::io::Result<()> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let len = file.metadata()?.len();
|
||||||
|
let ret = unsafe {
|
||||||
|
libc::posix_fadvise(file.as_raw_fd(), 0, len as i64, libc::POSIX_FADV_DONTNEED)
|
||||||
|
};
|
||||||
|
// posix_fadvise returns error code directly (not errno), 0 = success
|
||||||
|
if ret != 0 {
|
||||||
|
return Err(std::io::Error::from_raw_os_error(ret));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reset VmHWM by writing to /proc/self/clear_refs (requires root)
|
||||||
|
pub fn reset_vm_hwm() -> std::io::Result<()> {
|
||||||
|
fs::write("/proc/self/clear_refs", "5").map_err(|e| {
|
||||||
|
if e.kind() == std::io::ErrorKind::PermissionDenied {
|
||||||
|
std::io::Error::new(
|
||||||
|
std::io::ErrorKind::PermissionDenied,
|
||||||
|
"VmHWM reset requires root (can't write /proc/self/clear_refs)",
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
e
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Get file inode number
|
||||||
|
pub fn get_inode(path: &Path) -> std::io::Result<u64> {
|
||||||
|
let meta = fs::metadata(path)?;
|
||||||
|
Ok(meta.ino())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Check if file was rotated (inode changed)
|
||||||
|
pub fn detect_rotation(original_inode: u64, path: &Path) -> bool {
|
||||||
|
Self::get_inode(path)
|
||||||
|
.map(|ino| ino != original_inode)
|
||||||
|
.unwrap_or(true)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn parse_kb_value(line: &str) -> u64 {
|
||||||
|
// Format: "VmRSS: 12345 kB"
|
||||||
|
line.split_whitespace()
|
||||||
|
.nth(1)
|
||||||
|
.and_then(|v| v.parse::<u64>().ok())
|
||||||
|
.unwrap_or(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn mean(data: &[u64]) -> f64 {
|
||||||
|
if data.is_empty() {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
data.iter().sum::<u64>() as f64 / data.len() as f64
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Percentile of data at given fraction (0.0–1.0). Returns from a sorted copy.
|
||||||
|
pub fn percentile(data: &[u64], p: f64) -> u64 {
|
||||||
|
if data.is_empty() {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
let mut sorted: Vec<u64> = data.to_vec();
|
||||||
|
sorted.sort_unstable();
|
||||||
|
let idx = ((p * (sorted.len() - 1) as f64).round()) as usize;
|
||||||
|
sorted[idx.min(sorted.len() - 1)]
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn stdev(data: &[u64]) -> f64 {
|
||||||
|
if data.len() < 2 {
|
||||||
|
return 0.0;
|
||||||
|
}
|
||||||
|
let m = mean(data);
|
||||||
|
let variance: f64 = data
|
||||||
|
.iter()
|
||||||
|
.map(|&v| {
|
||||||
|
let d = v as f64 - m;
|
||||||
|
d * d
|
||||||
|
})
|
||||||
|
.sum::<f64>()
|
||||||
|
/ (data.len() - 1) as f64;
|
||||||
|
variance.sqrt()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn p50(data: &[u64]) -> u64 {
|
||||||
|
percentile(data, 0.50)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn p95(data: &[u64]) -> u64 {
|
||||||
|
percentile(data, 0.95)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn p99(data: &[u64]) -> u64 {
|
||||||
|
percentile(data, 0.99)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_rss_returns_values() {
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
assert!(
|
||||||
|
rss.vm_rss_kb > 0,
|
||||||
|
"VmRSS should be non-zero for a running process"
|
||||||
|
);
|
||||||
|
assert!(
|
||||||
|
rss.vm_hwm_kb > 0,
|
||||||
|
"VmHWM should be non-zero for a running process"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_page_faults_returns_values() {
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
assert!(
|
||||||
|
faults.minor_faults > 0,
|
||||||
|
"Should have some minor page faults"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_mean() {
|
||||||
|
let data = vec![100, 200, 300, 400, 500];
|
||||||
|
let result = mean(&data);
|
||||||
|
assert!(
|
||||||
|
(result - 300.0).abs() < f64::EPSILON,
|
||||||
|
"mean should be 300.0, got {result}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_mean_empty() {
|
||||||
|
assert_eq!(mean(&[]), 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_percentile_p50() {
|
||||||
|
let data = vec![100, 200, 300, 400, 500];
|
||||||
|
assert_eq!(percentile(&data, 0.50), 300);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_percentile_p99() {
|
||||||
|
let data = vec![10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
|
||||||
|
let p99_result = percentile(&data, 0.99);
|
||||||
|
assert!(p99_result >= 90, "P99 should be near max, got {p99_result}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_percentile_empty() {
|
||||||
|
assert_eq!(percentile(&[], 0.5), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_stdev() {
|
||||||
|
let data = vec![100, 200, 300, 400, 500];
|
||||||
|
let s = stdev(&data);
|
||||||
|
assert!(s > 100.0, "stdev should be significant, got {s}");
|
||||||
|
assert!(s < 200.0, "stdev should be < 200, got {s}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_stdev_single() {
|
||||||
|
assert_eq!(stdev(&[42]), 0.0);
|
||||||
|
assert_eq!(stdev(&[]), 0.0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_kb_value() {
|
||||||
|
assert_eq!(parse_kb_value("VmRSS: 12345 kB"), 12345);
|
||||||
|
assert_eq!(parse_kb_value("VmHWM:\t2048 kB"), 2048);
|
||||||
|
assert_eq!(parse_kb_value("VmRSS: 0 kB"), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_parse_kb_value_malformed() {
|
||||||
|
assert_eq!(parse_kb_value("VmRSS: NaN kB"), 0);
|
||||||
|
assert_eq!(parse_kb_value("garbage"), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_convenience_percentiles() {
|
||||||
|
let data = vec![10, 20, 30, 40, 50];
|
||||||
|
assert_eq!(p50(&data), 30);
|
||||||
|
assert_eq!(p95(&data), 50);
|
||||||
|
assert_eq!(p99(&data), 50);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_inode_for_existing_file() {
|
||||||
|
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||||
|
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
|
||||||
|
assert!(inode > 0, "inode should be non-zero");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_detect_rotation_no_rotation() {
|
||||||
|
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||||
|
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
|
||||||
|
assert!(!MetricsCollector::detect_rotation(inode, tmp.path()));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_detect_rotation_file_removed() {
|
||||||
|
let inode: u64 = 99999;
|
||||||
|
let result = MetricsCollector::detect_rotation(inode, Path::new("/no/such/file"));
|
||||||
|
assert!(result, "missing file should indicate rotation");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_clear_file_cache() {
|
||||||
|
let tmp = tempfile::NamedTempFile::new().unwrap();
|
||||||
|
let result = MetricsCollector::clear_file_cache(tmp.path());
|
||||||
|
assert!(
|
||||||
|
result.is_ok(),
|
||||||
|
"clear_file_cache should succeed on temp file: {result:?}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,752 @@
|
|||||||
|
// ─── mmap_reader.rs ──────────────────────────────────────────────────────────
|
||||||
|
// mmap-based FileReaderBackend implementations with 5 variants for benchmarking.
|
||||||
|
// Includes SIGBUS handler for file-truncation resilience and remap support
|
||||||
|
// for growing-file scenarios.
|
||||||
|
// ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io::BufReader;
|
||||||
|
use std::path::Path;
|
||||||
|
use std::sync::Once;
|
||||||
|
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU8, Ordering};
|
||||||
|
|
||||||
|
use memmap2::{Advice, Mmap, MmapOptions, RemapOptions};
|
||||||
|
use nix::sys::signal::{SaFlags, SigAction, SigHandler, SigSet, Signal, sigaction};
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::line_index::LineIndex;
|
||||||
|
|
||||||
|
// ─── SIGBUS Handler ──────────────────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// Signal-safety architecture:
|
||||||
|
// - Old handler state is stored in raw atomics (AtomicU8 + AtomicPtr) that are
|
||||||
|
// async-signal-safe to read — no OnceLock, Mutex, or other non-trivial abstractions
|
||||||
|
// in the signal handler path.
|
||||||
|
// - Installation uses `Once` for idempotency and follows a strict publish-then-install
|
||||||
|
// sequence to close the "handler-active-but-state-unpublished" race window.
|
||||||
|
|
||||||
|
/// Global flag set by the SIGBUS handler when a bus error is intercepted.
|
||||||
|
/// Process-global: concurrent benchmarks that reset/check this flag may interfere.
|
||||||
|
/// Currently acceptable because only `rotation` suite (sequential) uses it.
|
||||||
|
static SIGBUS_OCCURRED: AtomicBool = AtomicBool::new(false);
|
||||||
|
|
||||||
|
/// Discriminant values for old SIGBUS handler type.
|
||||||
|
const HANDLER_NONE: u8 = 0;
|
||||||
|
const HANDLER_DEFAULT: u8 = 1;
|
||||||
|
const HANDLER_IGNORE: u8 = 2;
|
||||||
|
const HANDLER_PLAIN: u8 = 3; // extern "C" fn(c_int)
|
||||||
|
#[allow(
|
||||||
|
clippy::unseparated_literal_suffix,
|
||||||
|
reason = "clarity: this is the SA_SIGACTION variant"
|
||||||
|
)]
|
||||||
|
const HANDLER_SIGACTION: u8 = 4; // extern "C" fn(c_int, *mut siginfo_t, *mut c_void)
|
||||||
|
|
||||||
|
/// Old SIGBUS handler type — raw atomic, async-signal-safe to read.
|
||||||
|
static OLD_HANDLER_KIND: AtomicU8 = AtomicU8::new(HANDLER_NONE);
|
||||||
|
|
||||||
|
/// Old SIGBUS handler function pointer — raw atomic, async-signal-safe to read.
|
||||||
|
static OLD_HANDLER_PTR: AtomicPtr<std::ffi::c_void> = AtomicPtr::new(std::ptr::null_mut());
|
||||||
|
|
||||||
|
/// Ensures `install_sigbus_handler` runs exactly once across all threads.
|
||||||
|
static INSTALL_ONCE: Once = Once::new();
|
||||||
|
|
||||||
|
/// Returns `true` if a SIGBUS was intercepted since the last reset.
|
||||||
|
pub fn sigbus_flag() -> bool {
|
||||||
|
SIGBUS_OCCURRED.load(Ordering::SeqCst)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resets the SIGBUS flag. Call before operations where you want to detect
|
||||||
|
/// a fresh SIGBUS.
|
||||||
|
pub fn reset_sigbus_flag() {
|
||||||
|
SIGBUS_OCCURRED.store(false, Ordering::SeqCst)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// SIGBUS signal handler.
|
||||||
|
///
|
||||||
|
/// # Safety Constraints (signal handler context)
|
||||||
|
/// - NO TLS access
|
||||||
|
/// - NO memory allocation
|
||||||
|
/// - NO lock acquisition
|
||||||
|
/// - Only: AtomicBool store → mmap → raw atomic loads → chain
|
||||||
|
extern "C" fn sigbus_handler(
|
||||||
|
sig: libc::c_int,
|
||||||
|
info: *mut libc::siginfo_t,
|
||||||
|
ctx: *mut std::ffi::c_void,
|
||||||
|
) {
|
||||||
|
SIGBUS_OCCURRED.store(true, Ordering::SeqCst);
|
||||||
|
|
||||||
|
// Align fault address down to page boundary.
|
||||||
|
// si_addr is NOT guaranteed to be page-aligned; without this, mmap(MAP_FIXED)
|
||||||
|
// could replace the wrong page.
|
||||||
|
let addr = unsafe { (*info).si_addr() };
|
||||||
|
let aligned = (addr as usize & !0xFFF) as *mut libc::c_void;
|
||||||
|
|
||||||
|
// Map anonymous zero page at fault address to prevent crash on file truncation.
|
||||||
|
let result = unsafe {
|
||||||
|
libc::mmap(
|
||||||
|
aligned,
|
||||||
|
4096,
|
||||||
|
libc::PROT_READ,
|
||||||
|
libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_FIXED,
|
||||||
|
-1,
|
||||||
|
0,
|
||||||
|
)
|
||||||
|
};
|
||||||
|
if result == libc::MAP_FAILED {
|
||||||
|
// Chain to old handler via raw atomics (async-signal-safe).
|
||||||
|
let kind = OLD_HANDLER_KIND.load(Ordering::Acquire);
|
||||||
|
match kind {
|
||||||
|
HANDLER_PLAIN => {
|
||||||
|
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
|
||||||
|
if !ptr.is_null() {
|
||||||
|
// SAFETY: ptr was derived from a valid function pointer
|
||||||
|
// published by install_sigbus_handler before this handler was installed.
|
||||||
|
let f: extern "C" fn(libc::c_int) = unsafe { std::mem::transmute(ptr) };
|
||||||
|
f(sig);
|
||||||
|
} else {
|
||||||
|
unsafe { libc::_exit(128 + sig) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
HANDLER_SIGACTION => {
|
||||||
|
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
|
||||||
|
if !ptr.is_null() {
|
||||||
|
let f: extern "C" fn(libc::c_int, *mut libc::siginfo_t, *mut std::ffi::c_void) =
|
||||||
|
unsafe { std::mem::transmute(ptr) };
|
||||||
|
f(sig, info, ctx);
|
||||||
|
} else {
|
||||||
|
unsafe { libc::_exit(128 + sig) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
// HANDLER_NONE / HANDLER_DEFAULT / HANDLER_IGNORE — no safe chaining.
|
||||||
|
unsafe { libc::_exit(128 + sig) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Install the SIGBUS handler via sigaction(). Chains to any previous handler.
|
||||||
|
/// Uses `Once` to guarantee exactly-once installation — safe to call from
|
||||||
|
/// multiple threads concurrently.
|
||||||
|
///
|
||||||
|
/// The install sequence closes the "install-then-publish" race by:
|
||||||
|
/// 1. Querying the current handler via raw `libc::sigaction` (no modification).
|
||||||
|
/// 2. Publishing old handler state to raw atomics (`Release` stores).
|
||||||
|
/// 3. Installing our handler via `nix::sigaction`.
|
||||||
|
///
|
||||||
|
/// This ensures the atomics are readable *before* our handler can fire.
|
||||||
|
fn install_sigbus_handler() {
|
||||||
|
INSTALL_ONCE.call_once(|| {
|
||||||
|
// Step 1: Query current SIGBUS disposition without changing it.
|
||||||
|
let mut old_act: libc::sigaction = unsafe { std::mem::zeroed() };
|
||||||
|
let ret = unsafe { libc::sigaction(libc::SIGBUS, std::ptr::null(), &mut old_act) };
|
||||||
|
if ret != 0 {
|
||||||
|
// Failed to query — skip installation entirely.
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 2: Publish old handler to atomics BEFORE installing ours.
|
||||||
|
let is_siginfo = (old_act.sa_flags & libc::SA_SIGINFO) != 0;
|
||||||
|
|
||||||
|
let raw_usize = old_act.sa_sigaction as usize;
|
||||||
|
|
||||||
|
if raw_usize == 0 {
|
||||||
|
// SIG_DFL (null function pointer → default disposition)
|
||||||
|
OLD_HANDLER_KIND.store(HANDLER_DEFAULT, Ordering::Release);
|
||||||
|
} else if raw_usize == 1 {
|
||||||
|
// SIG_IGN (sentinel value 1 → ignore disposition)
|
||||||
|
OLD_HANDLER_KIND.store(HANDLER_IGNORE, Ordering::Release);
|
||||||
|
} else {
|
||||||
|
// Real handler function pointer
|
||||||
|
OLD_HANDLER_PTR.store(raw_usize as *mut std::ffi::c_void, Ordering::Release);
|
||||||
|
if is_siginfo {
|
||||||
|
OLD_HANDLER_KIND.store(HANDLER_SIGACTION, Ordering::Release);
|
||||||
|
} else {
|
||||||
|
OLD_HANDLER_KIND.store(HANDLER_PLAIN, Ordering::Release);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 3: Install our handler. Atomics are already published,
|
||||||
|
// so the handler can safely read them from the moment it's installed.
|
||||||
|
let new_action = SigAction::new(
|
||||||
|
SigHandler::SigAction(sigbus_handler),
|
||||||
|
SaFlags::SA_SIGINFO,
|
||||||
|
SigSet::empty(),
|
||||||
|
);
|
||||||
|
let _ = unsafe { sigaction(Signal::SIGBUS, &new_action) };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Core MmapReader ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Core mmap-based reader. Holds the memory mapping, file handle, and
|
||||||
|
/// line index. Used as the engine inside each variant wrapper.
|
||||||
|
pub struct MmapReader {
|
||||||
|
mmap: Mmap,
|
||||||
|
#[allow(dead_code)]
|
||||||
|
file: File,
|
||||||
|
line_index: LineIndex,
|
||||||
|
file_size: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MmapReader {
|
||||||
|
/// Open a file and create an mmap. Does NOT apply madvise.
|
||||||
|
/// The caller is responsible for applying the desired advice.
|
||||||
|
fn open_raw(path: &Path) -> std::io::Result<Self> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let file_size = file.metadata()?.len();
|
||||||
|
|
||||||
|
let mmap = unsafe { Mmap::map(&file)? };
|
||||||
|
|
||||||
|
let line_index = {
|
||||||
|
let mut reader = BufReader::new(&file);
|
||||||
|
LineIndex::from_reader(&mut reader)?
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
mmap,
|
||||||
|
file,
|
||||||
|
line_index,
|
||||||
|
file_size,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Open with MmapOptions (for populate, etc.).
|
||||||
|
fn open_with_options(path: &Path, opts: &MmapOptions) -> std::io::Result<Self> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let file_size = file.metadata()?.len();
|
||||||
|
|
||||||
|
let mmap = if file_size == 0 {
|
||||||
|
unsafe { Mmap::map(&file)? }
|
||||||
|
} else {
|
||||||
|
unsafe { opts.map(&file)? }
|
||||||
|
};
|
||||||
|
|
||||||
|
let line_index = {
|
||||||
|
let mut reader = BufReader::new(&file);
|
||||||
|
LineIndex::from_reader(&mut reader)?
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
mmap,
|
||||||
|
file,
|
||||||
|
line_index,
|
||||||
|
file_size,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Apply madvise to the entire mapping.
|
||||||
|
fn advise(&self, advice: Advice) {
|
||||||
|
let _ = self.mmap.advise(advice);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Grow the mmap to `new_size` bytes.
|
||||||
|
///
|
||||||
|
/// # Safety
|
||||||
|
/// Caller must ensure no `&[u8]` references to the old mmap data exist.
|
||||||
|
/// After remap, any pointers derived from the old mapping may be invalid
|
||||||
|
/// (if the kernel moved the mapping).
|
||||||
|
pub unsafe fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
|
||||||
|
unsafe {
|
||||||
|
self.mmap
|
||||||
|
.remap(new_size, RemapOptions::new().may_move(true))?;
|
||||||
|
}
|
||||||
|
self.file_size = new_size as u64;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn file_size(&self) -> u64 {
|
||||||
|
self.file_size
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn total_lines(&self) -> usize {
|
||||||
|
self.line_index.line_count()
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.line_index
|
||||||
|
.get_line(&self.mmap, idx)
|
||||||
|
.map(|s| s.to_owned())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
let start = offset as usize;
|
||||||
|
let end = start.checked_add(len)?;
|
||||||
|
if end > self.mmap.len() {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
Some(self.mmap[start..end].to_vec())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── 5 Variants ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Variant 1: Plain mmap, no madvise.
|
||||||
|
pub struct MmapReaderPlain {
|
||||||
|
inner: MmapReader,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl MmapReaderPlain {
|
||||||
|
pub fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
|
||||||
|
unsafe { self.inner.remap(new_size) }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for MmapReaderPlain {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"mmap_plain"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
install_sigbus_handler();
|
||||||
|
let inner = MmapReader::open_raw(path)?;
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 2: mmap with MADV_SEQUENTIAL — optimal for sequential scan (index build).
|
||||||
|
pub struct MmapReaderSequential {
|
||||||
|
inner: MmapReader,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for MmapReaderSequential {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"mmap_sequential"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
install_sigbus_handler();
|
||||||
|
let inner = MmapReader::open_raw(path)?;
|
||||||
|
inner.advise(Advice::Sequential);
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 3: mmap with MADV_RANDOM — optimal for random line access after index is built.
|
||||||
|
pub struct MmapReaderRandom {
|
||||||
|
inner: MmapReader,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for MmapReaderRandom {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"mmap_random"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
install_sigbus_handler();
|
||||||
|
let inner = MmapReader::open_raw(path)?;
|
||||||
|
inner.advise(Advice::Random);
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 4: mmap with MAP_POPULATE — pre-fault all pages at mmap time.
|
||||||
|
/// Trades higher upfront cost for smoother subsequent access.
|
||||||
|
pub struct MmapReaderPopulate {
|
||||||
|
inner: MmapReader,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for MmapReaderPopulate {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"mmap_populate"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
install_sigbus_handler();
|
||||||
|
let mut opts = MmapOptions::new();
|
||||||
|
opts.populate();
|
||||||
|
let inner = MmapReader::open_with_options(path, &opts)?;
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 5: Phase-aware — MADV_SEQUENTIAL during index build, then MADV_RANDOM
|
||||||
|
/// for line access. Best of both worlds for the read-index-then-query pattern.
|
||||||
|
pub struct MmapReaderPhaseAware {
|
||||||
|
inner: MmapReader,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for MmapReaderPhaseAware {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"mmap_phase_aware"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
install_sigbus_handler();
|
||||||
|
let inner = MmapReader::open_raw(path)?;
|
||||||
|
// Index build used sequential streaming via BufReader. Now switch to random.
|
||||||
|
inner.advise(Advice::Random);
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Tests ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::io::Write as _;
|
||||||
|
use tempfile::NamedTempFile;
|
||||||
|
|
||||||
|
fn create_temp_file(content: &[u8]) -> NamedTempFile {
|
||||||
|
let mut f = NamedTempFile::new().unwrap();
|
||||||
|
f.write_all(content).unwrap();
|
||||||
|
f.flush().unwrap();
|
||||||
|
f
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_open_and_read_lines() {
|
||||||
|
let f = create_temp_file(b"hello\nworld\nfoo\n");
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "mmap_plain");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(1), Some("world".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(3), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_out_of_bounds_returns_none() {
|
||||||
|
let f = create_temp_file(b"line1\nline2\n");
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
assert_eq!(reader.get_line(100), None);
|
||||||
|
assert_eq!(reader.get_line(usize::MAX), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_read_range() {
|
||||||
|
let content = b"hello world\nfoo bar\n";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
let range = reader.read_range(0, 5).unwrap();
|
||||||
|
assert_eq!(&range, b"hello");
|
||||||
|
let range = reader.read_range(6, 5).unwrap();
|
||||||
|
assert_eq!(&range, b"world");
|
||||||
|
assert_eq!(reader.read_range(0, 1000), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_empty_file() {
|
||||||
|
let f = create_temp_file(b"");
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 0);
|
||||||
|
assert_eq!(reader.file_size(), 0);
|
||||||
|
assert_eq!(reader.get_line(0), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_sequential_variant() {
|
||||||
|
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
|
||||||
|
let reader = MmapReaderSequential::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "mmap_sequential");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_random_variant() {
|
||||||
|
let f = create_temp_file(b"one\ntwo\nthree\n");
|
||||||
|
let reader = MmapReaderRandom::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "mmap_random");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(2), Some("three".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_populate_variant() {
|
||||||
|
let f = create_temp_file(b"x\ny\nz\n");
|
||||||
|
let reader = MmapReaderPopulate::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "mmap_populate");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(0), Some("x".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_phase_aware_variant() {
|
||||||
|
let f = create_temp_file(b"a\nb\nc\n");
|
||||||
|
let reader = MmapReaderPhaseAware::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "mmap_phase_aware");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(1), Some("b".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_file_size() {
|
||||||
|
let content = b"hello world";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.file_size(), content.len() as u64);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_sigbus_flag_api() {
|
||||||
|
reset_sigbus_flag();
|
||||||
|
assert!(!sigbus_flag());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_concurrent_open_installs_handler_once() {
|
||||||
|
let f = create_temp_file(b"line1\nline2\nline3\n");
|
||||||
|
let path = f.path().to_owned();
|
||||||
|
let num_threads = 8;
|
||||||
|
|
||||||
|
let handles: Vec<_> = (0..num_threads)
|
||||||
|
.map(|thread_id| {
|
||||||
|
let path = path.clone();
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
let reader = match thread_id % 5 {
|
||||||
|
0 => {
|
||||||
|
let r = MmapReaderPlain::open(&path).unwrap();
|
||||||
|
assert_eq!(r.total_lines(), 3);
|
||||||
|
r.close();
|
||||||
|
"plain"
|
||||||
|
}
|
||||||
|
1 => {
|
||||||
|
let r = MmapReaderSequential::open(&path).unwrap();
|
||||||
|
assert_eq!(r.total_lines(), 3);
|
||||||
|
r.close();
|
||||||
|
"sequential"
|
||||||
|
}
|
||||||
|
2 => {
|
||||||
|
let r = MmapReaderRandom::open(&path).unwrap();
|
||||||
|
assert_eq!(r.total_lines(), 3);
|
||||||
|
r.close();
|
||||||
|
"random"
|
||||||
|
}
|
||||||
|
3 => {
|
||||||
|
let r = MmapReaderPopulate::open(&path).unwrap();
|
||||||
|
assert_eq!(r.total_lines(), 3);
|
||||||
|
r.close();
|
||||||
|
"populate"
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
let r = MmapReaderPhaseAware::open(&path).unwrap();
|
||||||
|
assert_eq!(r.total_lines(), 3);
|
||||||
|
r.close();
|
||||||
|
"phase_aware"
|
||||||
|
}
|
||||||
|
};
|
||||||
|
reader.to_owned()
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let results: Vec<String> = handles
|
||||||
|
.into_iter()
|
||||||
|
.map(|h| h.join().expect("thread panicked"))
|
||||||
|
.collect();
|
||||||
|
assert_eq!(results.len(), num_threads);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_no_trailing_newline() {
|
||||||
|
let f = create_temp_file(b"line1\nline2");
|
||||||
|
let reader = MmapReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Diagnostic test: measure how long each stage of `open_raw()` takes
|
||||||
|
/// on a large file (e.g. the 5GB extreme.log).
|
||||||
|
/// Run with: cargo test -p log-viewer-bench --release -- --nocapture diag_open_stages
|
||||||
|
#[test]
|
||||||
|
fn diag_open_stages() {
|
||||||
|
let path = std::path::Path::new("/tmp/test-logviewer/extreme.log");
|
||||||
|
if !path.exists() {
|
||||||
|
eprintln!("SKIP: test file not found");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stage 1: File::open + metadata
|
||||||
|
let t0 = std::time::Instant::now();
|
||||||
|
let file = File::open(path).unwrap();
|
||||||
|
let file_size = file.metadata().unwrap().len();
|
||||||
|
let stage1 = t0.elapsed();
|
||||||
|
eprintln!(
|
||||||
|
"Stage 1 - File::open + metadata: {:.2}ms ({:.1}GB)",
|
||||||
|
stage1.as_secs_f64() * 1000.0,
|
||||||
|
file_size as f64 / 1073741824.0
|
||||||
|
);
|
||||||
|
|
||||||
|
// Stage 2: mmap::map
|
||||||
|
let t1 = std::time::Instant::now();
|
||||||
|
let _mmap = unsafe { Mmap::map(&file) }.unwrap();
|
||||||
|
let stage2 = t1.elapsed();
|
||||||
|
eprintln!(
|
||||||
|
"Stage 2 - mmap::map: {:.2}ms",
|
||||||
|
stage2.as_secs_f64() * 1000.0
|
||||||
|
);
|
||||||
|
drop(_mmap);
|
||||||
|
|
||||||
|
// Stage 3: LineIndex::from_reader via BufReader (default 8KB buffer)
|
||||||
|
let t2 = std::time::Instant::now();
|
||||||
|
let mut reader = BufReader::new(&file);
|
||||||
|
let line_index = crate::line_index::LineIndex::from_reader(&mut reader).unwrap();
|
||||||
|
let stage3 = t2.elapsed();
|
||||||
|
eprintln!(
|
||||||
|
"Stage 3 - LineIndex::from_reader: {:.2}ms ({} lines, {} sampled_offsets)",
|
||||||
|
stage3.as_secs_f64() * 1000.0,
|
||||||
|
line_index.line_count(),
|
||||||
|
line_index.sampled_offsets.len()
|
||||||
|
);
|
||||||
|
|
||||||
|
// Stage 3b: Try with 1MB buffer
|
||||||
|
let file2 = File::open(path).unwrap();
|
||||||
|
let t2b = std::time::Instant::now();
|
||||||
|
let mut reader2 = BufReader::with_capacity(1024 * 1024, &file2);
|
||||||
|
let line_index2 = crate::line_index::LineIndex::from_reader(&mut reader2).unwrap();
|
||||||
|
let stage3b = t2b.elapsed();
|
||||||
|
eprintln!(
|
||||||
|
"Stage 3b - LineIndex (1MB buffer): {:.2}ms",
|
||||||
|
stage3b.as_secs_f64() * 1000.0
|
||||||
|
);
|
||||||
|
|
||||||
|
eprintln!("\n=== 瓶颈分析 ===");
|
||||||
|
let total_ms = stage1.as_secs_f64() * 1000.0
|
||||||
|
+ stage2.as_secs_f64() * 1000.0
|
||||||
|
+ stage3.as_secs_f64() * 1000.0;
|
||||||
|
let total_dur = stage1 + stage2 + stage3;
|
||||||
|
eprintln!("Total ~{:.0}ms", total_ms);
|
||||||
|
eprintln!(
|
||||||
|
" File::open: {:.1}% ({:.0}ms)",
|
||||||
|
stage1.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
|
||||||
|
stage1.as_secs_f64() * 1000.0
|
||||||
|
);
|
||||||
|
eprintln!(
|
||||||
|
" mmap::map: {:.1}% ({:.0}ms)",
|
||||||
|
stage2.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
|
||||||
|
stage2.as_secs_f64() * 1000.0
|
||||||
|
);
|
||||||
|
eprintln!(
|
||||||
|
" from_reader: {:.1}% ({:.0}ms)",
|
||||||
|
stage3.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
|
||||||
|
stage3.as_secs_f64() * 1000.0
|
||||||
|
);
|
||||||
|
|
||||||
|
// Suppress unused warnings
|
||||||
|
let _ = line_index2;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,533 @@
|
|||||||
|
// ─── pread_reader.rs ──────────────────────────────────────────────────────────
|
||||||
|
// pread-based FileReaderBackend implementations with 3 variants for benchmarking.
|
||||||
|
// Uses ReadCache for 4KB block caching to reduce syscalls, and posix_fadvise
|
||||||
|
// for kernel readahead control.
|
||||||
|
//
|
||||||
|
// CRITICAL: get_line() is custom — it does NOT use LineIndex::get_line() which
|
||||||
|
// requires a full &[u8] data slice. Instead it uses sampled_offsets directly
|
||||||
|
// and reads on-demand via pread.
|
||||||
|
// ──────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
use std::cell::RefCell;
|
||||||
|
use std::fs::File;
|
||||||
|
use std::io::{BufReader, Seek as _, SeekFrom};
|
||||||
|
use std::os::unix::fs::FileExt;
|
||||||
|
use std::os::unix::io::AsRawFd;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::line_index::LineIndex;
|
||||||
|
|
||||||
|
const BLOCK_SIZE: usize = 256;
|
||||||
|
const CACHE_CHUNK: usize = 4096;
|
||||||
|
|
||||||
|
// ─── ReadCache ────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Single-block read cache. Reduces syscalls by caching the last read.
|
||||||
|
/// Typical cache hit: sequential get_line() calls within the same 4KB block.
|
||||||
|
struct ReadCache {
|
||||||
|
buf: Vec<u8>,
|
||||||
|
buf_offset: u64,
|
||||||
|
buf_len: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ReadCache {
|
||||||
|
fn new() -> Self {
|
||||||
|
Self {
|
||||||
|
buf: vec![0u8; CACHE_CHUNK],
|
||||||
|
buf_offset: 0,
|
||||||
|
buf_len: 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read `len` bytes starting at `offset`. Returns a slice into the cache.
|
||||||
|
/// On cache hit (range fully within cached block), no syscall needed.
|
||||||
|
/// On miss, performs a `read_exact_at` syscall.
|
||||||
|
fn invalidate(&mut self) {
|
||||||
|
self.buf_len = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get(&mut self, file: &File, offset: u64, len: usize) -> std::io::Result<&[u8]> {
|
||||||
|
let end = offset + len as u64;
|
||||||
|
if offset >= self.buf_offset && end <= self.buf_offset + self.buf_len as u64 {
|
||||||
|
let start = (offset - self.buf_offset) as usize;
|
||||||
|
return Ok(&self.buf[start..start + len]);
|
||||||
|
}
|
||||||
|
let alloc_len = CACHE_CHUNK.max(len);
|
||||||
|
self.buf.resize(alloc_len, 0);
|
||||||
|
file.read_exact_at(&mut self.buf[..len], offset)?;
|
||||||
|
self.buf_offset = offset;
|
||||||
|
self.buf_len = len;
|
||||||
|
Ok(&self.buf[..len])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── PreadReaderCore ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Core pread-based reader. Uses ReadCache and sparse LineIndex for on-demand
|
||||||
|
/// line retrieval without mmap.
|
||||||
|
pub struct PreadReaderCore {
|
||||||
|
file: File,
|
||||||
|
line_index: LineIndex,
|
||||||
|
file_size: u64,
|
||||||
|
cache: RefCell<ReadCache>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PreadReaderCore {
|
||||||
|
/// Open file, build line index via streaming BufReader, prepare pread reader.
|
||||||
|
fn open_raw(path: &Path) -> std::io::Result<Self> {
|
||||||
|
let file = File::open(path)?;
|
||||||
|
let file_size = file.metadata()?.len();
|
||||||
|
|
||||||
|
let line_index = {
|
||||||
|
let mut reader = BufReader::new(&file);
|
||||||
|
LineIndex::from_reader(&mut reader)?
|
||||||
|
};
|
||||||
|
|
||||||
|
Ok(Self {
|
||||||
|
file,
|
||||||
|
line_index,
|
||||||
|
file_size,
|
||||||
|
cache: RefCell::new(ReadCache::new()),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Apply posix_fadvise to the entire file.
|
||||||
|
fn advise(&self, advice: libc::c_int) -> std::io::Result<()> {
|
||||||
|
let fd = self.file.as_raw_fd();
|
||||||
|
let ret = unsafe { libc::posix_fadvise(fd, 0, self.file_size as i64, advice) };
|
||||||
|
if ret != 0 {
|
||||||
|
return Err(std::io::Error::from_raw_os_error(ret));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Custom get_line using sparse index + pread.
|
||||||
|
/// Does NOT use LineIndex::get_line() (which needs full &[u8] slice).
|
||||||
|
///
|
||||||
|
/// Algorithm:
|
||||||
|
/// 1. Look up sampled_offsets[block] to get approximate byte position
|
||||||
|
/// 2. Scan forward through offset_in_block newlines via memchr
|
||||||
|
/// 3. Collect bytes from that position until next newline (may span 4KB blocks)
|
||||||
|
fn get_line_impl(&self, idx: usize) -> Option<String> {
|
||||||
|
let total = self.line_index.total_lines as usize;
|
||||||
|
if idx >= total || total == 0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
let block = idx / BLOCK_SIZE;
|
||||||
|
let offset_in_block = idx % BLOCK_SIZE;
|
||||||
|
|
||||||
|
let mut cache = self.cache.borrow_mut();
|
||||||
|
|
||||||
|
// Phase 1: Find byte position of the start of line `idx`.
|
||||||
|
// Start at sampled_offsets[block], scan forward through `offset_in_block` newlines.
|
||||||
|
let start_offset = self.line_index.sampled_offsets[block];
|
||||||
|
let mut pos = start_offset;
|
||||||
|
let mut newlines_found = 0;
|
||||||
|
|
||||||
|
while newlines_found < offset_in_block {
|
||||||
|
let remaining = self.file_size.saturating_sub(pos) as usize;
|
||||||
|
if remaining == 0 {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let to_read = CACHE_CHUNK.min(remaining);
|
||||||
|
let data = cache.get(&self.file, pos, to_read).ok()?;
|
||||||
|
|
||||||
|
for byte_pos in memchr::memchr_iter(b'\n', data) {
|
||||||
|
newlines_found += 1;
|
||||||
|
if newlines_found == offset_in_block {
|
||||||
|
pos = pos + byte_pos as u64 + 1;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if newlines_found < offset_in_block {
|
||||||
|
pos += to_read as u64;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Phase 2: Collect bytes from `pos` until next newline or EOF.
|
||||||
|
// Line data may span multiple 4KB cache blocks.
|
||||||
|
let mut result = Vec::new();
|
||||||
|
|
||||||
|
while pos < self.file_size {
|
||||||
|
let remaining = (self.file_size - pos) as usize;
|
||||||
|
let to_read = CACHE_CHUNK.min(remaining);
|
||||||
|
let data = cache.get(&self.file, pos, to_read).ok()?;
|
||||||
|
|
||||||
|
match memchr::memchr(b'\n', data) {
|
||||||
|
Some(rel) => {
|
||||||
|
result.extend_from_slice(&data[..rel]);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
None => {
|
||||||
|
result.extend_from_slice(data);
|
||||||
|
pos += to_read as u64;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
String::from_utf8(result)
|
||||||
|
.ok()
|
||||||
|
.map(|s| s.trim_end_matches(['\r', '\n']).to_owned())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Read a raw byte range from the file using pread.
|
||||||
|
fn read_range_impl(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
let end = offset.checked_add(len as u64)?;
|
||||||
|
if end > self.file_size {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
let mut buf = vec![0u8; len];
|
||||||
|
self.file.read_exact_at(&mut buf, offset).ok()?;
|
||||||
|
Some(buf)
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn file_size(&self) -> u64 {
|
||||||
|
self.file_size
|
||||||
|
}
|
||||||
|
|
||||||
|
#[inline]
|
||||||
|
pub fn total_lines(&self) -> usize {
|
||||||
|
self.line_index.line_count()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn refresh_index(&mut self) -> std::io::Result<()> {
|
||||||
|
let new_size = self.file.metadata()?.len();
|
||||||
|
self.file.seek(SeekFrom::Start(0))?;
|
||||||
|
let new_index = {
|
||||||
|
let mut reader = BufReader::new(&self.file);
|
||||||
|
LineIndex::from_reader(&mut reader)?
|
||||||
|
};
|
||||||
|
self.file_size = new_size;
|
||||||
|
self.line_index = new_index;
|
||||||
|
self.cache.get_mut().invalidate();
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── 3 Variants ───────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Variant 1: Plain pread, no fadvise.
|
||||||
|
pub struct PreadReaderPlain {
|
||||||
|
inner: PreadReaderCore,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PreadReaderPlain {
|
||||||
|
pub fn refresh_index(&mut self) -> std::io::Result<()> {
|
||||||
|
self.inner.refresh_index()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for PreadReaderPlain {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"pread_plain"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
let inner = PreadReaderCore::open_raw(path)?;
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line_impl(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range_impl(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 2: pread with POSIX_FADV_RANDOM — disable kernel readahead.
|
||||||
|
/// Best for random-access line lookup patterns.
|
||||||
|
pub struct PreadReaderRandom {
|
||||||
|
inner: PreadReaderCore,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for PreadReaderRandom {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"pread_random"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
let inner = PreadReaderCore::open_raw(path)?;
|
||||||
|
inner.advise(libc::POSIX_FADV_RANDOM)?;
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line_impl(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range_impl(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Variant 3: pread with POSIX_FADV_SEQUENTIAL — aggressive kernel readahead.
|
||||||
|
/// Best for sequential scan patterns.
|
||||||
|
pub struct PreadReaderSequential {
|
||||||
|
inner: PreadReaderCore,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FileReaderBackend for PreadReaderSequential {
|
||||||
|
fn name(&self) -> &str {
|
||||||
|
"pread_sequential"
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open(path: &Path) -> std::io::Result<Self>
|
||||||
|
where
|
||||||
|
Self: Sized,
|
||||||
|
{
|
||||||
|
let inner = PreadReaderCore::open_raw(path)?;
|
||||||
|
inner.advise(libc::POSIX_FADV_SEQUENTIAL)?;
|
||||||
|
Ok(Self { inner })
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file_size(&self) -> u64 {
|
||||||
|
self.inner.file_size()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn total_lines(&self) -> usize {
|
||||||
|
self.inner.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
self.inner.get_line_impl(idx)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
|
||||||
|
self.inner.read_range_impl(offset, len)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn close(self) {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── Tests ────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::io::Write as _;
|
||||||
|
use tempfile::NamedTempFile;
|
||||||
|
|
||||||
|
fn create_temp_file(content: &[u8]) -> NamedTempFile {
|
||||||
|
let mut f = NamedTempFile::new().unwrap();
|
||||||
|
f.write_all(content).unwrap();
|
||||||
|
f.flush().unwrap();
|
||||||
|
f
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_open_and_read_lines() {
|
||||||
|
let f = create_temp_file(b"hello\nworld\nfoo\n");
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "pread_plain");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(1), Some("world".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(3), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_out_of_bounds_returns_none() {
|
||||||
|
let f = create_temp_file(b"line1\nline2\n");
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
assert_eq!(reader.get_line(100), None);
|
||||||
|
assert_eq!(reader.get_line(usize::MAX), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_read_range() {
|
||||||
|
let content = b"hello world\nfoo bar\n";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
let range = reader.read_range(0, 5).unwrap();
|
||||||
|
assert_eq!(&range, b"hello");
|
||||||
|
let range = reader.read_range(6, 5).unwrap();
|
||||||
|
assert_eq!(&range, b"world");
|
||||||
|
assert_eq!(reader.read_range(0, 1000), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_plain_empty_file() {
|
||||||
|
let f = create_temp_file(b"");
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 0);
|
||||||
|
assert_eq!(reader.file_size(), 0);
|
||||||
|
assert_eq!(reader.get_line(0), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_random_variant() {
|
||||||
|
let f = create_temp_file(b"one\ntwo\nthree\n");
|
||||||
|
let reader = PreadReaderRandom::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "pread_random");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(2), Some("three".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_sequential_variant() {
|
||||||
|
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
|
||||||
|
let reader = PreadReaderSequential::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.name(), "pread_sequential");
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_file_size() {
|
||||||
|
let content = b"hello world";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.file_size(), content.len() as u64);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_no_trailing_newline() {
|
||||||
|
let f = create_temp_file(b"line1\nline2");
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_line_spanning_cache_boundary() {
|
||||||
|
// Create a line that spans a 4KB boundary
|
||||||
|
let mut content = vec![b'a'; 4090];
|
||||||
|
content.push(b'\n');
|
||||||
|
content.extend_from_slice(b"target\n");
|
||||||
|
let f = create_temp_file(&content);
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.get_line(0).unwrap().len(), 4090);
|
||||||
|
assert_eq!(reader.get_line(1), Some("target".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(2), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_many_lines_random_access() {
|
||||||
|
// Create 300 lines to test block boundary (256 lines per block)
|
||||||
|
let mut content = String::new();
|
||||||
|
for i in 0..300 {
|
||||||
|
content.push_str(&format!("line_{}\n", i));
|
||||||
|
}
|
||||||
|
let f = create_temp_file(content.as_bytes());
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 300);
|
||||||
|
// Test lines across the 256-line block boundary
|
||||||
|
assert_eq!(reader.get_line(0), Some("line_0".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(255), Some("line_255".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(256), Some("line_256".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(299), Some("line_299".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(300), None);
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_refresh_index_sees_appended_lines() {
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let path = dir.path().join("refresh_test.log");
|
||||||
|
|
||||||
|
// Phase 1: write 3 initial lines
|
||||||
|
{
|
||||||
|
let mut f = std::fs::File::create(&path).unwrap();
|
||||||
|
std::io::Write::write_all(&mut f, b"alpha\nbeta\ngamma\n").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut reader = PreadReaderPlain::open(&path).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 3);
|
||||||
|
assert_eq!(reader.get_line(0), Some("alpha".to_owned()));
|
||||||
|
assert_eq!(
|
||||||
|
reader.get_line(3),
|
||||||
|
None,
|
||||||
|
"should be out of bounds before append"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Phase 2: append 2 more lines
|
||||||
|
{
|
||||||
|
use std::io::Write as _;
|
||||||
|
let mut f = std::fs::OpenOptions::new()
|
||||||
|
.append(true)
|
||||||
|
.open(&path)
|
||||||
|
.unwrap();
|
||||||
|
f.write_all(b"delta\nepsilon\n").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Still stale — refresh needed
|
||||||
|
assert_eq!(reader.get_line(3), None, "stale index before refresh");
|
||||||
|
|
||||||
|
reader.refresh_index().unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 5);
|
||||||
|
assert_eq!(reader.get_line(3), Some("delta".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(4), Some("epsilon".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(5), None);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_refresh_index_no_change_is_noop() {
|
||||||
|
let f = create_temp_file(b"one\ntwo\n");
|
||||||
|
let mut reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
|
||||||
|
reader.refresh_index().unwrap();
|
||||||
|
assert_eq!(reader.total_lines(), 2);
|
||||||
|
assert_eq!(reader.get_line(0), Some("one".to_owned()));
|
||||||
|
assert_eq!(reader.get_line(1), Some("two".to_owned()));
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_truncation_graceful_error() {
|
||||||
|
// Verify that truncated file access returns None/Err instead of panicking
|
||||||
|
let f = create_temp_file(b"hello\nworld\n");
|
||||||
|
let reader = PreadReaderPlain::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
|
||||||
|
// Truncate behind the reader — pread may return zeros or error
|
||||||
|
let _ = reader.get_line(1); // must not panic
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,301 @@
|
|||||||
|
use crate::metrics;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
fn format_rss_mb(kb: u64) -> String {
|
||||||
|
format!("{:.1}MB", kb as f64 / 1024.0)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn format_faults(count: u64) -> String {
|
||||||
|
if count < 1000 {
|
||||||
|
count.to_string()
|
||||||
|
} else {
|
||||||
|
let s = count.to_string();
|
||||||
|
let mut result = String::new();
|
||||||
|
for (i, c) in s.chars().rev().enumerate() {
|
||||||
|
if i > 0 && i % 3 == 0 {
|
||||||
|
result.push(',');
|
||||||
|
}
|
||||||
|
result.push(c);
|
||||||
|
}
|
||||||
|
result.chars().rev().collect()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Format benchmark results as Markdown, grouped by category.
|
||||||
|
pub fn format_report(results: &[BenchmarkResult]) -> String {
|
||||||
|
let mut report = String::new();
|
||||||
|
report.push_str("# Benchmark: mmap vs pread\n\n");
|
||||||
|
|
||||||
|
let mut categories: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
|
||||||
|
std::collections::BTreeMap::new();
|
||||||
|
for r in results {
|
||||||
|
categories.entry(&r.category).or_default().push(r);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (category, category_results) in &categories {
|
||||||
|
report.push_str(&format!("## {}\n\n", capitalize(category)));
|
||||||
|
|
||||||
|
let mut tests: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
|
||||||
|
std::collections::BTreeMap::new();
|
||||||
|
for r in category_results {
|
||||||
|
tests.entry(&r.test_name).or_default().push(r);
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut variants: Vec<(String, String)> = Vec::new();
|
||||||
|
for r in category_results {
|
||||||
|
if !variants
|
||||||
|
.iter()
|
||||||
|
.any(|(b, v)| b == &r.backend && v == &r.variant)
|
||||||
|
{
|
||||||
|
variants.push((r.backend.clone(), r.variant.clone()));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
variants.sort();
|
||||||
|
|
||||||
|
report.push_str("### Latency\n\n");
|
||||||
|
report.push_str("| Test |");
|
||||||
|
for (backend, variant) in &variants {
|
||||||
|
report.push_str(&format!(" {} ({}) |", backend, variant));
|
||||||
|
}
|
||||||
|
report.push_str(" Winner |\n");
|
||||||
|
|
||||||
|
report.push_str("|------|");
|
||||||
|
for _ in &variants {
|
||||||
|
report.push_str("------|");
|
||||||
|
}
|
||||||
|
report.push_str("--------|\n");
|
||||||
|
|
||||||
|
for (test_name, test_results) in &tests {
|
||||||
|
report.push_str(&format!("| {} |", test_name));
|
||||||
|
|
||||||
|
let mut all_means: Vec<f64> = Vec::new();
|
||||||
|
for (backend, variant) in &variants {
|
||||||
|
if let Some(r) = test_results
|
||||||
|
.iter()
|
||||||
|
.find(|r| r.backend == *backend && r.variant == *variant)
|
||||||
|
{
|
||||||
|
let avg = if r.latency_us.is_empty() {
|
||||||
|
0.0
|
||||||
|
} else {
|
||||||
|
metrics::mean(&r.latency_us)
|
||||||
|
};
|
||||||
|
all_means.push(avg);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let use_ms = all_means.iter().any(|&v| v >= 1000.0);
|
||||||
|
|
||||||
|
let mut best_backend = String::new();
|
||||||
|
let mut best_latency = f64::MAX;
|
||||||
|
|
||||||
|
for (backend, variant) in &variants {
|
||||||
|
let matching = test_results
|
||||||
|
.iter()
|
||||||
|
.find(|r| r.backend == *backend && r.variant == *variant);
|
||||||
|
if let Some(r) = matching {
|
||||||
|
let avg = if r.latency_us.is_empty() {
|
||||||
|
0.0
|
||||||
|
} else {
|
||||||
|
metrics::mean(&r.latency_us)
|
||||||
|
};
|
||||||
|
let sd = metrics::stdev(&r.latency_us);
|
||||||
|
let p95_val = metrics::p95(&r.latency_us) as f64;
|
||||||
|
|
||||||
|
let cell = if use_ms {
|
||||||
|
format!(
|
||||||
|
"{:.2}\u{00b1}{:.2}ms (p95:{:.2}ms)",
|
||||||
|
avg / 1000.0,
|
||||||
|
sd / 1000.0,
|
||||||
|
p95_val / 1000.0
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!(
|
||||||
|
"{:.1}\u{00b1}{:.1}\u{b5}s (p95:{:.1}\u{b5}s)",
|
||||||
|
avg, sd, p95_val
|
||||||
|
)
|
||||||
|
};
|
||||||
|
report.push_str(&format!(" {} |", cell));
|
||||||
|
if avg > 0.0 && avg < best_latency {
|
||||||
|
best_latency = avg;
|
||||||
|
best_backend = format!("{} ({})", backend, variant);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
report.push_str(" - |");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
report.push_str(&format!(
|
||||||
|
" {} |\n",
|
||||||
|
if best_backend.is_empty() {
|
||||||
|
"-"
|
||||||
|
} else {
|
||||||
|
&best_backend
|
||||||
|
}
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
report.push('\n');
|
||||||
|
|
||||||
|
let has_memory = category_results
|
||||||
|
.iter()
|
||||||
|
.any(|r| r.rss_kb > 0 || r.rss_peak_kb > 0);
|
||||||
|
|
||||||
|
if has_memory {
|
||||||
|
report.push_str("### Memory\n\n");
|
||||||
|
report.push_str("| Test | Variant | RSS | Peak RSS | Page Faults |\n");
|
||||||
|
report.push_str("|------|---------|-----|----------|-------------|\n");
|
||||||
|
|
||||||
|
let mut mem_rows: Vec<&BenchmarkResult> = category_results.to_vec();
|
||||||
|
mem_rows.sort_by(|a, b| {
|
||||||
|
(&a.test_name, &a.backend, &a.variant).cmp(&(&b.test_name, &b.backend, &b.variant))
|
||||||
|
});
|
||||||
|
for r in mem_rows {
|
||||||
|
let variant_label = format!("{} ({})", r.backend, r.variant);
|
||||||
|
report.push_str(&format!(
|
||||||
|
"| {} | {} | {} | {} | {} |\n",
|
||||||
|
r.test_name,
|
||||||
|
variant_label,
|
||||||
|
format_rss_mb(r.rss_kb),
|
||||||
|
format_rss_mb(r.rss_peak_kb),
|
||||||
|
format_faults(r.page_faults),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
report.push('\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
type ExtraEntry = (String, f64);
|
||||||
|
type ExtraGroup = (String, String, Vec<ExtraEntry>);
|
||||||
|
|
||||||
|
let mut extras: Vec<ExtraGroup> = category_results
|
||||||
|
.iter()
|
||||||
|
.filter(|r| !r.extra.is_empty())
|
||||||
|
.map(|r| {
|
||||||
|
let mut pairs: Vec<(String, f64)> =
|
||||||
|
r.extra.iter().map(|(k, &v)| (k.clone(), v)).collect();
|
||||||
|
pairs.sort_by(|a, b| a.0.cmp(&b.0));
|
||||||
|
(
|
||||||
|
r.test_name.clone(),
|
||||||
|
format!("{} ({})", r.backend, r.variant),
|
||||||
|
pairs,
|
||||||
|
)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
extras.sort_by(|a, b| (&a.0, &a.1).cmp(&(&b.0, &b.1)));
|
||||||
|
|
||||||
|
if !extras.is_empty() {
|
||||||
|
report.push_str("### Extra Metrics\n\n");
|
||||||
|
report.push_str("| Test | Variant | Metric | Value |\n");
|
||||||
|
report.push_str("|------|---------|--------|-------|\n");
|
||||||
|
|
||||||
|
for (test_name, variant_label, pairs) in &extras {
|
||||||
|
for (key, val) in pairs {
|
||||||
|
report.push_str(&format!(
|
||||||
|
"| {} | {} | {} | {:.3} |\n",
|
||||||
|
test_name, variant_label, key, val
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
report.push('\n');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
report.push_str("## Summary\n\n");
|
||||||
|
report.push_str(&format!("- Total benchmarks: {}\n", results.len()));
|
||||||
|
report.push_str("- Categories: ");
|
||||||
|
report.push_str(&categories.keys().cloned().collect::<Vec<_>>().join(", "));
|
||||||
|
report.push('\n');
|
||||||
|
|
||||||
|
report
|
||||||
|
}
|
||||||
|
|
||||||
|
fn capitalize(s: &str) -> String {
|
||||||
|
let mut c = s.chars();
|
||||||
|
match c.next() {
|
||||||
|
None => String::new(),
|
||||||
|
Some(f) => f.to_uppercase().chain(c).collect(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
fn make_result(
|
||||||
|
category: &str,
|
||||||
|
test_name: &str,
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
latency_us: Vec<u64>,
|
||||||
|
) -> BenchmarkResult {
|
||||||
|
BenchmarkResult {
|
||||||
|
category: category.to_string(),
|
||||||
|
test_name: test_name.to_string(),
|
||||||
|
backend: backend.to_string(),
|
||||||
|
variant: variant.to_string(),
|
||||||
|
latency_us,
|
||||||
|
rss_kb: 0,
|
||||||
|
rss_peak_kb: 0,
|
||||||
|
page_faults: 0,
|
||||||
|
extra: HashMap::new(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn report_ordering_independent_of_input_order() {
|
||||||
|
let set_a = vec![
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_1mb",
|
||||||
|
"pread",
|
||||||
|
"default",
|
||||||
|
vec![100, 110, 105],
|
||||||
|
),
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_1mb",
|
||||||
|
"mmap",
|
||||||
|
"default",
|
||||||
|
vec![80, 85, 90],
|
||||||
|
),
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_4kb",
|
||||||
|
"pread",
|
||||||
|
"default",
|
||||||
|
vec![10, 12, 11],
|
||||||
|
),
|
||||||
|
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
|
||||||
|
];
|
||||||
|
|
||||||
|
let set_b = vec![
|
||||||
|
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_1mb",
|
||||||
|
"mmap",
|
||||||
|
"default",
|
||||||
|
vec![80, 85, 90],
|
||||||
|
),
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_4kb",
|
||||||
|
"pread",
|
||||||
|
"default",
|
||||||
|
vec![10, 12, 11],
|
||||||
|
),
|
||||||
|
make_result(
|
||||||
|
"sequential",
|
||||||
|
"read_1mb",
|
||||||
|
"pread",
|
||||||
|
"default",
|
||||||
|
vec![100, 110, 105],
|
||||||
|
),
|
||||||
|
];
|
||||||
|
|
||||||
|
let report_a = format_report(&set_a);
|
||||||
|
let report_b = format_report(&set_b);
|
||||||
|
assert_eq!(
|
||||||
|
report_a, report_b,
|
||||||
|
"Reports must be identical regardless of input order"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
use std::path::PathBuf;
|
||||||
|
use std::sync::atomic::{AtomicBool, Ordering};
|
||||||
|
|
||||||
|
/// All recognized benchmark suite identifiers.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum Suite {
|
||||||
|
Startup,
|
||||||
|
Render,
|
||||||
|
Jump,
|
||||||
|
Memory,
|
||||||
|
Growth,
|
||||||
|
Rotation,
|
||||||
|
Concurrent,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Suite {
|
||||||
|
/// All valid suite identifiers, in execution order.
|
||||||
|
pub const ALL: &[Suite] = &[
|
||||||
|
Suite::Startup,
|
||||||
|
Suite::Render,
|
||||||
|
Suite::Jump,
|
||||||
|
Suite::Memory,
|
||||||
|
Suite::Growth,
|
||||||
|
Suite::Rotation,
|
||||||
|
Suite::Concurrent,
|
||||||
|
];
|
||||||
|
|
||||||
|
#[must_use]
|
||||||
|
pub const fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
Suite::Startup => "startup",
|
||||||
|
Suite::Render => "render",
|
||||||
|
Suite::Jump => "jump",
|
||||||
|
Suite::Memory => "memory",
|
||||||
|
Suite::Growth => "growth",
|
||||||
|
Suite::Rotation => "rotation",
|
||||||
|
Suite::Concurrent => "concurrent",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::str::FromStr for Suite {
|
||||||
|
type Err = String;
|
||||||
|
|
||||||
|
fn from_str(s: &str) -> Result<Self, Self::Err> {
|
||||||
|
Suite::ALL
|
||||||
|
.iter()
|
||||||
|
.find(|suite| suite.as_str() == s)
|
||||||
|
.copied()
|
||||||
|
.ok_or_else(|| {
|
||||||
|
let valid = Suite::ALL
|
||||||
|
.iter()
|
||||||
|
.map(|s| s.as_str())
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(", ");
|
||||||
|
format!("invalid value '{s}' for '--suites': valid values are {valid}")
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct BenchConfig {
|
||||||
|
pub test_file: PathBuf,
|
||||||
|
pub quick_mode: bool,
|
||||||
|
pub suites: Option<Vec<Suite>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Track whether we have already warned about VmHWM reset failure.
|
||||||
|
/// Prevents duplicate warnings across multiple suite runs.
|
||||||
|
static HWM_WARNED: AtomicBool = AtomicBool::new(false);
|
||||||
|
|
||||||
|
pub fn warn_reset_hwm() {
|
||||||
|
if let Err(e) = MetricsCollector::reset_vm_hwm() {
|
||||||
|
if HWM_WARNED.swap(true, Ordering::Relaxed) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if e.kind() == std::io::ErrorKind::PermissionDenied {
|
||||||
|
eprintln!(
|
||||||
|
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
|
||||||
|
Memory peak values may be contaminated across benchmark suites. \
|
||||||
|
Try running as root."
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
eprintln!(
|
||||||
|
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
|
||||||
|
Memory peak values may be contaminated across benchmark suites."
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
fn reset_hwm_warned() {
|
||||||
|
HWM_WARNED.store(false, Ordering::Relaxed);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn run_all(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
let should_run = |suite: Suite| -> bool {
|
||||||
|
match &config.suites {
|
||||||
|
Some(suites) => suites.contains(&suite),
|
||||||
|
None => true,
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if should_run(Suite::Startup) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::startup::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Render) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::render::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Jump) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::jump::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Memory) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::memory::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Growth) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::growth::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Rotation) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::rotation::run(config));
|
||||||
|
}
|
||||||
|
if should_run(Suite::Concurrent) {
|
||||||
|
warn_reset_hwm();
|
||||||
|
results.extend(crate::suites::concurrent::run(config));
|
||||||
|
}
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::Suite;
|
||||||
|
use std::str::FromStr;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn parse_all_valid_suites() {
|
||||||
|
let expected = [
|
||||||
|
("startup", Suite::Startup),
|
||||||
|
("render", Suite::Render),
|
||||||
|
("jump", Suite::Jump),
|
||||||
|
("memory", Suite::Memory),
|
||||||
|
("growth", Suite::Growth),
|
||||||
|
("rotation", Suite::Rotation),
|
||||||
|
("concurrent", Suite::Concurrent),
|
||||||
|
];
|
||||||
|
for (s, expected_suite) in expected {
|
||||||
|
assert_eq!(
|
||||||
|
Suite::from_str(s).unwrap(),
|
||||||
|
expected_suite,
|
||||||
|
"failed to parse '{s}'"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn misspelled_suite_returns_error() {
|
||||||
|
let err = Suite::from_str("startp").unwrap_err();
|
||||||
|
assert!(
|
||||||
|
err.contains("invalid value 'startp'"),
|
||||||
|
"error should mention the bad value: {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn error_message_lists_all_valid_values() {
|
||||||
|
let err = Suite::from_str("bogus").unwrap_err();
|
||||||
|
for name in Suite::ALL.iter().map(|s| s.as_str()) {
|
||||||
|
assert!(
|
||||||
|
err.contains(name),
|
||||||
|
"error should list valid suite '{name}': {err}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn mixed_valid_invalid_stops_at_first_error() {
|
||||||
|
// "startup" is valid, "zzz" is not — collect hits the first Err
|
||||||
|
let names = ["startup".to_string(), "zzz".to_string()];
|
||||||
|
let result: Result<Vec<Suite>, _> = names.iter().map(|s| s.parse()).collect();
|
||||||
|
assert!(result.is_err());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn hwm_warned_flag_prevents_reentry() {
|
||||||
|
use super::reset_hwm_warned;
|
||||||
|
use std::sync::atomic::Ordering;
|
||||||
|
|
||||||
|
reset_hwm_warned();
|
||||||
|
assert!(
|
||||||
|
!super::HWM_WARNED.load(Ordering::Relaxed),
|
||||||
|
"flag should be false after reset"
|
||||||
|
);
|
||||||
|
|
||||||
|
// Simulate the flag being set (as warn_reset_hwm would do on error)
|
||||||
|
super::HWM_WARNED.store(true, Ordering::Relaxed);
|
||||||
|
|
||||||
|
// swap should now return true (old value), indicating already warned
|
||||||
|
assert!(
|
||||||
|
super::HWM_WARNED.swap(true, Ordering::Relaxed),
|
||||||
|
"swap should return the previous value (true)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn warn_reset_hwm_does_not_panic() {
|
||||||
|
use super::reset_hwm_warned;
|
||||||
|
|
||||||
|
reset_hwm_warned();
|
||||||
|
// Whether reset_vm_hwm succeeds or fails, warn_reset_hwm must not panic.
|
||||||
|
// Multiple calls must also be safe.
|
||||||
|
super::warn_reset_hwm();
|
||||||
|
super::warn_reset_hwm();
|
||||||
|
super::warn_reset_hwm();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{
|
||||||
|
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
|
||||||
|
MmapReaderSequential,
|
||||||
|
};
|
||||||
|
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
results.extend(bench_parallel_reads::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<MmapReaderRandom>(
|
||||||
|
"mmap", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_parallel_reads::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_parallel_reads<B: FileReaderBackend + Send + 'static>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = config.test_file.clone();
|
||||||
|
let iterations = if config.quick_mode { 250 } else { 1000 };
|
||||||
|
|
||||||
|
let total_lines = {
|
||||||
|
let reader = B::open(&path).expect("Failed to open file for line count");
|
||||||
|
let count = reader.total_lines();
|
||||||
|
reader.close();
|
||||||
|
count
|
||||||
|
};
|
||||||
|
|
||||||
|
let overall_start = std::time::Instant::now();
|
||||||
|
let num_threads = 4usize;
|
||||||
|
|
||||||
|
let handles: Vec<_> = (0..num_threads)
|
||||||
|
.map(|thread_id| {
|
||||||
|
let path = path.clone();
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
let reader = B::open(&path).expect("Failed to open file in thread");
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
|
||||||
|
for i in 0..iterations {
|
||||||
|
let line_idx = (thread_id * iterations + i) % total_lines.max(1);
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(line_idx);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
latencies
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let thread_latencies: Vec<Vec<u64>> = handles
|
||||||
|
.into_iter()
|
||||||
|
.map(|h| h.join().expect("Thread panicked"))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
let total_elapsed = overall_start.elapsed();
|
||||||
|
|
||||||
|
let all_latencies: Vec<u64> = thread_latencies.into_iter().flatten().collect();
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("num_threads".into(), num_threads as f64);
|
||||||
|
extra.insert("iterations_per_thread".into(), iterations as f64);
|
||||||
|
extra.insert("total_time_us".into(), total_elapsed.as_micros() as f64);
|
||||||
|
extra.insert("total_lines".into(), total_lines as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "concurrent".into(),
|
||||||
|
test_name: "parallel_reads".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: all_latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
@@ -0,0 +1,295 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::data_gen;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::MmapReaderPlain;
|
||||||
|
use crate::pread_reader::PreadReaderPlain;
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
let dir = tempfile::tempdir().expect("Failed to create temp dir");
|
||||||
|
results.extend(bench_append_visibility_mmap(config, dir.path()));
|
||||||
|
results.extend(bench_append_visibility_pread(config, dir.path()));
|
||||||
|
results.extend(bench_remap_cost(config, dir.path()));
|
||||||
|
results.extend(bench_scroll_during_append(config, dir.path()));
|
||||||
|
results.extend(bench_high_frequency_append(config, dir.path()));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_append_visibility_mmap(
|
||||||
|
config: &BenchConfig,
|
||||||
|
dir: &std::path::Path,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = data_gen::generate_growable_file(dir).expect("Failed to create growable file");
|
||||||
|
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
|
||||||
|
|
||||||
|
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open growable file");
|
||||||
|
let original_lines = reader.total_lines();
|
||||||
|
let original_size = reader.file_size();
|
||||||
|
|
||||||
|
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
|
||||||
|
|
||||||
|
let new_metadata = std::fs::metadata(&path).expect("Failed to read metadata");
|
||||||
|
let new_size = new_metadata.len() as usize;
|
||||||
|
|
||||||
|
let remap_start = std::time::Instant::now();
|
||||||
|
reader.remap(new_size).expect("Failed to remap");
|
||||||
|
let remap_elapsed = remap_start.elapsed();
|
||||||
|
|
||||||
|
let new_line_bytes =
|
||||||
|
reader.read_range(original_size, (new_size - original_size as usize).min(256));
|
||||||
|
let visible = new_line_bytes.is_some();
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("original_lines".into(), original_lines as f64);
|
||||||
|
extra.insert("appended_lines".into(), append_count as f64);
|
||||||
|
extra.insert("new_bytes_visible".into(), visible as u64 as f64);
|
||||||
|
extra.insert("original_size".into(), original_size as f64);
|
||||||
|
extra.insert("new_size".into(), new_size as f64);
|
||||||
|
extra.insert("remap_us".into(), remap_elapsed.as_micros() as f64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "growth".into(),
|
||||||
|
test_name: "append_visibility_mmap".into(),
|
||||||
|
backend: "mmap".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: vec![remap_elapsed.as_micros() as u64],
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_append_visibility_pread(
|
||||||
|
config: &BenchConfig,
|
||||||
|
dir: &std::path::Path,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("pread_growth");
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
|
||||||
|
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
|
||||||
|
|
||||||
|
let reader = PreadReaderPlain::open(&path).expect("Failed to open growable file");
|
||||||
|
let original_lines = reader.total_lines();
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
|
||||||
|
|
||||||
|
let reopen_start = std::time::Instant::now();
|
||||||
|
let new_reader = PreadReaderPlain::open(&path).expect("Failed to reopen file");
|
||||||
|
let reopen_elapsed = reopen_start.elapsed();
|
||||||
|
|
||||||
|
let new_lines = new_reader.total_lines();
|
||||||
|
let can_read_new = new_lines > original_lines;
|
||||||
|
if can_read_new {
|
||||||
|
let last_line = new_lines.saturating_sub(1);
|
||||||
|
let _ = new_reader.get_line(last_line);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("original_lines".into(), original_lines as f64);
|
||||||
|
extra.insert("appended_lines".into(), append_count as f64);
|
||||||
|
extra.insert("new_total_lines".into(), new_lines as f64);
|
||||||
|
extra.insert("reopen_us".into(), reopen_elapsed.as_micros() as f64);
|
||||||
|
|
||||||
|
new_reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "growth".into(),
|
||||||
|
test_name: "append_visibility_pread".into(),
|
||||||
|
backend: "pread".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: vec![reopen_elapsed.as_micros() as u64],
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_remap_cost(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("remap_cost");
|
||||||
|
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
|
||||||
|
let iterations: usize = if config.quick_mode { 5 } else { 20 };
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
|
||||||
|
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open file");
|
||||||
|
|
||||||
|
data_gen::append_lines(&path, append_count).expect("Failed to append");
|
||||||
|
|
||||||
|
let new_size = std::fs::metadata(&path).expect("metadata").len() as usize;
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
reader.remap(new_size).expect("remap failed");
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
let _ = std::fs::remove_file(&path);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("appended_per_iter".into(), append_count as f64);
|
||||||
|
extra.insert("iterations".into(), iterations as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "growth".into(),
|
||||||
|
test_name: "remap_cost".into(),
|
||||||
|
backend: "mmap".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_scroll_during_append(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("scroll_append");
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
|
||||||
|
let duration_secs: u64 = if config.quick_mode { 2 } else { 10 };
|
||||||
|
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
|
||||||
|
|
||||||
|
let bg_path = path.clone();
|
||||||
|
let bg_handle = std::thread::spawn(move || {
|
||||||
|
let batch_size = 100;
|
||||||
|
let batch_interval =
|
||||||
|
std::time::Duration::from_micros(1_000_000 / (append_rate / batch_size).max(1) as u64);
|
||||||
|
let start = std::time::Instant::now();
|
||||||
|
while start.elapsed().as_secs() < duration_secs {
|
||||||
|
data_gen::append_lines(&bg_path, batch_size).ok();
|
||||||
|
std::thread::sleep(batch_interval);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let mut reader = PreadReaderPlain::open(&path).expect("Failed to open file");
|
||||||
|
let initial_lines = reader.total_lines();
|
||||||
|
let mut frame_latencies = Vec::new();
|
||||||
|
let mut current_line = 0usize;
|
||||||
|
let mut refresh_count: u64 = 0;
|
||||||
|
let mut none_count: u64 = 0;
|
||||||
|
let scroll_start = std::time::Instant::now();
|
||||||
|
let mut last_refresh = std::time::Instant::now();
|
||||||
|
let refresh_interval = std::time::Duration::from_millis(250);
|
||||||
|
|
||||||
|
while scroll_start.elapsed().as_secs() < duration_secs {
|
||||||
|
if last_refresh.elapsed() >= refresh_interval {
|
||||||
|
reader.refresh_index().ok();
|
||||||
|
refresh_count += 1;
|
||||||
|
last_refresh = std::time::Instant::now();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(_line) = reader.get_line(current_line) {
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
frame_latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
current_line += 1;
|
||||||
|
} else {
|
||||||
|
none_count += 1;
|
||||||
|
std::thread::sleep(std::time::Duration::from_millis(1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let max_line = current_line;
|
||||||
|
assert!(
|
||||||
|
max_line > initial_lines,
|
||||||
|
"benchmark never read past initial {initial_lines} lines (max={max_line})"
|
||||||
|
);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
bg_handle.join().ok();
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("duration_secs".into(), duration_secs as f64);
|
||||||
|
extra.insert("append_rate_per_sec".into(), append_rate as f64);
|
||||||
|
extra.insert("frames_rendered".into(), frame_latencies.len() as f64);
|
||||||
|
extra.insert("refresh_count".into(), refresh_count as f64);
|
||||||
|
extra.insert("none_count".into(), none_count as f64);
|
||||||
|
extra.insert("initial_lines".into(), initial_lines as f64);
|
||||||
|
extra.insert("max_line_seen".into(), max_line as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "growth".into(),
|
||||||
|
test_name: "scroll_during_append".into(),
|
||||||
|
backend: "pread".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: frame_latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_high_frequency_append(
|
||||||
|
config: &BenchConfig,
|
||||||
|
dir: &std::path::Path,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("high_freq_append");
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
|
||||||
|
let duration_secs: u64 = if config.quick_mode { 3 } else { 30 };
|
||||||
|
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
|
||||||
|
let batch_size: usize = 100;
|
||||||
|
let batches_per_sec = append_rate / batch_size;
|
||||||
|
let total_batches = (duration_secs as usize * batches_per_sec).max(1);
|
||||||
|
|
||||||
|
let mut detect_latencies = Vec::with_capacity(total_batches);
|
||||||
|
|
||||||
|
for _ in 0..total_batches {
|
||||||
|
data_gen::append_lines(&path, batch_size).expect("Failed to append");
|
||||||
|
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
if let Ok(reader) = PreadReaderPlain::open(&path) {
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let _ = reader.get_line(total.saturating_sub(1));
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
detect_latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
|
||||||
|
std::thread::sleep(std::time::Duration::from_micros(
|
||||||
|
1_000_000 / batches_per_sec as u64,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("duration_secs".into(), duration_secs as f64);
|
||||||
|
extra.insert("append_rate_per_sec".into(), append_rate as f64);
|
||||||
|
extra.insert("batch_size".into(), batch_size as f64);
|
||||||
|
extra.insert("total_batches".into(), total_batches as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "growth".into(),
|
||||||
|
test_name: "high_frequency_append".into(),
|
||||||
|
backend: "pread".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: detect_latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
@@ -0,0 +1,251 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use super::FRAME_LINES;
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{
|
||||||
|
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
|
||||||
|
MmapReaderSequential,
|
||||||
|
};
|
||||||
|
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
results.extend(bench_near_jump::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_near_jump::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<MmapReaderRandom>(
|
||||||
|
"mmap", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_near_jump::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_far_jump::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_far_jump::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_far_jump::<MmapReaderRandom>("mmap", "random", config));
|
||||||
|
results.extend(bench_far_jump::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_far_jump::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_far_jump::<PreadReaderPlain>("pread", "plain", config));
|
||||||
|
results.extend(bench_far_jump::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_far_jump::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_jump_end::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_jump_end::<PreadReaderPlain>("pread", "plain", config));
|
||||||
|
|
||||||
|
results.extend(bench_reverse_scan::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_reverse_scan::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_near_jump<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let iterations: usize = if config.quick_mode { 10 } else { 100 };
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
let mut current = 0usize;
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let target = (current + 15).min(total.saturating_sub(1));
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(target);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
current = target;
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "jump".into(),
|
||||||
|
test_name: "near_jump".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra: HashMap::new(),
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_far_jump<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let repetitions: usize = if config.quick_mode { 3 } else { 10 };
|
||||||
|
|
||||||
|
let fractions = [0.25, 0.50, 0.75];
|
||||||
|
let mut latencies = Vec::new();
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
|
||||||
|
for &frac in &fractions {
|
||||||
|
let target = ((total as f64 * frac) as usize).min(total.saturating_sub(1));
|
||||||
|
for _ in 0..repetitions {
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(target);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
extra.insert("jump_positions".into(), 3.0);
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "jump".into(),
|
||||||
|
test_name: "far_jump".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_jump_end<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let iterations: usize = if config.quick_mode { 5 } else { 10 };
|
||||||
|
|
||||||
|
let last_line = total.saturating_sub(1);
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(last_line);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("last_line_idx".into(), last_line as f64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "jump".into(),
|
||||||
|
test_name: "jump_end".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_reverse_scan<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
|
||||||
|
if total <= FRAME_LINES {
|
||||||
|
reader.close();
|
||||||
|
return vec![];
|
||||||
|
}
|
||||||
|
|
||||||
|
let iterations: usize = if config.quick_mode { 5 } else { 10 };
|
||||||
|
let mut latencies = Vec::with_capacity(iterations * FRAME_LINES);
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let start = total - FRAME_LINES;
|
||||||
|
for i in (start..total).rev() {
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(i);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("lines_per_scan".into(), FRAME_LINES as f64);
|
||||||
|
extra.insert("iterations".into(), iterations as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "jump".into(),
|
||||||
|
test_name: "reverse_scan".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{
|
||||||
|
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
|
||||||
|
MmapReaderSequential,
|
||||||
|
};
|
||||||
|
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
results.extend(bench_idle_rss::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_idle_rss::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_idle_rss::<MmapReaderRandom>("mmap", "random", config));
|
||||||
|
results.extend(bench_idle_rss::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_idle_rss::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_idle_rss::<PreadReaderPlain>("pread", "plain", config));
|
||||||
|
results.extend(bench_idle_rss::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_idle_rss::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_scroll_rss::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_scroll_rss::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_jump_end_rss::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_jump_end_rss::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_rss_reclaim::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_rss_reclaim::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_idle_rss<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let reader = B::open(&config.test_file).expect("Failed to open file");
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("total_lines".into(), reader.total_lines() as f64);
|
||||||
|
extra.insert(
|
||||||
|
"file_size_mb".into(),
|
||||||
|
reader.file_size() as f64 / (1024.0 * 1024.0),
|
||||||
|
);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "memory".into(),
|
||||||
|
test_name: "idle_rss".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: vec![],
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_scroll_rss<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let reader = B::open(&config.test_file).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let sample_interval = 100_000;
|
||||||
|
let max_lines = if config.quick_mode { 100_000 } else { total };
|
||||||
|
|
||||||
|
let upper = max_lines.min(total);
|
||||||
|
let mut rss_samples = Vec::new();
|
||||||
|
let mut hwm_samples = Vec::new();
|
||||||
|
let mut lines_read = 0usize;
|
||||||
|
|
||||||
|
for i in (0..upper).step_by(sample_interval) {
|
||||||
|
if reader.get_line(i).is_some() {
|
||||||
|
lines_read += 1;
|
||||||
|
}
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
rss_samples.push(rss.vm_rss_kb);
|
||||||
|
hwm_samples.push(rss.vm_hwm_kb);
|
||||||
|
}
|
||||||
|
|
||||||
|
let final_rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("rss_samples_count".into(), rss_samples.len() as f64);
|
||||||
|
extra.insert(
|
||||||
|
"max_rss_kb".into(),
|
||||||
|
rss_samples.iter().copied().fold(0u64, u64::max) as f64,
|
||||||
|
);
|
||||||
|
extra.insert(
|
||||||
|
"max_hwm_kb".into(),
|
||||||
|
hwm_samples.iter().copied().fold(0u64, u64::max) as f64,
|
||||||
|
);
|
||||||
|
extra.insert("lines_read".into(), lines_read as f64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "memory".into(),
|
||||||
|
test_name: "scroll_rss".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: vec![],
|
||||||
|
rss_kb: final_rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: final_rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_jump_end_rss<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let reader = B::open(&config.test_file).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let last_line = total.saturating_sub(1);
|
||||||
|
|
||||||
|
let _ = reader.get_line(last_line);
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("last_line_idx".into(), last_line as f64);
|
||||||
|
extra.insert("total_lines".into(), total as f64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "memory".into(),
|
||||||
|
test_name: "jump_end_rss".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: vec![],
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_rss_reclaim<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let reader = B::open(&config.test_file).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let last_line = total.saturating_sub(1);
|
||||||
|
|
||||||
|
let _ = reader.get_line(last_line);
|
||||||
|
|
||||||
|
let wait_secs: u64 = if config.quick_mode { 5 } else { 30 };
|
||||||
|
let sample_interval: u64 = 5;
|
||||||
|
let num_samples = (wait_secs / sample_interval) as usize;
|
||||||
|
|
||||||
|
let mut rss_samples = Vec::with_capacity(num_samples);
|
||||||
|
let mut hwm_samples = Vec::with_capacity(num_samples);
|
||||||
|
|
||||||
|
for _ in 0..num_samples {
|
||||||
|
std::thread::sleep(std::time::Duration::from_secs(sample_interval));
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
rss_samples.push(rss.vm_rss_kb);
|
||||||
|
hwm_samples.push(rss.vm_hwm_kb);
|
||||||
|
}
|
||||||
|
|
||||||
|
let final_rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("wait_total_secs".into(), wait_secs as f64);
|
||||||
|
extra.insert("rss_samples".into(), rss_samples.len() as f64);
|
||||||
|
if let (Some(&first), Some(&last)) = (rss_samples.first(), rss_samples.last()) {
|
||||||
|
extra.insert("rss_first_kb".into(), first as f64);
|
||||||
|
extra.insert("rss_last_kb".into(), last as f64);
|
||||||
|
extra.insert(
|
||||||
|
"rss_change_pct".into(),
|
||||||
|
if first > 0 {
|
||||||
|
((last as f64 - first as f64) / first as f64) * 100.0
|
||||||
|
} else {
|
||||||
|
0.0
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "memory".into(),
|
||||||
|
test_name: "rss_reclaim".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: vec![],
|
||||||
|
rss_kb: final_rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: final_rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
pub mod concurrent;
|
||||||
|
pub mod growth;
|
||||||
|
pub mod jump;
|
||||||
|
pub mod memory;
|
||||||
|
pub mod render;
|
||||||
|
pub mod rotation;
|
||||||
|
pub mod startup;
|
||||||
|
|
||||||
|
/// Number of lines read per single-frame render benchmark.
|
||||||
|
/// Also used as the scan window size for reverse-scan benchmarks.
|
||||||
|
/// Shared across suites to ensure consistent workload sizing.
|
||||||
|
pub(crate) const FRAME_LINES: usize = 35;
|
||||||
@@ -0,0 +1,325 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use super::FRAME_LINES;
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{
|
||||||
|
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
|
||||||
|
MmapReaderSequential,
|
||||||
|
};
|
||||||
|
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
results.extend(bench_single_frame::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<MmapReaderRandom>(
|
||||||
|
"mmap", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_single_frame::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results.extend(bench_continuous_scroll::<MmapReaderPlain>(
|
||||||
|
"mmap", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<MmapReaderRandom>(
|
||||||
|
"mmap", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_continuous_scroll::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_single_frame<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
for (pos_name, start_line) in select_frame_positions(total) {
|
||||||
|
let mut latencies = Vec::with_capacity(FRAME_LINES);
|
||||||
|
for i in 0..FRAME_LINES {
|
||||||
|
let line_idx = start_line + i;
|
||||||
|
if line_idx >= total {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(line_idx);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
results.push(BenchmarkResult {
|
||||||
|
category: "render".into(),
|
||||||
|
test_name: format!("single_frame_{pos_name}"),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra: HashMap::new(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Select non-overlapping benchmark positions for single-frame tests.
|
||||||
|
///
|
||||||
|
/// Returns `(label, start_line)` pairs. Positions are chosen so that no
|
||||||
|
/// two frames overlap, skipping any position that cannot fit without
|
||||||
|
/// colliding with a previously selected range.
|
||||||
|
pub(crate) fn select_frame_positions(total: usize) -> Vec<(&'static str, usize)> {
|
||||||
|
let frame = FRAME_LINES;
|
||||||
|
if total == 0 {
|
||||||
|
return vec![];
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut positions: Vec<(&'static str, usize)> = vec![("head", 0)];
|
||||||
|
|
||||||
|
// Need at least 3 full frames to place head, middle, and tail without overlap.
|
||||||
|
if total >= 3 * frame {
|
||||||
|
let head_end = frame;
|
||||||
|
let tail_start = total - frame;
|
||||||
|
let gap = tail_start - head_end;
|
||||||
|
// Center the middle frame within the gap, ensuring it fits entirely
|
||||||
|
let middle_start = head_end + gap.saturating_sub(frame) / 2;
|
||||||
|
positions.push(("middle", middle_start));
|
||||||
|
positions.push(("tail", tail_start));
|
||||||
|
}
|
||||||
|
|
||||||
|
positions
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_continuous_scroll<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let path = &config.test_file;
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let total = reader.total_lines();
|
||||||
|
|
||||||
|
let iterations = if config.quick_mode { 100 } else { 1000 };
|
||||||
|
let timeout = if config.quick_mode {
|
||||||
|
std::time::Duration::from_secs(1)
|
||||||
|
} else {
|
||||||
|
std::time::Duration::from_secs(10)
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
let start = std::time::Instant::now();
|
||||||
|
|
||||||
|
for i in 0..iterations {
|
||||||
|
if start.elapsed() > timeout {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let line_idx = i % total.max(1);
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let _ = reader.get_line(line_idx);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("total_lines_scrolled".into(), latencies.len() as f64);
|
||||||
|
|
||||||
|
reader.close();
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "render".into(),
|
||||||
|
test_name: "continuous_scroll".into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
fn starts(pos: &[(&str, usize)]) -> Vec<usize> {
|
||||||
|
pos.iter().map(|&(_, s)| s).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn labels<'a>(pos: &[(&'a str, usize)]) -> Vec<&'a str> {
|
||||||
|
pos.iter().map(|&(l, _)| l).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn ranges_overlap(a_start: usize, b_start: usize) -> bool {
|
||||||
|
let a_end = a_start + FRAME_LINES;
|
||||||
|
let b_end = b_start + FRAME_LINES;
|
||||||
|
a_start < b_end && b_start < a_end
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_lines_no_positions() {
|
||||||
|
assert!(select_frame_positions(0).is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn one_line_head_only() {
|
||||||
|
let pos = select_frame_positions(1);
|
||||||
|
assert_eq!(labels(&pos), vec!["head"]);
|
||||||
|
assert_eq!(starts(&pos), vec![0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn at_frame_lines_head_only() {
|
||||||
|
let pos = select_frame_positions(FRAME_LINES);
|
||||||
|
assert_eq!(labels(&pos), vec!["head"]);
|
||||||
|
assert_eq!(starts(&pos), vec![0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn just_above_threshold_head_middle_tail() {
|
||||||
|
let total = 3 * FRAME_LINES;
|
||||||
|
let pos = select_frame_positions(total);
|
||||||
|
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
|
||||||
|
assert_eq!(starts(&pos)[0], 0);
|
||||||
|
assert_eq!(*starts(&pos).last().unwrap(), total - FRAME_LINES);
|
||||||
|
// Verify no overlap
|
||||||
|
for i in 0..pos.len() {
|
||||||
|
for j in (i + 1)..pos.len() {
|
||||||
|
assert!(
|
||||||
|
!ranges_overlap(pos[i].1, pos[j].1),
|
||||||
|
"overlap: {:?} @ {} vs {:?} @ {} (total={})",
|
||||||
|
pos[i].0,
|
||||||
|
pos[i].1,
|
||||||
|
pos[j].0,
|
||||||
|
pos[j].1,
|
||||||
|
total
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_overlap_at_total_70() {
|
||||||
|
// total=70 < 3*35=105, so only head is returned
|
||||||
|
let pos = select_frame_positions(70);
|
||||||
|
assert_eq!(labels(&pos), vec!["head"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_overlap_at_total_104() {
|
||||||
|
let pos = select_frame_positions(104);
|
||||||
|
for i in 0..pos.len() {
|
||||||
|
for j in (i + 1)..pos.len() {
|
||||||
|
assert!(
|
||||||
|
!ranges_overlap(pos[i].1, pos[j].1),
|
||||||
|
"overlap at total=104: {:?} @ {} vs {:?} @ {}",
|
||||||
|
pos[i].0,
|
||||||
|
pos[i].1,
|
||||||
|
pos[j].0,
|
||||||
|
pos[j].1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn no_overlap_at_total_105() {
|
||||||
|
let pos = select_frame_positions(105);
|
||||||
|
for i in 0..pos.len() {
|
||||||
|
for j in (i + 1)..pos.len() {
|
||||||
|
assert!(
|
||||||
|
!ranges_overlap(pos[i].1, pos[j].1),
|
||||||
|
"overlap at total=105: {:?} @ {} vs {:?} @ {}",
|
||||||
|
pos[i].0,
|
||||||
|
pos[i].1,
|
||||||
|
pos[j].0,
|
||||||
|
pos[j].1
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn middle_is_centered_between_head_and_tail() {
|
||||||
|
let total = 1000;
|
||||||
|
let pos = select_frame_positions(total);
|
||||||
|
assert_eq!(pos.len(), 3);
|
||||||
|
|
||||||
|
let (_, head_start) = pos[0];
|
||||||
|
let (_, middle_start) = pos[1];
|
||||||
|
let (_, tail_start) = pos[2];
|
||||||
|
|
||||||
|
assert_eq!(head_start, 0);
|
||||||
|
assert_eq!(tail_start, total - FRAME_LINES);
|
||||||
|
|
||||||
|
let head_end = head_start + FRAME_LINES;
|
||||||
|
let gap = tail_start - head_end;
|
||||||
|
assert_eq!(middle_start, head_end + gap.saturating_sub(FRAME_LINES) / 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn large_file_all_three_positions() {
|
||||||
|
let pos = select_frame_positions(1_000_000);
|
||||||
|
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::data_gen;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{self, MmapReaderPlain};
|
||||||
|
use crate::pread_reader::PreadReaderPlain;
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
let dir = tempfile::tempdir().expect("Failed to create temp dir");
|
||||||
|
results.extend(bench_truncate_safety_mmap(config, dir.path()));
|
||||||
|
results.extend(bench_truncate_safety_pread(config, dir.path()));
|
||||||
|
results.extend(bench_rotation_detection(config, dir.path()));
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_truncate_safety_mmap(
|
||||||
|
_config: &BenchConfig,
|
||||||
|
dir: &std::path::Path,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("trunc_mmap");
|
||||||
|
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
let mut sigbus_detected = 0usize;
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
|
||||||
|
|
||||||
|
mmap_reader::reset_sigbus_flag();
|
||||||
|
|
||||||
|
let reader = MmapReaderPlain::open(&path).expect("Failed to open file");
|
||||||
|
let original_size = reader.file_size();
|
||||||
|
|
||||||
|
let truncate_size = original_size / 2;
|
||||||
|
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
|
||||||
|
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let mid_offset = original_size as u64 / 2;
|
||||||
|
let _ = reader.read_range(mid_offset, 64);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
|
||||||
|
if mmap_reader::sigbus_flag() {
|
||||||
|
sigbus_detected += 1;
|
||||||
|
}
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("iterations".into(), iterations as f64);
|
||||||
|
extra.insert("sigbus_detected".into(), sigbus_detected as f64);
|
||||||
|
extra.insert("crashed".into(), 0.0);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "rotation".into(),
|
||||||
|
test_name: "truncate_safety_mmap".into(),
|
||||||
|
backend: "mmap".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_truncate_safety_pread(
|
||||||
|
_config: &BenchConfig,
|
||||||
|
dir: &std::path::Path,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("trunc_pread");
|
||||||
|
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
let mut error_count = 0usize;
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
|
||||||
|
|
||||||
|
let reader = PreadReaderPlain::open(&path).expect("Failed to open file");
|
||||||
|
let original_size = reader.file_size();
|
||||||
|
|
||||||
|
let truncate_size = original_size / 2;
|
||||||
|
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
|
||||||
|
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let mid_offset = original_size as u64 / 2;
|
||||||
|
let result = reader.read_range(mid_offset, 64);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
|
||||||
|
if result.is_none() {
|
||||||
|
error_count += 1;
|
||||||
|
}
|
||||||
|
reader.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("iterations".into(), iterations as f64);
|
||||||
|
extra.insert("errors_returned".into(), error_count as f64);
|
||||||
|
extra.insert("crashed".into(), 0.0);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "rotation".into(),
|
||||||
|
test_name: "truncate_safety_pread".into(),
|
||||||
|
backend: "pread".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_rotation_detection(_config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
|
||||||
|
let sub_dir = dir.join("rotation_detect");
|
||||||
|
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
|
||||||
|
|
||||||
|
let mut latencies = Vec::with_capacity(iterations);
|
||||||
|
let mut detected_count = 0usize;
|
||||||
|
|
||||||
|
for _ in 0..iterations {
|
||||||
|
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
|
||||||
|
let original_inode = MetricsCollector::get_inode(&path).expect("Failed to get inode");
|
||||||
|
|
||||||
|
let _rotated = data_gen::rotate_file(&path).expect("Failed to rotate file");
|
||||||
|
|
||||||
|
let t = std::time::Instant::now();
|
||||||
|
let detected = MetricsCollector::detect_rotation(original_inode, &path);
|
||||||
|
latencies.push(t.elapsed().as_micros() as u64);
|
||||||
|
|
||||||
|
if detected {
|
||||||
|
detected_count += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let _ = std::fs::remove_file(&path);
|
||||||
|
let rotated = sub_dir.join("growable.log.1");
|
||||||
|
let _ = std::fs::remove_file(&rotated);
|
||||||
|
}
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let mut extra = HashMap::new();
|
||||||
|
extra.insert("iterations".into(), iterations as f64);
|
||||||
|
extra.insert("rotations_detected".into(), detected_count as f64);
|
||||||
|
|
||||||
|
vec![BenchmarkResult {
|
||||||
|
category: "rotation".into(),
|
||||||
|
test_name: "rotation_detection".into(),
|
||||||
|
backend: "both".into(),
|
||||||
|
variant: "plain".into(),
|
||||||
|
latency_us: latencies,
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra,
|
||||||
|
}]
|
||||||
|
}
|
||||||
@@ -0,0 +1,143 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use crate::FileReaderBackend;
|
||||||
|
use crate::metrics::MetricsCollector;
|
||||||
|
use crate::mmap_reader::{
|
||||||
|
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
|
||||||
|
MmapReaderSequential,
|
||||||
|
};
|
||||||
|
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
|
||||||
|
use crate::runner::BenchConfig;
|
||||||
|
use crate::types::BenchmarkResult;
|
||||||
|
|
||||||
|
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
|
||||||
|
let mut results = Vec::new();
|
||||||
|
|
||||||
|
results.extend(bench_hot_open::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_hot_open::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_hot_open::<MmapReaderRandom>("mmap", "random", config));
|
||||||
|
results.extend(bench_hot_open::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_hot_open::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_hot_open::<PreadReaderPlain>("pread", "plain", config));
|
||||||
|
results.extend(bench_hot_open::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_hot_open::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
|
||||||
|
if !config.quick_mode {
|
||||||
|
match MetricsCollector::clear_file_cache(&config.test_file) {
|
||||||
|
Ok(()) => {
|
||||||
|
results.extend(bench_cold_open::<MmapReaderPlain>("mmap", "plain", config));
|
||||||
|
results.extend(bench_cold_open::<MmapReaderSequential>(
|
||||||
|
"mmap",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<MmapReaderRandom>(
|
||||||
|
"mmap", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<MmapReaderPopulate>(
|
||||||
|
"mmap", "populate", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<MmapReaderPhaseAware>(
|
||||||
|
"mmap",
|
||||||
|
"phase_aware",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<PreadReaderPlain>(
|
||||||
|
"pread", "plain", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<PreadReaderRandom>(
|
||||||
|
"pread", "random", config,
|
||||||
|
));
|
||||||
|
results.extend(bench_cold_open::<PreadReaderSequential>(
|
||||||
|
"pread",
|
||||||
|
"sequential",
|
||||||
|
config,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!(
|
||||||
|
"WARNING: Failed to clear file cache; skipping cold startup benchmarks because results would not be cold: {e}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
results
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_hot_open<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
open_and_measure::<B>(backend, variant, &config.test_file, "hot_open")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn bench_cold_open<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
config: &BenchConfig,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
match MetricsCollector::clear_file_cache(&config.test_file) {
|
||||||
|
Ok(()) => open_and_measure::<B>(backend, variant, &config.test_file, "cold_open"),
|
||||||
|
Err(e) => {
|
||||||
|
eprintln!(
|
||||||
|
"WARNING: Failed to clear file cache for {backend}/{variant}; skipping cold benchmark: {e}"
|
||||||
|
);
|
||||||
|
Vec::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn open_and_measure<B: FileReaderBackend>(
|
||||||
|
backend: &str,
|
||||||
|
variant: &str,
|
||||||
|
path: &Path,
|
||||||
|
test_name: &str,
|
||||||
|
) -> Vec<BenchmarkResult> {
|
||||||
|
let start = std::time::Instant::now();
|
||||||
|
let reader = B::open(path).expect("Failed to open file");
|
||||||
|
let elapsed = start.elapsed();
|
||||||
|
|
||||||
|
let rss = MetricsCollector::read_rss();
|
||||||
|
let faults = MetricsCollector::read_page_faults();
|
||||||
|
|
||||||
|
let result = BenchmarkResult {
|
||||||
|
category: "startup".into(),
|
||||||
|
test_name: test_name.into(),
|
||||||
|
backend: backend.into(),
|
||||||
|
variant: variant.into(),
|
||||||
|
latency_us: vec![elapsed.as_micros() as u64],
|
||||||
|
rss_kb: rss.vm_rss_kb,
|
||||||
|
rss_peak_kb: rss.vm_hwm_kb,
|
||||||
|
page_faults: faults.minor_faults + faults.major_faults,
|
||||||
|
extra: {
|
||||||
|
let mut m = HashMap::new();
|
||||||
|
m.insert("total_lines".into(), reader.total_lines() as f64);
|
||||||
|
m.insert(
|
||||||
|
"file_size_mb".into(),
|
||||||
|
reader.file_size() as f64 / (1024.0 * 1024.0),
|
||||||
|
);
|
||||||
|
m
|
||||||
|
},
|
||||||
|
};
|
||||||
|
reader.close();
|
||||||
|
vec![result]
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
pub struct BenchmarkResult {
|
||||||
|
pub category: String,
|
||||||
|
pub test_name: String,
|
||||||
|
pub backend: String,
|
||||||
|
pub variant: String,
|
||||||
|
pub latency_us: Vec<u64>,
|
||||||
|
pub rss_kb: u64,
|
||||||
|
pub rss_peak_kb: u64,
|
||||||
|
pub page_faults: u64,
|
||||||
|
pub extra: HashMap<String, f64>,
|
||||||
|
}
|
||||||
@@ -17,7 +17,8 @@ memmap2.workspace = true
|
|||||||
directories.workspace = true
|
directories.workspace = true
|
||||||
xxhash-rust.workspace = true
|
xxhash-rust.workspace = true
|
||||||
bincode.workspace = true
|
bincode.workspace = true
|
||||||
|
unicode-width.workspace = true
|
||||||
|
tempfile.workspace = true
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
insta.workspace = true
|
insta.workspace = true
|
||||||
tempfile.workspace = true
|
|
||||||
|
|||||||
@@ -9,6 +9,14 @@ use crate::io::index_cache::IndexCache;
|
|||||||
use crate::io::line_index::LineIndex;
|
use crate::io::line_index::LineIndex;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||||
|
pub enum AppendStatus {
|
||||||
|
Unchanged,
|
||||||
|
Appended(u64),
|
||||||
|
/// File shrank; mmap and index rebuilt via reload().
|
||||||
|
Reloaded,
|
||||||
|
}
|
||||||
|
|
||||||
pub struct FileReader {
|
pub struct FileReader {
|
||||||
path: PathBuf,
|
path: PathBuf,
|
||||||
mmap: Option<memmap2::Mmap>,
|
mmap: Option<memmap2::Mmap>,
|
||||||
@@ -25,20 +33,23 @@ impl FileReader {
|
|||||||
} else {
|
} else {
|
||||||
// SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。
|
// SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。
|
||||||
// memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。
|
// memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。
|
||||||
//
|
let m =
|
||||||
// ⚠️ Known limitation (Phase 5): 如果文件在 mmap 期间被外部进程截断,
|
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
|
||||||
// 访问截断区域的内存会触发 SIGBUS(致命信号,无法恢复)。
|
|
||||||
// FileWatcher Phase 将添加文件修改检测和 re-mmap 机制来处理此情况。
|
// Layer 3: mmap 后立即 stat 同一 fd,检测截断(TOCTOU 缓解,非安全证明)
|
||||||
// 在 Phase 5 中,假设打开的文件不会被外部修改。
|
let current_size = file.metadata()?.len();
|
||||||
Some(unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?)
|
if current_size < m.len() as u64 {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
Some(m)
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let line_index = {
|
// 直接从 mmap 快照构建行索引,确保索引与数据来自同一内存映射,
|
||||||
let mut reader = std::io::BufReader::new(&file);
|
// 消除 mmap + BufReader 双读之间的 TOCTOU 竞态窗口。
|
||||||
LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io {
|
let line_index = match &mmap {
|
||||||
source: e,
|
Some(m) => LineIndex::from_bytes(m.as_ref()),
|
||||||
context: "building line index".into(),
|
None => LineIndex::from_bytes(&[]),
|
||||||
})?
|
|
||||||
};
|
};
|
||||||
|
|
||||||
Ok(Self {
|
Ok(Self {
|
||||||
@@ -76,9 +87,8 @@ impl FileReader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Save the line index cache to disk.
|
|
||||||
pub fn save_cache(&self) -> std::io::Result<()> {
|
pub fn save_cache(&self) -> std::io::Result<()> {
|
||||||
IndexCache::save(&self.path, &self.line_index)?;
|
IndexCache::save_with_hash(&self.path, &self.line_index, self.data())?;
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -94,27 +104,35 @@ impl FileReader {
|
|||||||
self.mmap = None;
|
self.mmap = None;
|
||||||
|
|
||||||
if file_size > 0 {
|
if file_size > 0 {
|
||||||
self.mmap = Some(
|
let m =
|
||||||
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?,
|
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
|
||||||
);
|
let current_size = file.metadata()?.len();
|
||||||
|
self.mmap = if current_size < m.len() as u64 {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
Some(m)
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
let mut reader = std::io::BufReader::new(&file);
|
self.line_index = match &self.mmap {
|
||||||
self.line_index = LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io {
|
Some(m) => LineIndex::from_bytes(m.as_ref()),
|
||||||
source: e,
|
None => LineIndex::from_bytes(&[]),
|
||||||
context: "rebuilding line index on reload".into(),
|
};
|
||||||
})?;
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn update_for_append(&mut self) -> Result<u64> {
|
pub fn update_for_append(&mut self) -> Result<AppendStatus> {
|
||||||
let file = std::fs::File::open(&self.path)?;
|
let file = std::fs::File::open(&self.path)?;
|
||||||
let new_size = file.metadata()?.len();
|
let new_size = file.metadata()?.len();
|
||||||
let old_size = self.mmap.as_ref().map_or(0u64, |m| m.len() as u64);
|
let old_size = self.mmap.as_ref().map_or(0u64, |m| m.len() as u64);
|
||||||
|
|
||||||
if new_size <= old_size {
|
if new_size < old_size {
|
||||||
return Ok(0);
|
self.reload()?;
|
||||||
|
return Ok(AppendStatus::Reloaded);
|
||||||
|
}
|
||||||
|
if new_size == old_size {
|
||||||
|
return Ok(AppendStatus::Unchanged);
|
||||||
}
|
}
|
||||||
|
|
||||||
let old_lines = self.line_index.line_count() as u64;
|
let old_lines = self.line_index.line_count() as u64;
|
||||||
@@ -123,11 +141,19 @@ impl FileReader {
|
|||||||
let mmap =
|
let mmap =
|
||||||
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
|
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
|
||||||
|
|
||||||
|
let current_size = file.metadata()?.len();
|
||||||
|
if current_size < mmap.len() as u64 {
|
||||||
|
self.line_index = LineIndex::from_bytes(&[]);
|
||||||
|
return Ok(AppendStatus::Reloaded);
|
||||||
|
}
|
||||||
|
|
||||||
self.line_index
|
self.line_index
|
||||||
.extend_from_bytes(&mmap[old_size as usize..], old_size);
|
.extend_from_bytes(&mmap[old_size as usize..], old_size);
|
||||||
self.mmap = Some(mmap);
|
self.mmap = Some(mmap);
|
||||||
|
|
||||||
Ok(self.line_index.line_count() as u64 - old_lines)
|
Ok(AppendStatus::Appended(
|
||||||
|
self.line_index.line_count() as u64 - old_lines,
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -305,8 +331,8 @@ mod tests {
|
|||||||
file.write_all(b"ccc\nddd\n").unwrap();
|
file.write_all(b"ccc\nddd\n").unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
let new_lines = reader.update_for_append().unwrap();
|
let status = reader.update_for_append().unwrap();
|
||||||
assert_eq!(new_lines, 2);
|
assert_eq!(status, AppendStatus::Appended(2));
|
||||||
assert_eq!(reader.line_count(), 4);
|
assert_eq!(reader.line_count(), 4);
|
||||||
assert_eq!(reader.get_line(0), Some("aaa"));
|
assert_eq!(reader.get_line(0), Some("aaa"));
|
||||||
assert_eq!(reader.get_line(1), Some("bbb"));
|
assert_eq!(reader.get_line(1), Some("bbb"));
|
||||||
@@ -320,8 +346,8 @@ mod tests {
|
|||||||
let mut reader = FileReader::open(f.path()).unwrap();
|
let mut reader = FileReader::open(f.path()).unwrap();
|
||||||
assert_eq!(reader.line_count(), 1);
|
assert_eq!(reader.line_count(), 1);
|
||||||
|
|
||||||
let new_lines = reader.update_for_append().unwrap();
|
let status = reader.update_for_append().unwrap();
|
||||||
assert_eq!(new_lines, 0);
|
assert_eq!(status, AppendStatus::Unchanged);
|
||||||
assert_eq!(reader.line_count(), 1);
|
assert_eq!(reader.line_count(), 1);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -341,8 +367,11 @@ mod tests {
|
|||||||
file.write_all(b"x\n").unwrap();
|
file.write_all(b"x\n").unwrap();
|
||||||
}
|
}
|
||||||
|
|
||||||
let new_lines = reader.update_for_append().unwrap();
|
let status = reader.update_for_append().unwrap();
|
||||||
assert_eq!(new_lines, 0);
|
assert_eq!(status, AppendStatus::Reloaded);
|
||||||
|
assert_eq!(reader.line_count(), 1);
|
||||||
|
assert_eq!(reader.file_size(), 2);
|
||||||
|
assert_eq!(reader.get_line(0), Some("x"));
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -368,4 +397,164 @@ mod tests {
|
|||||||
let idx = reader.line_index();
|
let idx = reader.line_index();
|
||||||
assert_eq!(idx.line_count(), 2);
|
assert_eq!(idx.line_count(), 2);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─── TOCTOU 修复验证 ─────────────────────────────────────────────────
|
||||||
|
// open() 和 reload() 现在从 mmap 快照构建索引(from_bytes),
|
||||||
|
// 而非从独立 BufReader 读取(from_reader)。以下测试验证:
|
||||||
|
// 1. from_bytes 与 from_reader 产出完全一致的索引
|
||||||
|
// 2. 修改文件后 reload 仍返回正确内容(无残留旧状态)
|
||||||
|
// 3. 极端情况下(空文件、单行、跨块)open+reload 一致性
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_open_from_bytes_matches_from_reader() {
|
||||||
|
let cases: Vec<&[u8]> = vec![
|
||||||
|
b"",
|
||||||
|
b"hello",
|
||||||
|
b"hello\n",
|
||||||
|
b"aaa\nbbb\nccc",
|
||||||
|
b"\n",
|
||||||
|
b"a\n\nb",
|
||||||
|
];
|
||||||
|
|
||||||
|
let mut large_cases = Vec::new();
|
||||||
|
for &n in &[256usize, 300, 512] {
|
||||||
|
let mut buf = Vec::new();
|
||||||
|
for i in 0..n {
|
||||||
|
buf.extend_from_slice(format!("line{}\n", i).as_bytes());
|
||||||
|
}
|
||||||
|
large_cases.push(buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
let all_contents: Vec<&[u8]> = cases
|
||||||
|
.iter()
|
||||||
|
.copied()
|
||||||
|
.chain(large_cases.iter().map(Vec::as_slice))
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
for content in all_contents {
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let reader = FileReader::open(f.path()).unwrap();
|
||||||
|
|
||||||
|
let from_bytes_idx = LineIndex::from_bytes(content);
|
||||||
|
let mut buf_reader = std::io::BufReader::new(content);
|
||||||
|
let from_reader_idx = LineIndex::from_reader(&mut buf_reader).unwrap();
|
||||||
|
|
||||||
|
assert_eq!(
|
||||||
|
reader.line_index().total_lines(),
|
||||||
|
from_bytes_idx.total_lines(),
|
||||||
|
"open() line_count should match from_bytes for {:?}B file",
|
||||||
|
content.len()
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
from_bytes_idx.total_lines(),
|
||||||
|
from_reader_idx.total_lines(),
|
||||||
|
"from_bytes should match from_reader total_lines for {:?}B file",
|
||||||
|
content.len()
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
from_bytes_idx.sampled_offsets(),
|
||||||
|
from_reader_idx.sampled_offsets(),
|
||||||
|
"from_bytes should match from_reader sampled_offsets for {:?}B file",
|
||||||
|
content.len()
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
from_bytes_idx.has_trailing_newline(),
|
||||||
|
from_reader_idx.has_trailing_newline(),
|
||||||
|
"from_bytes should match from_reader has_trailing_newline for {:?}B file",
|
||||||
|
content.len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_reload_after_external_modify_returns_correct_content() {
|
||||||
|
let f = create_temp_file(b"aaa\nbbb\n");
|
||||||
|
let mut reader = FileReader::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.get_line(0), Some("aaa"));
|
||||||
|
assert_eq!(reader.get_line(1), Some("bbb"));
|
||||||
|
assert_eq!(reader.line_count(), 2);
|
||||||
|
|
||||||
|
// 外部修改:覆盖写入新内容
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
std::fs::write(f.path(), b"xxx\nyyy\nzzz\n").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
reader.reload().unwrap();
|
||||||
|
assert_eq!(reader.line_count(), 3);
|
||||||
|
assert_eq!(reader.get_line(0), Some("xxx"));
|
||||||
|
assert_eq!(reader.get_line(1), Some("yyy"));
|
||||||
|
assert_eq!(reader.get_line(2), Some("zzz"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_reload_after_truncate_then_rewrite_no_stale_data() {
|
||||||
|
let f = create_temp_file(b"line0\nline1\nline2\nline3\n");
|
||||||
|
let mut reader = FileReader::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.line_count(), 4);
|
||||||
|
|
||||||
|
// 截断后写入更短内容
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
let mut file = std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(f.path())
|
||||||
|
.unwrap();
|
||||||
|
file.write_all(b"new\n").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
reader.reload().unwrap();
|
||||||
|
assert_eq!(reader.line_count(), 1);
|
||||||
|
assert_eq!(reader.get_line(0), Some("new"));
|
||||||
|
assert_eq!(
|
||||||
|
reader.get_line(1),
|
||||||
|
None,
|
||||||
|
"old line should not be accessible"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_open_reload_idempotent_cross_block() {
|
||||||
|
let mut content = Vec::new();
|
||||||
|
for i in 0..600u32 {
|
||||||
|
content.extend_from_slice(format!("line{}\n", i).as_bytes());
|
||||||
|
}
|
||||||
|
let f = create_temp_file(&content);
|
||||||
|
|
||||||
|
let reader = FileReader::open(f.path()).unwrap();
|
||||||
|
let mut reloaded = FileReader::open(f.path()).unwrap();
|
||||||
|
reloaded.reload().unwrap();
|
||||||
|
|
||||||
|
assert_eq!(reader.line_count(), reloaded.line_count());
|
||||||
|
for i in 0..600 {
|
||||||
|
assert_eq!(
|
||||||
|
reader.get_line(i),
|
||||||
|
reloaded.get_line(i),
|
||||||
|
"line {i} mismatch between open and reload"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_open_stat_after_mmap_detects_truncation() {
|
||||||
|
let content = b"line0\nline1\nline2\nline3\n";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
|
||||||
|
let reader = FileReader::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.line_count(), 4);
|
||||||
|
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
let _ = std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(f.path())
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let reader = FileReader::open(f.path()).unwrap();
|
||||||
|
assert_eq!(reader.line_count(), 0);
|
||||||
|
assert_eq!(reader.file_size(), 0);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,19 +1,37 @@
|
|||||||
use std::io::{Read as _, Write as _};
|
use std::io::{Read as _, Write as _};
|
||||||
use std::path::Path;
|
use std::path::Path;
|
||||||
|
|
||||||
use crate::io::cache_util::{cache_path, CACHE_VERSION};
|
use crate::io::cache_util::{CACHE_VERSION, cache_path};
|
||||||
use crate::io::line_index::LineIndex;
|
use crate::io::line_index::LineIndex;
|
||||||
|
|
||||||
pub struct IndexCache;
|
pub struct IndexCache;
|
||||||
|
|
||||||
impl IndexCache {
|
/// Write `buf` to `dest` atomically using a unique temporary file in the same directory.
|
||||||
/// Save a `LineIndex` to disk using atomic write (write to .tmp, then rename).
|
///
|
||||||
pub fn save(file_path: &Path, index: &LineIndex) -> std::io::Result<()> {
|
/// Each call creates its own temp file via `tempfile::Builder`, eliminating collisions
|
||||||
let dest = cache_path(file_path).ok_or_else(|| {
|
/// when multiple threads (or processes) save to the same cache path concurrently.
|
||||||
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
|
/// The temp file is created in `dest.parent()` so the final `rename` stays on the
|
||||||
|
/// same filesystem and remains atomic.
|
||||||
|
fn write_cache_atomically(dest: &Path, buf: &[u8]) -> std::io::Result<()> {
|
||||||
|
let dir = dest.parent().ok_or_else(|| {
|
||||||
|
std::io::Error::new(
|
||||||
|
std::io::ErrorKind::InvalidInput,
|
||||||
|
"cache destination has no parent directory",
|
||||||
|
)
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
let file_hash = compute_file_hash(file_path)?;
|
let mut tmp = tempfile::Builder::new()
|
||||||
|
.prefix("index-cache-")
|
||||||
|
.suffix(".tmp")
|
||||||
|
.tempfile_in(dir)?;
|
||||||
|
|
||||||
|
tmp.as_file_mut().write_all(buf)?;
|
||||||
|
tmp.as_file_mut().sync_all()?;
|
||||||
|
|
||||||
|
tmp.persist(dest).map(|_| ()).map_err(|e| e.error)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn encode_cache(file_hash: u64, index: &LineIndex) -> std::io::Result<Vec<u8>> {
|
||||||
let index_bytes = bincode::serialize(index)
|
let index_bytes = bincode::serialize(index)
|
||||||
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
|
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
|
||||||
|
|
||||||
@@ -21,16 +39,36 @@ impl IndexCache {
|
|||||||
buf.push(CACHE_VERSION);
|
buf.push(CACHE_VERSION);
|
||||||
buf.extend_from_slice(&file_hash.to_le_bytes());
|
buf.extend_from_slice(&file_hash.to_le_bytes());
|
||||||
buf.extend_from_slice(&index_bytes);
|
buf.extend_from_slice(&index_bytes);
|
||||||
|
Ok(buf)
|
||||||
|
}
|
||||||
|
|
||||||
let tmp_path = dest.with_extension("index.tmp");
|
impl IndexCache {
|
||||||
{
|
/// Save a `LineIndex` to disk using atomic write (unique temp file, then rename).
|
||||||
let mut f = std::fs::File::create(&tmp_path)?;
|
///
|
||||||
f.write_all(&buf)?;
|
/// The file hash is derived from `data` (the same byte slice used to build the index),
|
||||||
f.sync_all()?;
|
/// avoiding TOCTOU issues from re-reading the file from disk.
|
||||||
|
pub fn save_with_hash(file_path: &Path, index: &LineIndex, data: &[u8]) -> std::io::Result<()> {
|
||||||
|
let dest = cache_path(file_path).ok_or_else(|| {
|
||||||
|
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let file_hash = compute_data_hash(data);
|
||||||
|
let buf = encode_cache(file_hash, index)?;
|
||||||
|
write_cache_atomically(&dest, &buf)
|
||||||
}
|
}
|
||||||
std::fs::rename(&tmp_path, &dest)?;
|
|
||||||
|
|
||||||
Ok(())
|
/// Save a `LineIndex` to disk, computing the hash by re-reading the file.
|
||||||
|
///
|
||||||
|
/// Prefer [`save_with_hash`] when the source data is already in memory
|
||||||
|
/// to avoid a TOCTOU race between indexing and hash computation.
|
||||||
|
pub fn save(file_path: &Path, index: &LineIndex) -> std::io::Result<()> {
|
||||||
|
let dest = cache_path(file_path).ok_or_else(|| {
|
||||||
|
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let file_hash = compute_file_hash(file_path)?;
|
||||||
|
let buf = encode_cache(file_hash, index)?;
|
||||||
|
write_cache_atomically(&dest, &buf)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Load a cached `LineIndex` from disk.
|
/// Load a cached `LineIndex` from disk.
|
||||||
@@ -60,6 +98,32 @@ impl IndexCache {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Compute a fingerprint from in-memory data, using the same algorithm as `compute_file_hash`.
|
||||||
|
/// Returns 0 for empty data.
|
||||||
|
pub fn compute_data_hash(data: &[u8]) -> u64 {
|
||||||
|
let len = data.len();
|
||||||
|
if len == 0 {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
let head_size = 4096.min(len);
|
||||||
|
let tail_size = 4096.min(len);
|
||||||
|
|
||||||
|
let mut hasher_state = xxhash_rust::xxh3::Xxh3::new();
|
||||||
|
hasher_state.update(&data[..head_size]);
|
||||||
|
|
||||||
|
if len > head_size + tail_size {
|
||||||
|
hasher_state.update(&data[len - tail_size..]);
|
||||||
|
} else {
|
||||||
|
// For small files the tail overlaps with the head; hash from the real tail start.
|
||||||
|
let tail_start = len.saturating_sub(tail_size);
|
||||||
|
hasher_state.update(&data[tail_start..]);
|
||||||
|
}
|
||||||
|
hasher_state.update(&(len as u64).to_le_bytes());
|
||||||
|
|
||||||
|
hasher_state.digest()
|
||||||
|
}
|
||||||
|
|
||||||
/// Compute a fast fingerprint of the file: xxhash of (head 4KB + tail 4KB + file size).
|
/// Compute a fast fingerprint of the file: xxhash of (head 4KB + tail 4KB + file size).
|
||||||
/// Returns 0 for empty files.
|
/// Returns 0 for empty files.
|
||||||
fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> {
|
fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> {
|
||||||
@@ -243,4 +307,81 @@ mod tests {
|
|||||||
|
|
||||||
assert_ne!(h1, h2, "hash should change when content changes");
|
assert_ne!(h1, h2, "hash should change when content changes");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_concurrent_save_with_hash_no_corruption() {
|
||||||
|
use std::sync::{Arc, Barrier};
|
||||||
|
|
||||||
|
let file = make_test_file(300);
|
||||||
|
let data = std::fs::read(file.path()).unwrap();
|
||||||
|
|
||||||
|
let num_threads = 8;
|
||||||
|
let iterations = 50;
|
||||||
|
let barrier = Arc::new(Barrier::new(num_threads));
|
||||||
|
let path = file.path().to_path_buf();
|
||||||
|
|
||||||
|
let handles: Vec<_> = (0..num_threads)
|
||||||
|
.map(|_| {
|
||||||
|
let barrier = Arc::clone(&barrier);
|
||||||
|
let path = path.clone();
|
||||||
|
let data = data.clone();
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
let index = LineIndex::from_bytes(&data);
|
||||||
|
barrier.wait();
|
||||||
|
for _ in 0..iterations {
|
||||||
|
IndexCache::save_with_hash(&path, &index, &data)
|
||||||
|
.expect("concurrent save_with_hash should succeed");
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
for h in handles {
|
||||||
|
h.join().expect("thread should not panic");
|
||||||
|
}
|
||||||
|
|
||||||
|
let loaded = IndexCache::load(file.path()).expect("final cache should load successfully");
|
||||||
|
let expected = LineIndex::from_bytes(&data);
|
||||||
|
assert_eq!(loaded.line_count(), expected.line_count());
|
||||||
|
assert_eq!(
|
||||||
|
loaded.sampled_offsets(),
|
||||||
|
expected.sampled_offsets(),
|
||||||
|
"concurrent writes must not produce interleaved data"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_concurrent_save_same_dest_all_succeed() {
|
||||||
|
use std::sync::{Arc, Barrier};
|
||||||
|
|
||||||
|
let dir = tempfile::tempdir().unwrap();
|
||||||
|
let dest = dir.path().join("test.index");
|
||||||
|
|
||||||
|
let payloads: Vec<Vec<u8>> = (0..8).map(|i| vec![i; 64 * 1024]).collect();
|
||||||
|
let barrier = Arc::new(Barrier::new(8));
|
||||||
|
|
||||||
|
let handles: Vec<_> = payloads
|
||||||
|
.into_iter()
|
||||||
|
.map(|payload| {
|
||||||
|
let barrier = Arc::clone(&barrier);
|
||||||
|
let dest = dest.clone();
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
barrier.wait();
|
||||||
|
write_cache_atomically(&dest, &payload)
|
||||||
|
.expect("concurrent atomic write should succeed");
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
for h in handles {
|
||||||
|
h.join().expect("thread should not panic");
|
||||||
|
}
|
||||||
|
|
||||||
|
let final_data = std::fs::read(&dest).expect("dest file should exist");
|
||||||
|
assert_eq!(
|
||||||
|
final_data.len(),
|
||||||
|
64 * 1024,
|
||||||
|
"final file must be exactly one payload"
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
@@ -20,14 +20,14 @@ const BLOCK_SIZE: usize = 256;
|
|||||||
pub struct LineIndex {
|
pub struct LineIndex {
|
||||||
// 采样偏移量:每 BLOCK_SIZE 行记录一个起始字节偏移。
|
// 采样偏移量:每 BLOCK_SIZE 行记录一个起始字节偏移。
|
||||||
// sampled_offsets[i] 存储第 (i * BLOCK_SIZE) 行的字节起始位置。
|
// sampled_offsets[i] 存储第 (i * BLOCK_SIZE) 行的字节起始位置。
|
||||||
sampled_offsets: Vec<u64>,
|
pub(crate) sampled_offsets: Vec<u64>,
|
||||||
|
|
||||||
// 文件总行数。
|
// 文件总行数。
|
||||||
total_lines: u64,
|
pub(crate) total_lines: u64,
|
||||||
|
|
||||||
// 文件最后一个字节是否是换行符 \n。
|
// 文件最后一个字节是否是换行符 \n。
|
||||||
#[allow(dead_code)]
|
#[allow(dead_code)]
|
||||||
has_trailing_newline: bool,
|
pub(crate) has_trailing_newline: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl LineIndex {
|
impl LineIndex {
|
||||||
@@ -158,7 +158,7 @@ impl LineIndex {
|
|||||||
|
|
||||||
// If the junction falls on a block boundary, record the start offset
|
// If the junction falls on a block boundary, record the start offset
|
||||||
// (analogous to from_bytes always pushing offset 0 for line 0).
|
// (analogous to from_bytes always pushing offset 0 for line 0).
|
||||||
if starts_new_line && (old_total as usize) % BLOCK_SIZE == 0 {
|
if starts_new_line && (old_total as usize).is_multiple_of(BLOCK_SIZE) {
|
||||||
self.sampled_offsets.push(start_offset);
|
self.sampled_offsets.push(start_offset);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -212,15 +212,17 @@ impl LineIndex {
|
|||||||
self.total_lines as usize
|
self.total_lines as usize
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── getter 方法 ────────────────────────────────────────────────────
|
#[cfg(test)]
|
||||||
pub(crate) fn sampled_offsets(&self) -> &[u64] {
|
pub(crate) fn sampled_offsets(&self) -> &[u64] {
|
||||||
&self.sampled_offsets
|
&self.sampled_offsets
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
pub(crate) fn total_lines(&self) -> u64 {
|
pub(crate) fn total_lines(&self) -> u64 {
|
||||||
self.total_lines
|
self.total_lines
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
pub(crate) fn has_trailing_newline(&self) -> bool {
|
pub(crate) fn has_trailing_newline(&self) -> bool {
|
||||||
self.has_trailing_newline
|
self.has_trailing_newline
|
||||||
}
|
}
|
||||||
@@ -233,7 +235,7 @@ impl LineIndex {
|
|||||||
}
|
}
|
||||||
let block = idx / BLOCK_SIZE;
|
let block = idx / BLOCK_SIZE;
|
||||||
let offset_in_block = idx % BLOCK_SIZE;
|
let offset_in_block = idx % BLOCK_SIZE;
|
||||||
let mut pos = self.sampled_offsets[block] as usize;
|
let mut pos = (*self.sampled_offsets.get(block)?) as usize;
|
||||||
for _ in 0..offset_in_block {
|
for _ in 0..offset_in_block {
|
||||||
match memchr::memchr(b'\n', &data[pos..]) {
|
match memchr::memchr(b'\n', &data[pos..]) {
|
||||||
Some(rel) => pos = pos + rel + 1,
|
Some(rel) => pos = pos + rel + 1,
|
||||||
|
|||||||
@@ -18,4 +18,5 @@ pub mod cache_util;
|
|||||||
pub mod index_cache;
|
pub mod index_cache;
|
||||||
pub mod line_sampler;
|
pub mod line_sampler;
|
||||||
pub mod progressive_reader;
|
pub mod progressive_reader;
|
||||||
|
pub mod read_cache;
|
||||||
pub mod wrap;
|
pub mod wrap;
|
||||||
|
|||||||
@@ -1,13 +1,32 @@
|
|||||||
use std::cell::RefCell;
|
use std::cell::RefCell;
|
||||||
use std::fmt;
|
use std::fmt;
|
||||||
|
use std::io::BufRead;
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
|
|
||||||
use crate::error::{CoreError, Result};
|
use crate::error::{CoreError, Result};
|
||||||
use crate::io::file_reader::FileReader;
|
use crate::io::file_reader::{AppendStatus, FileReader};
|
||||||
use crate::io::index_cache::IndexCache;
|
use crate::io::index_cache::IndexCache;
|
||||||
use crate::io::line_index::LineIndex;
|
use crate::io::line_index::LineIndex;
|
||||||
use crate::io::line_sampler::sample_line_count;
|
use crate::io::line_sampler::sample_line_count;
|
||||||
use crate::io::wrap::{format_json_line, wrap_line_chars, MAX_WRAP_INPUT_LEN};
|
use crate::io::wrap::{MAX_WRAP_INPUT_LEN, format_json_line};
|
||||||
|
|
||||||
|
// ─── Cancel-aware channel helpers ────────────────────────────────────────────
|
||||||
|
|
||||||
|
/// Send a message on `tx`, but abort if `cancel_rx` fires first.
|
||||||
|
///
|
||||||
|
/// Uses `crossbeam_channel::select!` so the thread sleeps efficiently instead
|
||||||
|
/// of busy-looping. Used for terminal messages (Complete / Error) that must
|
||||||
|
/// not be silently dropped while the receiver is still alive.
|
||||||
|
fn send_cancelable<T>(
|
||||||
|
tx: &crossbeam_channel::Sender<T>,
|
||||||
|
msg: T,
|
||||||
|
cancel_rx: &crossbeam_channel::Receiver<()>,
|
||||||
|
) {
|
||||||
|
crossbeam_channel::select! {
|
||||||
|
send(tx, msg) -> _ => {}
|
||||||
|
recv(cancel_rx) -> _ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ─── IndexerMessage ──────────────────────────────────────────────────────────
|
// ─── IndexerMessage ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
@@ -101,7 +120,7 @@ impl VisualHeightIndex {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self {
|
pub fn with_params(mut self, json_format: bool, terminal_width: usize) -> Self {
|
||||||
self.json_format = json_format;
|
self.json_format = json_format;
|
||||||
self.terminal_width = terminal_width;
|
self.terminal_width = terminal_width;
|
||||||
self
|
self
|
||||||
@@ -158,6 +177,31 @@ impl VisualHeightIndex {
|
|||||||
self.total_visual_rows += h as u64;
|
self.total_visual_rows += h as u64;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Replace the visual height of the last logical line. O(1).
|
||||||
|
///
|
||||||
|
/// Must be called **before** `extend_from_heights` so that the last line
|
||||||
|
/// index still refers to the pre-extension line.
|
||||||
|
pub fn replace_last_line_height(&mut self, new_height: usize) {
|
||||||
|
let n = self.prefix_sums.len();
|
||||||
|
if n < 2 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let last_line = n - 2;
|
||||||
|
let old_height = self.prefix_sums[last_line + 1] - self.prefix_sums[last_line];
|
||||||
|
let new_height = new_height as u64;
|
||||||
|
if new_height == old_height {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let delta = new_height.abs_diff(old_height);
|
||||||
|
if new_height > old_height {
|
||||||
|
self.prefix_sums[last_line + 1] += delta;
|
||||||
|
self.total_visual_rows += delta;
|
||||||
|
} else {
|
||||||
|
self.prefix_sums[last_line + 1] -= delta;
|
||||||
|
self.total_visual_rows -= delta;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── VisualHeightRebuildResult ────────────────────────────────────────────────
|
// ─── VisualHeightRebuildResult ────────────────────────────────────────────────
|
||||||
@@ -179,6 +223,9 @@ pub fn compute_line_visual_height(
|
|||||||
}
|
}
|
||||||
if json_format {
|
if json_format {
|
||||||
let formatted = format_json_line(line_text);
|
let formatted = format_json_line(line_text);
|
||||||
|
if formatted.len() > MAX_WRAP_INPUT_LEN {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
compute_text_visual_height(&formatted, terminal_width)
|
compute_text_visual_height(&formatted, terminal_width)
|
||||||
} else {
|
} else {
|
||||||
compute_text_visual_height(line_text, terminal_width)
|
compute_text_visual_height(line_text, terminal_width)
|
||||||
@@ -188,29 +235,11 @@ pub fn compute_line_visual_height(
|
|||||||
fn compute_text_visual_height(text: &str, width: usize) -> usize {
|
fn compute_text_visual_height(text: &str, width: usize) -> usize {
|
||||||
let mut height = 0;
|
let mut height = 0;
|
||||||
for sub_line in text.split('\n') {
|
for sub_line in text.split('\n') {
|
||||||
height += wrap_line_chars(sub_line, width).len();
|
height += crate::io::wrap::wrap_line_count(sub_line, width);
|
||||||
}
|
}
|
||||||
height.max(1)
|
height.max(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn compute_visual_heights(
|
|
||||||
reader: &FileReader,
|
|
||||||
terminal_width: usize,
|
|
||||||
json_format: bool,
|
|
||||||
) -> Vec<usize> {
|
|
||||||
let line_count = reader.line_count();
|
|
||||||
let mut visual_heights = Vec::with_capacity(line_count);
|
|
||||||
for i in 0..line_count {
|
|
||||||
let line_text = reader.get_line(i).unwrap_or("");
|
|
||||||
visual_heights.push(compute_line_visual_height(
|
|
||||||
line_text,
|
|
||||||
terminal_width,
|
|
||||||
json_format,
|
|
||||||
));
|
|
||||||
}
|
|
||||||
visual_heights
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── ReaderState ─────────────────────────────────────────────────────────────
|
// ─── ReaderState ─────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
pub enum ReaderState {
|
pub enum ReaderState {
|
||||||
@@ -234,8 +263,8 @@ pub enum ReaderState {
|
|||||||
pub fn spawn_indexer(
|
pub fn spawn_indexer(
|
||||||
path: PathBuf,
|
path: PathBuf,
|
||||||
generation: u64,
|
generation: u64,
|
||||||
terminal_width: usize,
|
_terminal_width: usize,
|
||||||
json_format: bool,
|
_json_format: bool,
|
||||||
cancel_rx: crossbeam_channel::Receiver<()>,
|
cancel_rx: crossbeam_channel::Receiver<()>,
|
||||||
) -> crossbeam_channel::Receiver<IndexerMessage> {
|
) -> crossbeam_channel::Receiver<IndexerMessage> {
|
||||||
let (tx, rx) = crossbeam_channel::bounded(10);
|
let (tx, rx) = crossbeam_channel::bounded(10);
|
||||||
@@ -244,62 +273,84 @@ pub fn spawn_indexer(
|
|||||||
let file = match std::fs::File::open(&path) {
|
let file = match std::fs::File::open(&path) {
|
||||||
Ok(f) => f,
|
Ok(f) => f,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let _ = tx.send(IndexerMessage::Error {
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
IndexerMessage::Error {
|
||||||
generation,
|
generation,
|
||||||
message: e.to_string(),
|
message: e.to_string(),
|
||||||
});
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
let file_size = match file.metadata() {
|
let target_len = match file.metadata() {
|
||||||
Ok(m) => m.len(),
|
Ok(m) => m.len(),
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let _ = tx.send(IndexerMessage::Error {
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
IndexerMessage::Error {
|
||||||
generation,
|
generation,
|
||||||
message: e.to_string(),
|
message: e.to_string(),
|
||||||
});
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
let mmap = if file_size == 0 {
|
let mut buf_reader = std::io::BufReader::with_capacity(64 * 1024, file);
|
||||||
None
|
let mut sampled_offsets: Vec<u64> = vec![0];
|
||||||
} else {
|
let mut next_line_idx: usize = 1;
|
||||||
match unsafe { memmap2::Mmap::map(&file) } {
|
|
||||||
Ok(m) => Some(m),
|
|
||||||
Err(e) => {
|
|
||||||
let _ = tx.send(IndexerMessage::Error {
|
|
||||||
generation,
|
|
||||||
message: e.to_string(),
|
|
||||||
});
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
let data = mmap.as_deref().unwrap_or(&[]);
|
|
||||||
|
|
||||||
if !data.is_empty() {
|
|
||||||
let mut newline_count: usize = 0;
|
let mut newline_count: usize = 0;
|
||||||
let mut chars_since_check: usize = 0;
|
let mut chunk_offset: u64 = 0;
|
||||||
let mut prev_pos: usize = 0;
|
let mut last_byte: Option<u8> = None;
|
||||||
|
let mut bytes_since_check: usize = 0;
|
||||||
|
|
||||||
for pos in memchr::memchr_iter(b'\n', data) {
|
loop {
|
||||||
chars_since_check += pos - prev_pos;
|
let buf = match buf_reader.fill_buf() {
|
||||||
prev_pos = pos;
|
Ok(b) => b,
|
||||||
|
Err(e) => {
|
||||||
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
IndexerMessage::Error {
|
||||||
|
generation,
|
||||||
|
message: e.to_string(),
|
||||||
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if buf.is_empty() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
if chars_since_check >= 1_000_000 {
|
if let Some(&b) = buf.last() {
|
||||||
chars_since_check = 0;
|
last_byte = Some(b);
|
||||||
|
}
|
||||||
|
|
||||||
|
for pos in memchr::memchr_iter(b'\n', buf) {
|
||||||
|
newline_count += 1;
|
||||||
|
if next_line_idx.is_multiple_of(256) {
|
||||||
|
sampled_offsets.push(chunk_offset + pos as u64 + 1);
|
||||||
|
}
|
||||||
|
next_line_idx += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let consumed = buf.len();
|
||||||
|
bytes_since_check += consumed;
|
||||||
|
chunk_offset += consumed as u64;
|
||||||
|
buf_reader.consume(consumed);
|
||||||
|
|
||||||
|
if bytes_since_check >= 1_000_000 {
|
||||||
|
bytes_since_check = 0;
|
||||||
if cancel_rx.try_recv().is_ok() {
|
if cancel_rx.try_recv().is_ok() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
}
|
if target_len > 0 {
|
||||||
|
let percent = (chunk_offset as f64 / target_len as f64) * 100.0;
|
||||||
newline_count += 1;
|
let _ = tx.try_send(IndexerMessage::Progress {
|
||||||
|
|
||||||
if newline_count % 256_000 == 0 {
|
|
||||||
let percent = (pos as f64 / file_size as f64) * 100.0;
|
|
||||||
let _ = tx.send(IndexerMessage::Progress {
|
|
||||||
generation,
|
generation,
|
||||||
percent,
|
percent,
|
||||||
lines_scanned: newline_count as u64,
|
lines_scanned: newline_count as u64,
|
||||||
@@ -312,24 +363,84 @@ pub fn spawn_indexer(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
let line_index = LineIndex::from_bytes(data);
|
let line_index = if chunk_offset == 0 {
|
||||||
|
LineIndex {
|
||||||
|
sampled_offsets: vec![],
|
||||||
|
total_lines: 0,
|
||||||
|
has_trailing_newline: false,
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
|
||||||
|
let total_lines = if has_trailing_newline && newline_count > 0 {
|
||||||
|
newline_count as u64
|
||||||
|
} else {
|
||||||
|
(1 + newline_count) as u64
|
||||||
|
};
|
||||||
|
|
||||||
let _ = IndexCache::save(&path, &line_index);
|
if has_trailing_newline && newline_count > 0 {
|
||||||
|
let trailing_line_idx = newline_count;
|
||||||
|
if trailing_line_idx.is_multiple_of(256) {
|
||||||
|
sampled_offsets.pop();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
LineIndex {
|
||||||
|
sampled_offsets,
|
||||||
|
total_lines,
|
||||||
|
has_trailing_newline,
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let mmap = if target_len == 0 {
|
||||||
|
None
|
||||||
|
} else {
|
||||||
|
match std::fs::File::open(&path) {
|
||||||
|
Ok(mmap_file) => match unsafe { memmap2::Mmap::map(&mmap_file) } {
|
||||||
|
Ok(m) => match mmap_file.metadata() {
|
||||||
|
Ok(metadata) if metadata.len() >= m.len() as u64 => Some(m),
|
||||||
|
Ok(_) | Err(_) => None,
|
||||||
|
},
|
||||||
|
Err(e) => {
|
||||||
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
IndexerMessage::Error {
|
||||||
|
generation,
|
||||||
|
message: e.to_string(),
|
||||||
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
Err(e) => {
|
||||||
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
IndexerMessage::Error {
|
||||||
|
generation,
|
||||||
|
message: e.to_string(),
|
||||||
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if let Some(data) = mmap.as_deref() {
|
||||||
|
let _ = IndexCache::save_with_hash(&path, &line_index, data);
|
||||||
|
}
|
||||||
|
|
||||||
let reader = FileReader::from_parts(path, mmap, line_index);
|
let reader = FileReader::from_parts(path, mmap, line_index);
|
||||||
|
|
||||||
let visual_height_index = if terminal_width > 0 {
|
send_cancelable(
|
||||||
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format);
|
&tx,
|
||||||
Some(VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width))
|
IndexerMessage::Complete {
|
||||||
} else {
|
|
||||||
None
|
|
||||||
};
|
|
||||||
|
|
||||||
let _ = tx.send(IndexerMessage::Complete {
|
|
||||||
generation,
|
generation,
|
||||||
reader,
|
reader,
|
||||||
visual_height_index,
|
visual_height_index: None,
|
||||||
});
|
},
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
rx
|
rx
|
||||||
@@ -355,30 +466,67 @@ pub fn spawn_visual_height_rebuild(
|
|||||||
Err(_) => return,
|
Err(_) => return,
|
||||||
};
|
};
|
||||||
|
|
||||||
let file_size = match file.metadata() {
|
let mmap = match unsafe { memmap2::Mmap::map(&file) } {
|
||||||
Ok(m) => m.len(),
|
Ok(m) => match file.metadata() {
|
||||||
|
Ok(meta) if meta.len() >= m.len() as u64 => m,
|
||||||
|
_ => return,
|
||||||
|
},
|
||||||
Err(_) => return,
|
Err(_) => return,
|
||||||
};
|
};
|
||||||
|
let data = mmap.as_ref();
|
||||||
|
let total_lines = line_index.line_count();
|
||||||
|
let mut visual_heights: Vec<usize> = Vec::with_capacity(total_lines);
|
||||||
|
|
||||||
let mmap = if file_size == 0 {
|
let mut line_start = 0usize;
|
||||||
None
|
let mut bytes_since_cancel_check = 0usize;
|
||||||
} else {
|
|
||||||
match unsafe { memmap2::Mmap::map(&file) } {
|
|
||||||
Ok(m) => Some(m),
|
|
||||||
Err(_) => return,
|
|
||||||
}
|
|
||||||
};
|
|
||||||
|
|
||||||
|
loop {
|
||||||
|
if bytes_since_cancel_check >= 1_000_000 {
|
||||||
|
bytes_since_cancel_check = 0;
|
||||||
if cancel_rx.try_recv().is_ok() {
|
if cancel_rx.try_recv().is_ok() {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let newline_rel = memchr::memchr(b'\n', &data[line_start..]);
|
||||||
|
let line_end = match newline_rel {
|
||||||
|
Some(p) => line_start + p,
|
||||||
|
None => data.len(),
|
||||||
|
};
|
||||||
|
|
||||||
|
let mut slice_end = line_end;
|
||||||
|
if slice_end > line_start && data[slice_end - 1] == b'\r' {
|
||||||
|
slice_end -= 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let line_text = std::str::from_utf8(&data[line_start..slice_end]).unwrap_or("");
|
||||||
|
visual_heights.push(compute_line_visual_height(
|
||||||
|
line_text,
|
||||||
|
terminal_width,
|
||||||
|
json_format,
|
||||||
|
));
|
||||||
|
|
||||||
|
bytes_since_cancel_check =
|
||||||
|
bytes_since_cancel_check.saturating_add(line_end - line_start + 1);
|
||||||
|
|
||||||
|
match newline_rel {
|
||||||
|
Some(_) => line_start = line_end + 1,
|
||||||
|
None => break,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if visual_heights.len() != total_lines {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
let reader = FileReader::from_parts(path, mmap, line_index);
|
|
||||||
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format);
|
|
||||||
let index =
|
let index =
|
||||||
VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width);
|
VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width);
|
||||||
|
|
||||||
let _ = tx.send(VisualHeightRebuildResult { generation, index });
|
send_cancelable(
|
||||||
|
&tx,
|
||||||
|
VisualHeightRebuildResult { generation, index },
|
||||||
|
&cancel_rx,
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
rx
|
rx
|
||||||
@@ -656,10 +804,10 @@ impl ProgressiveFileReader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn update_for_append(&mut self) -> Result<u64> {
|
pub fn update_for_append(&mut self) -> Result<AppendStatus> {
|
||||||
match &mut self.state {
|
match &mut self.state {
|
||||||
ReaderState::Ready { reader, .. } => reader.update_for_append(),
|
ReaderState::Ready { reader, .. } => reader.update_for_append(),
|
||||||
_ => Ok(0),
|
_ => Ok(AppendStatus::Unchanged),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -675,7 +823,7 @@ impl ProgressiveFileReader {
|
|||||||
|
|
||||||
pub fn start_visual_height_rebuild(&mut self, terminal_width: usize, json_format: bool) {
|
pub fn start_visual_height_rebuild(&mut self, terminal_width: usize, json_format: bool) {
|
||||||
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
|
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
|
||||||
let _ = tx.send(());
|
let _ = tx.try_send(());
|
||||||
}
|
}
|
||||||
|
|
||||||
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
@@ -730,10 +878,10 @@ impl ProgressiveFileReader {
|
|||||||
impl Drop for ProgressiveFileReader {
|
impl Drop for ProgressiveFileReader {
|
||||||
fn drop(&mut self) {
|
fn drop(&mut self) {
|
||||||
if let Some(tx) = &self.cancel_tx {
|
if let Some(tx) = &self.cancel_tx {
|
||||||
let _ = tx.send(());
|
let _ = tx.try_send(());
|
||||||
}
|
}
|
||||||
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
|
if let Some(tx) = self.vh_rebuild_cancel_tx.take() {
|
||||||
let _ = tx.send(());
|
let _ = tx.try_send(());
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -991,12 +1139,10 @@ mod tests {
|
|||||||
assert_eq!(reader.get_line(0), Some("line1"));
|
assert_eq!(reader.get_line(0), Some("line1"));
|
||||||
assert_eq!(reader.get_line(1), Some("line2"));
|
assert_eq!(reader.get_line(1), Some("line2"));
|
||||||
assert_eq!(reader.get_line(2), Some("line3"));
|
assert_eq!(reader.get_line(2), Some("line3"));
|
||||||
assert!(visual_height_index.is_some());
|
assert!(
|
||||||
let idx = visual_height_index.unwrap();
|
visual_height_index.is_none(),
|
||||||
assert_eq!(idx.total_visual_rows(), 3);
|
"spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
|
||||||
assert_eq!(idx.visual_row_to_logical_row(0), 0);
|
);
|
||||||
assert_eq!(idx.visual_row_to_logical_row(1), 1);
|
|
||||||
assert_eq!(idx.visual_row_to_logical_row(2), 2);
|
|
||||||
}
|
}
|
||||||
other => panic!("expected Complete, got {:?}", other),
|
other => panic!("expected Complete, got {:?}", other),
|
||||||
}
|
}
|
||||||
@@ -1193,20 +1339,25 @@ mod tests {
|
|||||||
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
let rx = spawn_indexer(temp.path().to_path_buf(), 1, 80, false, cancel_rx);
|
let rx = spawn_indexer(temp.path().to_path_buf(), 1, 80, false, cancel_rx);
|
||||||
let msg = rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap();
|
loop {
|
||||||
|
match rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap() {
|
||||||
match msg {
|
IndexerMessage::Progress { .. } => continue,
|
||||||
IndexerMessage::Complete {
|
IndexerMessage::Complete {
|
||||||
|
reader,
|
||||||
visual_height_index,
|
visual_height_index,
|
||||||
..
|
..
|
||||||
} => {
|
} => {
|
||||||
let idx = visual_height_index.expect("should have visual height index");
|
assert_eq!(reader.line_count(), 2);
|
||||||
assert_eq!(idx.visual_height_of_line(0), 1);
|
assert!(
|
||||||
assert_eq!(idx.visual_height_of_line(1), 1);
|
visual_height_index.is_none(),
|
||||||
|
"spawn_indexer no longer builds VHI inline; UI triggers rebuild post-layout"
|
||||||
|
);
|
||||||
|
break;
|
||||||
}
|
}
|
||||||
other => panic!("expected Complete, got {:?}", other),
|
other => panic!("expected Complete, got {:?}", other),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_visual_height_index_is_valid_for() {
|
fn test_visual_height_index_is_valid_for() {
|
||||||
@@ -1258,6 +1409,247 @@ mod tests {
|
|||||||
idx.extend_from_heights(&[1, 2, 3]);
|
idx.extend_from_heights(&[1, 2, 3]);
|
||||||
|
|
||||||
assert_eq!(idx.total_visual_rows(), 6);
|
assert_eq!(idx.total_visual_rows(), 6);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_increase() {
|
||||||
|
let heights = [2, 3];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 5);
|
||||||
|
assert_eq!(idx.visual_height_of_line(1), 3);
|
||||||
|
|
||||||
|
idx.replace_last_line_height(7);
|
||||||
|
|
||||||
|
assert_eq!(idx.visual_height_of_line(0), 2);
|
||||||
|
assert_eq!(idx.visual_height_of_line(1), 7);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 9);
|
||||||
|
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
|
||||||
|
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_decrease() {
|
||||||
|
let heights = [2, 5];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
|
||||||
|
idx.replace_last_line_height(1);
|
||||||
|
|
||||||
|
assert_eq!(idx.visual_height_of_line(0), 2);
|
||||||
|
assert_eq!(idx.visual_height_of_line(1), 1);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 3);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_same_is_noop() {
|
||||||
|
let heights = [2, 3];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
let total_before = idx.total_visual_rows();
|
||||||
|
|
||||||
|
idx.replace_last_line_height(3);
|
||||||
|
|
||||||
|
assert_eq!(idx.total_visual_rows(), total_before);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_then_extend() {
|
||||||
|
let heights = [2, 1];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 3);
|
||||||
|
|
||||||
|
idx.replace_last_line_height(4);
|
||||||
|
idx.extend_from_heights(&[3]);
|
||||||
|
|
||||||
assert_eq!(idx.line_count(), 3);
|
assert_eq!(idx.line_count(), 3);
|
||||||
|
assert_eq!(idx.visual_height_of_line(0), 2);
|
||||||
|
assert_eq!(idx.visual_height_of_line(1), 4);
|
||||||
|
assert_eq!(idx.visual_height_of_line(2), 3);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 9);
|
||||||
|
assert_eq!(idx.cursor_to_first_visual_row(0), 0);
|
||||||
|
assert_eq!(idx.cursor_to_first_visual_row(1), 2);
|
||||||
|
assert_eq!(idx.cursor_to_first_visual_row(2), 6);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_single_line() {
|
||||||
|
let heights = [5];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
|
||||||
|
idx.replace_last_line_height(2);
|
||||||
|
|
||||||
|
assert_eq!(idx.visual_height_of_line(0), 2);
|
||||||
|
assert_eq!(idx.total_visual_rows(), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_replace_last_line_height_empty_index() {
|
||||||
|
let heights: [usize; 0] = [];
|
||||||
|
let mut idx = VisualHeightIndex::build(&heights);
|
||||||
|
|
||||||
|
idx.replace_last_line_height(5);
|
||||||
|
|
||||||
|
assert_eq!(idx.total_visual_rows(), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_spawn_indexer_file_truncated_during_scan() {
|
||||||
|
let mut content = Vec::new();
|
||||||
|
for i in 0..100_000 {
|
||||||
|
writeln!(content, "line number {:08}", i).unwrap();
|
||||||
|
}
|
||||||
|
let f = create_temp_file(&content);
|
||||||
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
|
||||||
|
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
let _ = std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(f.path())
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let result = rx.recv_timeout(std::time::Duration::from_secs(10));
|
||||||
|
match result {
|
||||||
|
Ok(IndexerMessage::Complete { reader, .. }) => {
|
||||||
|
assert!(reader.line_count() <= 100_000);
|
||||||
|
}
|
||||||
|
Ok(IndexerMessage::Error { .. }) => {}
|
||||||
|
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
|
||||||
|
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
|
||||||
|
other => panic!("unexpected: {:?}", other),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_spawn_visual_height_rebuild_line_count_mismatch_discards() {
|
||||||
|
let content = b"line0\nline1\nline2\n";
|
||||||
|
let f = create_temp_file(content);
|
||||||
|
let data = std::fs::read(f.path()).unwrap();
|
||||||
|
let index = LineIndex::from_bytes(&data);
|
||||||
|
IndexCache::save(f.path(), &index).unwrap();
|
||||||
|
|
||||||
|
{
|
||||||
|
use std::io::Write;
|
||||||
|
let mut file = std::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(f.path())
|
||||||
|
.unwrap();
|
||||||
|
file.write_all(b"only_one_line\n").unwrap();
|
||||||
|
}
|
||||||
|
|
||||||
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
let rx = spawn_visual_height_rebuild(f.path().to_path_buf(), 1, 80, false, cancel_rx);
|
||||||
|
|
||||||
|
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
|
||||||
|
match result {
|
||||||
|
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
|
||||||
|
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
|
||||||
|
Ok(_) => panic!("should have been discarded due to line count mismatch"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_send_cancelable_delivers_on_empty_channel() {
|
||||||
|
let (tx, rx) = crossbeam_channel::bounded(2);
|
||||||
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
send_cancelable(&tx, 42, &cancel_rx);
|
||||||
|
|
||||||
|
assert_eq!(rx.try_recv(), Ok(42));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_send_cancelable_aborts_on_cancel() {
|
||||||
|
let (tx, rx) = crossbeam_channel::bounded(1);
|
||||||
|
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
tx.send("filler").unwrap();
|
||||||
|
|
||||||
|
let handle = std::thread::spawn(move || {
|
||||||
|
send_cancelable(&tx, "important", &cancel_rx);
|
||||||
|
});
|
||||||
|
|
||||||
|
cancel_tx.send(()).unwrap();
|
||||||
|
handle.join().unwrap();
|
||||||
|
|
||||||
|
assert_eq!(rx.try_recv(), Ok("filler"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_send_cancelable_drains_when_room_available() {
|
||||||
|
let (tx, rx) = crossbeam_channel::bounded(1);
|
||||||
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
tx.send("first").unwrap();
|
||||||
|
|
||||||
|
let rx_clone = rx.clone();
|
||||||
|
let handle = std::thread::spawn(move || {
|
||||||
|
std::thread::sleep(std::time::Duration::from_millis(50));
|
||||||
|
let _ = rx_clone.try_recv();
|
||||||
|
});
|
||||||
|
|
||||||
|
send_cancelable(&tx, "second", &cancel_rx);
|
||||||
|
handle.join().unwrap();
|
||||||
|
|
||||||
|
assert_eq!(rx.try_recv(), Ok("second"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_progress_try_send_does_not_block_full_channel() {
|
||||||
|
let mut content = Vec::new();
|
||||||
|
for i in 0..50_000 {
|
||||||
|
writeln!(content, "line number {:08}", i).unwrap();
|
||||||
|
}
|
||||||
|
let f = create_temp_file(&content);
|
||||||
|
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
|
||||||
|
|
||||||
|
let mut got_complete = false;
|
||||||
|
let timeout = std::time::Duration::from_secs(15);
|
||||||
|
let start = std::time::Instant::now();
|
||||||
|
while start.elapsed() < timeout {
|
||||||
|
match rx.recv_timeout(std::time::Duration::from_secs(1)) {
|
||||||
|
Ok(IndexerMessage::Progress { .. }) => {}
|
||||||
|
Ok(IndexerMessage::Complete { .. }) => {
|
||||||
|
got_complete = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
Ok(IndexerMessage::Error { message, .. }) => {
|
||||||
|
panic!("unexpected error: {}", message);
|
||||||
|
}
|
||||||
|
Err(e) => panic!("recv error: {:?}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert!(
|
||||||
|
got_complete,
|
||||||
|
"indexer should complete even when Progress fills channel"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_indexer_cancel_with_full_channel() {
|
||||||
|
let mut content = Vec::new();
|
||||||
|
for i in 0..500_000 {
|
||||||
|
writeln!(content, "line number {:08}", i).unwrap();
|
||||||
|
}
|
||||||
|
let f = create_temp_file(&content);
|
||||||
|
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
|
||||||
|
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
|
||||||
|
|
||||||
|
cancel_tx.send(()).unwrap();
|
||||||
|
|
||||||
|
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
|
||||||
|
match result {
|
||||||
|
Err(crossbeam_channel::RecvTimeoutError::Timeout)
|
||||||
|
| Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
|
||||||
|
Ok(IndexerMessage::Complete { .. }) => {}
|
||||||
|
Ok(IndexerMessage::Error { .. }) => {}
|
||||||
|
Ok(IndexerMessage::Progress { .. }) => {}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -56,9 +56,15 @@ impl LruReadCache {
|
|||||||
/// on a hit, or fills a cache slot on a miss. Cross-block reads go through
|
/// on a hit, or fills a cache slot on a miss. Cross-block reads go through
|
||||||
/// the spill buffer and are not cached.
|
/// the spill buffer and are not cached.
|
||||||
pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> {
|
pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> {
|
||||||
|
if len == 0 {
|
||||||
|
return Ok(&[]);
|
||||||
|
}
|
||||||
|
|
||||||
let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1);
|
let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1);
|
||||||
let request_end = offset.saturating_add(len as u64);
|
let request_end = offset.checked_add(len as u64).ok_or_else(|| {
|
||||||
let block_end = aligned_key + BLOCK_ALIGN as u64;
|
io::Error::new(io::ErrorKind::InvalidInput, "read range overflows u64")
|
||||||
|
})?;
|
||||||
|
let block_end = aligned_key.saturating_add(BLOCK_ALIGN as u64);
|
||||||
|
|
||||||
if request_end > block_end {
|
if request_end > block_end {
|
||||||
self.spill_buf.resize(len, 0);
|
self.spill_buf.resize(len, 0);
|
||||||
@@ -74,7 +80,8 @@ impl LruReadCache {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let hit_idx = self.slots.iter().position(|slot| {
|
let hit_idx = self.slots.iter().position(|slot| {
|
||||||
slot.block_offset == aligned_key && request_end <= slot.block_offset + slot.len as u64
|
let slot_end = slot.block_offset.saturating_add(slot.len as u64);
|
||||||
|
slot.len > 0 && slot.block_offset == aligned_key && request_end <= slot_end
|
||||||
});
|
});
|
||||||
|
|
||||||
if let Some(idx) = hit_idx {
|
if let Some(idx) = hit_idx {
|
||||||
@@ -96,8 +103,8 @@ impl LruReadCache {
|
|||||||
let slot = &mut self.slots[evict_idx];
|
let slot = &mut self.slots[evict_idx];
|
||||||
let bytes_read = file.read_at(&mut slot.buf, aligned_key)?;
|
let bytes_read = file.read_at(&mut slot.buf, aligned_key)?;
|
||||||
|
|
||||||
// Note: get(file, 0, 0) on an empty file now returns Err (old code returned Ok(&[])).
|
// Non-empty reads that return 0 are EOF. Zero-length reads are handled above
|
||||||
// No callers pass len == 0, so this is a safe semantic change.
|
// as a successful no-op.
|
||||||
if bytes_read == 0 {
|
if bytes_read == 0 {
|
||||||
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
|
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
|
||||||
}
|
}
|
||||||
@@ -107,7 +114,8 @@ impl LruReadCache {
|
|||||||
slot.last_access = self.tick;
|
slot.last_access = self.tick;
|
||||||
self.tick += 1;
|
self.tick += 1;
|
||||||
|
|
||||||
if request_end > aligned_key + bytes_read as u64 {
|
let bytes_end = aligned_key.saturating_add(bytes_read as u64);
|
||||||
|
if request_end > bytes_end {
|
||||||
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
|
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -118,7 +126,9 @@ impl LruReadCache {
|
|||||||
/// Invalidate all cache slots and the spill buffer.
|
/// Invalidate all cache slots and the spill buffer.
|
||||||
pub fn clear(&mut self) {
|
pub fn clear(&mut self) {
|
||||||
for slot in &mut self.slots {
|
for slot in &mut self.slots {
|
||||||
|
slot.block_offset = 0;
|
||||||
slot.len = 0;
|
slot.len = 0;
|
||||||
|
slot.last_access = 0;
|
||||||
}
|
}
|
||||||
self.spill_len = 0;
|
self.spill_len = 0;
|
||||||
}
|
}
|
||||||
@@ -314,9 +324,11 @@ mod tests {
|
|||||||
|
|
||||||
cache.clear();
|
cache.clear();
|
||||||
|
|
||||||
// All slots should have len == 0.
|
// All slots should be fully reset.
|
||||||
for slot in &cache.slots {
|
for slot in &cache.slots {
|
||||||
|
assert_eq!(slot.block_offset, 0);
|
||||||
assert_eq!(slot.len, 0);
|
assert_eq!(slot.len, 0);
|
||||||
|
assert_eq!(slot.last_access, 0);
|
||||||
}
|
}
|
||||||
assert_eq!(cache.spill_len, 0);
|
assert_eq!(cache.spill_len, 0);
|
||||||
|
|
||||||
@@ -432,4 +444,52 @@ mod tests {
|
|||||||
assert_eq!(&line2[..4090], &[b'B'; 4090]);
|
assert_eq!(&line2[..4090], &[b'B'; 4090]);
|
||||||
assert_eq!(line2[4090], b'\n');
|
assert_eq!(line2[4090], b'\n');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_len_read_is_noop_on_fresh_cache() {
|
||||||
|
let f = make_file(b"");
|
||||||
|
let file = File::open(f.path()).unwrap();
|
||||||
|
let mut cache = ReadCache::new();
|
||||||
|
|
||||||
|
let result = cache.get(&file, 0, 0).unwrap();
|
||||||
|
assert!(result.is_empty());
|
||||||
|
assert_eq!(cache.tick, 0);
|
||||||
|
assert!(cache.slots.iter().all(|s| s.len == 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_len_read_is_noop_on_populated_cache() {
|
||||||
|
let f = make_file(b"abc");
|
||||||
|
let file = File::open(f.path()).unwrap();
|
||||||
|
let mut cache = ReadCache::new();
|
||||||
|
|
||||||
|
cache.get(&file, 0, 1).unwrap();
|
||||||
|
let tick_before = cache.tick;
|
||||||
|
|
||||||
|
let result = cache.get(&file, 0, 0).unwrap();
|
||||||
|
assert!(result.is_empty());
|
||||||
|
assert_eq!(cache.tick, tick_before);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn zero_len_read_at_max_offset_is_ok() {
|
||||||
|
let f = make_file(b"");
|
||||||
|
let file = File::open(f.path()).unwrap();
|
||||||
|
let mut cache = ReadCache::new();
|
||||||
|
|
||||||
|
let result = cache.get(&file, u64::MAX, 0).unwrap();
|
||||||
|
assert!(result.is_empty());
|
||||||
|
assert_eq!(cache.tick, 0);
|
||||||
|
assert!(cache.slots.iter().all(|s| s.len == 0));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn nonzero_read_range_overflow_returns_invalid_input() {
|
||||||
|
let f = make_file(b"abc");
|
||||||
|
let file = File::open(f.path()).unwrap();
|
||||||
|
let mut cache = ReadCache::new();
|
||||||
|
|
||||||
|
let err = cache.get(&file, u64::MAX, 1).unwrap_err();
|
||||||
|
assert_eq!(err.kind(), io::ErrorKind::InvalidInput);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
+302
-21
@@ -1,10 +1,21 @@
|
|||||||
|
use std::borrow::Cow;
|
||||||
|
|
||||||
/// Maximum input length for wrap/format operations (10 MB).
|
/// Maximum input length for wrap/format operations (10 MB).
|
||||||
/// Lines exceeding this are returned as-is to avoid pathological cases.
|
/// Callers should check against this constant before invoking `wrap_line_chars`
|
||||||
|
/// to avoid pathological cases on oversized lines.
|
||||||
pub const MAX_WRAP_INPUT_LEN: usize = 10 * 1024 * 1024;
|
pub const MAX_WRAP_INPUT_LEN: usize = 10 * 1024 * 1024;
|
||||||
|
|
||||||
/// Split a line into chunks of exactly `width` characters (display columns).
|
/// Column spacing for tab stop alignment.
|
||||||
|
const TAB_WIDTH: usize = 4;
|
||||||
|
|
||||||
|
/// Split a line into chunks of exactly `width` display columns.
|
||||||
/// For a log viewer, we want character-level wrapping, not word-level.
|
/// For a log viewer, we want character-level wrapping, not word-level.
|
||||||
|
/// Uses `unicode-width` for correct CJK/emoji/zero-width handling.
|
||||||
|
/// Tab characters expand to the next tab-stop boundary and split across
|
||||||
|
/// rows when the expansion exceeds the remaining width.
|
||||||
pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
|
pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
|
||||||
|
use unicode_width::UnicodeWidthChar;
|
||||||
|
|
||||||
if width == 0 {
|
if width == 0 {
|
||||||
return vec![String::new()];
|
return vec![String::new()];
|
||||||
}
|
}
|
||||||
@@ -15,23 +26,42 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
|
|||||||
let mut row = String::new();
|
let mut row = String::new();
|
||||||
let mut col = 0;
|
let mut col = 0;
|
||||||
for ch in line.chars() {
|
for ch in line.chars() {
|
||||||
let w = if ch == '\t' { 4 } else { 1 };
|
if ch == '\t' {
|
||||||
|
let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
|
||||||
|
let mut remaining = tab_stop;
|
||||||
|
while remaining > 0 {
|
||||||
|
let avail = width.saturating_sub(col);
|
||||||
|
let fill = remaining.min(avail);
|
||||||
|
for _ in 0..fill {
|
||||||
|
row.push(' ');
|
||||||
|
}
|
||||||
|
col += fill;
|
||||||
|
remaining -= fill;
|
||||||
|
if col >= width {
|
||||||
|
result.push(std::mem::take(&mut row));
|
||||||
|
col = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let w = if ch.is_control() {
|
||||||
|
// Control characters (except tab): width 0, still pushed to preserve content.
|
||||||
|
// Visible rendering is the caller's responsibility.
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
ch.width().unwrap_or(0)
|
||||||
|
};
|
||||||
if col + w > width && !row.is_empty() {
|
if col + w > width && !row.is_empty() {
|
||||||
result.push(std::mem::take(&mut row));
|
result.push(std::mem::take(&mut row));
|
||||||
col = 0;
|
col = 0;
|
||||||
}
|
}
|
||||||
if ch == '\t' {
|
|
||||||
row.push_str(" ");
|
|
||||||
col += 4;
|
|
||||||
} else {
|
|
||||||
row.push(ch);
|
row.push(ch);
|
||||||
col += w;
|
col += w;
|
||||||
}
|
|
||||||
if col >= width {
|
if col >= width {
|
||||||
result.push(std::mem::take(&mut row));
|
result.push(std::mem::take(&mut row));
|
||||||
col = 0;
|
col = 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
if !row.is_empty() {
|
if !row.is_empty() {
|
||||||
result.push(row);
|
result.push(row);
|
||||||
}
|
}
|
||||||
@@ -41,21 +71,81 @@ pub fn wrap_line_chars(line: &str, width: usize) -> Vec<String> {
|
|||||||
result
|
result
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Format a line as pretty-printed JSON if it's a JSON Object.
|
/// Count wrapped rows for a line without allocating the wrapped strings.
|
||||||
/// Returns the original line unchanged for non-JSON or non-Object content.
|
/// MUST produce the same count as `wrap_line_chars(line, width).len()`.
|
||||||
pub fn format_json_line(line: &str) -> String {
|
pub fn wrap_line_count(line: &str, width: usize) -> usize {
|
||||||
if line.trim().is_empty() {
|
use unicode_width::UnicodeWidthChar;
|
||||||
return String::new();
|
|
||||||
|
if width == 0 || line.is_empty() {
|
||||||
|
return 1;
|
||||||
}
|
}
|
||||||
// Quick pre-check: only try parsing if it starts with '{'
|
let mut count = 0usize;
|
||||||
if !line.trim_start().starts_with('{') {
|
let mut col = 0usize;
|
||||||
return line.to_string();
|
let mut row_has_content = false;
|
||||||
|
|
||||||
|
for ch in line.chars() {
|
||||||
|
if ch == '\t' {
|
||||||
|
let tab_stop = TAB_WIDTH - (col % TAB_WIDTH);
|
||||||
|
let mut remaining = tab_stop;
|
||||||
|
while remaining > 0 {
|
||||||
|
let avail = width.saturating_sub(col);
|
||||||
|
let fill = remaining.min(avail);
|
||||||
|
col += fill;
|
||||||
|
remaining -= fill;
|
||||||
|
if col >= width {
|
||||||
|
count += 1;
|
||||||
|
col = 0;
|
||||||
|
row_has_content = false;
|
||||||
|
} else {
|
||||||
|
row_has_content = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let w = if ch.is_control() {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
ch.width().unwrap_or(0)
|
||||||
|
};
|
||||||
|
if col + w > width && row_has_content {
|
||||||
|
count += 1;
|
||||||
|
col = 0;
|
||||||
|
}
|
||||||
|
col += w;
|
||||||
|
if col >= width {
|
||||||
|
count += 1;
|
||||||
|
col = 0;
|
||||||
|
row_has_content = false;
|
||||||
|
} else {
|
||||||
|
row_has_content = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if row_has_content {
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
if count == 0 {
|
||||||
|
count = 1;
|
||||||
|
}
|
||||||
|
count
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Format a line as pretty-printed JSON if it's a JSON Object.
|
||||||
|
/// Returns the original line borrowed for non-JSON or non-Object content,
|
||||||
|
/// only allocating when pretty-printing actually happens.
|
||||||
|
pub fn format_json_line(line: &str) -> Cow<'_, str> {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.is_empty() {
|
||||||
|
return Cow::Borrowed("");
|
||||||
|
}
|
||||||
|
if !trimmed.starts_with('{') {
|
||||||
|
return Cow::Borrowed(line);
|
||||||
}
|
}
|
||||||
match serde_json::from_str::<serde_json::Value>(line) {
|
match serde_json::from_str::<serde_json::Value>(line) {
|
||||||
Ok(value) if value.is_object() => {
|
Ok(value) if value.is_object() => match serde_json::to_string_pretty(&value) {
|
||||||
serde_json::to_string_pretty(&value).unwrap_or_else(|_| line.to_string())
|
Ok(s) => Cow::Owned(s),
|
||||||
}
|
Err(_) => Cow::Borrowed(line),
|
||||||
_ => line.to_string(),
|
},
|
||||||
|
_ => Cow::Borrowed(line),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -96,7 +186,7 @@ mod tests {
|
|||||||
#[test]
|
#[test]
|
||||||
fn test_wrap_with_tab() {
|
fn test_wrap_with_tab() {
|
||||||
let result = wrap_line_chars("a\tb", 4);
|
let result = wrap_line_chars("a\tb", 4);
|
||||||
assert_eq!(result, vec!["a", " ", "b"]);
|
assert_eq!(result, vec!["a ", "b"]);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
@@ -132,4 +222,195 @@ mod tests {
|
|||||||
fn test_max_wrap_input_len_constant() {
|
fn test_max_wrap_input_len_constant() {
|
||||||
assert_eq!(MAX_WRAP_INPUT_LEN, 10 * 1024 * 1024);
|
assert_eq!(MAX_WRAP_INPUT_LEN, 10 * 1024 * 1024);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_cjk_chars() {
|
||||||
|
let result = wrap_line_chars("你好", 3);
|
||||||
|
assert_eq!(result, vec!["你", "好"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_cjk_ascii_mixed() {
|
||||||
|
let result = wrap_line_chars("a你好", 4);
|
||||||
|
assert_eq!(result, vec!["a你", "好"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_zero_width_char() {
|
||||||
|
let result = wrap_line_chars("a\u{200B}b", 2);
|
||||||
|
assert_eq!(result, vec!["a\u{200B}b"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_emoji() {
|
||||||
|
let result = wrap_line_chars("😀a", 3);
|
||||||
|
assert_eq!(result, vec!["😀a"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_emoji_exact_wrap() {
|
||||||
|
let result = wrap_line_chars("😀a", 2);
|
||||||
|
assert_eq!(result, vec!["😀", "a"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_combining_mark() {
|
||||||
|
// Scalar-width wrapping: combining mark (width 0) stays with next base char,
|
||||||
|
// not the preceding one, because the base char already triggered a flush.
|
||||||
|
let result = wrap_line_chars("a\u{0301}b", 1);
|
||||||
|
assert_eq!(result, vec!["a", "\u{0301}b"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_cjk_width_one() {
|
||||||
|
let result = wrap_line_chars("你好", 1);
|
||||||
|
assert_eq!(result, vec!["你", "好"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_tab_narrow_width() {
|
||||||
|
let result = wrap_line_chars("\t", 2);
|
||||||
|
assert_eq!(result, vec![" ", " "]);
|
||||||
|
let result = wrap_line_chars("\t", 1);
|
||||||
|
assert_eq!(result, vec![" ", " ", " ", " "]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_tab_stop_alignment() {
|
||||||
|
assert_eq!(wrap_line_chars("a\tb", 8), vec!["a b"]);
|
||||||
|
assert_eq!(wrap_line_chars("ab\t", 4), vec!["ab "]);
|
||||||
|
assert_eq!(wrap_line_chars("abc\tb", 8), vec!["abc b"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_tab_at_line_boundary() {
|
||||||
|
let result = wrap_line_chars("a\tb", 4);
|
||||||
|
assert_eq!(result, vec!["a ", "b"]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_tab_regression_ab_tab() {
|
||||||
|
let result = wrap_line_chars("ab\t", 4);
|
||||||
|
assert_eq!(result, vec!["ab "]);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_empty() {
|
||||||
|
assert_eq!(wrap_line_count("", 80), wrap_line_chars("", 80).len());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_zero_width() {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count("hello", 0),
|
||||||
|
wrap_line_chars("hello", 0).len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_ascii_short() {
|
||||||
|
for width in 1..20 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count("hello world", width),
|
||||||
|
wrap_line_chars("hello world", width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_cjk() {
|
||||||
|
for width in 1..10 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count("你好世界", width),
|
||||||
|
wrap_line_chars("你好世界", width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_emoji() {
|
||||||
|
for width in 1..6 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count("😀👨👩👧", width),
|
||||||
|
wrap_line_chars("😀👨👩👧", width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_combining_mark() {
|
||||||
|
for width in 1..6 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count("a\u{0301}b\u{0301}c", width),
|
||||||
|
wrap_line_chars("a\u{0301}b\u{0301}c", width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_tab_variants() {
|
||||||
|
let inputs = ["\t", "a\tb", "ab\t", "\t\t", "abc\tb", "a\t\t\tb"];
|
||||||
|
for input in inputs {
|
||||||
|
for width in 1..9 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count(input, width),
|
||||||
|
wrap_line_chars(input, width).len(),
|
||||||
|
"input={input:?} width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_mixed() {
|
||||||
|
let input = "log: 你好 😀\tvalue=true [ERROR]";
|
||||||
|
for width in 1..40 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count(input, width),
|
||||||
|
wrap_line_chars(input, width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_zero_width_chars() {
|
||||||
|
let input = "a\u{200B}b\u{200C}c\u{200D}d";
|
||||||
|
for width in 1..8 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count(input, width),
|
||||||
|
wrap_line_chars(input, width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_control_chars() {
|
||||||
|
let input = "a\x07b\x1Fc";
|
||||||
|
for width in 1..8 {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count(input, width),
|
||||||
|
wrap_line_chars(input, width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_wrap_count_matches_chars_long_log_line() {
|
||||||
|
let input = "2025-01-14T10:23:45.123Z [INFO] [auth] request handled status=200 \
|
||||||
|
latency=0.234s endpoint=/api/v1/users user=user_42 request_id=req_abc123def456";
|
||||||
|
for width in [10, 20, 40, 60, 80, 100, 120] {
|
||||||
|
assert_eq!(
|
||||||
|
wrap_line_count(input, width),
|
||||||
|
wrap_line_chars(input, width).len(),
|
||||||
|
"width={width}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+306
-90
@@ -16,7 +16,14 @@
|
|||||||
// 类似于 Python 的 dict、JavaScript 的 Object/Map、Java 的 HashMap。
|
// 类似于 Python 的 dict、JavaScript 的 Object/Map、Java 的 HashMap。
|
||||||
// 它存储键值对(key-value pairs),可以通过键快速查找对应的值。
|
// 它存储键值对(key-value pairs),可以通过键快速查找对应的值。
|
||||||
// 这里用 HashMap<String, Value> 来存储 JSON 中除 timestamp/level 之外的其他字段。
|
// 这里用 HashMap<String, Value> 来存储 JSON 中除 timestamp/level 之外的其他字段。
|
||||||
use std::collections::HashMap;
|
use std::collections::{HashMap, HashSet};
|
||||||
|
|
||||||
|
// serde::de 中的 Visitor / MapAccess 允许我们自定义 JSON 对象的反序列化过程。
|
||||||
|
// 默认的 serde_json::from_str::<HashMap<_, _>>() 遇到重复键时会采用"后者覆盖前者"(last-wins),
|
||||||
|
// 前面的值被静默丢弃。这里我们通过自定义 Visitor 在反序列化过程中逐个观察 key-value 对,
|
||||||
|
// 在保持 last-wins 行为的同时,将重复 key 的所有值记录到 DuplicateKey 中。
|
||||||
|
use serde::Deserializer;
|
||||||
|
use serde::de::{MapAccess, Visitor};
|
||||||
|
|
||||||
// serde_json::Value — 来自 serde_json 库(Rust 中最流行的 JSON 处理库)。
|
// serde_json::Value — 来自 serde_json 库(Rust 中最流行的 JSON 处理库)。
|
||||||
// Value 是一个枚举类型,可以表示任意 JSON 值:
|
// Value 是一个枚举类型,可以表示任意 JSON 值:
|
||||||
@@ -32,7 +39,20 @@ use serde_json::Value;
|
|||||||
// ─── 引入项目内部类型 ──────────────────────────────────────────────────────
|
// ─── 引入项目内部类型 ──────────────────────────────────────────────────────
|
||||||
// crate 表示"当前项目(crate)"。
|
// crate 表示"当前项目(crate)"。
|
||||||
// types 模块中定义了 LogEntry(一条日志记录)和 LogLevel(日志级别,如 INFO/ERROR)。
|
// types 模块中定义了 LogEntry(一条日志记录)和 LogLevel(日志级别,如 INFO/ERROR)。
|
||||||
use crate::types::{LogEntry, LogLevel};
|
use crate::types::{DuplicateKey, LogEntry, LogLevel};
|
||||||
|
|
||||||
|
// ─── strip_bom 辅助函数 ──────────────────────────────────────────────────
|
||||||
|
// 剥离行首的 UTF-8 BOM(Byte Order Mark, U+FEFF)。
|
||||||
|
//
|
||||||
|
// Windows 环境和某些导出工具生成的文件会在行首插入 BOM,
|
||||||
|
// 而 serde_json 不接受 BOM 前缀的 JSON 文本(会报 "expected value" 错误)。
|
||||||
|
// 只剥离一个前导 BOM,不处理多个 BOM 或行内 BOM(那些是畸形输入)。
|
||||||
|
//
|
||||||
|
// 参数: line: &str — 输入字符串切片。
|
||||||
|
// 返回: &str — 去掉 BOM 后的字符串切片(借用原始字符串,零分配)。
|
||||||
|
fn strip_bom(line: &str) -> &str {
|
||||||
|
line.strip_prefix('\u{FEFF}').unwrap_or(line)
|
||||||
|
}
|
||||||
|
|
||||||
// ─── detect_json_log 函数 ──────────────────────────────────────────────────
|
// ─── detect_json_log 函数 ──────────────────────────────────────────────────
|
||||||
// 检测一行文本是否是一个 JSON 对象。
|
// 检测一行文本是否是一个 JSON 对象。
|
||||||
@@ -63,7 +83,92 @@ pub fn detect_json_log(line: &str) -> bool {
|
|||||||
// 则匹配成功。_ 是通配符,表示"不关心对象里面的具体内容"。
|
// 则匹配成功。_ 是通配符,表示"不关心对象里面的具体内容"。
|
||||||
//
|
//
|
||||||
// 如果匹配到 Ok(Value::Object(_)) 返回 true,否则返回 false。
|
// 如果匹配到 Ok(Value::Object(_)) 返回 true,否则返回 false。
|
||||||
matches!(serde_json::from_str::<Value>(line), Ok(Value::Object(_)))
|
matches!(
|
||||||
|
serde_json::from_str::<Value>(strip_bom(line)),
|
||||||
|
Ok(Value::Object(_))
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── DuplicateKeyVisitor ──────────────────────────────────────────────────
|
||||||
|
// 自定义 serde Visitor,在反序列化 JSON 对象时检测重复 key。
|
||||||
|
//
|
||||||
|
// 工作原理:
|
||||||
|
// serde 的 MapAccess trait 允许我们逐个遍历 JSON 对象的 key-value 对。
|
||||||
|
// 每读到一个 (key, value),我们:
|
||||||
|
// 1. 检查这个 key 是否已经见过(通过 HashSet)
|
||||||
|
// 2. 如果是重复 key,记录到 Vec<DuplicateKey> 中(包含所有出现过的值)
|
||||||
|
// 3. 将 key-value 插入 Map(last-wins,与 serde_json 默认行为一致)
|
||||||
|
//
|
||||||
|
// 这样既保持了兼容性(last-wins),又不丢失信息(所有值都记录在 DuplicateKey 中)。
|
||||||
|
struct DuplicateKeyVisitor;
|
||||||
|
|
||||||
|
impl<'de> Visitor<'de> for DuplicateKeyVisitor {
|
||||||
|
// 返回类型:(serde_json::Map, 重复 key 列表)
|
||||||
|
type Value = (serde_json::Map<String, Value>, Vec<DuplicateKey>);
|
||||||
|
|
||||||
|
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
||||||
|
f.write_str("a JSON object")
|
||||||
|
}
|
||||||
|
|
||||||
|
fn visit_map<A>(self, mut access: A) -> Result<Self::Value, A::Error>
|
||||||
|
where
|
||||||
|
A: MapAccess<'de>,
|
||||||
|
{
|
||||||
|
let mut map = serde_json::Map::new();
|
||||||
|
let mut seen = HashSet::new();
|
||||||
|
let mut duplicates: Vec<DuplicateKey> = Vec::new();
|
||||||
|
|
||||||
|
while let Some((key, value)) = access.next_entry::<String, Value>()? {
|
||||||
|
if !seen.insert(key.clone()) {
|
||||||
|
// 重复 key:将之前 map 中的值和当前值都记录下来
|
||||||
|
if let Some(existing) = duplicates.iter_mut().find(|d| d.key == key) {
|
||||||
|
// 同一个 key 第三次及以上出现:追加当前值
|
||||||
|
existing.values.push(value.clone());
|
||||||
|
} else {
|
||||||
|
// 同一个 key 第二次出现:记录第一次的值 + 当前值
|
||||||
|
let prev_value = map.get(&key).cloned().unwrap_or(Value::Null);
|
||||||
|
duplicates.push(DuplicateKey {
|
||||||
|
key: key.clone(),
|
||||||
|
values: vec![prev_value, value.clone()],
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// last-wins:后出现的值覆盖前面的值,与 serde_json 默认行为一致
|
||||||
|
map.insert(key, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok((map, duplicates))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// 使用自定义 Visitor 解析 JSON 对象,同时检测重复 key。
|
||||||
|
///
|
||||||
|
/// 返回 (serde_json::Map, Vec<DuplicateKey>):
|
||||||
|
/// - Map 中存储所有 key-value(重复 key 取 last-wins)
|
||||||
|
/// - Vec 中记录所有重复 key 及其全部值
|
||||||
|
fn parse_json_object_with_duplicates(
|
||||||
|
json: &str,
|
||||||
|
) -> Option<(serde_json::Map<String, Value>, Vec<DuplicateKey>)> {
|
||||||
|
let mut deserializer = serde_json::Deserializer::from_str(json);
|
||||||
|
deserializer.deserialize_map(DuplicateKeyVisitor).ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── take_string_field_from_map 辅助函数 ──────────────────────────────────
|
||||||
|
// 从 serde_json::Map 中安全提取字符串字段。
|
||||||
|
// 功能与原 take_string_field 相同,但操作 serde_json::Map 而非 HashMap。
|
||||||
|
fn take_string_field_from_map(
|
||||||
|
obj: &mut serde_json::Map<String, Value>,
|
||||||
|
keys: &[&str],
|
||||||
|
) -> Option<String> {
|
||||||
|
for key in keys {
|
||||||
|
if obj.get(*key).is_some_and(Value::is_string) {
|
||||||
|
let Some(Value::String(v)) = obj.remove(*key) else {
|
||||||
|
unreachable!("value was checked as string");
|
||||||
|
};
|
||||||
|
return Some(v);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── parse_line 函数 ──────────────────────────────────────────────────────
|
// ─── parse_line 函数 ──────────────────────────────────────────────────────
|
||||||
@@ -73,107 +178,35 @@ pub fn detect_json_log(line: &str) -> bool {
|
|||||||
// 返回: Option<LogEntry> — 解析成功返回 Some(LogEntry),失败或不合法返回 None。
|
// 返回: Option<LogEntry> — 解析成功返回 Some(LogEntry),失败或不合法返回 None。
|
||||||
// Option 是 Rust 的可选类型:Some(值) 表示有值,None 表示没有值。
|
// Option 是 Rust 的可选类型:Some(值) 表示有值,None 表示没有值。
|
||||||
pub fn parse_line(line: &str) -> Option<LogEntry> {
|
pub fn parse_line(line: &str) -> Option<LogEntry> {
|
||||||
// ─── 跳过空行 ──────────────────────────────────────────────────────────
|
let line = strip_bom(line);
|
||||||
// line.trim() 去除首尾空白字符(空格、制表符、换行符等)。
|
|
||||||
// .is_empty() 检查是否为空字符串。
|
|
||||||
// 如果去除空白后是空的,说明是空行,不需要解析,直接返回 None。
|
|
||||||
if line.trim().is_empty() {
|
if line.trim().is_empty() {
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── 解析 JSON 为 HashMap ──────────────────────────────────────────────
|
// ─── 使用自定义 Visitor 解析 JSON ──────────────────────────────────
|
||||||
// serde_json::from_str(line) 尝试将字符串解析为 JSON。
|
// 通过 DuplicateKeyVisitor 反序列化,在保持 last-wins 的同时检测重复 key。
|
||||||
// 由于我们声明了 HashMap<String, Value> 类型,Rust 会自动将 JSON 对象
|
// 返回的 (serde_json::Map, Vec<DuplicateKey>) 中:
|
||||||
// 转换为 HashMap,其中每个键是 String,每个值是 serde_json::Value。
|
// - Map 包含所有 key-value(重复 key 取最后一个值)
|
||||||
//
|
// - Vec 记录了所有重复 key 及其出现过的全部值
|
||||||
// .ok() 将 Result 转换为 Option:
|
let (mut obj, duplicate_keys) = parse_json_object_with_duplicates(line)?;
|
||||||
// Ok(值) → Some(值)
|
|
||||||
// Err(_) → None
|
|
||||||
//
|
|
||||||
// 末尾的 ? 是"问号操作符"(try operator),在这里的作用是:
|
|
||||||
// 如果 .ok() 返回 None(即 JSON 解析失败),则整个函数直接返回 None。
|
|
||||||
// 如果返回 Some(hashmap),则将 hashmap 取出并绑定到 fields 变量。
|
|
||||||
//
|
|
||||||
// let mut 表示这是一个"可变变量"(mutable variable),
|
|
||||||
// 后续代码会修改这个 HashMap(从中删除已识别的字段)。
|
|
||||||
let mut fields: HashMap<String, Value> = serde_json::from_str(line).ok()?;
|
|
||||||
|
|
||||||
// ─── 保存原始行内容 ──────────────────────────────────────────────────
|
|
||||||
// line.to_string() 将 &str(字符串切片引用)转换为 String(拥有所有权的字符串)。
|
|
||||||
// 保存原始行是为了在 UI 中显示未经修改的原始日志内容。
|
|
||||||
let raw_line = line.to_string();
|
let raw_line = line.to_string();
|
||||||
|
let timestamp =
|
||||||
// ─── 提取时间戳字段 ──────────────────────────────────────────────────
|
take_string_field_from_map(&mut obj, &["timestamp", "time", "ts", "@timestamp"]);
|
||||||
// 这段代码尝试从 JSON 中提取时间戳,逻辑如下:
|
let level = take_string_field_from_map(&mut obj, &["level", "lvl", "severity"])
|
||||||
//
|
|
||||||
// 1. ["timestamp", "time", "ts", "@timestamp"] — 候选键名数组。
|
|
||||||
// 不同日志系统使用不同的时间戳字段名,这里列出常见的几种。
|
|
||||||
//
|
|
||||||
// 2. .iter() — 创建数组的迭代器,可以逐个遍历元素。
|
|
||||||
//
|
|
||||||
// 3. .find_map(|key| fields.remove(*key)) — 对每个候选键名:
|
|
||||||
// - fields.remove(*key): 尝试从 HashMap 中移除该键并返回对应的值。
|
|
||||||
// 如果键不存在,remove 返回 None。
|
|
||||||
// *key 是解引用(deref),将 &str(引用)转换为 str,因为 remove 接受 &str 类型。
|
|
||||||
// - find_map: 遍历所有候选键,返回第一个 Some(值) 的结果。
|
|
||||||
// 即找到第一个存在的键就停止。
|
|
||||||
//
|
|
||||||
// 4. .and_then(|v| v.as_str().map(String::from)) — 如果找到了时间戳值:
|
|
||||||
// - v.as_str(): 尝试将 serde_json::Value 转换为 &str(字符串切片)。
|
|
||||||
// 如果 Value 不是字符串类型(比如是数字),返回 None。
|
|
||||||
// - .map(String::from): 如果是字符串,将其转换为 String(拥有所有权的字符串)。
|
|
||||||
// - and_then: 类似于 map,但用于"扁平化"嵌套的 Option。
|
|
||||||
// 如果 as_str() 返回 None,整个链返回 None。
|
|
||||||
let timestamp = ["timestamp", "time", "ts", "@timestamp"]
|
|
||||||
.iter()
|
|
||||||
.find_map(|key| fields.remove(*key))
|
|
||||||
.and_then(|v| v.as_str().map(String::from));
|
|
||||||
|
|
||||||
// ─── 提取日志级别字段 ──────────────────────────────────────────────
|
|
||||||
// 与时间戳提取类似,但多了一步:将字符串解析为 LogLevel 枚举。
|
|
||||||
let level = ["level", "lvl", "severity"]
|
|
||||||
.iter()
|
|
||||||
.find_map(|key| fields.remove(*key))
|
|
||||||
.and_then(|v| v.as_str().map(String::from))
|
|
||||||
// .map(|s| s.parse::<LogLevel>(...)) — 尝试将字符串解析为 LogLevel 枚举。
|
|
||||||
// parse::<LogLevel> 中的 ::<LogLevel> 是泛型参数(turbofish 语法),
|
|
||||||
// 指定我们要将字符串解析为 LogLevel 类型。
|
|
||||||
//
|
|
||||||
// .unwrap_or_else(|e| match e {}) — 错误处理:
|
|
||||||
// - 如果解析成功,直接返回 LogLevel 值。
|
|
||||||
// - 如果解析失败(字符串不匹配任何已知的日志级别),执行闭包。
|
|
||||||
// - |e| match e {}: 这个闭包接收解析错误 e,用 match e {} 进行"穷尽匹配"。
|
|
||||||
// 由于 LogLevel 的 parse 错误类型是一个空枚举(没有任何变体),
|
|
||||||
// match e {} 意味着"这个分支永远不会执行"(unreachable)。
|
|
||||||
// 但实际上,如果 parse 失败,unwrap_or_else 不会执行这个闭包——
|
|
||||||
// 等等,这里有个细微之处:
|
|
||||||
// unwrap_or_else 只在 Err 时执行闭包,但 match e {} 对空枚举是合法的
|
|
||||||
// (因为空枚举没有任何可能的值,所以 match 是穷尽的)。
|
|
||||||
// 不过这里的实际效果是:如果 parse 失败,整个 .map() 返回 None
|
|
||||||
// (因为 unwrap_or_else 返回的类型是 LogLevel,而空 match 不会有返回值)。
|
|
||||||
//
|
|
||||||
// 实际上更准确的解释:parse() 的错误类型是 Infallible(不可失败的),
|
|
||||||
// 即解析总是成功。所以 unwrap_or_else 永远不会被执行。
|
|
||||||
// 但即使如此,unwrap_or_else 的闭包也需要类型正确,match e {} 满足这一点。
|
|
||||||
.map(|s| s.parse::<LogLevel>().unwrap_or_else(|e| match e {}));
|
.map(|s| s.parse::<LogLevel>().unwrap_or_else(|e| match e {}));
|
||||||
|
|
||||||
// ─── 构建 LogEntry 并返回 ──────────────────────────────────────────
|
// serde_json::Map → HashMap:剩余字段转为 HashMap 存入 fields
|
||||||
// 此时 fields HashMap 中还剩下未被提取的字段(如 message、自定义字段等)。
|
let fields: HashMap<String, Value> = obj.into_iter().collect();
|
||||||
// timestamp 和 level 已经从 fields 中移除了(通过 remove)。
|
|
||||||
//
|
|
||||||
// Some(LogEntry { ... }) — 使用结构体字面量创建 LogEntry 实例,
|
|
||||||
// 并用 Some() 包裹表示"有值"。
|
|
||||||
Some(LogEntry {
|
Some(LogEntry {
|
||||||
// line_number 设为 0,由调用者(如 parse_line_with_number)设置正确的值。
|
|
||||||
line_number: 0,
|
line_number: 0,
|
||||||
// 原始行内容。
|
|
||||||
raw_line,
|
raw_line,
|
||||||
// 时间戳(可能为 None,如果 JSON 中没有时间戳字段)。
|
|
||||||
timestamp,
|
timestamp,
|
||||||
// 日志级别(可能为 None,如果 JSON 中没有级别字段)。
|
|
||||||
level,
|
level,
|
||||||
// 剩余的 JSON 字段(已移除 timestamp 和 level)。
|
|
||||||
fields,
|
fields,
|
||||||
|
duplicate_keys,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -317,6 +350,13 @@ mod tests {
|
|||||||
assert_eq!(parse_line(warn_line).unwrap().level, Some(LogLevel::Warn));
|
assert_eq!(parse_line(warn_line).unwrap().level, Some(LogLevel::Warn));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
// Regression: level field with surrounding whitespace should still be recognized.
|
||||||
|
fn test_level_whitespace_in_json() {
|
||||||
|
let line = r#"{"level":" WARN ","message":"test"}"#;
|
||||||
|
assert_eq!(parse_line(line).unwrap().level, Some(LogLevel::Warn));
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
// 测试:所有候选时间戳键名(timestamp, time, ts, @timestamp)都能被识别。
|
// 测试:所有候选时间戳键名(timestamp, time, ts, @timestamp)都能被识别。
|
||||||
fn test_timestamp_key_names() {
|
fn test_timestamp_key_names() {
|
||||||
@@ -357,4 +397,180 @@ mod tests {
|
|||||||
assert_eq!(entry.level, Some(LogLevel::Info), "failed for key: {key}");
|
assert_eq!(entry.level, Some(LogLevel::Info), "failed for key: {key}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
// 测试:数字类型的 level 值不应被提取,应保留在 fields 中。
|
||||||
|
fn test_numeric_level_preserved_in_fields() {
|
||||||
|
let line = r#"{"level":30,"message":"hello"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// level 不是字符串,应返回 None。
|
||||||
|
assert!(entry.level.is_none());
|
||||||
|
// 数字 level 应保留在 fields 中,不被静默丢弃。
|
||||||
|
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
|
||||||
|
// message 仍正常存在。
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("message"),
|
||||||
|
Some(&Value::String("hello".to_string()))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
// 测试:数字类型的 timestamp 值不应被提取,应保留在 fields 中。
|
||||||
|
fn test_numeric_timestamp_preserved_in_fields() {
|
||||||
|
let line = r#"{"timestamp":1718000000,"message":"hello"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// timestamp 不是字符串,应返回 None。
|
||||||
|
assert!(entry.timestamp.is_none());
|
||||||
|
// 数字 timestamp 应保留在 fields 中。
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("timestamp"),
|
||||||
|
Some(&Value::Number(1718000000.into()))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
// 测试:当第一个候选键是数字时,应回退到下一个字符串类型的候选键。
|
||||||
|
fn test_fallback_to_string_key() {
|
||||||
|
let line = r#"{"level":30,"lvl":"INFO","message":"hello"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// "level" 是数字,应跳过;"lvl" 是字符串,应成功提取。
|
||||||
|
assert_eq!(entry.level, Some(LogLevel::Info));
|
||||||
|
// 数字 "level" 保留在 fields 中。
|
||||||
|
assert_eq!(entry.fields.get("level"), Some(&Value::Number(30.into())));
|
||||||
|
// "lvl" 已被成功提取并从 fields 中移除。
|
||||||
|
assert!(entry.fields.get("lvl").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
// 测试:timestamp 的 fallback 行为 — 数字 timestamp 被保留,字符串 time 被提取。
|
||||||
|
fn test_timestamp_fallback_preserves_numeric() {
|
||||||
|
let line = r#"{"timestamp":1718000000,"time":"2024-01-01T00:00:00Z"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// "timestamp" 是数字,跳过;"time" 是字符串,成功提取。
|
||||||
|
assert_eq!(entry.timestamp, Some("2024-01-01T00:00:00Z".to_string()));
|
||||||
|
// 数字 "timestamp" 保留在 fields 中。
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("timestamp"),
|
||||||
|
Some(&Value::Number(1718000000.into()))
|
||||||
|
);
|
||||||
|
// "time" 已被提取并从 fields 中移除。
|
||||||
|
assert!(entry.fields.get("time").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_bom_prefixed_json() {
|
||||||
|
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
assert_eq!(entry.level, Some(LogLevel::Info));
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("message"),
|
||||||
|
Some(&Value::String("hello".into()))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_bom_prefixed_detect() {
|
||||||
|
assert!(detect_json_log("\u{FEFF}{\"level\":\"INFO\"}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_bom_only_whitespace() {
|
||||||
|
assert!(parse_line("\u{FEFF} ").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_bom_stripped_from_raw_line() {
|
||||||
|
let line = "\u{FEFF}{\"level\":\"INFO\",\"message\":\"hello\"}";
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
assert_eq!(entry.raw_line, "{\"level\":\"INFO\",\"message\":\"hello\"}");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_internal_bom_not_stripped() {
|
||||||
|
let line = "{\"message\":\"\u{FEFF}hello\"}";
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("message"),
|
||||||
|
Some(&Value::String("\u{FEFF}hello".into()))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ─── 重复 key 检测测试 ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_no_duplicate_keys_normal_json() {
|
||||||
|
let line = r#"{"level":"INFO","message":"hello"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
assert!(entry.duplicate_keys.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_duplicate_message_key_detected() {
|
||||||
|
let line = r#"{"level":"INFO","message":"first","message":"second"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// last-wins: fields 中保留第二个值
|
||||||
|
assert_eq!(
|
||||||
|
entry.fields.get("message"),
|
||||||
|
Some(&Value::String("second".into()))
|
||||||
|
);
|
||||||
|
// 重复 key 记录中包含所有值
|
||||||
|
assert_eq!(entry.duplicate_keys.len(), 1);
|
||||||
|
assert_eq!(entry.duplicate_keys[0].key, "message");
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
|
||||||
|
assert_eq!(
|
||||||
|
entry.duplicate_keys[0].values[0],
|
||||||
|
Value::String("first".into())
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
entry.duplicate_keys[0].values[1],
|
||||||
|
Value::String("second".into())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_duplicate_key_last_wins() {
|
||||||
|
let line = r#"{"msg":"a","msg":"b","msg":"c"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// last-wins: 最终值是 "c"
|
||||||
|
assert_eq!(entry.fields.get("msg"), Some(&Value::String("c".into())));
|
||||||
|
// 三个值都被记录
|
||||||
|
assert_eq!(entry.duplicate_keys.len(), 1);
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values.len(), 3);
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values[0], Value::String("a".into()));
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values[1], Value::String("b".into()));
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values[2], Value::String("c".into()));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_multiple_different_duplicate_keys() {
|
||||||
|
let line = r#"{"a":"1","b":"2","a":"3","b":"4"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
assert_eq!(entry.duplicate_keys.len(), 2);
|
||||||
|
let dup_a = entry.duplicate_keys.iter().find(|d| d.key == "a").unwrap();
|
||||||
|
let dup_b = entry.duplicate_keys.iter().find(|d| d.key == "b").unwrap();
|
||||||
|
assert_eq!(dup_a.values.len(), 2);
|
||||||
|
assert_eq!(dup_b.values.len(), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_duplicate_level_key_detected() {
|
||||||
|
let line = r#"{"level":"INFO","level":"ERROR","message":"hello"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// last-wins: level 被提取为 ERROR
|
||||||
|
assert_eq!(entry.level, Some(LogLevel::Error));
|
||||||
|
// 重复 key 被记录
|
||||||
|
assert_eq!(entry.duplicate_keys.len(), 1);
|
||||||
|
assert_eq!(entry.duplicate_keys[0].key, "level");
|
||||||
|
assert_eq!(entry.duplicate_keys[0].values.len(), 2);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_duplicate_timestamp_key_detected() {
|
||||||
|
let line = r#"{"timestamp":"2024-01-01","timestamp":"2024-06-01"}"#;
|
||||||
|
let entry = parse_line(line).unwrap();
|
||||||
|
// last-wins: timestamp 提取为后者
|
||||||
|
assert_eq!(entry.timestamp, Some("2024-06-01".to_string()));
|
||||||
|
assert_eq!(entry.duplicate_keys.len(), 1);
|
||||||
|
assert_eq!(entry.duplicate_keys[0].key, "timestamp");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -100,13 +100,21 @@ fn detect_level_from_text(line: &str) -> Option<LogLevel> {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─── is_ident_char ─────────────────────────────────────────────────────────
|
||||||
|
/// Whether a byte looks like an ASCII identifier continuation character
|
||||||
|
/// (letter / digit / underscore). Log-level keywords must NOT be adjacent to
|
||||||
|
/// such characters to count as a valid word boundary.
|
||||||
|
fn is_ident_char(b: u8) -> bool {
|
||||||
|
b.is_ascii_alphanumeric() || b == b'_'
|
||||||
|
}
|
||||||
|
|
||||||
// ─── is_word_boundary ───────────────────────────────────────────────────────
|
// ─── is_word_boundary ───────────────────────────────────────────────────────
|
||||||
/// Check that the match at `start..start+len` is surrounded by non-alphabetic
|
/// Check that the match at `start..start+len` is surrounded by non-identifier
|
||||||
/// characters (or the string edge).
|
/// characters (or the string edge).
|
||||||
fn is_word_boundary(text: &str, start: usize, len: usize) -> bool {
|
fn is_word_boundary(text: &str, start: usize, len: usize) -> bool {
|
||||||
let before_ok = start == 0 || !text.as_bytes()[start - 1].is_ascii_alphabetic();
|
let before_ok = start == 0 || !is_ident_char(text.as_bytes()[start - 1]);
|
||||||
let after_idx = start + len;
|
let after_idx = start + len;
|
||||||
let after_ok = after_idx >= text.len() || !text.as_bytes()[after_idx].is_ascii_alphabetic();
|
let after_ok = after_idx >= text.len() || !is_ident_char(text.as_bytes()[after_idx]);
|
||||||
before_ok && after_ok
|
before_ok && after_ok
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -209,4 +217,36 @@ mod tests {
|
|||||||
let line = format!("{prefix} ERROR something");
|
let line = format!("{prefix} ERROR something");
|
||||||
assert_eq!(detect_level(&line), Some(LogLevel::Error));
|
assert_eq!(detect_level(&line), Some(LogLevel::Error));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_boundary_rejects_trailing_digits() {
|
||||||
|
assert_eq!(detect_level("ERROR123"), None);
|
||||||
|
assert_eq!(detect_level("WARN2: bad"), None);
|
||||||
|
assert_eq!(detect_level("ERR2"), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_boundary_rejects_underscore() {
|
||||||
|
assert_eq!(detect_level("INFO_foo"), None);
|
||||||
|
assert_eq!(detect_level("DBG_value=5"), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_boundary_rejects_leading_digits_and_underscore() {
|
||||||
|
assert_eq!(detect_level("123ERROR: fail"), None);
|
||||||
|
assert_eq!(detect_level("foo_ERROR: fail"), None);
|
||||||
|
assert_eq!(detect_level("1WRN"), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_boundary_accepts_valid_suffixes() {
|
||||||
|
assert_eq!(detect_level("ERROR: fail"), Some(LogLevel::Error));
|
||||||
|
assert_eq!(detect_level("[ERROR] fail"), Some(LogLevel::Error));
|
||||||
|
assert_eq!(detect_level("ERROR fail"), Some(LogLevel::Error));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_boundary_camel_case_regression() {
|
||||||
|
assert_eq!(detect_level("errorLevel"), None);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -71,13 +71,17 @@ impl FromStr for LogLevel {
|
|||||||
// 接收一个字符串切片 &str,返回 Result<LogLevel, Infallible>。
|
// 接收一个字符串切片 &str,返回 Result<LogLevel, Infallible>。
|
||||||
// 由于 Err 类型是 Infallible,实际上返回值总是 Ok(LogLevel)。
|
// 由于 Err 类型是 Infallible,实际上返回值总是 Ok(LogLevel)。
|
||||||
fn from_str(s: &str) -> Result<Self, Self::Err> {
|
fn from_str(s: &str) -> Result<Self, Self::Err> {
|
||||||
// `s.to_uppercase()` — 将字符串转换为大写,实现不区分大小写的匹配。
|
// `s.trim()` — 去除字符串前后的 Unicode 空白字符。
|
||||||
|
// 例如 " WARN " → "WARN","\tINFO\n" → "INFO"。
|
||||||
|
let trimmed = s.trim();
|
||||||
|
|
||||||
|
// `trimmed.to_uppercase()` — 将 trimmed 后的字符串转换为大写,实现不区分大小写的匹配。
|
||||||
// 例如 "info"、"Info"、"INFO" 都会被转换为 "INFO"。
|
// 例如 "info"、"Info"、"INFO" 都会被转换为 "INFO"。
|
||||||
// 返回一个新的 String(堆分配)。
|
// 返回一个新的 String(堆分配)。
|
||||||
//
|
//
|
||||||
// `.as_str()` — 将 String 转换回 &str(字符串切片引用)。
|
// `.as_str()` — 将 String 转换回 &str(字符串切片引用)。
|
||||||
// 因为 match 需要匹配 &str 而不是 String。
|
// 因为 match 需要匹配 &str 而不是 String。
|
||||||
match s.to_uppercase().as_str() {
|
match trimmed.to_uppercase().as_str() {
|
||||||
// `|` 在 match 分支中表示"或"(multiple patterns)。
|
// `|` 在 match 分支中表示"或"(multiple patterns)。
|
||||||
// "ERROR" | "ERR" | "SEVERE" | "FATAL" 都匹配到 LogLevel::Error。
|
// "ERROR" | "ERR" | "SEVERE" | "FATAL" 都匹配到 LogLevel::Error。
|
||||||
"ERROR" | "ERR" | "SEVERE" | "FATAL" => Ok(LogLevel::Error),
|
"ERROR" | "ERR" | "SEVERE" | "FATAL" => Ok(LogLevel::Error),
|
||||||
@@ -86,9 +90,9 @@ impl FromStr for LogLevel {
|
|||||||
"DEBUG" | "DBG" => Ok(LogLevel::Debug),
|
"DEBUG" | "DBG" => Ok(LogLevel::Debug),
|
||||||
"TRACE" | "TRC" => Ok(LogLevel::Trace),
|
"TRACE" | "TRC" => Ok(LogLevel::Trace),
|
||||||
// `_` 是通配符,匹配所有未被上面分支捕获的值。
|
// `_` 是通配符,匹配所有未被上面分支捕获的值。
|
||||||
// 对于未知级别,包装为 Unknown 并保存原始字符串。
|
// 对于未知级别,包装为 Unknown 并保存 trimmed 后的字符串。
|
||||||
// s.to_string() 将 &str 转换为 String(注意这里用原始的 s,不是大写后的)。
|
// s.to_string() 将 &str 转换为 String(注意这里用 trimmed,不是原始 s)。
|
||||||
_ => Ok(LogLevel::Unknown(s.to_string())),
|
_ => Ok(LogLevel::Unknown(trimmed.to_string())),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -119,6 +123,18 @@ impl fmt::Display for LogLevel {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ─── LogEntry 结构体 ────────────────────────────────────────────────────────
|
// ─── LogEntry 结构体 ────────────────────────────────────────────────────────
|
||||||
|
/// 记录 JSON 日志中出现的重复 key 信息
|
||||||
|
///
|
||||||
|
/// 当 JSON 对象中同一个 key 出现多次时,serde_json 默认 last-wins(后值覆盖前值),
|
||||||
|
/// 前面的值会静默丢失。此结构记录所有重复出现的 key 及其全部值,供 UI 展示警告。
|
||||||
|
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||||
|
pub struct DuplicateKey {
|
||||||
|
/// 重复的 key 名称
|
||||||
|
pub key: String,
|
||||||
|
/// 该 key 出现的所有值(按出现顺序排列,最后一个值是 fields 中的最终值)
|
||||||
|
pub values: Vec<Value>,
|
||||||
|
}
|
||||||
|
|
||||||
/// 一行解析后的日志
|
/// 一行解析后的日志
|
||||||
///
|
///
|
||||||
/// 表示日志文件中经过解析器处理后的一行内容。
|
/// 表示日志文件中经过解析器处理后的一行内容。
|
||||||
@@ -144,6 +160,9 @@ pub struct LogEntry {
|
|||||||
/// HashMap<String, Value> 是一个字典,键是字段名,值是 JSON 值。
|
/// HashMap<String, Value> 是一个字典,键是字段名,值是 JSON 值。
|
||||||
/// 例如 {"message": "hello", "request_id": "abc123"}。
|
/// 例如 {"message": "hello", "request_id": "abc123"}。
|
||||||
pub fields: HashMap<String, Value>,
|
pub fields: HashMap<String, Value>,
|
||||||
|
|
||||||
|
/// JSON 中重复出现的 key 记录(正常日志为空 Vec)
|
||||||
|
pub duplicate_keys: Vec<DuplicateKey>,
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── SearchResult 结构体 ────────────────────────────────────────────────────
|
// ─── SearchResult 结构体 ────────────────────────────────────────────────────
|
||||||
@@ -288,6 +307,31 @@ mod tests {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_from_str_whitespace_trimmed() {
|
||||||
|
assert_eq!(" WARN ".parse::<LogLevel>(), Ok(LogLevel::Warn));
|
||||||
|
assert_eq!("\tINFO".parse::<LogLevel>(), Ok(LogLevel::Info));
|
||||||
|
assert_eq!("ERROR\n".parse::<LogLevel>(), Ok(LogLevel::Error));
|
||||||
|
assert_eq!(" debug ".parse::<LogLevel>(), Ok(LogLevel::Debug));
|
||||||
|
assert_eq!("\tTRACE\t".parse::<LogLevel>(), Ok(LogLevel::Trace));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_from_str_whitespace_unknown_trimmed() {
|
||||||
|
// Unknown stores the trimmed value, not the original.
|
||||||
|
assert_eq!(
|
||||||
|
" CUSTOM ".parse::<LogLevel>(),
|
||||||
|
Ok(LogLevel::Unknown("CUSTOM".into()))
|
||||||
|
);
|
||||||
|
// Pure whitespace becomes Unknown("").
|
||||||
|
assert_eq!(" ".parse::<LogLevel>(), Ok(LogLevel::Unknown("".into())));
|
||||||
|
// Internal whitespace is NOT collapsed.
|
||||||
|
assert_eq!(
|
||||||
|
"W ARN".parse::<LogLevel>(),
|
||||||
|
Ok(LogLevel::Unknown("W ARN".into()))
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
// 测试:LogLevel 的 Display 输出格式是否正确。
|
// 测试:LogLevel 的 Display 输出格式是否正确。
|
||||||
fn test_display_output() {
|
fn test_display_output() {
|
||||||
@@ -319,6 +363,7 @@ mod tests {
|
|||||||
timestamp: Some("2024-01-01T00:00:00".to_string()),
|
timestamp: Some("2024-01-01T00:00:00".to_string()),
|
||||||
level: Some(LogLevel::Info),
|
level: Some(LogLevel::Info),
|
||||||
fields,
|
fields,
|
||||||
|
duplicate_keys: vec![],
|
||||||
};
|
};
|
||||||
|
|
||||||
// 逐字段验证。
|
// 逐字段验证。
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::sync::{Arc, Mutex};
|
use std::sync::{Arc, Mutex};
|
||||||
|
|
||||||
use crossbeam_channel::{bounded, Receiver, Sender};
|
use crossbeam_channel::{Receiver, Sender, bounded};
|
||||||
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
|
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
|
||||||
|
|
||||||
use crate::error::Result;
|
use crate::error::Result;
|
||||||
@@ -13,6 +13,7 @@ pub enum FileEvent {
|
|||||||
Truncated { new_size: u64 },
|
Truncated { new_size: u64 },
|
||||||
Rotated { new_inode: u64 },
|
Rotated { new_inode: u64 },
|
||||||
Removed,
|
Removed,
|
||||||
|
WatcherError { message: String },
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─── get_inode ──────────────────────────────────────────────────────────────
|
// ─── get_inode ──────────────────────────────────────────────────────────────
|
||||||
@@ -34,6 +35,43 @@ struct WatchState {
|
|||||||
last_inode: u64,
|
last_inode: u64,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn process_event(event: Event, watch_path: &Path, state: &mut WatchState) -> Option<FileEvent> {
|
||||||
|
if !event.paths.iter().any(|p| p == watch_path) {
|
||||||
|
return None;
|
||||||
|
}
|
||||||
|
|
||||||
|
match event.kind {
|
||||||
|
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
|
||||||
|
EventKind::Remove(_) => return Some(FileEvent::Removed),
|
||||||
|
_ => return None,
|
||||||
|
}
|
||||||
|
|
||||||
|
let current_inode = get_inode(watch_path).unwrap_or(0);
|
||||||
|
let current_size = std::fs::metadata(watch_path).map(|m| m.len()).unwrap_or(0);
|
||||||
|
|
||||||
|
if current_inode != 0 && state.last_inode != 0 && current_inode != state.last_inode {
|
||||||
|
state.last_inode = current_inode;
|
||||||
|
state.last_size = current_size;
|
||||||
|
Some(FileEvent::Rotated {
|
||||||
|
new_inode: current_inode,
|
||||||
|
})
|
||||||
|
} else if current_size > state.last_size {
|
||||||
|
state.last_size = current_size;
|
||||||
|
state.last_inode = current_inode;
|
||||||
|
Some(FileEvent::Appended {
|
||||||
|
new_size: current_size,
|
||||||
|
})
|
||||||
|
} else if current_size < state.last_size {
|
||||||
|
state.last_size = current_size;
|
||||||
|
state.last_inode = current_inode;
|
||||||
|
Some(FileEvent::Truncated {
|
||||||
|
new_size: current_size,
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ─── FileWatcher ────────────────────────────────────────────────────────────
|
// ─── FileWatcher ────────────────────────────────────────────────────────────
|
||||||
pub struct FileWatcher {
|
pub struct FileWatcher {
|
||||||
rx: Receiver<FileEvent>,
|
rx: Receiver<FileEvent>,
|
||||||
@@ -56,50 +94,22 @@ impl FileWatcher {
|
|||||||
notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| {
|
notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| {
|
||||||
let event = match res {
|
let event = match res {
|
||||||
Ok(e) => e,
|
Ok(e) => e,
|
||||||
Err(_) => return,
|
Err(error) => {
|
||||||
|
let _ = tx.try_send(FileEvent::WatcherError {
|
||||||
|
message: error.to_string(),
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
match event.kind {
|
|
||||||
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
|
|
||||||
EventKind::Remove(_) => {
|
|
||||||
let _ = tx.send(FileEvent::Removed);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
_ => return,
|
|
||||||
}
|
|
||||||
|
|
||||||
if !event.paths.iter().any(|p| p == &watch_path) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
let current_inode = get_inode(&watch_path).unwrap_or(0);
|
|
||||||
let current_size = std::fs::metadata(&watch_path).map(|m| m.len()).unwrap_or(0);
|
|
||||||
|
|
||||||
let mut st = state.lock().unwrap_or_else(|poison| {
|
let mut st = state.lock().unwrap_or_else(|poison| {
|
||||||
// Recover from poisoned mutex — state only tracks last_size
|
// Recover from poisoned mutex — state only tracks last_size
|
||||||
// and last_inode for event dedup. Stale values at worst
|
// and last_inode for event dedup. Stale values at worst
|
||||||
// cause a duplicate event, which is harmless.
|
// cause a duplicate event, which is harmless.
|
||||||
poison.into_inner()
|
poison.into_inner()
|
||||||
});
|
});
|
||||||
|
if let Some(fe) = process_event(event, &watch_path, &mut st) {
|
||||||
if current_inode != 0 && st.last_inode != 0 && current_inode != st.last_inode {
|
let _ = tx.try_send(fe);
|
||||||
let _ = tx.send(FileEvent::Rotated {
|
|
||||||
new_inode: current_inode,
|
|
||||||
});
|
|
||||||
st.last_inode = current_inode;
|
|
||||||
st.last_size = current_size;
|
|
||||||
} else if current_size > st.last_size {
|
|
||||||
let _ = tx.send(FileEvent::Appended {
|
|
||||||
new_size: current_size,
|
|
||||||
});
|
|
||||||
st.last_size = current_size;
|
|
||||||
st.last_inode = current_inode;
|
|
||||||
} else if current_size < st.last_size {
|
|
||||||
let _ = tx.send(FileEvent::Truncated {
|
|
||||||
new_size: current_size,
|
|
||||||
});
|
|
||||||
st.last_size = current_size;
|
|
||||||
st.last_inode = current_inode;
|
|
||||||
}
|
}
|
||||||
})?;
|
})?;
|
||||||
|
|
||||||
@@ -144,6 +154,56 @@ mod tests {
|
|||||||
events
|
events
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_remove_wrong_path_ignored() {
|
||||||
|
let dir = tempfile::tempdir().expect("create temp dir");
|
||||||
|
let watched = dir.path().join("watched.log");
|
||||||
|
let other = dir.path().join("other.log");
|
||||||
|
std::fs::write(&watched, b"hello\n").expect("write watched");
|
||||||
|
std::fs::write(&other, b"other\n").expect("write other");
|
||||||
|
|
||||||
|
let mut state = WatchState {
|
||||||
|
last_size: 6,
|
||||||
|
last_inode: get_inode(&watched).unwrap_or(0),
|
||||||
|
};
|
||||||
|
|
||||||
|
let event = Event {
|
||||||
|
kind: EventKind::Remove(notify::event::RemoveKind::File),
|
||||||
|
paths: vec![other.clone()],
|
||||||
|
attrs: Default::default(),
|
||||||
|
};
|
||||||
|
|
||||||
|
let result = process_event(event, &watched, &mut state);
|
||||||
|
assert_eq!(
|
||||||
|
result, None,
|
||||||
|
"Remove for non-watched path should be ignored"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_remove_correct_path_emits_removed() {
|
||||||
|
let dir = tempfile::tempdir().expect("create temp dir");
|
||||||
|
let watched = dir.path().join("watched.log");
|
||||||
|
std::fs::write(&watched, b"hello\n").expect("write watched");
|
||||||
|
|
||||||
|
let mut state = WatchState {
|
||||||
|
last_size: 6,
|
||||||
|
last_inode: get_inode(&watched).unwrap_or(0),
|
||||||
|
};
|
||||||
|
|
||||||
|
let event = Event {
|
||||||
|
kind: EventKind::Remove(notify::event::RemoveKind::File),
|
||||||
|
paths: vec![watched.clone()],
|
||||||
|
attrs: Default::default(),
|
||||||
|
};
|
||||||
|
|
||||||
|
let result = process_event(event, &watched, &mut state);
|
||||||
|
assert!(
|
||||||
|
matches!(result, Some(FileEvent::Removed)),
|
||||||
|
"Remove for watched path should emit Removed"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_watcher_append() {
|
fn test_watcher_append() {
|
||||||
let dir = tempfile::tempdir().expect("create temp dir");
|
let dir = tempfile::tempdir().expect("create temp dir");
|
||||||
@@ -232,5 +292,19 @@ mod tests {
|
|||||||
|
|
||||||
let d = FileEvent::Rotated { new_inode: 42 };
|
let d = FileEvent::Rotated { new_inode: 42 };
|
||||||
assert_ne!(a, d);
|
assert_ne!(a, d);
|
||||||
|
|
||||||
|
let e1 = FileEvent::WatcherError {
|
||||||
|
message: "io error".into(),
|
||||||
|
};
|
||||||
|
let e2 = FileEvent::WatcherError {
|
||||||
|
message: "io error".into(),
|
||||||
|
};
|
||||||
|
assert_eq!(e1, e2);
|
||||||
|
assert_ne!(
|
||||||
|
e1,
|
||||||
|
FileEvent::WatcherError {
|
||||||
|
message: "other".into()
|
||||||
|
}
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -11,6 +11,7 @@ anyhow.workspace = true
|
|||||||
log-viewer-core.workspace = true
|
log-viewer-core.workspace = true
|
||||||
serde_json.workspace = true
|
serde_json.workspace = true
|
||||||
crossbeam-channel.workspace = true
|
crossbeam-channel.workspace = true
|
||||||
|
unicode-width.workspace = true
|
||||||
|
|
||||||
[dev-dependencies]
|
[dev-dependencies]
|
||||||
tempfile = { workspace = true }
|
tempfile = { workspace = true }
|
||||||
|
|||||||
@@ -0,0 +1,138 @@
|
|||||||
|
use std::time::Instant;
|
||||||
|
|
||||||
|
use crossterm::event::{KeyCode, KeyEvent, KeyEventKind, KeyModifiers};
|
||||||
|
|
||||||
|
use super::{App, AppMode};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
pub fn handle_key(&mut self, key: KeyEvent) {
|
||||||
|
let should_handle = match key.kind {
|
||||||
|
KeyEventKind::Press => true,
|
||||||
|
KeyEventKind::Repeat => self.is_repeatable_key(&key),
|
||||||
|
KeyEventKind::Release => false,
|
||||||
|
};
|
||||||
|
|
||||||
|
if !should_handle {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
match self.mode {
|
||||||
|
AppMode::Normal => self.handle_normal_key(key),
|
||||||
|
AppMode::Settings => self.handle_settings_key(key),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Keys that should auto-repeat when held (scroll/navigation only).
|
||||||
|
fn is_repeatable_key(&self, key: &KeyEvent) -> bool {
|
||||||
|
let plain = key.modifiers.is_empty();
|
||||||
|
let ctrl = key.modifiers == KeyModifiers::CONTROL;
|
||||||
|
|
||||||
|
match self.mode {
|
||||||
|
AppMode::Normal => {
|
||||||
|
(plain
|
||||||
|
&& matches!(
|
||||||
|
key.code,
|
||||||
|
KeyCode::Char('j')
|
||||||
|
| KeyCode::Down
|
||||||
|
| KeyCode::Char('k')
|
||||||
|
| KeyCode::Up
|
||||||
|
| KeyCode::PageDown
|
||||||
|
| KeyCode::PageUp
|
||||||
|
))
|
||||||
|
|| (ctrl
|
||||||
|
&& matches!(
|
||||||
|
key.code,
|
||||||
|
KeyCode::Char('d')
|
||||||
|
| KeyCode::Char('u')
|
||||||
|
| KeyCode::Char('f')
|
||||||
|
| KeyCode::Char('b')
|
||||||
|
))
|
||||||
|
}
|
||||||
|
AppMode::Settings => {
|
||||||
|
plain
|
||||||
|
&& matches!(
|
||||||
|
key.code,
|
||||||
|
KeyCode::Char('j')
|
||||||
|
| KeyCode::Down
|
||||||
|
| KeyCode::Char('k')
|
||||||
|
| KeyCode::Up
|
||||||
|
| KeyCode::Left
|
||||||
|
| KeyCode::Right
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn handle_normal_key(&mut self, key: KeyEvent) {
|
||||||
|
match key.code {
|
||||||
|
KeyCode::Char('q') | KeyCode::Esc => {
|
||||||
|
self.should_quit = true;
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('j') | KeyCode::Down => {
|
||||||
|
self.scroll_down_line();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('k') | KeyCode::Up => {
|
||||||
|
self.scroll_up_line();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('d') if key.modifiers.contains(KeyModifiers::CONTROL) => {
|
||||||
|
self.scroll_down_half_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('u') if key.modifiers.contains(KeyModifiers::CONTROL) => {
|
||||||
|
self.scroll_up_half_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('f') if key.modifiers.contains(KeyModifiers::CONTROL) => {
|
||||||
|
self.scroll_down_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::PageDown => {
|
||||||
|
self.scroll_down_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('b') if key.modifiers.contains(KeyModifiers::CONTROL) => {
|
||||||
|
self.scroll_up_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::PageUp => {
|
||||||
|
self.scroll_up_page();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('G') | KeyCode::End => {
|
||||||
|
self.scroll_to_bottom();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('g') => {
|
||||||
|
if let Some(instant) = self.last_g_press
|
||||||
|
&& instant.elapsed().as_millis() < 500
|
||||||
|
{
|
||||||
|
self.scroll_to_top();
|
||||||
|
self.last_g_press = None;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
self.last_g_press = Some(Instant::now());
|
||||||
|
}
|
||||||
|
KeyCode::Home => {
|
||||||
|
self.scroll_to_top();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Tab => {
|
||||||
|
self.toggle_json_format();
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
KeyCode::Char('s') | KeyCode::Char('S')
|
||||||
|
if !key.modifiers.contains(KeyModifiers::CONTROL) =>
|
||||||
|
{
|
||||||
|
self.settings_draft = self.color_config.clone();
|
||||||
|
self.settings_error = None;
|
||||||
|
self.mode = AppMode::Settings;
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
self.last_g_press = None;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
use std::path::Path;
|
||||||
|
|
||||||
|
use log_viewer_core::io::progressive_reader::{
|
||||||
|
IndexerMessage, ProgressiveFileReader, spawn_indexer,
|
||||||
|
};
|
||||||
|
use log_viewer_core::watcher::file_watcher::FileWatcher;
|
||||||
|
|
||||||
|
use super::{App, AppLoadingState};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
pub fn load_file(&mut self, path: &str) -> anyhow::Result<()> {
|
||||||
|
// ── Phase 1: Pure computation, no mutation of self ──────────
|
||||||
|
// If any step fails and returns ?, self remains completely untouched,
|
||||||
|
// preserving the old file's watcher, loading_state, and file_path.
|
||||||
|
let mut pfr =
|
||||||
|
ProgressiveFileReader::open(Path::new(path)).map_err(|e| anyhow::anyhow!("{e}"))?;
|
||||||
|
|
||||||
|
let new_loading_state = if pfr.is_sampling() {
|
||||||
|
// Cache miss: spawn background indexer
|
||||||
|
let (cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
|
||||||
|
let generation = pfr.generation();
|
||||||
|
let indexer_rx =
|
||||||
|
spawn_indexer(pfr.path().to_path_buf(), generation, 80, false, cancel_rx);
|
||||||
|
pfr = ProgressiveFileReader::with_channels(
|
||||||
|
Path::new(path),
|
||||||
|
cancel_tx,
|
||||||
|
indexer_rx,
|
||||||
|
generation,
|
||||||
|
)
|
||||||
|
.map_err(|e| anyhow::anyhow!("{e}"))?;
|
||||||
|
|
||||||
|
let estimated = pfr.line_count() as u64;
|
||||||
|
AppLoadingState::Loading {
|
||||||
|
reader: pfr,
|
||||||
|
estimated_lines: estimated,
|
||||||
|
progress_percent: 0.0,
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Cache hit: Ready state
|
||||||
|
AppLoadingState::Ready { reader: pfr }
|
||||||
|
};
|
||||||
|
|
||||||
|
let new_watcher = FileWatcher::watch(Path::new(path)).ok();
|
||||||
|
|
||||||
|
// ── Phase 2: Commit — swap self to new state ───────────────
|
||||||
|
// SAFETY: Do NOT add any fallible operations (with ?) below this point.
|
||||||
|
// The old file_watcher and loading_state are dropped here, cancelling
|
||||||
|
// any background indexer for the previous file.
|
||||||
|
self.file_watcher = new_watcher;
|
||||||
|
self.loading_state = new_loading_state;
|
||||||
|
self.file_path = Some(path.to_string());
|
||||||
|
|
||||||
|
// Reset UI state for the new file
|
||||||
|
self.cursor_line = 0;
|
||||||
|
self.v_offset = 0;
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
self.last_g_press = None;
|
||||||
|
self.json_format = false;
|
||||||
|
self.mode = super::AppMode::Normal;
|
||||||
|
self.reload_after_loading = false;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Poll the background indexer for progress/completion.
|
||||||
|
/// Transitions Loading → Ready when indexing completes.
|
||||||
|
/// Must be called every frame in the event loop.
|
||||||
|
pub fn poll_background_indexer(&mut self) {
|
||||||
|
// Poll visual height rebuild (Ready state only).
|
||||||
|
// Recalibrate v_offset only when VHI actually changed.
|
||||||
|
if let AppLoadingState::Ready { reader } = &mut self.loading_state
|
||||||
|
&& let Some(index) = reader.poll_visual_height_rebuild()
|
||||||
|
{
|
||||||
|
if let log_viewer_core::io::progressive_reader::ReaderState::Ready {
|
||||||
|
visual_height_index,
|
||||||
|
..
|
||||||
|
} = &mut reader.state
|
||||||
|
{
|
||||||
|
*visual_height_index = Some(index);
|
||||||
|
}
|
||||||
|
|
||||||
|
let logical_top = self.v_offset.min(self.total_lines().saturating_sub(1));
|
||||||
|
let sub = self.v_sub_offset;
|
||||||
|
let first_visual = self.cursor_to_first_visual_row(logical_top);
|
||||||
|
let line_height = self
|
||||||
|
.get_visual_height_index()
|
||||||
|
.map_or(1, |idx| idx.visual_height_of_line(logical_top));
|
||||||
|
self.v_offset = first_visual.saturating_add(sub.min(line_height.saturating_sub(1)));
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.clamp_cursor_sub_offset_no_vhi();
|
||||||
|
self.clamp_v_offset();
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Poll main indexer (Loading state)
|
||||||
|
let old_state = std::mem::replace(&mut self.loading_state, AppLoadingState::Empty);
|
||||||
|
|
||||||
|
if let AppLoadingState::Loading {
|
||||||
|
mut reader,
|
||||||
|
estimated_lines,
|
||||||
|
mut progress_percent,
|
||||||
|
} = old_state
|
||||||
|
{
|
||||||
|
if let Some(msg) = reader.poll_indexer() {
|
||||||
|
match msg {
|
||||||
|
IndexerMessage::Progress { percent, .. } => {
|
||||||
|
progress_percent = percent;
|
||||||
|
self.loading_state = AppLoadingState::Loading {
|
||||||
|
reader,
|
||||||
|
estimated_lines,
|
||||||
|
progress_percent,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
IndexerMessage::Complete {
|
||||||
|
reader: fr,
|
||||||
|
visual_height_index,
|
||||||
|
..
|
||||||
|
} => {
|
||||||
|
let saved_cursor = self.cursor_line;
|
||||||
|
|
||||||
|
reader.set_ready(fr, visual_height_index);
|
||||||
|
self.loading_state = AppLoadingState::Ready { reader };
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
|
||||||
|
// Clamp cursor if exact count < estimated
|
||||||
|
self.cursor_line = saved_cursor.min(self.total_lines().saturating_sub(1));
|
||||||
|
|
||||||
|
// Loading uses 1:1 logical-line offsets; Ready uses visual-row
|
||||||
|
// offsets derived from the prefix-sum index. Recompute v_offset
|
||||||
|
// so the same logical line stays visible (falls back to 1:1 when
|
||||||
|
// the index is absent, which is the case right after invalidate).
|
||||||
|
self.v_offset = self.cursor_to_first_visual_row(self.cursor_line);
|
||||||
|
self.clamp_v_offset();
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
|
||||||
|
// Gutter width changes (~N → N) shift content_width, so any
|
||||||
|
// VisualHeightIndex built with the old width is stale.
|
||||||
|
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
|
||||||
|
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
}
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = new_sub;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
|
||||||
|
if self.reload_after_loading {
|
||||||
|
self.reload_after_loading = false;
|
||||||
|
self.reload_ready_reader();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
IndexerMessage::Error { message, .. } => {
|
||||||
|
self.loading_state = AppLoadingState::Error(message);
|
||||||
|
self.reload_after_loading = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
self.loading_state = AppLoadingState::Loading {
|
||||||
|
reader,
|
||||||
|
estimated_lines,
|
||||||
|
progress_percent,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
self.loading_state = old_state;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,209 @@
|
|||||||
|
use log_viewer_core::config::ColorConfig;
|
||||||
|
use log_viewer_core::io::progressive_reader::VisualHeightIndex;
|
||||||
|
|
||||||
|
use super::viewport_cache::gutter_width_for;
|
||||||
|
use super::{App, AppLoadingState, AppMode, ViewportRenderRow};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub fn get_line(&self, idx: usize) -> Option<String> {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Ready { reader } => reader.get_line(idx),
|
||||||
|
AppLoadingState::Loading { reader, .. } => reader.get_line(idx),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub fn file_name(&self) -> Option<&str> {
|
||||||
|
self.file_path
|
||||||
|
.as_ref()
|
||||||
|
.and_then(|p| std::path::Path::new(p).file_name().and_then(|n| n.to_str()))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn total_lines(&self) -> usize {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Ready { reader } => reader.line_count(),
|
||||||
|
AppLoadingState::Loading {
|
||||||
|
reader,
|
||||||
|
estimated_lines,
|
||||||
|
..
|
||||||
|
} => {
|
||||||
|
// Use estimated total lines (not sampled_line_count) so the user can
|
||||||
|
// scroll freely during indexing. get_line() incrementally scans
|
||||||
|
// forward on demand, so lines beyond the initial 64KB are still
|
||||||
|
// accessible. The .max() guards against under-estimates.
|
||||||
|
(*estimated_lines as usize).max(reader.sampled_line_count())
|
||||||
|
}
|
||||||
|
_ => 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn is_loaded(&self) -> bool {
|
||||||
|
matches!(
|
||||||
|
self.loading_state,
|
||||||
|
AppLoadingState::Ready { .. } | AppLoadingState::Loading { .. }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn is_loading(&self) -> bool {
|
||||||
|
matches!(self.loading_state, AppLoadingState::Loading { .. })
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn is_error(&self) -> bool {
|
||||||
|
matches!(self.loading_state, AppLoadingState::Error(_))
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn get_visual_height_index(&self) -> Option<&VisualHeightIndex> {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Ready { reader } => match &reader.state {
|
||||||
|
log_viewer_core::io::progressive_reader::ReaderState::Ready {
|
||||||
|
visual_height_index,
|
||||||
|
..
|
||||||
|
} => visual_height_index.as_ref(),
|
||||||
|
_ => None,
|
||||||
|
},
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn error_message(&self) -> Option<&str> {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Error(msg) => Some(msg),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn loading_progress(&self) -> Option<f64> {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Loading {
|
||||||
|
progress_percent, ..
|
||||||
|
} => Some(*progress_percent),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn estimated_lines(&self) -> Option<u64> {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Loading {
|
||||||
|
estimated_lines, ..
|
||||||
|
} => Some(*estimated_lines),
|
||||||
|
_ => None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn mode(&self) -> AppMode {
|
||||||
|
self.mode
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn settings_error(&self) -> Option<&str> {
|
||||||
|
self.settings_error.as_deref()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn settings_draft(&self) -> &ColorConfig {
|
||||||
|
&self.settings_draft
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn settings_cursor(&self) -> usize {
|
||||||
|
self.settings_cursor
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn color_config(&self) -> &ColorConfig {
|
||||||
|
&self.color_config
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn cursor_line(&self) -> usize {
|
||||||
|
self.cursor_line
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn cursor_sub_offset(&self) -> usize {
|
||||||
|
self.cursor_sub_offset
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub(crate) fn content_width(&self) -> u16 {
|
||||||
|
self.content_width
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub(crate) fn content_height(&self) -> u16 {
|
||||||
|
self.content_height
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn set_content_area(&mut self, width: u16, height: u16) {
|
||||||
|
self.content_width = width;
|
||||||
|
self.content_height = height;
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(crate) fn viewport_rows(
|
||||||
|
&self,
|
||||||
|
start_logical: usize,
|
||||||
|
offset_in_line: usize,
|
||||||
|
available_rows: usize,
|
||||||
|
) -> Vec<ViewportRenderRow<'_>> {
|
||||||
|
let mut rows = Vec::new();
|
||||||
|
|
||||||
|
for (entry_idx, entry) in self.viewport_cache.entries.iter().enumerate() {
|
||||||
|
let logical_line = self.viewport_cache.logical_start + entry_idx;
|
||||||
|
let start_row = if logical_line == start_logical {
|
||||||
|
offset_in_line
|
||||||
|
} else {
|
||||||
|
0
|
||||||
|
};
|
||||||
|
|
||||||
|
for (visual_row, text) in entry.wrapped_rows.iter().enumerate().skip(start_row) {
|
||||||
|
if rows.len() >= available_rows {
|
||||||
|
return rows;
|
||||||
|
}
|
||||||
|
rows.push(ViewportRenderRow {
|
||||||
|
logical_line,
|
||||||
|
visual_row,
|
||||||
|
text,
|
||||||
|
level: entry.level.as_ref(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rows
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) fn enter_settings_mode_for_test(&mut self) {
|
||||||
|
self.mode = AppMode::Settings;
|
||||||
|
self.settings_draft = self.color_config.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
pub(crate) fn set_cursor_for_test(&mut self, line: usize, sub_offset: usize) {
|
||||||
|
self.cursor_line = line;
|
||||||
|
self.cursor_sub_offset = sub_offset;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
pub fn set_error_state(&mut self, msg: impl Into<String>) {
|
||||||
|
self.loading_state = AppLoadingState::Error(msg.into());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub fn file_size(&self) -> u64 {
|
||||||
|
match &self.loading_state {
|
||||||
|
AppLoadingState::Ready { reader } => reader.reader().map_or(0, |r| r.file_size()),
|
||||||
|
AppLoadingState::Loading { reader, .. } => reader.reader().map_or(0, |r| r.file_size()),
|
||||||
|
_ => 0,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// MUST match the renderer's `gutter_width` formula in `ui.rs::render_content`.
|
||||||
|
pub(crate) fn gutter_width(&self) -> usize {
|
||||||
|
gutter_width_for(self.total_lines(), self.is_loading())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn get_content_width(&self) -> usize {
|
||||||
|
let total = if self.content_width > 0 {
|
||||||
|
self.content_width as usize
|
||||||
|
} else {
|
||||||
|
80
|
||||||
|
};
|
||||||
|
total.saturating_sub(gutter_width_for(self.total_lines(), self.is_loading()))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,359 @@
|
|||||||
|
use super::App;
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
fn line_visual_height(&self, line: usize, width: usize) -> usize {
|
||||||
|
if line >= self.total_lines() {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
self.compute_visual_height(line, width).max(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn advance_visual_pos(
|
||||||
|
&self,
|
||||||
|
mut line: usize,
|
||||||
|
mut sub: usize,
|
||||||
|
mut n: usize,
|
||||||
|
width: usize,
|
||||||
|
) -> (usize, usize) {
|
||||||
|
let last = self.total_lines().saturating_sub(1);
|
||||||
|
while n > 0 && line < last {
|
||||||
|
let h = self.line_visual_height(line, width);
|
||||||
|
let remaining_in_line = h.saturating_sub(sub);
|
||||||
|
if n < remaining_in_line {
|
||||||
|
sub += n;
|
||||||
|
return (line, sub);
|
||||||
|
}
|
||||||
|
n -= remaining_in_line;
|
||||||
|
line += 1;
|
||||||
|
sub = 0;
|
||||||
|
}
|
||||||
|
if line >= last {
|
||||||
|
let h = self.line_visual_height(last, width);
|
||||||
|
sub = (sub + n).min(h.saturating_sub(1));
|
||||||
|
line = last;
|
||||||
|
}
|
||||||
|
(line, sub)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn retreat_visual_pos(
|
||||||
|
&self,
|
||||||
|
mut line: usize,
|
||||||
|
mut sub: usize,
|
||||||
|
mut n: usize,
|
||||||
|
width: usize,
|
||||||
|
) -> (usize, usize) {
|
||||||
|
while n > 0 {
|
||||||
|
if sub >= n {
|
||||||
|
sub -= n;
|
||||||
|
return (line, sub);
|
||||||
|
}
|
||||||
|
n -= sub + 1;
|
||||||
|
if line == 0 {
|
||||||
|
return (0, 0);
|
||||||
|
}
|
||||||
|
line -= 1;
|
||||||
|
sub = self.line_visual_height(line, width).saturating_sub(1);
|
||||||
|
}
|
||||||
|
(line, sub)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn clamp_cursor_sub_offset_no_vhi(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let h = self.line_visual_height(self.cursor_line, width);
|
||||||
|
if self.cursor_sub_offset >= h {
|
||||||
|
self.cursor_sub_offset = h.saturating_sub(1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn ensure_cursor_visible_no_vhi(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 || self.content_height == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let content_h = self.content_height as usize;
|
||||||
|
let cur_line = self.cursor_line;
|
||||||
|
let cur_sub = self.cursor_sub_offset;
|
||||||
|
let v_top_line = self.v_offset;
|
||||||
|
let v_top_sub = self.v_sub_offset;
|
||||||
|
|
||||||
|
if cur_line < v_top_line || (cur_line == v_top_line && cur_sub < v_top_sub) {
|
||||||
|
self.v_offset = cur_line;
|
||||||
|
self.v_sub_offset = cur_sub;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut line = v_top_line;
|
||||||
|
let mut sub = v_top_sub;
|
||||||
|
let mut walked = 0usize;
|
||||||
|
let total = self.total_lines();
|
||||||
|
while walked < content_h {
|
||||||
|
if line == cur_line && sub == cur_sub {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let h = self.line_visual_height(line, width);
|
||||||
|
if sub + 1 < h {
|
||||||
|
sub += 1;
|
||||||
|
} else if line + 1 < total {
|
||||||
|
line += 1;
|
||||||
|
sub = 0;
|
||||||
|
} else {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
walked += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let target_distance = content_h.saturating_sub(1);
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.retreat_visual_pos(cur_line, cur_sub, target_distance, width);
|
||||||
|
self.v_offset = new_line;
|
||||||
|
self.v_sub_offset = new_sub;
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_down_line(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let last = self.total_lines().saturating_sub(1);
|
||||||
|
let current_height = index.visual_height_of_line(self.cursor_line);
|
||||||
|
if self.cursor_sub_offset + 1 < current_height {
|
||||||
|
self.cursor_sub_offset += 1;
|
||||||
|
} else if self.cursor_line < last {
|
||||||
|
self.cursor_line += 1;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
}
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 && self.total_lines() > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, 1, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_up_line(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
if self.cursor_sub_offset > 0 {
|
||||||
|
self.cursor_sub_offset -= 1;
|
||||||
|
} else if self.cursor_line > 0 {
|
||||||
|
let previous_line = self.cursor_line - 1;
|
||||||
|
let previous_line_last_sub =
|
||||||
|
index.visual_height_of_line(previous_line).saturating_sub(1);
|
||||||
|
self.cursor_line = previous_line;
|
||||||
|
self.cursor_sub_offset = previous_line_last_sub;
|
||||||
|
}
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, 1, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_down_half_page(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let half = self.content_height as usize / 2;
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let total_vr = index.total_visual_rows();
|
||||||
|
if total_vr == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let cur_vr = self.cursor_visual_row();
|
||||||
|
let target_vr = cur_vr.saturating_add(half as u64).min(total_vr - 1);
|
||||||
|
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 && half > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, half, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_up_half_page(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let half = self.content_height as usize / 2;
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let total_vr = index.total_visual_rows();
|
||||||
|
if total_vr == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let cur_vr = self.cursor_visual_row();
|
||||||
|
let target_vr = cur_vr.saturating_sub(half as u64);
|
||||||
|
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 && half > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, half, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_down_page(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let page = self.content_height as usize;
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let total_vr = index.total_visual_rows();
|
||||||
|
if total_vr == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let cur_vr = self.cursor_visual_row();
|
||||||
|
let target_vr = cur_vr.saturating_add(page as u64).min(total_vr - 1);
|
||||||
|
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 && page > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.advance_visual_pos(self.cursor_line, self.cursor_sub_offset, page, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_up_page(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let page = self.content_height as usize;
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let total_vr = index.total_visual_rows();
|
||||||
|
if total_vr == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let cur_vr = self.cursor_visual_row();
|
||||||
|
let target_vr = cur_vr.saturating_sub(page as u64);
|
||||||
|
let (new_line, new_sub) = index.visual_row_to_logical_row_with_offset(target_vr);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
if width > 0 && page > 0 {
|
||||||
|
let (new_line, new_sub) =
|
||||||
|
self.retreat_visual_pos(self.cursor_line, self.cursor_sub_offset, page, width);
|
||||||
|
self.cursor_line = new_line;
|
||||||
|
self.cursor_sub_offset = new_sub;
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_to_top(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
self.cursor_line = 0;
|
||||||
|
self.v_offset = 0;
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn scroll_to_bottom(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
self.cursor_line = self.total_lines().saturating_sub(1);
|
||||||
|
if let Some(idx) = self.get_visual_height_index() {
|
||||||
|
self.cursor_sub_offset = idx
|
||||||
|
.visual_height_of_line(self.cursor_line)
|
||||||
|
.saturating_sub(1);
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
self.clamp_v_offset();
|
||||||
|
} else {
|
||||||
|
let width = self.get_content_width();
|
||||||
|
self.cursor_sub_offset = if width > 0 {
|
||||||
|
self.line_visual_height(self.cursor_line, width)
|
||||||
|
.saturating_sub(1)
|
||||||
|
} else {
|
||||||
|
0
|
||||||
|
};
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn ensure_cursor_visible(&mut self) {
|
||||||
|
if !self.is_loaded() || self.total_lines() == 0 || self.content_height == 0 {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if self.is_loading() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let cursor_visual = self.cursor_visual_row() as usize;
|
||||||
|
|
||||||
|
let content_h = self.content_height as usize;
|
||||||
|
|
||||||
|
if cursor_visual < self.v_offset {
|
||||||
|
self.v_offset = cursor_visual;
|
||||||
|
} else if cursor_visual >= self.v_offset.saturating_add(content_h) {
|
||||||
|
self.v_offset = cursor_visual.saturating_sub(content_h).saturating_add(1);
|
||||||
|
}
|
||||||
|
self.clamp_v_offset();
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn clamp_v_offset(&mut self) {
|
||||||
|
let max_offset = self
|
||||||
|
.total_visual_rows()
|
||||||
|
.saturating_sub(self.content_height as usize);
|
||||||
|
self.v_offset = self.v_offset.min(max_offset);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cursor_visual_row(&self) -> u64 {
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
let first = index.cursor_to_first_visual_row(self.cursor_line);
|
||||||
|
let height = index.visual_height_of_line(self.cursor_line);
|
||||||
|
let max_sub = height.saturating_sub(1);
|
||||||
|
let sub = self.cursor_sub_offset.min(max_sub) as u64;
|
||||||
|
first + sub
|
||||||
|
} else {
|
||||||
|
self.cursor_line as u64
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
use crossterm::event::{KeyCode, KeyEvent};
|
||||||
|
|
||||||
|
use crate::color::AVAILABLE_COLORS;
|
||||||
|
|
||||||
|
use super::{App, AppMode};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
pub(super) fn handle_settings_key(&mut self, key: KeyEvent) {
|
||||||
|
match key.code {
|
||||||
|
KeyCode::Esc | KeyCode::Char('q') => {
|
||||||
|
self.settings_error = None;
|
||||||
|
self.mode = AppMode::Normal;
|
||||||
|
}
|
||||||
|
KeyCode::Enter => {
|
||||||
|
let draft = self.settings_draft.clone();
|
||||||
|
match draft.save() {
|
||||||
|
Ok(()) => {
|
||||||
|
self.color_config = draft;
|
||||||
|
self.settings_error = None;
|
||||||
|
self.mode = AppMode::Normal;
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
self.settings_error = Some(format!("Failed to save settings: {e}"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
KeyCode::Char('j') | KeyCode::Down => {
|
||||||
|
if self.settings_cursor < 5 {
|
||||||
|
self.settings_cursor += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
KeyCode::Char('k') | KeyCode::Up => {
|
||||||
|
self.settings_cursor = self.settings_cursor.saturating_sub(1);
|
||||||
|
}
|
||||||
|
KeyCode::Left => {
|
||||||
|
self.cycle_color(self.settings_cursor, false);
|
||||||
|
}
|
||||||
|
KeyCode::Right => {
|
||||||
|
self.cycle_color(self.settings_cursor, true);
|
||||||
|
}
|
||||||
|
KeyCode::Char(c) if ('1'..='8').contains(&c) => {
|
||||||
|
let idx = (c as usize) - ('1' as usize);
|
||||||
|
self.set_color(self.settings_cursor, AVAILABLE_COLORS[idx]);
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cycle_color(&mut self, level_idx: usize, forward: bool) {
|
||||||
|
let current = self.get_settings_color(level_idx).to_string();
|
||||||
|
let colors = AVAILABLE_COLORS;
|
||||||
|
let pos = colors.iter().position(|&c| c == current);
|
||||||
|
let new_pos = match pos {
|
||||||
|
Some(p) => {
|
||||||
|
if forward {
|
||||||
|
(p + 1) % colors.len()
|
||||||
|
} else if p == 0 {
|
||||||
|
colors.len() - 1
|
||||||
|
} else {
|
||||||
|
p - 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None => {
|
||||||
|
if forward {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
colors.len() - 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
self.set_color(level_idx, colors[new_pos]);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_settings_color(&self, level_idx: usize) -> &str {
|
||||||
|
match level_idx {
|
||||||
|
0 => &self.settings_draft.error,
|
||||||
|
1 => &self.settings_draft.warn,
|
||||||
|
2 => &self.settings_draft.info,
|
||||||
|
3 => &self.settings_draft.debug,
|
||||||
|
4 => &self.settings_draft.trace,
|
||||||
|
5 => &self.settings_draft.unknown,
|
||||||
|
_ => "white",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn set_color(&mut self, level_idx: usize, color_name: &str) {
|
||||||
|
self.settings_error = None;
|
||||||
|
match level_idx {
|
||||||
|
0 => self.settings_draft.error = color_name.to_string(),
|
||||||
|
1 => self.settings_draft.warn = color_name.to_string(),
|
||||||
|
2 => self.settings_draft.info = color_name.to_string(),
|
||||||
|
3 => self.settings_draft.debug = color_name.to_string(),
|
||||||
|
4 => self.settings_draft.trace = color_name.to_string(),
|
||||||
|
5 => self.settings_draft.unknown = color_name.to_string(),
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
use std::borrow::Cow;
|
||||||
|
|
||||||
|
use log_viewer_core::io::wrap::{MAX_WRAP_INPUT_LEN, format_json_line, wrap_line_chars};
|
||||||
|
|
||||||
|
use super::viewport_cache::{ViewportEntry, truncate_to_columns};
|
||||||
|
use super::{App, AppLoadingState};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
/// Compute a single line's viewport entry (wrapped rows + level + height).
|
||||||
|
pub(super) fn compute_line_entry(&self, line: usize, width: usize) -> ViewportEntry {
|
||||||
|
let raw = self.get_line(line).unwrap_or_default();
|
||||||
|
|
||||||
|
// Guard 1: oversized raw input — skip detect_level and JSON formatting
|
||||||
|
// to avoid O(n) parsing overhead on huge lines.
|
||||||
|
if raw.len() > MAX_WRAP_INPUT_LEN {
|
||||||
|
return ViewportEntry {
|
||||||
|
wrapped_rows: vec![truncate_to_columns(&raw, width)],
|
||||||
|
level: None,
|
||||||
|
visual_height: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
let level = log_viewer_core::parser::level::detect_level(&raw);
|
||||||
|
let display_text: Cow<'_, str> = if self.json_format {
|
||||||
|
format_json_line(&raw)
|
||||||
|
} else {
|
||||||
|
Cow::Borrowed(raw.as_str())
|
||||||
|
};
|
||||||
|
|
||||||
|
// Guard 2: JSON pretty-printing may expand a line beyond the limit.
|
||||||
|
if display_text.len() > MAX_WRAP_INPUT_LEN {
|
||||||
|
return ViewportEntry {
|
||||||
|
wrapped_rows: vec![truncate_to_columns(&display_text, width)],
|
||||||
|
level,
|
||||||
|
visual_height: 1,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut wrapped = Vec::new();
|
||||||
|
for sub_line in display_text.split('\n') {
|
||||||
|
wrapped.extend(wrap_line_chars(sub_line, width));
|
||||||
|
}
|
||||||
|
let visual_height = wrapped.len().max(1);
|
||||||
|
ViewportEntry {
|
||||||
|
wrapped_rows: wrapped,
|
||||||
|
level,
|
||||||
|
visual_height,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute visual height for a single line without storing it.
|
||||||
|
pub(super) fn compute_visual_height(&self, line: usize, width: usize) -> usize {
|
||||||
|
let raw = self.get_line(line).unwrap_or_default();
|
||||||
|
|
||||||
|
// Guard 1: oversized raw input.
|
||||||
|
if raw.len() > MAX_WRAP_INPUT_LEN {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let display_text: Cow<'_, str> = if self.json_format {
|
||||||
|
format_json_line(&raw)
|
||||||
|
} else {
|
||||||
|
Cow::Borrowed(raw.as_str())
|
||||||
|
};
|
||||||
|
|
||||||
|
if display_text.len() > MAX_WRAP_INPUT_LEN {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut height = 0;
|
||||||
|
for sub_line in display_text.split('\n') {
|
||||||
|
height += wrap_line_chars(sub_line, width).len();
|
||||||
|
}
|
||||||
|
height.max(1)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Find (logical_line, offset_in_line) for a given visual row offset.
|
||||||
|
fn find_logical_line_at_visual_row(&self, visual_row: usize, _width: usize) -> (usize, usize) {
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
return index.visual_row_to_logical_row_with_offset(visual_row as u64);
|
||||||
|
}
|
||||||
|
(visual_row.min(self.total_lines().saturating_sub(1)), 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ensure the viewport cache covers the visible range.
|
||||||
|
/// Returns (start_logical, offset_in_line) for rendering.
|
||||||
|
pub(crate) fn ensure_viewport_cache(&mut self, width: usize) -> (usize, usize) {
|
||||||
|
let viewport_height = self.content_height as usize;
|
||||||
|
|
||||||
|
if !self.is_loaded() || width == 0 || viewport_height == 0 {
|
||||||
|
return (0, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
let params_changed = self.viewport_cache.needs_recompute(width, self.json_format);
|
||||||
|
|
||||||
|
if params_changed {
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
self.viewport_cache.width = width;
|
||||||
|
self.viewport_cache.set_json_format(self.json_format);
|
||||||
|
self.ensure_visual_height_index(width);
|
||||||
|
|
||||||
|
if self.get_visual_height_index().is_some() {
|
||||||
|
self.ensure_cursor_visible();
|
||||||
|
} else {
|
||||||
|
self.ensure_cursor_visible_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find start logical line from v_offset
|
||||||
|
let (start_logical, offset_in_line) = if self.is_loading() {
|
||||||
|
(
|
||||||
|
self.v_offset.min(self.total_lines().saturating_sub(1)),
|
||||||
|
self.v_sub_offset,
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
self.find_logical_line_at_visual_row(self.v_offset, width)
|
||||||
|
};
|
||||||
|
|
||||||
|
// Compute viewport entries
|
||||||
|
self.viewport_cache.entries.clear();
|
||||||
|
self.viewport_cache.logical_start = start_logical;
|
||||||
|
|
||||||
|
let total = self.total_lines();
|
||||||
|
let mut rows_remaining = viewport_height + offset_in_line;
|
||||||
|
|
||||||
|
for line_idx in start_logical..total {
|
||||||
|
if rows_remaining == 0 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
let entry = self.compute_line_entry(line_idx, width);
|
||||||
|
rows_remaining = rows_remaining.saturating_sub(entry.visual_height);
|
||||||
|
self.viewport_cache.entries.push(entry);
|
||||||
|
}
|
||||||
|
|
||||||
|
(start_logical, offset_in_line)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute total visual rows (cached, lazily evaluated).
|
||||||
|
/// Returns `total_lines` for sampling mode (1:1 mapping).
|
||||||
|
pub(super) fn total_visual_rows(&mut self) -> usize {
|
||||||
|
if self.is_loading() {
|
||||||
|
return self.total_lines();
|
||||||
|
}
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
return index.total_visual_rows() as usize;
|
||||||
|
}
|
||||||
|
self.total_lines()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn cursor_to_first_visual_row(&self, line: usize) -> usize {
|
||||||
|
if self.is_loading() {
|
||||||
|
return line;
|
||||||
|
}
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
return index.cursor_to_first_visual_row(line) as usize;
|
||||||
|
}
|
||||||
|
line
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn visual_row_to_logical_row(&self, visual_row: usize) -> usize {
|
||||||
|
if self.is_loading() {
|
||||||
|
return visual_row.min(self.total_lines().saturating_sub(1));
|
||||||
|
}
|
||||||
|
if let Some(index) = self.get_visual_height_index() {
|
||||||
|
return index.visual_row_to_logical_row(visual_row as u64);
|
||||||
|
}
|
||||||
|
visual_row.min(self.total_lines().saturating_sub(1))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Compute the rebased offset pair (logical_line, sub_row) from the
|
||||||
|
/// current visual-row `v_offset`. Returns `(v_offset, v_sub_offset)`
|
||||||
|
/// suitable for the no-VHI fallback scrolling path.
|
||||||
|
///
|
||||||
|
/// MUST be called before borrowing `&mut self.loading_state` for
|
||||||
|
/// `invalidate_visual_height_index`, because it reads VHI through
|
||||||
|
/// `&self`.
|
||||||
|
pub(super) fn rebase_offset_for_invalidate(&self) -> (usize, usize) {
|
||||||
|
if self.get_visual_height_index().is_some() {
|
||||||
|
let top_visual = self.v_offset;
|
||||||
|
let top_line = self.visual_row_to_logical_row(top_visual);
|
||||||
|
let line_first_visual = self.cursor_to_first_visual_row(top_line);
|
||||||
|
let sub = top_visual.saturating_sub(line_first_visual);
|
||||||
|
(top_line, sub)
|
||||||
|
} else {
|
||||||
|
(self.v_offset, self.v_sub_offset)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn ensure_visual_height_index(&mut self, width: usize) {
|
||||||
|
let needs_rebuild = match self.get_visual_height_index() {
|
||||||
|
Some(idx) => !idx.is_valid_for(self.json_format, width),
|
||||||
|
None => true,
|
||||||
|
};
|
||||||
|
|
||||||
|
if needs_rebuild {
|
||||||
|
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
|
||||||
|
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
reader.start_visual_height_rebuild(width, self.json_format);
|
||||||
|
}
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = new_sub;
|
||||||
|
self.clamp_cursor_sub_offset_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn toggle_json_format(&mut self) {
|
||||||
|
self.json_format = !self.json_format;
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
let width = self.viewport_cache.width;
|
||||||
|
let (new_offset, new_sub) = self.rebase_offset_for_invalidate();
|
||||||
|
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
if width > 0 {
|
||||||
|
reader.start_visual_height_rebuild(width, self.json_format);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = new_sub;
|
||||||
|
self.clamp_cursor_sub_offset_no_vhi();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
use log_viewer_core::types::LogLevel;
|
||||||
|
use unicode_width::UnicodeWidthChar;
|
||||||
|
|
||||||
|
pub(super) struct ViewportEntry {
|
||||||
|
pub(super) wrapped_rows: Vec<String>,
|
||||||
|
pub(super) level: Option<LogLevel>,
|
||||||
|
pub(super) visual_height: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) struct ViewportCache {
|
||||||
|
pub(super) entries: Vec<ViewportEntry>,
|
||||||
|
pub(super) logical_start: usize,
|
||||||
|
pub(super) width: usize,
|
||||||
|
json_format: bool,
|
||||||
|
pub(super) cached_total_visual_rows: Option<usize>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ViewportCache {
|
||||||
|
pub(super) fn new() -> Self {
|
||||||
|
Self {
|
||||||
|
entries: Vec::new(),
|
||||||
|
logical_start: 0,
|
||||||
|
width: 0,
|
||||||
|
json_format: false,
|
||||||
|
cached_total_visual_rows: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn invalidate(&mut self) {
|
||||||
|
self.entries.clear();
|
||||||
|
self.logical_start = 0;
|
||||||
|
self.width = 0;
|
||||||
|
self.cached_total_visual_rows = None;
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn needs_recompute(&self, width: usize, json_format: bool) -> bool {
|
||||||
|
self.width != width || self.json_format != json_format
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn set_json_format(&mut self, json_format: bool) {
|
||||||
|
self.json_format = json_format;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
|
pub(super) fn get_entry(&self, logical_line: usize) -> Option<&ViewportEntry> {
|
||||||
|
if logical_line >= self.logical_start {
|
||||||
|
let idx = logical_line - self.logical_start;
|
||||||
|
self.entries.get(idx)
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) const TRUNCATE_TAB_WIDTH: usize = 4;
|
||||||
|
|
||||||
|
pub(super) fn gutter_width_for(total_lines: usize, is_loading: bool) -> usize {
|
||||||
|
if total_lines == 0 {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
let line_num_width = total_lines.to_string().len();
|
||||||
|
let loading_extra = if is_loading { 1 } else { 0 };
|
||||||
|
line_num_width + loading_extra + 1 + 1
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn truncate_to_columns(s: &str, max_cols: usize) -> String {
|
||||||
|
if max_cols == 0 || s.is_empty() {
|
||||||
|
return String::new();
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut out = String::new();
|
||||||
|
let mut col = 0;
|
||||||
|
|
||||||
|
for ch in s.chars() {
|
||||||
|
if ch == '\t' {
|
||||||
|
let tab_stop = TRUNCATE_TAB_WIDTH - (col % TRUNCATE_TAB_WIDTH);
|
||||||
|
if col + tab_stop > max_cols {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
for _ in 0..tab_stop {
|
||||||
|
out.push(' ');
|
||||||
|
}
|
||||||
|
col += tab_stop;
|
||||||
|
} else {
|
||||||
|
let w = if ch.is_control() {
|
||||||
|
0
|
||||||
|
} else {
|
||||||
|
ch.width().unwrap_or(0)
|
||||||
|
};
|
||||||
|
if col + w > max_cols {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
out.push(ch);
|
||||||
|
col += w;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
out
|
||||||
|
}
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
use log_viewer_core::io::file_reader::AppendStatus;
|
||||||
|
use log_viewer_core::io::progressive_reader::{ReaderState, compute_line_visual_height};
|
||||||
|
use log_viewer_core::watcher::file_watcher::FileEvent;
|
||||||
|
|
||||||
|
use super::viewport_cache::gutter_width_for;
|
||||||
|
use super::{App, AppLoadingState};
|
||||||
|
|
||||||
|
impl App {
|
||||||
|
pub fn poll_file_watcher(&mut self) {
|
||||||
|
let events: Vec<FileEvent> = match &mut self.file_watcher {
|
||||||
|
Some(w) => std::iter::from_fn(|| w.try_recv()).collect(),
|
||||||
|
None => return,
|
||||||
|
};
|
||||||
|
|
||||||
|
for event in events {
|
||||||
|
match event {
|
||||||
|
FileEvent::Appended { new_size: _ } => {
|
||||||
|
self.handle_file_appended();
|
||||||
|
}
|
||||||
|
FileEvent::Truncated { new_size: _ } => {
|
||||||
|
self.handle_file_truncated();
|
||||||
|
}
|
||||||
|
FileEvent::Rotated { new_inode: _ } => {
|
||||||
|
// Don't auto-switch; old content preserved.
|
||||||
|
// User can reload manually if desired.
|
||||||
|
}
|
||||||
|
FileEvent::Removed => {
|
||||||
|
self.loading_state = AppLoadingState::Error("File has been deleted".into());
|
||||||
|
}
|
||||||
|
FileEvent::WatcherError { message: _ } => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn handle_file_appended(&mut self) {
|
||||||
|
let rebased = self.rebase_offset_for_invalidate();
|
||||||
|
match &mut self.loading_state {
|
||||||
|
AppLoadingState::Ready { reader } => {
|
||||||
|
let old_reader_line_count = reader.line_count();
|
||||||
|
let status = reader.update_for_append();
|
||||||
|
let width = {
|
||||||
|
let total = if self.content_width > 0 {
|
||||||
|
self.content_width as usize
|
||||||
|
} else {
|
||||||
|
80
|
||||||
|
};
|
||||||
|
total.saturating_sub(gutter_width_for(reader.line_count(), false))
|
||||||
|
};
|
||||||
|
match status {
|
||||||
|
Ok(AppendStatus::Appended(_new_lines)) => {
|
||||||
|
let _ = reader.save_cache();
|
||||||
|
|
||||||
|
let (old_line_count, can_extend) = {
|
||||||
|
match &reader.state {
|
||||||
|
ReaderState::Ready {
|
||||||
|
visual_height_index: Some(idx),
|
||||||
|
..
|
||||||
|
} => (idx.line_count(), idx.is_valid_for(self.json_format, width)),
|
||||||
|
_ => (0, false),
|
||||||
|
}
|
||||||
|
};
|
||||||
|
let new_line_count = reader.line_count();
|
||||||
|
|
||||||
|
if can_extend && old_line_count == old_reader_line_count {
|
||||||
|
if let ReaderState::Ready {
|
||||||
|
visual_height_index: Some(index),
|
||||||
|
reader: fr,
|
||||||
|
} = &mut reader.state
|
||||||
|
{
|
||||||
|
if old_line_count > 0 {
|
||||||
|
let last_old_line_text =
|
||||||
|
fr.get_line(old_line_count - 1).unwrap_or("");
|
||||||
|
let new_h = compute_line_visual_height(
|
||||||
|
last_old_line_text,
|
||||||
|
width,
|
||||||
|
self.json_format,
|
||||||
|
);
|
||||||
|
index.replace_last_line_height(new_h);
|
||||||
|
}
|
||||||
|
let mut new_heights = Vec::with_capacity(
|
||||||
|
new_line_count.saturating_sub(old_line_count),
|
||||||
|
);
|
||||||
|
for i in old_line_count..new_line_count {
|
||||||
|
let line_text = fr.get_line(i).unwrap_or("");
|
||||||
|
new_heights.push(compute_line_visual_height(
|
||||||
|
line_text,
|
||||||
|
width,
|
||||||
|
self.json_format,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
index.extend_from_heights(&new_heights);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let (new_offset, new_sub) = rebased;
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = new_sub;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
reader.start_visual_height_rebuild(width, self.json_format);
|
||||||
|
}
|
||||||
|
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
}
|
||||||
|
Ok(AppendStatus::Reloaded) => {
|
||||||
|
let _ = reader.save_cache();
|
||||||
|
let (new_offset, _new_sub) = rebased;
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
reader.start_visual_height_rebuild(width, self.json_format);
|
||||||
|
self.cursor_line =
|
||||||
|
self.cursor_line.min(self.total_lines().saturating_sub(1));
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.cursor_sub_offset = 0;
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
self.clamp_v_offset();
|
||||||
|
}
|
||||||
|
Ok(AppendStatus::Unchanged) | Err(_) => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
AppLoadingState::Loading { .. } => {
|
||||||
|
self.reload_after_loading = true;
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn reload_ready_reader(&mut self) {
|
||||||
|
let (new_offset, _new_sub) = self.rebase_offset_for_invalidate();
|
||||||
|
if let AppLoadingState::Ready { reader } = &mut self.loading_state {
|
||||||
|
let _ = reader.reload();
|
||||||
|
let width = {
|
||||||
|
let total = if self.content_width > 0 {
|
||||||
|
self.content_width as usize
|
||||||
|
} else {
|
||||||
|
80
|
||||||
|
};
|
||||||
|
total.saturating_sub(gutter_width_for(reader.line_count(), false))
|
||||||
|
};
|
||||||
|
let _ = reader.save_cache();
|
||||||
|
reader.invalidate_visual_height_index();
|
||||||
|
reader.start_visual_height_rebuild(width, self.json_format);
|
||||||
|
self.cursor_line = self.cursor_line.min(self.total_lines().saturating_sub(1));
|
||||||
|
self.v_offset = new_offset;
|
||||||
|
self.v_sub_offset = 0;
|
||||||
|
self.clamp_cursor_sub_offset_no_vhi();
|
||||||
|
self.viewport_cache.invalidate();
|
||||||
|
self.clamp_v_offset();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub(super) fn handle_file_truncated(&mut self) {
|
||||||
|
match &mut self.loading_state {
|
||||||
|
AppLoadingState::Ready { .. } => {
|
||||||
|
self.reload_ready_reader();
|
||||||
|
}
|
||||||
|
AppLoadingState::Loading { .. } => {
|
||||||
|
self.reload_after_loading = true;
|
||||||
|
}
|
||||||
|
_ => {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
+73
-21
@@ -1,3 +1,6 @@
|
|||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
use anyhow::Context;
|
||||||
use clap::Parser;
|
use clap::Parser;
|
||||||
|
|
||||||
mod app;
|
mod app;
|
||||||
@@ -11,44 +14,93 @@ struct Cli {
|
|||||||
files: Vec<String>,
|
files: Vec<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── RAII terminal guard ────────────────────────────────────────────
|
||||||
|
//
|
||||||
|
// Holds the ratatui Terminal and guarantees terminal state restoration
|
||||||
|
// (disable raw mode, leave alternate screen, show cursor) on **every**
|
||||||
|
// exit path: normal return, `?` error propagation, and panic unwind.
|
||||||
|
//
|
||||||
|
// IMPORTANT: `Drop` does **not** run on `std::process::exit` or
|
||||||
|
// `panic = "abort"`. Therefore we must never call `process::exit`
|
||||||
|
// inside the guarded scope — use `?` to return `Err` instead.
|
||||||
|
|
||||||
|
type Backend = ratatui::backend::CrosstermBackend<std::io::Stdout>;
|
||||||
|
|
||||||
|
struct TerminalGuard {
|
||||||
|
terminal: ratatui::Terminal<Backend>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TerminalGuard {
|
||||||
|
/// Enable raw mode, enter alternate screen, and create the terminal.
|
||||||
|
/// On partial failure, rolls back any steps that already succeeded.
|
||||||
|
fn enter() -> anyhow::Result<Self> {
|
||||||
|
crossterm::terminal::enable_raw_mode()
|
||||||
|
.map_err(|e| anyhow::anyhow!("enable_raw_mode: {e}"))?;
|
||||||
|
|
||||||
|
let mut stdout = std::io::stdout();
|
||||||
|
if let Err(e) = crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen) {
|
||||||
|
let _ = crossterm::terminal::disable_raw_mode();
|
||||||
|
return Err(anyhow::anyhow!("EnterAlternateScreen: {e}"));
|
||||||
|
}
|
||||||
|
|
||||||
|
let backend = ratatui::backend::CrosstermBackend::new(stdout);
|
||||||
|
match ratatui::Terminal::new(backend) {
|
||||||
|
Ok(terminal) => Ok(Self { terminal }),
|
||||||
|
Err(e) => {
|
||||||
|
// Roll back: leave alternate screen + disable raw mode
|
||||||
|
let _ = crossterm::execute!(
|
||||||
|
std::io::stdout(),
|
||||||
|
crossterm::terminal::LeaveAlternateScreen
|
||||||
|
);
|
||||||
|
let _ = crossterm::terminal::disable_raw_mode();
|
||||||
|
Err(anyhow::anyhow!("Terminal::new: {e}"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn terminal(&mut self) -> &mut ratatui::Terminal<Backend> {
|
||||||
|
&mut self.terminal
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Drop for TerminalGuard {
|
||||||
|
fn drop(&mut self) {
|
||||||
|
// Best-effort cleanup; suppress errors (Drop must not panic).
|
||||||
|
let _ = crossterm::terminal::disable_raw_mode();
|
||||||
|
let _ = crossterm::execute!(
|
||||||
|
self.terminal.backend_mut(),
|
||||||
|
crossterm::terminal::LeaveAlternateScreen
|
||||||
|
);
|
||||||
|
let _ = self.terminal.show_cursor();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── main ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
fn main() -> anyhow::Result<()> {
|
fn main() -> anyhow::Result<()> {
|
||||||
let cli = Cli::parse();
|
let cli = Cli::parse();
|
||||||
|
|
||||||
crossterm::terminal::enable_raw_mode()?;
|
let mut guard = TerminalGuard::enter()?;
|
||||||
let mut stdout = std::io::stdout();
|
|
||||||
crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen)?;
|
|
||||||
let backend = ratatui::backend::CrosstermBackend::new(stdout);
|
|
||||||
let mut terminal = ratatui::Terminal::new(backend)?;
|
|
||||||
|
|
||||||
let mut app = app::App::new();
|
let mut app = app::App::new();
|
||||||
app.color_config = log_viewer_core::config::ColorConfig::load();
|
app.color_config = log_viewer_core::config::ColorConfig::load();
|
||||||
|
|
||||||
if let Some(file) = cli.files.first()
|
if let Some(file) = cli.files.first() {
|
||||||
&& let Err(e) = app.load_file(file)
|
app.load_file(file)
|
||||||
{
|
.with_context(|| format!("loading file {file}"))?;
|
||||||
eprintln!("Error loading file: {e}");
|
|
||||||
std::process::exit(1);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
while !app.should_quit {
|
while !app.should_quit {
|
||||||
app.poll_background_indexer();
|
app.poll_background_indexer();
|
||||||
app.poll_file_watcher();
|
app.poll_file_watcher();
|
||||||
terminal.draw(|frame| ui::render(frame, &mut app))?;
|
guard.terminal().draw(|frame| ui::render(frame, &mut app))?;
|
||||||
if crossterm::event::poll(std::time::Duration::from_millis(100))? {
|
if crossterm::event::poll(Duration::from_millis(100))? {
|
||||||
match crossterm::event::read()? {
|
match crossterm::event::read()? {
|
||||||
crossterm::event::Event::Key(key) => app.handle_key(key),
|
crossterm::event::Event::Key(key) => app.handle_key(key),
|
||||||
crossterm::event::Event::Resize(_w, _h) => {}
|
crossterm::event::Event::Resize(_, _) => {}
|
||||||
_ => {}
|
_ => {}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
crossterm::terminal::disable_raw_mode()?;
|
|
||||||
crossterm::execute!(
|
|
||||||
terminal.backend_mut(),
|
|
||||||
crossterm::terminal::LeaveAlternateScreen
|
|
||||||
)?;
|
|
||||||
terminal.show_cursor()?;
|
|
||||||
|
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|||||||
+147
-55
@@ -6,6 +6,7 @@ use crate::color::level_fg;
|
|||||||
use log_viewer_core::config::ColorConfig;
|
use log_viewer_core::config::ColorConfig;
|
||||||
use log_viewer_core::types::LogLevel;
|
use log_viewer_core::types::LogLevel;
|
||||||
|
|
||||||
|
#[allow(dead_code)]
|
||||||
pub(crate) fn build_line_spans(
|
pub(crate) fn build_line_spans(
|
||||||
gutter_text: String,
|
gutter_text: String,
|
||||||
content_text: String,
|
content_text: String,
|
||||||
@@ -28,6 +29,10 @@ pub(crate) fn build_line_spans(
|
|||||||
])
|
])
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn is_cursor_visual_row(app: &App, logical_line: usize, visual_row: usize) -> bool {
|
||||||
|
logical_line == app.cursor_line() && visual_row == app.cursor_sub_offset()
|
||||||
|
}
|
||||||
|
|
||||||
pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
||||||
use ratatui::layout::{Constraint, Layout};
|
use ratatui::layout::{Constraint, Layout};
|
||||||
use ratatui::widgets::Paragraph;
|
use ratatui::widgets::Paragraph;
|
||||||
@@ -40,7 +45,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
|||||||
.split(frame.area());
|
.split(frame.area());
|
||||||
|
|
||||||
// ── Title bar ──────────────────────────────────────────────────
|
// ── Title bar ──────────────────────────────────────────────────
|
||||||
let title_text = if app.mode == AppMode::Settings {
|
let title_text = if app.mode() == AppMode::Settings {
|
||||||
" Color Settings".to_string()
|
" Color Settings".to_string()
|
||||||
} else if app.is_loading() {
|
} else if app.is_loading() {
|
||||||
let name = app.file_name().unwrap_or("unknown");
|
let name = app.file_name().unwrap_or("unknown");
|
||||||
@@ -51,7 +56,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
|||||||
let cursor_display = if app.total_lines() == 0 {
|
let cursor_display = if app.total_lines() == 0 {
|
||||||
0
|
0
|
||||||
} else {
|
} else {
|
||||||
app.cursor_line + 1
|
app.cursor_line() + 1
|
||||||
};
|
};
|
||||||
format!(" {} [{}/{}]", name, cursor_display, app.total_lines())
|
format!(" {} [{}/{}]", name, cursor_display, app.total_lines())
|
||||||
} else {
|
} else {
|
||||||
@@ -63,7 +68,7 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
|||||||
);
|
);
|
||||||
|
|
||||||
// ── Content area ───────────────────────────────────────────────
|
// ── Content area ───────────────────────────────────────────────
|
||||||
if app.mode == AppMode::Settings {
|
if app.mode() == AppMode::Settings {
|
||||||
render_settings(frame, app, outer[1]);
|
render_settings(frame, app, outer[1]);
|
||||||
} else if app.is_error() {
|
} else if app.is_error() {
|
||||||
let msg = app.error_message().unwrap_or_default();
|
let msg = app.error_message().unwrap_or_default();
|
||||||
@@ -88,9 +93,21 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ── Status bar ─────────────────────────────────────────────────
|
// ── Status bar ─────────────────────────────────────────────────
|
||||||
let status_text = if app.mode == AppMode::Settings {
|
if app.mode() == AppMode::Settings {
|
||||||
" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"
|
if let Some(err) = app.settings_error() {
|
||||||
} else if app.is_error() {
|
frame.render_widget(
|
||||||
|
Paragraph::new(err).style(Style::default().fg(Color::Red)),
|
||||||
|
outer[2],
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
frame.render_widget(
|
||||||
|
Paragraph::new(" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"),
|
||||||
|
outer[2],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let status_text = if app.is_error() {
|
||||||
" Press q to quit"
|
" Press q to quit"
|
||||||
} else if app.is_loading() {
|
} else if app.is_loading() {
|
||||||
let pct = app.loading_progress().map_or(0, |p| p as usize);
|
let pct = app.loading_progress().map_or(0, |p| p as usize);
|
||||||
@@ -110,8 +127,11 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
|
|||||||
} else if app.is_loaded() {
|
} else if app.is_loaded() {
|
||||||
let name = app.file_name().unwrap_or("unknown");
|
let name = app.file_name().unwrap_or("unknown");
|
||||||
let total = app.total_lines();
|
let total = app.total_lines();
|
||||||
let cursor_display = if total == 0 { 0 } else { app.cursor_line + 1 };
|
let cursor_display = if total == 0 { 0 } else { app.cursor_line() + 1 };
|
||||||
let status = format!(" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit", name, cursor_display, total);
|
let status = format!(
|
||||||
|
" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit",
|
||||||
|
name, cursor_display, total
|
||||||
|
);
|
||||||
frame.render_widget(Paragraph::new(status), outer[2]);
|
frame.render_widget(Paragraph::new(status), outer[2]);
|
||||||
return;
|
return;
|
||||||
} else {
|
} else {
|
||||||
@@ -130,26 +150,31 @@ pub fn render_settings(frame: &mut ratatui::Frame, app: &mut App, area: ratatui:
|
|||||||
|
|
||||||
let popup_w = ((area.width as u32 * 4 / 5).max(40)).min(area.width as u32) as u16;
|
let popup_w = ((area.width as u32 * 4 / 5).max(40)).min(area.width as u32) as u16;
|
||||||
let popup_h = ((area.height as u32 * 4 / 5).max(14)).min(area.height as u32) as u16;
|
let popup_h = ((area.height as u32 * 4 / 5).max(14)).min(area.height as u32) as u16;
|
||||||
let popup_x = area.width.saturating_sub(popup_w) / 2;
|
let popup_x = area
|
||||||
let popup_y = area.height.saturating_sub(popup_h) / 2;
|
.x
|
||||||
|
.saturating_add(area.width.saturating_sub(popup_w) / 2);
|
||||||
|
let popup_y = area
|
||||||
|
.y
|
||||||
|
.saturating_add(area.height.saturating_sub(popup_h) / 2);
|
||||||
let popup = ratatui::layout::Rect::new(popup_x, popup_y, popup_w, popup_h);
|
let popup = ratatui::layout::Rect::new(popup_x, popup_y, popup_w, popup_h);
|
||||||
|
|
||||||
let block = Block::new().borders(Borders::ALL).title(" Color Settings ");
|
let block = Block::new().borders(Borders::ALL).title(" Color Settings ");
|
||||||
let inner = block.inner(popup);
|
let inner = block.inner(popup);
|
||||||
frame.render_widget(block, popup);
|
frame.render_widget(block, popup);
|
||||||
|
|
||||||
|
let settings_draft = app.settings_draft();
|
||||||
let levels = [
|
let levels = [
|
||||||
("ERROR", &app.settings_draft.error),
|
("ERROR", &settings_draft.error),
|
||||||
("WARN", &app.settings_draft.warn),
|
("WARN", &settings_draft.warn),
|
||||||
("INFO", &app.settings_draft.info),
|
("INFO", &settings_draft.info),
|
||||||
("DEBUG", &app.settings_draft.debug),
|
("DEBUG", &settings_draft.debug),
|
||||||
("TRACE", &app.settings_draft.trace),
|
("TRACE", &settings_draft.trace),
|
||||||
("UNKNOWN", &app.settings_draft.unknown),
|
("UNKNOWN", &settings_draft.unknown),
|
||||||
];
|
];
|
||||||
|
|
||||||
let mut lines = Vec::new();
|
let mut lines = Vec::new();
|
||||||
for (i, (level_name, color_name)) in levels.iter().enumerate() {
|
for (i, (level_name, color_name)) in levels.iter().enumerate() {
|
||||||
let is_selected = i == app.settings_cursor;
|
let is_selected = i == app.settings_cursor();
|
||||||
let cursor_marker = if is_selected { "▶ " } else { " " };
|
let cursor_marker = if is_selected { "▶ " } else { " " };
|
||||||
|
|
||||||
let preview_color = color_name.parse::<Color>().unwrap_or(Color::White);
|
let preview_color = color_name.parse::<Color>().unwrap_or(Color::White);
|
||||||
@@ -200,8 +225,7 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
|
|||||||
let content_width = area.width as usize;
|
let content_width = area.width as usize;
|
||||||
let content_height = area.height as usize;
|
let content_height = area.height as usize;
|
||||||
|
|
||||||
app.content_height = area.height;
|
app.set_content_area(area.width, area.height);
|
||||||
app.content_width = area.width;
|
|
||||||
|
|
||||||
let total_lines = app.total_lines();
|
let total_lines = app.total_lines();
|
||||||
let line_num_width = if total_lines > 0 {
|
let line_num_width = if total_lines > 0 {
|
||||||
@@ -209,14 +233,8 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
|
|||||||
} else {
|
} else {
|
||||||
0
|
0
|
||||||
};
|
};
|
||||||
|
|
||||||
let is_loading = app.is_loading();
|
let is_loading = app.is_loading();
|
||||||
let gutter_prefix_extra = if is_loading { 1 } else { 0 };
|
let gutter_width = app.gutter_width();
|
||||||
let gutter_width = if total_lines > 0 {
|
|
||||||
line_num_width + gutter_prefix_extra + 1 + 1
|
|
||||||
} else {
|
|
||||||
0
|
|
||||||
};
|
|
||||||
|
|
||||||
let actual_content_width = content_width.saturating_sub(gutter_width);
|
let actual_content_width = content_width.saturating_sub(gutter_width);
|
||||||
|
|
||||||
@@ -227,7 +245,6 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
|
|||||||
let (start_logical, offset_in_line) = app.ensure_viewport_cache(actual_content_width);
|
let (start_logical, offset_in_line) = app.ensure_viewport_cache(actual_content_width);
|
||||||
|
|
||||||
let mut lines: Vec<Line> = Vec::new();
|
let mut lines: Vec<Line> = Vec::new();
|
||||||
let mut current_visual_offset: usize = 0;
|
|
||||||
let available_rows = content_height;
|
let available_rows = content_height;
|
||||||
|
|
||||||
let gutter_style = if is_loading {
|
let gutter_style = if is_loading {
|
||||||
@@ -238,40 +255,28 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
|
|||||||
Style::default().fg(Color::DarkGray)
|
Style::default().fg(Color::DarkGray)
|
||||||
};
|
};
|
||||||
|
|
||||||
for (entry_idx, entry) in app.viewport_cache.entries.iter().enumerate() {
|
for row in app.viewport_rows(start_logical, offset_in_line, available_rows) {
|
||||||
let logical_line = app.viewport_cache.logical_start + entry_idx;
|
let is_cursor = is_cursor_visual_row(app, row.logical_line, row.visual_row);
|
||||||
let start_row = if logical_line == start_logical {
|
let level = row.level;
|
||||||
offset_in_line
|
|
||||||
} else {
|
|
||||||
0
|
|
||||||
};
|
|
||||||
|
|
||||||
for (visual_row, text) in entry.wrapped_rows.iter().enumerate().skip(start_row) {
|
|
||||||
if current_visual_offset >= available_rows {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
let is_cursor = logical_line == app.cursor_line;
|
|
||||||
let level = entry.level.as_ref();
|
|
||||||
|
|
||||||
let bg_color = if is_cursor {
|
let bg_color = if is_cursor {
|
||||||
Color::DarkGray
|
Color::DarkGray
|
||||||
} else {
|
} else {
|
||||||
Color::Reset
|
Color::Reset
|
||||||
};
|
};
|
||||||
let level_fg = level_fg(level, &app.color_config).unwrap_or(Color::White);
|
let level_fg = level_fg(level, app.color_config()).unwrap_or(Color::White);
|
||||||
|
|
||||||
let gutter_text = if visual_row == 0 {
|
let gutter_text = if row.visual_row == 0 {
|
||||||
if is_loading {
|
if is_loading {
|
||||||
format!(
|
format!(
|
||||||
"~{:>width$} \u{2502}",
|
"~{:>width$} \u{2502}",
|
||||||
logical_line + 1,
|
row.logical_line + 1,
|
||||||
width = line_num_width
|
width = line_num_width
|
||||||
)
|
)
|
||||||
} else {
|
} else {
|
||||||
format!(
|
format!(
|
||||||
"{:>width$} \u{2502}",
|
"{:>width$} \u{2502}",
|
||||||
logical_line + 1,
|
row.logical_line + 1,
|
||||||
width = line_num_width
|
width = line_num_width
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -289,15 +294,11 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
|
|||||||
|
|
||||||
lines.push(Line::from(vec![
|
lines.push(Line::from(vec![
|
||||||
Span::styled(gutter_text, effective_gutter_style),
|
Span::styled(gutter_text, effective_gutter_style),
|
||||||
Span::styled(text.clone(), Style::default().fg(level_fg).bg(bg_color)),
|
Span::styled(
|
||||||
|
row.text.to_string(),
|
||||||
|
Style::default().fg(level_fg).bg(bg_color),
|
||||||
|
),
|
||||||
]));
|
]));
|
||||||
|
|
||||||
current_visual_offset += 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
if current_visual_offset >= available_rows {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
while lines.len() < available_rows {
|
while lines.len() < available_rows {
|
||||||
@@ -363,6 +364,16 @@ mod tests {
|
|||||||
assert_eq!(line.spans[1].style.bg, Some(Color::Reset));
|
assert_eq!(line.spans[1].style.bg, Some(Color::Reset));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_cursor_highlight_predicate_uses_cursor_sub_offset() {
|
||||||
|
let mut app = App::new();
|
||||||
|
app.set_cursor_for_test(4, 2);
|
||||||
|
|
||||||
|
assert!(!is_cursor_visual_row(&app, 4, 0));
|
||||||
|
assert!(!is_cursor_visual_row(&app, 3, 2));
|
||||||
|
assert!(is_cursor_visual_row(&app, 4, 2));
|
||||||
|
}
|
||||||
|
|
||||||
fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer {
|
fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer {
|
||||||
let backend = ratatui::backend::TestBackend::new(width, height);
|
let backend = ratatui::backend::TestBackend::new(width, height);
|
||||||
let mut terminal = ratatui::Terminal::new(backend).unwrap();
|
let mut terminal = ratatui::Terminal::new(backend).unwrap();
|
||||||
@@ -469,7 +480,8 @@ mod tests {
|
|||||||
let result = std::panic::catch_unwind(|| {
|
let result = std::panic::catch_unwind(|| {
|
||||||
let data = std::fs::read(&path).unwrap();
|
let data = std::fs::read(&path).unwrap();
|
||||||
let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data);
|
let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data);
|
||||||
let _ = log_viewer_core::io::index_cache::IndexCache::save(&path, &index);
|
let _ =
|
||||||
|
log_viewer_core::io::index_cache::IndexCache::save_with_hash(&path, &index, &data);
|
||||||
|
|
||||||
let mut app = App::new();
|
let mut app = App::new();
|
||||||
app.load_file(path.to_str().unwrap()).unwrap();
|
app.load_file(path.to_str().unwrap()).unwrap();
|
||||||
@@ -493,4 +505,84 @@ mod tests {
|
|||||||
let _ = std::fs::remove_file(&path);
|
let _ = std::fs::remove_file(&path);
|
||||||
assert!(result.is_ok());
|
assert!(result.is_ok());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Issue #31: Settings popup area offset tests ────────────────
|
||||||
|
|
||||||
|
/// Helper: enter settings mode and render to buffer.
|
||||||
|
fn render_settings_to_buffer(
|
||||||
|
app: &mut App,
|
||||||
|
width: u16,
|
||||||
|
height: u16,
|
||||||
|
) -> ratatui::buffer::Buffer {
|
||||||
|
app.enter_settings_mode_for_test();
|
||||||
|
render_to_buffer(app, width, height)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Find the top-left corner of the popup border by scanning for '┌'.
|
||||||
|
fn find_popup_top_left(
|
||||||
|
buf: &ratatui::buffer::Buffer,
|
||||||
|
width: u16,
|
||||||
|
height: u16,
|
||||||
|
) -> Option<(u16, u16)> {
|
||||||
|
for row in 0..height {
|
||||||
|
for col in 0..width {
|
||||||
|
if buf.cell((col, row)).unwrap().symbol() == "┌" {
|
||||||
|
return Some((col, row));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
None
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_settings_popup_includes_area_offset() {
|
||||||
|
// In an 80x24 frame with Layout [Length(1), Min(1), Length(1)]:
|
||||||
|
// outer[0] = title bar -> y=0
|
||||||
|
// outer[1] = content -> y=1, height=22
|
||||||
|
// outer[2] = status bar -> y=23
|
||||||
|
// The popup is centered within outer[1], so its y must be >= outer[1].y (which is 1).
|
||||||
|
let mut app = App::new();
|
||||||
|
let buf = render_settings_to_buffer(&mut app, 80, 24);
|
||||||
|
|
||||||
|
let (_px, py) =
|
||||||
|
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
|
||||||
|
|
||||||
|
// outer[1].y == 1; the popup is centered inside a 22-row area,
|
||||||
|
// so popup_y must be at least 1 (not 0).
|
||||||
|
assert!(
|
||||||
|
py >= 1,
|
||||||
|
"popup top row should account for area.y offset, got y={py} (expected >= 1)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_settings_popup_horizontal_centering_uses_area_x() {
|
||||||
|
// outer[1].x is 0 for this layout, so this mainly verifies the popup
|
||||||
|
// is centered and not shifted left. A non-zero area.x layout would
|
||||||
|
// need a different layout to trigger, but the formula is the same.
|
||||||
|
let mut app = App::new();
|
||||||
|
let buf = render_settings_to_buffer(&mut app, 80, 24);
|
||||||
|
|
||||||
|
let (px, _py) =
|
||||||
|
find_popup_top_left(&buf, 80, 24).expect("popup border '┌' should be rendered");
|
||||||
|
|
||||||
|
// popup_w = 80*4/5 = 64, centered: (80-64)/2 = 8
|
||||||
|
assert_eq!(
|
||||||
|
px, 8,
|
||||||
|
"popup should start at x=8 (centered 64-wide popup in 80-col area)"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn test_settings_popup_small_frame_no_panic() {
|
||||||
|
// Frame smaller than the min popup size (40x14) should not panic.
|
||||||
|
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
|
||||||
|
let mut app = App::new();
|
||||||
|
let _buf = render_settings_to_buffer(&mut app, 30, 10);
|
||||||
|
}));
|
||||||
|
assert!(
|
||||||
|
result.is_ok(),
|
||||||
|
"rendering settings in a small frame should not panic"
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user