logViewer/README.md

logViewer

中文文档

A high-performance terminal log file viewer built in Rust, designed to handle multi-gigabyte files with minimal memory overhead.

Uses memory-mapped I/O with a sparse line index and progressive background loading to open files instantly — even 5GB+ logs scroll smoothly from the first keystroke.

Features

• Instant file open — mmap-backed reader with background progressive indexing; start scrolling before indexing finishes
• Handles huge files — sparse line index (1 entry per 256 lines) keeps memory usage at ~8MB even for 5GB files
• Live file tracking — watches for appends, truncations, and log rotation via notify
• JSON log support — NDJSON parsing with BOM handling, duplicate key detection, and toggleable pretty-printing
• Unicode-aware wrapping — correct line wrapping for CJK, emoji, and tabs
• Persistent index cache — line indexes saved to disk with xxh3 content hashing; re-opens are near-instant
• Vim-like keybindings — familiar navigation for terminal users
• Customizable colors — per-log-level colors via TOML config, adjustable from within the TUI

Quick Start

# Build and run
cargo run -p log-viewer-tui -- path/to/logfile.log

# Or build first
cargo build --release -p log-viewer-tui
./target/release/log-viewer-tui path/to/logfile.log

Requires Rust 1.92+ (see rust-toolchain.toml).

Keybindings

Key	Action
j / ↓	Scroll down one line
k / ↑	Scroll up one line
Ctrl+d	Scroll down half page
Ctrl+u	Scroll up half page
Ctrl+f / PgDn	Scroll down full page
Ctrl+b / PgUp	Scroll up full page
G / End	Jump to end of file
gg / Home	Jump to start of file
Tab	Toggle JSON pretty-printing
S	Open color settings
q / Esc	Quit

Settings panel: use j/k to select a log level, ←/→ to cycle colors, Enter to save, Esc to cancel.

Workspace Structure

crates/core  — Shared library: I/O, parsing, types, config, file watching
crates/tui   — Terminal UI (ratatui + crossterm)
crates/gui   — GUI (egui + eframe) — placeholder, not yet functional
crates/bench — mmap vs pread benchmark harness

Development

# Check, test, lint
cargo check --workspace
cargo test --workspace
cargo fmt --check --all
cargo clippy --workspace -- -D warnings

# Run benchmarks (requires a pre-generated ~5GB test file)
mkdir -p /tmp/test-logviewer
dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000
cargo run -p log-viewer-bench
cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md

# Test a single crate
cargo test -p log-viewer-core
cargo test -p log-viewer-tui

CI

The CI gate runs on ubuntu-latest and windows-latest:

1. cargo fmt --check --all
2. cargo check --workspace
3. cargo test --workspace
4. cargo clippy --workspace -- -D warnings

No custom rustfmt or clippy config — uses toolchain defaults.

Architecture

The core library (log-viewer-core) owns all I/O, parsing, and data types. Key design decisions:

• Sparse line index — samples every 256 lines using memchr SIMD acceleration. A 1M-line file produces a ~32KB index instead of ~8MB.
• Progressive loading — ProgressiveFileReader starts with a quick head+tail sample for instant line estimates, then builds the full index in a background thread via crossbeam-channel.
• mmap with TOCTOU mitigation — post-mmap stat check to detect file changes. read_cache module exists as a future pread-based alternative to eliminate SIGBUS risk entirely.
• Persistent cache — line indexes are serialized to disk with xxh3 content hashing for invalidation. Atomic writes via temp files.
• Visual height index — prefix-sum over wrapped-line heights, enabling O(log n) scroll-to-line mapping for long wrapped lines.

Planned but not yet built: filtering, bookmarks, sessions, search engine (stub modules exist).

Benchmarks

Custom harness comparing mmap vs pread backends across 7 suites (startup, render, jump, memory, growth, rotation, concurrent). Uses wall-clock timing with /proc/self/ RSS and page fault metrics. Results are written as markdown tables to benchmark-report.md.

The harness expects /tmp/test-logviewer/extreme.log to already exist. It does not generate the primary 5GB test file automatically; create it first with the dd command shown above.

The benchmark binary includes ~75 unit tests for the reader backends and metrics collection.