diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f6ae39b..d91cbf6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -2,9 +2,9 @@ name: CI on: push: - branches: [main] + branches: [master] pull_request: - branches: [main] + branches: [master] env: CARGO_TERM_COLOR: always diff --git a/README-zh.md b/README-zh.md new file mode 100644 index 0000000..1776aa7 --- /dev/null +++ b/README-zh.md @@ -0,0 +1,105 @@ +# logViewer + +**[English](README.md)** + +用 Rust 构建的高性能终端日志查看器,专为处理超大日志文件(GB 级别)而设计,内存占用极低。 + +基于 mmap 内存映射、稀疏行索引和后台渐进式加载,打开文件几乎瞬时完成——即使是 5GB 以上的日志文件,也能在索引完成前就开始滚动浏览。 + +## 特性 + +- **瞬时打开** — mmap 读取 + 后台渐进式索引,索引未完成即可开始浏览 +- **超大文件支持** — 稀疏行索引(每 256 行采样一次),5GB 文件仅占用约 8MB 内存 +- **实时文件追踪** — 通过 `notify` 监控文件追加、截断和日志轮转 +- **JSON 日志解析** — 支持 NDJSON 格式,自动处理 BOM、检测重复键,可切换格式化显示 +- **Unicode 感知的换行** — 正确处理中文、emoji 和制表符的行宽计算 +- **持久化索引缓存** — 行索引保存到磁盘,使用 xxh3 哈希校验内容变更,再次打开几乎无需等待 +- **Vim 风格快捷键** — 终端用户无学习成本 +- **可自定义配色** — 每个日志级别独立配色,通过 TOML 配置文件管理,TUI 内直接调整 + +## 快速开始 + +```bash +# 直接编译运行 +cargo run -p log-viewer-tui -- path/to/logfile.log + +# 或先编译再运行 +cargo build --release -p log-viewer-tui +./target/release/log-viewer-tui path/to/logfile.log +``` + +需要 Rust 1.92 及以上版本(见 `rust-toolchain.toml`)。 + +## 快捷键 + +| 按键 | 操作 | +|------|------| +| `j` / `↓` | 向下滚动一行 | +| `k` / `↑` | 向上滚动一行 | +| `Ctrl+d` | 向下滚动半页 | +| `Ctrl+u` | 向上滚动半页 | +| `Ctrl+f` / `PgDn` | 向下滚动一页 | +| `Ctrl+b` / `PgUp` | 向上滚动一页 | +| `G` / `End` | 跳转到文件末尾 | +| `gg` / `Home` | 跳转到文件开头 | +| `Tab` | 切换 JSON 格式化显示 | +| `S` | 打开配色设置 | +| `q` / `Esc` | 退出 | + +设置面板:`j`/`k` 选择日志级别,`←`/`→` 切换颜色,`Enter` 保存,`Esc` 取消。 + +## 项目结构 + +``` +crates/core — 核心库:I/O、解析、数据类型、配置、文件监控 +crates/tui — 终端界面(ratatui + crossterm) +crates/gui — 图形界面(egui + eframe)— 占位模块,尚未实现 +crates/bench — mmap 与 pread 性能对比基准测试 +``` + +## 开发 + +```bash +# 检查、测试、代码规范 +cargo check --workspace +cargo test --workspace +cargo fmt --check --all +cargo clippy --workspace -- -D warnings + +# 运行基准测试(首次运行会生成约 5GB 测试文件) +cargo run -p log-viewer-bench +cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md + +# 单独测试某个 crate +cargo test -p log-viewer-core +cargo test -p log-viewer-tui +``` + +### CI + +CI 在 ubuntu-latest 和 windows-latest 上运行以下检查: + +1. `cargo fmt --check --all` +2. `cargo check --workspace` +3. `cargo test --workspace` +4. `cargo clippy --workspace -- -D warnings` + +无自定义 rustfmt 或 clippy 配置——使用工具链默认值。 + +## 架构 + +核心库 `log-viewer-core` 负责所有 I/O、解析和数据类型。关键设计决策: + +- **稀疏行索引** — 每 256 行采样一次,使用 memchr SIMD 加速。100 万行的文件仅产生约 32KB 的索引,而非 8MB。 +- **渐进式加载** — `ProgressiveFileReader` 先读取文件头尾进行快速行数估算,再通过 crossbeam-channel 在后台线程构建完整索引。 +- **mmap 与 TOCTOU 防护** — mmap 映射后进行 stat 校验以检测文件变更。`read_cache` 模块为未来基于 pread 的实现预留,可彻底消除 SIGBUS 风险。 +- **持久化缓存** — 行索引序列化到磁盘,使用 xxh3 内容哈希进行失效校验,通过临时文件实现原子写入。 +- **视觉高度索引** — 基于换行后行高的前缀和数组,支持 O(log n) 的滚动定位,适用于长行换行场景。 + +计划中但尚未实现的功能:过滤、书签、会话管理、搜索引擎(stub 模块已创建)。 + +## 基准测试 + +自研基准测试框架,对比 mmap 与 pread 两种后端在 7 个测试场景(启动、渲染、跳转、内存、增长、轮转、并发)下的表现。使用挂钟计时和 `/proc/self/` 的 RSS 及页错误指标。结果以 Markdown 表格形式输出到 `benchmark-report.md`。 + +基准测试二进制包含约 75 个单元测试,覆盖读取器后端和指标采集逻辑。 diff --git a/README.md b/README.md new file mode 100644 index 0000000..c89b788 --- /dev/null +++ b/README.md @@ -0,0 +1,104 @@ +# logViewer + +**[中文文档](README-zh.md)** + +A high-performance terminal log file viewer built in Rust, designed to handle multi-gigabyte files with minimal memory overhead. + +Uses memory-mapped I/O with a sparse line index and progressive background loading to open files instantly — even 5GB+ logs scroll smoothly from the first keystroke. + +## Features + +- **Instant file open** — mmap-backed reader with background progressive indexing; start scrolling before indexing finishes +- **Handles huge files** — sparse line index (1 entry per 256 lines) keeps memory usage at ~8MB even for 5GB files +- **Live file tracking** — watches for appends, truncations, and log rotation via `notify` +- **JSON log support** — NDJSON parsing with BOM handling, duplicate key detection, and toggleable pretty-printing +- **Unicode-aware wrapping** — correct line wrapping for CJK, emoji, and tabs +- **Persistent index cache** — line indexes saved to disk with xxh3 content hashing; re-opens are near-instant +- **Vim-like keybindings** — familiar navigation for terminal users +- **Customizable colors** — per-log-level colors via TOML config, adjustable from within the TUI + +## Quick Start + +```bash +# Build and run +cargo run -p log-viewer-tui -- path/to/logfile.log + +# Or build first +cargo build --release -p log-viewer-tui +./target/release/log-viewer-tui path/to/logfile.log +``` + +Requires Rust 1.92+ (see `rust-toolchain.toml`). + +## Keybindings + +| Key | Action | +|-----|--------| +| `j` / `↓` | Scroll down one line | +| `k` / `↑` | Scroll up one line | +| `Ctrl+d` | Scroll down half page | +| `Ctrl+u` | Scroll up half page | +| `Ctrl+f` / `PgDn` | Scroll down full page | +| `Ctrl+b` / `PgUp` | Scroll up full page | +| `G` / `End` | Jump to end of file | +| `gg` / `Home` | Jump to start of file | +| `Tab` | Toggle JSON pretty-printing | +| `S` | Open color settings | +| `q` / `Esc` | Quit | + +Settings panel: use `j`/`k` to select a log level, `←`/`→` to cycle colors, `Enter` to save, `Esc` to cancel. + +## Workspace Structure + +``` +crates/core — Shared library: I/O, parsing, types, config, file watching +crates/tui — Terminal UI (ratatui + crossterm) +crates/gui — GUI (egui + eframe) — placeholder, not yet functional +crates/bench — mmap vs pread benchmark harness +``` + +## Development + +```bash +# Check, test, lint +cargo check --workspace +cargo test --workspace +cargo fmt --check --all +cargo clippy --workspace -- -D warnings + +# Run benchmarks (generates ~5GB test file on first run) +cargo run -p log-viewer-bench +cargo run -p log-viewer-bench -- --quick --suites startup,render --output results.md + +# Test a single crate +cargo test -p log-viewer-core +cargo test -p log-viewer-tui +``` + +### CI + +The CI gate runs on ubuntu-latest and windows-latest: +1. `cargo fmt --check --all` +2. `cargo check --workspace` +3. `cargo test --workspace` +4. `cargo clippy --workspace -- -D warnings` + +No custom rustfmt or clippy config — uses toolchain defaults. + +## Architecture + +The core library (`log-viewer-core`) owns all I/O, parsing, and data types. Key design decisions: + +- **Sparse line index** — samples every 256 lines using memchr SIMD acceleration. A 1M-line file produces a ~32KB index instead of ~8MB. +- **Progressive loading** — `ProgressiveFileReader` starts with a quick head+tail sample for instant line estimates, then builds the full index in a background thread via crossbeam-channel. +- **mmap with TOCTOU mitigation** — post-mmap stat check to detect file changes. `read_cache` module exists as a future pread-based alternative to eliminate SIGBUS risk entirely. +- **Persistent cache** — line indexes are serialized to disk with xxh3 content hashing for invalidation. Atomic writes via temp files. +- **Visual height index** — prefix-sum over wrapped-line heights, enabling O(log n) scroll-to-line mapping for long wrapped lines. + +Planned but not yet built: filtering, bookmarks, sessions, search engine (stub modules exist). + +## Benchmarks + +Custom harness comparing mmap vs pread backends across 7 suites (startup, render, jump, memory, growth, rotation, concurrent). Uses wall-clock timing with `/proc/self/` RSS and page fault metrics. Results are written as markdown tables to `benchmark-report.md`. + +The benchmark binary includes ~75 unit tests for the reader backends and metrics collection.