Author SHA1 Message Date
dailz 6a2f8ecb66 fix(bench): resolve pre-existing clippy warnings in report.rs and mmap_reader.rs 2026-06-07 09:15:34 +08:00
dailz f6081b9fe9 fix(bench): propagate I/O errors in append_lines instead of silently defaulting to 0 (closes #45) 2026-06-07 09:13:37 +08:00
dailz 97a2c6a925 fix(bench): regenerate growable file each iteration in truncate safety benchmarks (closes #44) 2026-06-07 09:02:19 +08:00
dailz e6e0e2cc90 fix(bench): correct lines_read to actual successful reads in bench_scroll_rss
lines_read was incorrectly set to max_lines.min(total) (the loop upper bound)
instead of the actual number of successfully read lines. Now tracks lines_read
via get_line(i).is_some() counter and uses max_lines.min(total) as the correct
loop upper bound to handle empty file edge case.

Fixes #43
2026-06-07 08:50:20 +08:00
dailz ffaf462bae Merge fix/m23-single-frame-tail-overlap: [M23] small file single_frame_tail overlap fix 2026-06-07 08:32:58 +08:00
dailz a8dc067cd4 fix(bench): [M23] prevent single_frame_tail/head overlap for small files
- Extract shared FRAME_LINES constant into suites/mod.rs
- Add select_frame_positions() helper with 3*FRAME_LINES threshold
  to guarantee non-overlapping head/middle/tail ranges
- Guard bench_reverse_scan against total <= FRAME_LINES
- Add 9 boundary tests for position selection (0..1M lines)

Closes #42
2026-06-07 08:31:00 +08:00
dailz 502479677b fix(bench): warn when clear_file_cache fails instead of silently skipping cold benchmarks (closes #41) 2026-06-05 17:28:57 +08:00
dailz 5656b26d7b refactor(bench): unify line counting in get_file_info to use count_existing_lines
Reuse the existing count_existing_lines() (reader.lines().count())
instead of a manual read_until loop, eliminating duplicate line-counting
logic in data_gen.rs.

Closes #40
2026-06-05 17:04:46 +08:00
dailz a8b64e78bd fix(bench): stabilize report column/row ordering across input permutations (#39)
- variants: use direct tuple comparison instead of format! string, add sort() after dedup
- Memory section: sort rows by (test_name, backend, variant) before output
- Extra Metrics section: sort rows by (test_name, variant_label) before output
- add [lib] target to Cargo.toml to enable unit tests
- add regression test: same data in different input order produces identical report
2026-06-05 16:24:41 +08:00
dailz e945a357f7 fix(bench): warn on all reset_vm_hwm errors, not just PermissionDenied
Issue #38: warn_reset_hwm() silently swallowed non-permission I/O errors
from /proc/self/clear_refs (e.g. missing /proc, read-only procfs, kernel
incompatibility). This left users unaware that VmHWM reset failed and
memory peak data could be contaminated across suites.

Changes:
- runner.rs: all errors now produce a warning with specific failure reason;
  PermissionDenied retains 'try running as root' hint; AtomicBool warn-once
  prevents duplicate output across 7 suite runs
- main.rs: preflight check now uses warn_reset_hwm() instead of the vague
  can_reset_vm_hwm(), sharing the same warn-once mechanism
- metrics.rs: remove dead can_reset_vm_hwm() (no callers remaining)
- tests: add hwm_warned_flag_prevents_reentry and warn_reset_hwm_does_not_panic
2026-06-05 15:52:01 +08:00
dailz fb57584546 fix(bench): validate --suites names, reject unknown suites at CLI boundary
Introduce Suite enum (runner.rs) replacing stringly-typed suite matching.
BenchConfig.suites is now Option<Vec<Suite>>, making invalid states
unrepresentable. Unknown suite names produce a clear error listing all
valid values.

Fixes: #37
2026-06-05 15:20:24 +08:00
dailz 9baec5ab69 fix(bench): refresh PreadReader index periodically in scroll_during_append (closes #36)
The reader's line_index and file_size were frozen at open time.
After current_line exceeded the initial 150K lines, get_line_impl
returned None for all subsequent reads. With the background thread
appending ~10K lines/sec, ~40% of measured frame latencies were
actually the cost of a None return, not real I/O.

- Add PreadReaderCore::refresh_index(&mut self): seek to start,
  rebuild LineIndex, update file_size, invalidate read cache
- Add PreadReaderPlain::refresh_index forwarding method
- Add ReadCache::invalidate to force cache miss after reindex
- Rewrite bench_scroll_during_append: time-based refresh (250ms),
  only record latencies for successful reads, assert max_line > initial
- Add regression tests for refresh_index with appended lines
2026-06-05 14:40:32 +08:00
dailz 6dd87d2872 fix(bench): wrap file writes with BufWriter to reduce syscall overhead
Add BufWriter::with_capacity(64KB) to generate_test_file,
generate_growable_file, and append_lines in data_gen.rs.

Previously each writeln! triggered an individual write syscall,
making 5GB/74M-line benchmark data generation extremely slow.
BufWriter batches writes into 64KB chunks, reducing syscalls
by ~1000x.

Explicit flush()? + drop before subsequent reads ensures data
visibility and propagates flush errors (BufWriter::drop swallows
them).

Closes #35
2026-06-05 14:01:35 +08:00
dailz 83f633a562 fix(bench): make can_reset_vm_hwm side-effect-free with open probe (closes #34) 2026-06-05 13:34:01 +08:00
dailz dad5f5a635 fix(bench): eliminate SIGBUS handler static mut UB with Once + raw atomics (closes #33)
Replace `static mut OLD_SIGBUS_HANDLER` with AtomicU8 + AtomicPtr to
remove data race UB when concurrent benchmarks call open() from multiple
threads.

Key changes:
- Use `Once::call_once` to guarantee single handler installation
- Publish old handler to atomics BEFORE installing new handler (closes
  the handler-active-but-state-unpublished race window)
- Read atomics with Acquire in signal handler (async-signal-safe)
- Align si_addr to page boundary before mmap(MAP_FIXED)
- Add concurrent test: 8 threads open all 5 variants simultaneously
2026-06-05 13:22:02 +08:00
dailz 534a089b58 fix(tui): defer file-change events during Loading state to prevent stale reader (closes #10)
Loading state silently dropped FileEvent::Appended/Truncated via _ => {}.
After Loading→Ready transition the FileReader was based on a stale snapshot.

- Add reload_after_loading flag to defer reload until Ready state
- Extract reload_ready_reader() from handle_file_truncated
- Explicit 3-branch match: Ready handles, Loading sets flag, rest ignores
- Clear flag on IndexerMessage::Error to prevent stale dirty bit
- 4 regression tests covering append/truncate/collapse/error paths
2026-06-04 17:32:58 +08:00
dailz b7938e069d fix(tui): RAII TerminalGuard prevents terminal corruption on error exit (closes #8)
The previous code called std::process::exit(1) on file load failure,
bypassing all terminal restoration (disable_raw_mode, LeaveAlternateScreen,
show_cursor). This left the user's shell in a broken state with no echo
and no visible cursor.

Introduce TerminalGuard with Drop-based cleanup that fires on every exit
path: normal return, ? error propagation, and panic unwind. The guard
also handles partial initialization rollback (e.g. raw mode enabled but
alternate screen fails). Remove all process::exit calls from the guarded
scope.
2026-06-04 16:11:16 +08:00
dailz d40d70c600 fix(watcher): use try_send to prevent blocking notify callback
Replace tx.send() with tx.try_send() in file watcher notify callback
to avoid blocking the notify thread when the bounded channel (cap 100)
is full. Events are silently dropped instead of blocking, preventing
shutdown delays and event loss during consumer stalls.

Closes #7
2026-06-04 14:33:40 +08:00
dailz 1350f659fa fix(io): eliminate SIGBUS risk in background indexer threads (closes #6)
Background threads (spawn_indexer, spawn_visual_height_rebuild) previously
held mmap during entire file scan, risking SIGBUS if file was truncated
externally. Now uses BufReader streaming scan with mmap created only
after scan completes, plus stat validation.

Changes:
- spawn_indexer: replace mmap scan with BufReader fill_buf/consume loop,
  create mmap post-scan with fd stat validation
- spawn_visual_height_rebuild: replace mmap/FileReader with sequential
  BufReader scan, discard results on line count mismatch
- FileReader::open/reload/update_for_append: add stat-after-mmap check
- LineIndex: make fields pub(crate) for direct construction from scan loop
- Add 3 regression tests for truncation scenarios
2026-06-04 14:10:51 +08:00
dailz 1bb6b2e9f3 fix: eliminate TOCTOU race in IndexCache::save (closes #5)
spawn_indexer builds LineIndex from mmap snapshot but IndexCache::save()
re-opened the file to compute the hash. If the file changed between those
two steps, the cached index would be stored under the wrong hash.

- Add IndexCache::save_with_hash() that computes hash from in-memory data
- Add compute_data_hash() public function (same algorithm as compute_file_hash)
- Update spawn_indexer, FileReader::save_cache, and test callers
2026-06-04 13:30:16 +08:00
dailz bef0b44e91 fix(io): guard sampled_offsets index in get_line() to prevent panic on corrupt cache
Replace direct indexing with .get()? to return None instead of panicking
when sampled_offsets is shorter than expected (e.g. corrupt bincode cache).

Closes #3
2026-06-03 15:39:05 +08:00
dailz 24fe97a457 fix(io): eliminate TOCTOU race in FileReader open/reload
Replace LineIndex::from_reader(BufReader) with LineIndex::from_bytes(&mmap)
in both open() and reload(), ensuring the line index is always built from
the same mmap snapshot rather than a separate read through the file descriptor.

This closes the race window where an external file modification between
mmap() and from_reader() could cause line_index offsets to disagree with
the mmap data, leading to get_line() returning wrong content or panicking.

Closes #2
2026-06-03 15:08:22 +08:00
dailz b6e655bff6 fix(io): handle file shrink in update_for_append to prevent SIGBUS
File truncation/rotation during mmap lifetime caused SIGBUS crash
because update_for_append() ignored new_size < old_size, leaving
a stale mapping that would fault on access.

Introduce AppendStatus enum (Unchanged/Appended/Reloaded) so the
caller can distinguish shrink events from normal appends. On shrink,
reload() rebuilds the mmap and line index. The TUI layer clamps
cursor and invalidates viewport cache on Reloaded, matching the
existing handle_file_truncated() behavior.

Fixes #1
2026-06-03 14:47:23 +08:00
dailz b3256b2917 fix: audit fixes for 4 medium-severity bugs: index_cache tail hash, read_cache doc, mutex poison recovery, Remove event handling 2026-05-10 17:03:07 +08:00
dailz fb23e4c7cb fix(tui): smooth visual-row scrolling during Loading state
During Loading state (before VHI is built), j/k used to jump by logical
line, visually skipping multiple wrapped rows. Now uses v_sub_offset to
track position within a wrapped line, enabling smooth 1-visual-row scroll.

- Add v_sub_offset field to App for sub-line visual position tracking
- scroll_down/up_line else branch: advance v_sub_offset, wrap to next line
- ensure_viewport_cache Loading path: pass v_sub_offset as offset_in_line
- ensure_cursor_visible: skip during Loading (scroll functions manage it)
- Reset v_sub_offset on Loading→Ready, scroll_to_top, scroll_to_bottom
- Add 3 tests for Loading-state sub-offset scrolling behavior
2026-04-24 19:04:30 +08:00
dailz 8c5a838db0 fix(tui): j/k scroll by visual row instead of logical line
When a JSON line wraps to many visual rows (e.g. 73 rows), pressing j
would skip the entire logical line, making wrapped content unreadable.
Now j/k scroll by 1 visual row when a VisualHeightIndex is available,
with cursor tracking the viewport center. Falls back to logical-line
scroll in Loading/no-index modes or when all content fits the viewport.

Adds 4 tests: visual scroll down, visual scroll up, small-file fallback,
j/k roundtrip.
2026-04-24 07:41:30 +08:00
dailz 81cc72bd84 test(tui): add Loading + JSON expansion unit tests 2026-04-14 17:38:35 +08:00
dailz 06b2a39816 fix(tui): post-check cursor visibility in viewport during Loading 2026-04-14 17:09:50 +08:00
dailz 0941092b07 feat(tui): enable JSON expansion during Loading state 2026-04-14 16:52:17 +08:00
dailzandSisyphus 4ec3eb7cee fix(core): cap incremental scan in get_line() to prevent O(N) blocking
Add SCAN_AHEAD_LIMIT (10000 lines) to get_line() in Sampling state. Without this, jumping to end-of-file (G) during progressive loading would scan the entire file byte-by-byte on the main thread, blocking the UI and consuming excessive memory. Lines beyond the scanned region + limit now return None, which the TUI renders as empty.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-14 09:58:20 +08:00
dailzandSisyphus 05491304bf fix(tui): allow free scrolling during progressive loading
total_lines() was returning sampled_line_count() (only ~300 lines from the initial 64KB scan) during the Loading state, capping the scroll range. Use estimated_lines instead so the user can scroll to any position while indexing runs in the background. get_line() already supports incremental forward scanning on demand.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-14 09:36:03 +08:00
dailzandSisyphus a03af7e74e feat(core): implement FileWatcher for live file tailing
Complete FileWatcher implementation using notify 8.x crate with get_inode() for cross-platform file identity. Support append detection for incremental index updates and truncate detection for full reloads.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-14 09:07:27 +08:00
dailzandSisyphus bab3d9078a feat(tui): replace O(N) scanning with progressive loading and VisualHeightIndex
Replace synchronous file loading with AppLoadingState state machine (Empty/Loading/Ready/Error) for instant interactivity. Add ViewportCache for on-demand viewport computation, replacing global wrap/level caches. Integrate background indexer polling and file watcher events into the TUI event loop. Add loading UI with progress percentage, estimated line numbers with ~ prefix, and error state display. Eliminate all O(N) linear scans using VisualHeightIndex binary search.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
2026-04-14 09:07:18 +08:00
29 changed files with 7212 additions and 390 deletions
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "log-viewer-bench"
version = "0.1.0"
edition = "2024"
[lib]
name = "log_viewer_bench"
path = "src/lib.rs"
[[bin]]
name = "log-viewer-bench"
path = "src/main.rs"
[dependencies]
memmap2 = "0.9"
nix = { version = "0.30", features = ["signal", "resource", "mman", "fs"] }
libc = "0.2"
memchr = "2"
serde_json = "1"
clap = { version = "4", features = ["derive"] }
crossbeam-channel = "0.5"
tempfile = "3"
+223
View File
@@ -0,0 +1,223 @@
use std::fs;
use std::io::{BufRead, BufReader, BufWriter, Write};
use std::path::{Path, PathBuf};
pub struct TestFileInfo {
pub path: PathBuf,
pub size_bytes: u64,
pub line_count: u64,
pub avg_line_length: f64,
}
/// Check if test file exists and return its info, or generate it
pub fn ensure_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
if path.exists() {
return get_file_info(path);
}
generate_test_file(path)
}
/// Get info about an existing test file
fn get_file_info(path: &Path) -> std::io::Result<TestFileInfo> {
let metadata = fs::metadata(path)?;
let size_bytes = metadata.len();
let line_count = count_existing_lines(path)?;
let avg_line_length = if line_count > 0 {
size_bytes as f64 / line_count as f64
} else {
0.0
};
Ok(TestFileInfo {
path: path.to_path_buf(),
size_bytes,
line_count,
avg_line_length,
})
}
/// Generate a large test file (~5GB / ~74M lines) if it doesn't exist
fn generate_test_file(path: &Path) -> std::io::Result<TestFileInfo> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(path)?);
let target_lines: u64 = 74_000_000;
for i in 0..target_lines {
writeln!(
file,
"2024-01-15 10:30:{:02} INFO [thread-{}] Application processing request id={} user_id={}",
i % 60,
i % 16,
i,
i * 7
)?;
}
file.flush()?;
drop(file);
get_file_info(path)
}
/// Generate a smaller file (~10MB / ~150K lines) for growth/rotation tests
pub fn generate_growable_file(dir: &Path) -> std::io::Result<PathBuf> {
fs::create_dir_all(dir)?;
let path = dir.join("growable.log");
let mut file = BufWriter::with_capacity(64 * 1024, fs::File::create(&path)?);
for i in 0..150_000u64 {
writeln!(
file,
"2024-01-15 10:30:{:02} INFO [thread-{}] Appending test line {}",
i % 60,
i % 16,
i
)?;
}
file.flush()?;
Ok(path)
}
/// Append `count` lines to the file
pub fn append_lines(path: &Path, count: usize) -> std::io::Result<()> {
let existing_lines = count_existing_lines(path)?;
let mut file = BufWriter::with_capacity(
64 * 1024,
fs::OpenOptions::new().append(true).open(path)?,
);
for i in 0..count {
writeln!(
file,
"2024-01-15 10:30:00 INFO Appended line {}",
existing_lines + i as u64
)?;
}
file.flush()?;
Ok(())
}
/// Truncate file to specified size
pub fn truncate_file(path: &Path, size: u64) -> std::io::Result<()> {
let file = fs::OpenOptions::new().write(true).open(path)?;
file.set_len(size)
}
/// Rotate file: rename existing file, create new empty file
pub fn rotate_file(path: &Path) -> std::io::Result<PathBuf> {
let rotated = path.with_extension("log.1");
fs::rename(path, &rotated)?;
fs::File::create(path)?;
Ok(rotated)
}
/// Count lines in a file (helper)
fn count_existing_lines(path: &Path) -> std::io::Result<u64> {
let file = fs::File::open(path)?;
let reader = BufReader::new(file);
let mut count = 0u64;
for line in reader.lines() {
line?;
count += 1;
}
Ok(count)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_generate_growable_file_creates_approximately_correct_size() {
let dir = tempfile::tempdir().unwrap();
let path = generate_growable_file(dir.path()).unwrap();
assert!(path.exists());
let metadata = fs::metadata(&path).unwrap();
let size_mb = metadata.len() as f64 / (1024.0 * 1024.0);
// ~150K lines × ~67 bytes ≈ ~10MB; allow 5MB15MB range
assert!(
(5.0..=15.0).contains(&size_mb),
"Expected ~10MB, got {size_mb:.1}MB"
);
}
#[test]
fn test_append_lines_increases_line_count() {
let dir = tempfile::tempdir().unwrap();
let path = {
let mut f = fs::File::create(dir.path().join("test.log")).unwrap();
for i in 0..10u64 {
writeln!(f, "line {i}").unwrap();
}
dir.path().join("test.log")
};
let before = count_existing_lines(&path).unwrap();
append_lines(&path, 5).unwrap();
let after = count_existing_lines(&path).unwrap();
assert_eq!(after, before + 5);
}
#[test]
fn test_truncate_file_reduces_size() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("trunc.log");
{
let mut f = fs::File::create(&path).unwrap();
write!(f, "{}", "A".repeat(1024)).unwrap();
}
let before = fs::metadata(&path).unwrap().len();
assert_eq!(before, 1024);
truncate_file(&path, 512).unwrap();
let after = fs::metadata(&path).unwrap().len();
assert_eq!(after, 512);
}
#[test]
fn test_rotate_file_renames_and_creates_empty() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("rotate.log");
{
let mut f = fs::File::create(&path).unwrap();
write!(f, "original content").unwrap();
}
let rotated = rotate_file(&path).unwrap();
// Rotated file has the old content
assert!(rotated.exists());
assert_eq!(fs::read_to_string(&rotated).unwrap(), "original content");
// New file is empty
assert!(path.exists());
assert_eq!(fs::metadata(&path).unwrap().len(), 0);
}
#[test]
fn test_ensure_test_file_generates_when_missing() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("fresh.log");
assert!(!path.exists());
// Override the generator to use a small file for test speed:
// We'll test ensure_test_file indirectly by checking it calls generate_test_file.
// Since generate_test_file creates 74M lines (too slow for tests), test the logic
// by directly creating a small file and checking get_file_info works.
{
let mut f = fs::File::create(&path).unwrap();
for i in 0..100u64 {
writeln!(f, "2024-01-15 10:30:00 INFO line {i}").unwrap();
}
}
let info = ensure_test_file(&path).unwrap();
assert_eq!(info.line_count, 100);
assert!(info.size_bytes > 0);
assert!(info.avg_line_length > 0.0);
}
}
+24
View File
@@ -0,0 +1,24 @@
pub mod data_gen;
pub mod line_index;
pub mod metrics;
pub mod mmap_reader;
pub mod pread_reader;
pub mod report;
pub mod runner;
pub mod suites;
pub mod types;
use std::path::Path;
/// A single reader backend (mmap or pread)
pub trait FileReaderBackend {
fn name(&self) -> &str;
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized;
fn file_size(&self) -> u64;
fn total_lines(&self) -> usize;
fn get_line(&self, idx: usize) -> Option<String>;
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>>;
fn close(self);
}
+110
View File
@@ -0,0 +1,110 @@
// ─── line_index.rs ───────────────────────────────────────────────────────────
// Vendored from crates/core/src/io/line_index.rs
// Sparse line index: sample every 256 lines to reduce memory usage.
// ──────────────────────────────────────────────────────────────────────────────
const BLOCK_SIZE: usize = 256;
pub struct LineIndex {
pub(crate) sampled_offsets: Vec<u64>,
pub(crate) total_lines: u64,
#[allow(dead_code)]
pub(crate) has_trailing_newline: bool,
}
impl LineIndex {
/// Build sparse line index from a streaming reader.
/// Uses fill_buf()/consume() to avoid loading the entire file into memory.
/// RSS stays at ~64KB (BufReader buffer size), independent of file size.
pub fn from_reader(reader: &mut impl std::io::BufRead) -> std::io::Result<Self> {
let mut sampled_offsets: Vec<u64> = vec![0]; // line 0 starts at offset 0
let mut next_line_idx: usize = 1;
let mut newline_count: usize = 0;
let mut chunk_offset: u64 = 0;
let mut last_byte: Option<u8> = None;
loop {
let buf = reader.fill_buf()?;
if buf.is_empty() {
break;
}
if let Some(&b) = buf.last() {
last_byte = Some(b);
}
for pos in memchr::memchr_iter(b'\n', buf) {
newline_count += 1;
if next_line_idx.is_multiple_of(BLOCK_SIZE) {
sampled_offsets.push(chunk_offset + pos as u64 + 1);
}
next_line_idx += 1;
}
let consumed = buf.len();
chunk_offset += consumed as u64;
reader.consume(consumed);
}
// Empty file: no data at all
if chunk_offset == 0 {
return Ok(LineIndex {
sampled_offsets: vec![],
total_lines: 0,
has_trailing_newline: false,
});
}
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
let total_lines: u64 = if has_trailing_newline && newline_count > 0 {
newline_count as u64
} else {
(1 + newline_count) as u64
};
// Trailing \n pop logic
if has_trailing_newline && newline_count > 0 {
let trailing_line_idx = newline_count;
if trailing_line_idx.is_multiple_of(BLOCK_SIZE) {
sampled_offsets.pop();
}
}
Ok(LineIndex {
sampled_offsets,
total_lines,
has_trailing_newline,
})
}
/// Return total line count.
pub fn line_count(&self) -> usize {
self.total_lines as usize
}
/// Retrieve the content of line `idx` from the given data slice.
/// Uses sparse index to locate the block start, then scans forward
/// a small number of newlines to find the target line.
pub fn get_line<'a>(&self, data: &'a [u8], idx: usize) -> Option<&'a str> {
if idx >= self.total_lines as usize || data.is_empty() {
return None;
}
let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE;
let mut pos = self.sampled_offsets[block] as usize;
for _ in 0..offset_in_block {
match memchr::memchr(b'\n', &data[pos..]) {
Some(rel) => pos = pos + rel + 1,
None => return None,
}
}
let end = memchr::memchr(b'\n', &data[pos..])
.map(|rel| pos + rel)
.unwrap_or(data.len());
let line_bytes = &data[pos..end];
std::str::from_utf8(line_bytes)
.map(|s| s.trim_end_matches(['\r', '\n']))
.ok()
}
}
+81
View File
@@ -0,0 +1,81 @@
use clap::Parser;
use std::path::PathBuf;
/// Benchmark: mmap vs pread for large file reading
#[derive(Parser, Debug)]
#[command(name = "log-viewer-bench", version, about = "Benchmark mmap vs pread")]
struct Args {
/// Path to the test file (default: /tmp/test-logviewer/extreme.log)
#[arg(default_value = "/tmp/test-logviewer/extreme.log")]
test_file: PathBuf,
/// Quick mode: use smaller iterations and skip cold cache tests
#[arg(long)]
quick: bool,
/// Output report path (default: benchmark-report.md)
#[arg(long, default_value = "benchmark-report.md")]
output: PathBuf,
/// Only run specified suites (comma-separated: startup,render,jump,memory,growth,rotation,concurrent)
#[arg(long, value_delimiter = ',')]
suites: Option<Vec<String>>,
}
fn main() {
let args = Args::parse();
let suites = match args.suites {
Some(names) => {
let parsed: Result<Vec<_>, _> =
names.iter().map(|s| s.parse::<log_viewer_bench::runner::Suite>()).collect();
match parsed {
Ok(s) => Some(s),
Err(e) => {
eprintln!("error: {e}");
std::process::exit(1);
}
}
}
None => None,
};
println!("=== Benchmark: mmap vs pread ===");
println!("Test file: {}", args.test_file.display());
println!("Quick mode: {}", args.quick);
println!();
let config = log_viewer_bench::runner::BenchConfig {
test_file: args.test_file.clone(),
quick_mode: args.quick,
suites,
};
if !config.test_file.exists() {
eprintln!("ERROR: Test file not found: {}", config.test_file.display());
eprintln!(
"Generate one with: dd if=/dev/urandom of=/tmp/test-logviewer/extreme.log bs=1M count=5000"
);
std::process::exit(1);
}
log_viewer_bench::runner::warn_reset_hwm();
println!("Running benchmarks...");
let results = log_viewer_bench::runner::run_all(&config);
println!("Completed {} benchmarks.\n", results.len());
let report = log_viewer_bench::report::format_report(&results);
println!("{}", report);
if let Err(e) = std::fs::write(&args.output, &report) {
eprintln!(
"WARNING: Failed to save report to {}: {}",
args.output.display(),
e
);
} else {
eprintln!("Report saved to {}", args.output.display());
}
}
+275
View File
@@ -0,0 +1,275 @@
use std::fs::{self, File};
use std::os::unix::fs::MetadataExt;
use std::os::unix::io::AsRawFd;
use std::path::Path;
pub struct RssMetrics {
pub vm_rss_kb: u64,
pub vm_hwm_kb: u64,
}
pub struct PageFaultMetrics {
pub minor_faults: u64,
pub major_faults: u64,
}
pub struct MetricsCollector;
impl MetricsCollector {
/// Read VmRSS and VmHWM from /proc/self/status
pub fn read_rss() -> RssMetrics {
let status = fs::read_to_string("/proc/self/status").unwrap_or_default();
let mut vm_rss_kb: u64 = 0;
let mut vm_hwm_kb: u64 = 0;
for line in status.lines() {
if line.starts_with("VmRSS:") {
vm_rss_kb = parse_kb_value(line);
} else if line.starts_with("VmHWM:") {
vm_hwm_kb = parse_kb_value(line);
}
}
RssMetrics {
vm_rss_kb,
vm_hwm_kb,
}
}
/// Read page fault counts from getrusage
pub fn read_page_faults() -> PageFaultMetrics {
let usage =
nix::sys::resource::getrusage(nix::sys::resource::UsageWho::RUSAGE_SELF).unwrap();
PageFaultMetrics {
// getrusage() returns c_long (i64 on 64-bit Linux) — explicit as u64 conversion
minor_faults: usage.minor_page_faults() as u64,
major_faults: usage.major_page_faults() as u64,
}
}
/// Clear page cache (requires root: sync + drop_caches)
/// Falls back to doing nothing if no permission
pub fn clear_page_cache() -> std::io::Result<()> {
let _ = std::process::Command::new("sync").status();
fs::write("/proc/sys/vm/drop_caches", "1")
}
/// Clear file cache using posix_fadvise(DONTNEED) — no root required
pub fn clear_file_cache(path: &Path) -> std::io::Result<()> {
let file = File::open(path)?;
let len = file.metadata()?.len();
let ret = unsafe {
libc::posix_fadvise(file.as_raw_fd(), 0, len as i64, libc::POSIX_FADV_DONTNEED)
};
// posix_fadvise returns error code directly (not errno), 0 = success
if ret != 0 {
return Err(std::io::Error::from_raw_os_error(ret));
}
Ok(())
}
/// Reset VmHWM by writing to /proc/self/clear_refs (requires root)
pub fn reset_vm_hwm() -> std::io::Result<()> {
fs::write("/proc/self/clear_refs", "5").map_err(|e| {
if e.kind() == std::io::ErrorKind::PermissionDenied {
std::io::Error::new(
std::io::ErrorKind::PermissionDenied,
"VmHWM reset requires root (can't write /proc/self/clear_refs)",
)
} else {
e
}
})
}
/// Get file inode number
pub fn get_inode(path: &Path) -> std::io::Result<u64> {
let meta = fs::metadata(path)?;
Ok(meta.ino())
}
/// Check if file was rotated (inode changed)
pub fn detect_rotation(original_inode: u64, path: &Path) -> bool {
Self::get_inode(path)
.map(|ino| ino != original_inode)
.unwrap_or(true)
}
}
fn parse_kb_value(line: &str) -> u64 {
// Format: "VmRSS: 12345 kB"
line.split_whitespace()
.nth(1)
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(0)
}
pub fn mean(data: &[u64]) -> f64 {
if data.is_empty() {
return 0.0;
}
data.iter().sum::<u64>() as f64 / data.len() as f64
}
/// Percentile of data at given fraction (0.01.0). Returns from a sorted copy.
pub fn percentile(data: &[u64], p: f64) -> u64 {
if data.is_empty() {
return 0;
}
let mut sorted: Vec<u64> = data.to_vec();
sorted.sort_unstable();
let idx = ((p * (sorted.len() - 1) as f64).round()) as usize;
sorted[idx.min(sorted.len() - 1)]
}
pub fn stdev(data: &[u64]) -> f64 {
if data.len() < 2 {
return 0.0;
}
let m = mean(data);
let variance: f64 = data
.iter()
.map(|&v| {
let d = v as f64 - m;
d * d
})
.sum::<f64>()
/ (data.len() - 1) as f64;
variance.sqrt()
}
pub fn p50(data: &[u64]) -> u64 {
percentile(data, 0.50)
}
pub fn p95(data: &[u64]) -> u64 {
percentile(data, 0.95)
}
pub fn p99(data: &[u64]) -> u64 {
percentile(data, 0.99)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_rss_returns_values() {
let rss = MetricsCollector::read_rss();
assert!(
rss.vm_rss_kb > 0,
"VmRSS should be non-zero for a running process"
);
assert!(
rss.vm_hwm_kb > 0,
"VmHWM should be non-zero for a running process"
);
}
#[test]
fn test_page_faults_returns_values() {
let faults = MetricsCollector::read_page_faults();
assert!(
faults.minor_faults > 0,
"Should have some minor page faults"
);
}
#[test]
fn test_mean() {
let data = vec![100, 200, 300, 400, 500];
let result = mean(&data);
assert!(
(result - 300.0).abs() < f64::EPSILON,
"mean should be 300.0, got {result}"
);
}
#[test]
fn test_mean_empty() {
assert_eq!(mean(&[]), 0.0);
}
#[test]
fn test_percentile_p50() {
let data = vec![100, 200, 300, 400, 500];
assert_eq!(percentile(&data, 0.50), 300);
}
#[test]
fn test_percentile_p99() {
let data = vec![10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
let p99_result = percentile(&data, 0.99);
assert!(p99_result >= 90, "P99 should be near max, got {p99_result}");
}
#[test]
fn test_percentile_empty() {
assert_eq!(percentile(&[], 0.5), 0);
}
#[test]
fn test_stdev() {
let data = vec![100, 200, 300, 400, 500];
let s = stdev(&data);
assert!(s > 100.0, "stdev should be significant, got {s}");
assert!(s < 200.0, "stdev should be < 200, got {s}");
}
#[test]
fn test_stdev_single() {
assert_eq!(stdev(&[42]), 0.0);
assert_eq!(stdev(&[]), 0.0);
}
#[test]
fn test_parse_kb_value() {
assert_eq!(parse_kb_value("VmRSS: 12345 kB"), 12345);
assert_eq!(parse_kb_value("VmHWM:\t2048 kB"), 2048);
assert_eq!(parse_kb_value("VmRSS: 0 kB"), 0);
}
#[test]
fn test_parse_kb_value_malformed() {
assert_eq!(parse_kb_value("VmRSS: NaN kB"), 0);
assert_eq!(parse_kb_value("garbage"), 0);
}
#[test]
fn test_convenience_percentiles() {
let data = vec![10, 20, 30, 40, 50];
assert_eq!(p50(&data), 30);
assert_eq!(p95(&data), 50);
assert_eq!(p99(&data), 50);
}
#[test]
fn test_inode_for_existing_file() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
assert!(inode > 0, "inode should be non-zero");
}
#[test]
fn test_detect_rotation_no_rotation() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let inode = MetricsCollector::get_inode(tmp.path()).unwrap();
assert!(!MetricsCollector::detect_rotation(inode, tmp.path()));
}
#[test]
fn test_detect_rotation_file_removed() {
let inode: u64 = 99999;
let result = MetricsCollector::detect_rotation(inode, Path::new("/no/such/file"));
assert!(result, "missing file should indicate rotation");
}
#[test]
fn test_clear_file_cache() {
let tmp = tempfile::NamedTempFile::new().unwrap();
let result = MetricsCollector::clear_file_cache(tmp.path());
assert!(
result.is_ok(),
"clear_file_cache should succeed on temp file: {result:?}"
);
}
}
+752
View File
@@ -0,0 +1,752 @@
// ─── mmap_reader.rs ──────────────────────────────────────────────────────────
// mmap-based FileReaderBackend implementations with 5 variants for benchmarking.
// Includes SIGBUS handler for file-truncation resilience and remap support
// for growing-file scenarios.
// ──────────────────────────────────────────────────────────────────────────────
use std::fs::File;
use std::io::BufReader;
use std::path::Path;
use std::sync::atomic::{AtomicBool, AtomicPtr, AtomicU8, Ordering};
use std::sync::Once;
use memmap2::{Advice, Mmap, MmapOptions, RemapOptions};
use nix::sys::signal::{sigaction, SaFlags, SigAction, SigHandler, SigSet, Signal};
use crate::line_index::LineIndex;
use crate::FileReaderBackend;
// ─── SIGBUS Handler ──────────────────────────────────────────────────────────
//
// Signal-safety architecture:
// - Old handler state is stored in raw atomics (AtomicU8 + AtomicPtr) that are
// async-signal-safe to read — no OnceLock, Mutex, or other non-trivial abstractions
// in the signal handler path.
// - Installation uses `Once` for idempotency and follows a strict publish-then-install
// sequence to close the "handler-active-but-state-unpublished" race window.
/// Global flag set by the SIGBUS handler when a bus error is intercepted.
/// Process-global: concurrent benchmarks that reset/check this flag may interfere.
/// Currently acceptable because only `rotation` suite (sequential) uses it.
static SIGBUS_OCCURRED: AtomicBool = AtomicBool::new(false);
/// Discriminant values for old SIGBUS handler type.
const HANDLER_NONE: u8 = 0;
const HANDLER_DEFAULT: u8 = 1;
const HANDLER_IGNORE: u8 = 2;
const HANDLER_PLAIN: u8 = 3; // extern "C" fn(c_int)
#[allow(clippy::unseparated_literal_suffix, reason = "clarity: this is the SA_SIGACTION variant")]
const HANDLER_SIGACTION: u8 = 4; // extern "C" fn(c_int, *mut siginfo_t, *mut c_void)
/// Old SIGBUS handler type — raw atomic, async-signal-safe to read.
static OLD_HANDLER_KIND: AtomicU8 = AtomicU8::new(HANDLER_NONE);
/// Old SIGBUS handler function pointer — raw atomic, async-signal-safe to read.
static OLD_HANDLER_PTR: AtomicPtr<std::ffi::c_void> = AtomicPtr::new(std::ptr::null_mut());
/// Ensures `install_sigbus_handler` runs exactly once across all threads.
static INSTALL_ONCE: Once = Once::new();
/// Returns `true` if a SIGBUS was intercepted since the last reset.
pub fn sigbus_flag() -> bool {
SIGBUS_OCCURRED.load(Ordering::SeqCst)
}
/// Resets the SIGBUS flag. Call before operations where you want to detect
/// a fresh SIGBUS.
pub fn reset_sigbus_flag() {
SIGBUS_OCCURRED.store(false, Ordering::SeqCst)
}
/// SIGBUS signal handler.
///
/// # Safety Constraints (signal handler context)
/// - NO TLS access
/// - NO memory allocation
/// - NO lock acquisition
/// - Only: AtomicBool store → mmap → raw atomic loads → chain
extern "C" fn sigbus_handler(
sig: libc::c_int,
info: *mut libc::siginfo_t,
ctx: *mut std::ffi::c_void,
) {
SIGBUS_OCCURRED.store(true, Ordering::SeqCst);
// Align fault address down to page boundary.
// si_addr is NOT guaranteed to be page-aligned; without this, mmap(MAP_FIXED)
// could replace the wrong page.
let addr = unsafe { (*info).si_addr() };
let aligned = (addr as usize & !0xFFF) as *mut libc::c_void;
// Map anonymous zero page at fault address to prevent crash on file truncation.
let result = unsafe {
libc::mmap(
aligned,
4096,
libc::PROT_READ,
libc::MAP_PRIVATE | libc::MAP_ANONYMOUS | libc::MAP_FIXED,
-1,
0,
)
};
if result == libc::MAP_FAILED {
// Chain to old handler via raw atomics (async-signal-safe).
let kind = OLD_HANDLER_KIND.load(Ordering::Acquire);
match kind {
HANDLER_PLAIN => {
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
if !ptr.is_null() {
// SAFETY: ptr was derived from a valid function pointer
// published by install_sigbus_handler before this handler was installed.
let f: extern "C" fn(libc::c_int) = unsafe { std::mem::transmute(ptr) };
f(sig);
} else {
unsafe { libc::_exit(128 + sig) };
}
}
HANDLER_SIGACTION => {
let ptr = OLD_HANDLER_PTR.load(Ordering::Acquire);
if !ptr.is_null() {
let f: extern "C" fn(
libc::c_int,
*mut libc::siginfo_t,
*mut std::ffi::c_void,
) = unsafe { std::mem::transmute(ptr) };
f(sig, info, ctx);
} else {
unsafe { libc::_exit(128 + sig) };
}
}
_ => {
// HANDLER_NONE / HANDLER_DEFAULT / HANDLER_IGNORE — no safe chaining.
unsafe { libc::_exit(128 + sig) };
}
}
}
}
/// Install the SIGBUS handler via sigaction(). Chains to any previous handler.
/// Uses `Once` to guarantee exactly-once installation — safe to call from
/// multiple threads concurrently.
///
/// The install sequence closes the "install-then-publish" race by:
/// 1. Querying the current handler via raw `libc::sigaction` (no modification).
/// 2. Publishing old handler state to raw atomics (`Release` stores).
/// 3. Installing our handler via `nix::sigaction`.
///
/// This ensures the atomics are readable *before* our handler can fire.
fn install_sigbus_handler() {
INSTALL_ONCE.call_once(|| {
// Step 1: Query current SIGBUS disposition without changing it.
let mut old_act: libc::sigaction = unsafe { std::mem::zeroed() };
let ret = unsafe { libc::sigaction(libc::SIGBUS, std::ptr::null(), &mut old_act) };
if ret != 0 {
// Failed to query — skip installation entirely.
return;
}
// Step 2: Publish old handler to atomics BEFORE installing ours.
let is_siginfo = (old_act.sa_flags & libc::SA_SIGINFO) != 0;
let raw_usize = old_act.sa_sigaction as usize;
if raw_usize == 0 {
// SIG_DFL (null function pointer → default disposition)
OLD_HANDLER_KIND.store(HANDLER_DEFAULT, Ordering::Release);
} else if raw_usize == 1 {
// SIG_IGN (sentinel value 1 → ignore disposition)
OLD_HANDLER_KIND.store(HANDLER_IGNORE, Ordering::Release);
} else {
// Real handler function pointer
OLD_HANDLER_PTR.store(raw_usize as *mut std::ffi::c_void, Ordering::Release);
if is_siginfo {
OLD_HANDLER_KIND.store(HANDLER_SIGACTION, Ordering::Release);
} else {
OLD_HANDLER_KIND.store(HANDLER_PLAIN, Ordering::Release);
}
}
// Step 3: Install our handler. Atomics are already published,
// so the handler can safely read them from the moment it's installed.
let new_action = SigAction::new(
SigHandler::SigAction(sigbus_handler),
SaFlags::SA_SIGINFO,
SigSet::empty(),
);
let _ = unsafe { sigaction(Signal::SIGBUS, &new_action) };
});
}
// ─── Core MmapReader ─────────────────────────────────────────────────────────
/// Core mmap-based reader. Holds the memory mapping, file handle, and
/// line index. Used as the engine inside each variant wrapper.
pub struct MmapReader {
mmap: Mmap,
#[allow(dead_code)]
file: File,
line_index: LineIndex,
file_size: u64,
}
impl MmapReader {
/// Open a file and create an mmap. Does NOT apply madvise.
/// The caller is responsible for applying the desired advice.
fn open_raw(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let mmap = unsafe { Mmap::map(&file)? };
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
mmap,
file,
line_index,
file_size,
})
}
/// Open with MmapOptions (for populate, etc.).
fn open_with_options(path: &Path, opts: &MmapOptions) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let mmap = if file_size == 0 {
unsafe { Mmap::map(&file)? }
} else {
unsafe { opts.map(&file)? }
};
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
mmap,
file,
line_index,
file_size,
})
}
/// Apply madvise to the entire mapping.
fn advise(&self, advice: Advice) {
let _ = self.mmap.advise(advice);
}
/// Grow the mmap to `new_size` bytes.
///
/// # Safety
/// Caller must ensure no `&[u8]` references to the old mmap data exist.
/// After remap, any pointers derived from the old mapping may be invalid
/// (if the kernel moved the mapping).
pub unsafe fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
unsafe {
self.mmap
.remap(new_size, RemapOptions::new().may_move(true))?;
}
self.file_size = new_size as u64;
Ok(())
}
#[inline]
pub fn file_size(&self) -> u64 {
self.file_size
}
#[inline]
pub fn total_lines(&self) -> usize {
self.line_index.line_count()
}
#[inline]
pub fn get_line(&self, idx: usize) -> Option<String> {
self.line_index
.get_line(&self.mmap, idx)
.map(|s| s.to_owned())
}
#[inline]
pub fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
let start = offset as usize;
let end = start.checked_add(len)?;
if end > self.mmap.len() {
return None;
}
Some(self.mmap[start..end].to_vec())
}
}
// ─── 5 Variants ──────────────────────────────────────────────────────────────
/// Variant 1: Plain mmap, no madvise.
pub struct MmapReaderPlain {
inner: MmapReader,
}
impl MmapReaderPlain {
pub fn remap(&mut self, new_size: usize) -> std::io::Result<()> {
unsafe { self.inner.remap(new_size) }
}
}
impl FileReaderBackend for MmapReaderPlain {
fn name(&self) -> &str {
"mmap_plain"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 2: mmap with MADV_SEQUENTIAL — optimal for sequential scan (index build).
pub struct MmapReaderSequential {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderSequential {
fn name(&self) -> &str {
"mmap_sequential"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
inner.advise(Advice::Sequential);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 3: mmap with MADV_RANDOM — optimal for random line access after index is built.
pub struct MmapReaderRandom {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderRandom {
fn name(&self) -> &str {
"mmap_random"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
inner.advise(Advice::Random);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 4: mmap with MAP_POPULATE — pre-fault all pages at mmap time.
/// Trades higher upfront cost for smoother subsequent access.
pub struct MmapReaderPopulate {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderPopulate {
fn name(&self) -> &str {
"mmap_populate"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let mut opts = MmapOptions::new();
opts.populate();
let inner = MmapReader::open_with_options(path, &opts)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
/// Variant 5: Phase-aware — MADV_SEQUENTIAL during index build, then MADV_RANDOM
/// for line access. Best of both worlds for the read-index-then-query pattern.
pub struct MmapReaderPhaseAware {
inner: MmapReader,
}
impl FileReaderBackend for MmapReaderPhaseAware {
fn name(&self) -> &str {
"mmap_phase_aware"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
install_sigbus_handler();
let inner = MmapReader::open_raw(path)?;
// Index build used sequential streaming via BufReader. Now switch to random.
inner.advise(Advice::Random);
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range(offset, len)
}
fn close(self) {}
}
// ─── Tests ───────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write as _;
use tempfile::NamedTempFile;
fn create_temp_file(content: &[u8]) -> NamedTempFile {
let mut f = NamedTempFile::new().unwrap();
f.write_all(content).unwrap();
f.flush().unwrap();
f
}
#[test]
fn test_plain_open_and_read_lines() {
let f = create_temp_file(b"hello\nworld\nfoo\n");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_plain");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
assert_eq!(reader.get_line(1), Some("world".to_owned()));
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
assert_eq!(reader.get_line(3), None);
reader.close();
}
#[test]
fn test_plain_out_of_bounds_returns_none() {
let f = create_temp_file(b"line1\nline2\n");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(100), None);
assert_eq!(reader.get_line(usize::MAX), None);
reader.close();
}
#[test]
fn test_plain_read_range() {
let content = b"hello world\nfoo bar\n";
let f = create_temp_file(content);
let reader = MmapReaderPlain::open(f.path()).unwrap();
let range = reader.read_range(0, 5).unwrap();
assert_eq!(&range, b"hello");
let range = reader.read_range(6, 5).unwrap();
assert_eq!(&range, b"world");
assert_eq!(reader.read_range(0, 1000), None);
reader.close();
}
#[test]
fn test_plain_empty_file() {
let f = create_temp_file(b"");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 0);
assert_eq!(reader.file_size(), 0);
assert_eq!(reader.get_line(0), None);
reader.close();
}
#[test]
fn test_sequential_variant() {
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
let reader = MmapReaderSequential::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_sequential");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
reader.close();
}
#[test]
fn test_random_variant() {
let f = create_temp_file(b"one\ntwo\nthree\n");
let reader = MmapReaderRandom::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_random");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(2), Some("three".to_owned()));
reader.close();
}
#[test]
fn test_populate_variant() {
let f = create_temp_file(b"x\ny\nz\n");
let reader = MmapReaderPopulate::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_populate");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("x".to_owned()));
reader.close();
}
#[test]
fn test_phase_aware_variant() {
let f = create_temp_file(b"a\nb\nc\n");
let reader = MmapReaderPhaseAware::open(f.path()).unwrap();
assert_eq!(reader.name(), "mmap_phase_aware");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("b".to_owned()));
reader.close();
}
#[test]
fn test_file_size() {
let content = b"hello world";
let f = create_temp_file(content);
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.file_size(), content.len() as u64);
reader.close();
}
#[test]
fn test_sigbus_flag_api() {
reset_sigbus_flag();
assert!(!sigbus_flag());
}
#[test]
fn test_concurrent_open_installs_handler_once() {
let f = create_temp_file(b"line1\nline2\nline3\n");
let path = f.path().to_owned();
let num_threads = 8;
let handles: Vec<_> = (0..num_threads)
.map(|thread_id| {
let path = path.clone();
std::thread::spawn(move || {
let reader = match thread_id % 5 {
0 => {
let r = MmapReaderPlain::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"plain"
}
1 => {
let r = MmapReaderSequential::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"sequential"
}
2 => {
let r = MmapReaderRandom::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"random"
}
3 => {
let r = MmapReaderPopulate::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"populate"
}
_ => {
let r = MmapReaderPhaseAware::open(&path).unwrap();
assert_eq!(r.total_lines(), 3);
r.close();
"phase_aware"
}
};
reader.to_owned()
})
})
.collect();
let results: Vec<String> = handles
.into_iter()
.map(|h| h.join().expect("thread panicked"))
.collect();
assert_eq!(results.len(), num_threads);
}
#[test]
fn test_no_trailing_newline() {
let f = create_temp_file(b"line1\nline2");
let reader = MmapReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
reader.close();
}
/// Diagnostic test: measure how long each stage of `open_raw()` takes
/// on a large file (e.g. the 5GB extreme.log).
/// Run with: cargo test -p log-viewer-bench --release -- --nocapture diag_open_stages
#[test]
fn diag_open_stages() {
let path = std::path::Path::new("/tmp/test-logviewer/extreme.log");
if !path.exists() {
eprintln!("SKIP: test file not found");
return;
}
// Stage 1: File::open + metadata
let t0 = std::time::Instant::now();
let file = File::open(path).unwrap();
let file_size = file.metadata().unwrap().len();
let stage1 = t0.elapsed();
eprintln!(
"Stage 1 - File::open + metadata: {:.2}ms ({:.1}GB)",
stage1.as_secs_f64() * 1000.0,
file_size as f64 / 1073741824.0
);
// Stage 2: mmap::map
let t1 = std::time::Instant::now();
let _mmap = unsafe { Mmap::map(&file) }.unwrap();
let stage2 = t1.elapsed();
eprintln!(
"Stage 2 - mmap::map: {:.2}ms",
stage2.as_secs_f64() * 1000.0
);
drop(_mmap);
// Stage 3: LineIndex::from_reader via BufReader (default 8KB buffer)
let t2 = std::time::Instant::now();
let mut reader = BufReader::new(&file);
let line_index = crate::line_index::LineIndex::from_reader(&mut reader).unwrap();
let stage3 = t2.elapsed();
eprintln!(
"Stage 3 - LineIndex::from_reader: {:.2}ms ({} lines, {} sampled_offsets)",
stage3.as_secs_f64() * 1000.0,
line_index.line_count(),
line_index.sampled_offsets.len()
);
// Stage 3b: Try with 1MB buffer
let file2 = File::open(path).unwrap();
let t2b = std::time::Instant::now();
let mut reader2 = BufReader::with_capacity(1024 * 1024, &file2);
let line_index2 = crate::line_index::LineIndex::from_reader(&mut reader2).unwrap();
let stage3b = t2b.elapsed();
eprintln!(
"Stage 3b - LineIndex (1MB buffer): {:.2}ms",
stage3b.as_secs_f64() * 1000.0
);
eprintln!("\n=== 瓶颈分析 ===");
let total_ms = stage1.as_secs_f64() * 1000.0
+ stage2.as_secs_f64() * 1000.0
+ stage3.as_secs_f64() * 1000.0;
let total_dur = stage1 + stage2 + stage3;
eprintln!("Total ~{:.0}ms", total_ms);
eprintln!(
" File::open: {:.1}% ({:.0}ms)",
stage1.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage1.as_secs_f64() * 1000.0
);
eprintln!(
" mmap::map: {:.1}% ({:.0}ms)",
stage2.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage2.as_secs_f64() * 1000.0
);
eprintln!(
" from_reader: {:.1}% ({:.0}ms)",
stage3.as_secs_f64() * 1000.0 / total_dur.as_secs_f64() / 1000.0 * 100.0,
stage3.as_secs_f64() * 1000.0
);
// Suppress unused warnings
let _ = line_index2;
}
}
+526
View File
@@ -0,0 +1,526 @@
// ─── pread_reader.rs ──────────────────────────────────────────────────────────
// pread-based FileReaderBackend implementations with 3 variants for benchmarking.
// Uses ReadCache for 4KB block caching to reduce syscalls, and posix_fadvise
// for kernel readahead control.
//
// CRITICAL: get_line() is custom — it does NOT use LineIndex::get_line() which
// requires a full &[u8] data slice. Instead it uses sampled_offsets directly
// and reads on-demand via pread.
// ──────────────────────────────────────────────────────────────────────────────
use std::cell::RefCell;
use std::fs::File;
use std::io::{BufReader, Seek as _, SeekFrom};
use std::os::unix::fs::FileExt;
use std::os::unix::io::AsRawFd;
use std::path::Path;
use crate::line_index::LineIndex;
use crate::FileReaderBackend;
const BLOCK_SIZE: usize = 256;
const CACHE_CHUNK: usize = 4096;
// ─── ReadCache ────────────────────────────────────────────────────────────────
/// Single-block read cache. Reduces syscalls by caching the last read.
/// Typical cache hit: sequential get_line() calls within the same 4KB block.
struct ReadCache {
buf: Vec<u8>,
buf_offset: u64,
buf_len: usize,
}
impl ReadCache {
fn new() -> Self {
Self {
buf: vec![0u8; CACHE_CHUNK],
buf_offset: 0,
buf_len: 0,
}
}
/// Read `len` bytes starting at `offset`. Returns a slice into the cache.
/// On cache hit (range fully within cached block), no syscall needed.
/// On miss, performs a `read_exact_at` syscall.
fn invalidate(&mut self) {
self.buf_len = 0;
}
fn get(&mut self, file: &File, offset: u64, len: usize) -> std::io::Result<&[u8]> {
let end = offset + len as u64;
if offset >= self.buf_offset && end <= self.buf_offset + self.buf_len as u64 {
let start = (offset - self.buf_offset) as usize;
return Ok(&self.buf[start..start + len]);
}
let alloc_len = CACHE_CHUNK.max(len);
self.buf.resize(alloc_len, 0);
file.read_exact_at(&mut self.buf[..len], offset)?;
self.buf_offset = offset;
self.buf_len = len;
Ok(&self.buf[..len])
}
}
// ─── PreadReaderCore ──────────────────────────────────────────────────────────
/// Core pread-based reader. Uses ReadCache and sparse LineIndex for on-demand
/// line retrieval without mmap.
pub struct PreadReaderCore {
file: File,
line_index: LineIndex,
file_size: u64,
cache: RefCell<ReadCache>,
}
impl PreadReaderCore {
/// Open file, build line index via streaming BufReader, prepare pread reader.
fn open_raw(path: &Path) -> std::io::Result<Self> {
let file = File::open(path)?;
let file_size = file.metadata()?.len();
let line_index = {
let mut reader = BufReader::new(&file);
LineIndex::from_reader(&mut reader)?
};
Ok(Self {
file,
line_index,
file_size,
cache: RefCell::new(ReadCache::new()),
})
}
/// Apply posix_fadvise to the entire file.
fn advise(&self, advice: libc::c_int) -> std::io::Result<()> {
let fd = self.file.as_raw_fd();
let ret = unsafe { libc::posix_fadvise(fd, 0, self.file_size as i64, advice) };
if ret != 0 {
return Err(std::io::Error::from_raw_os_error(ret));
}
Ok(())
}
/// Custom get_line using sparse index + pread.
/// Does NOT use LineIndex::get_line() (which needs full &[u8] slice).
///
/// Algorithm:
/// 1. Look up sampled_offsets[block] to get approximate byte position
/// 2. Scan forward through offset_in_block newlines via memchr
/// 3. Collect bytes from that position until next newline (may span 4KB blocks)
fn get_line_impl(&self, idx: usize) -> Option<String> {
let total = self.line_index.total_lines as usize;
if idx >= total || total == 0 {
return None;
}
let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE;
let mut cache = self.cache.borrow_mut();
// Phase 1: Find byte position of the start of line `idx`.
// Start at sampled_offsets[block], scan forward through `offset_in_block` newlines.
let start_offset = self.line_index.sampled_offsets[block];
let mut pos = start_offset;
let mut newlines_found = 0;
while newlines_found < offset_in_block {
let remaining = self.file_size.saturating_sub(pos) as usize;
if remaining == 0 {
return None;
}
let to_read = CACHE_CHUNK.min(remaining);
let data = cache.get(&self.file, pos, to_read).ok()?;
for byte_pos in memchr::memchr_iter(b'\n', data) {
newlines_found += 1;
if newlines_found == offset_in_block {
pos = pos + byte_pos as u64 + 1;
break;
}
}
if newlines_found < offset_in_block {
pos += to_read as u64;
}
}
// Phase 2: Collect bytes from `pos` until next newline or EOF.
// Line data may span multiple 4KB cache blocks.
let mut result = Vec::new();
while pos < self.file_size {
let remaining = (self.file_size - pos) as usize;
let to_read = CACHE_CHUNK.min(remaining);
let data = cache.get(&self.file, pos, to_read).ok()?;
match memchr::memchr(b'\n', data) {
Some(rel) => {
result.extend_from_slice(&data[..rel]);
break;
}
None => {
result.extend_from_slice(data);
pos += to_read as u64;
}
}
}
String::from_utf8(result)
.ok()
.map(|s| s.trim_end_matches(['\r', '\n']).to_owned())
}
/// Read a raw byte range from the file using pread.
fn read_range_impl(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
let end = offset.checked_add(len as u64)?;
if end > self.file_size {
return None;
}
let mut buf = vec![0u8; len];
self.file.read_exact_at(&mut buf, offset).ok()?;
Some(buf)
}
#[inline]
pub fn file_size(&self) -> u64 {
self.file_size
}
#[inline]
pub fn total_lines(&self) -> usize {
self.line_index.line_count()
}
pub fn refresh_index(&mut self) -> std::io::Result<()> {
let new_size = self.file.metadata()?.len();
self.file.seek(SeekFrom::Start(0))?;
let new_index = {
let mut reader = BufReader::new(&self.file);
LineIndex::from_reader(&mut reader)?
};
self.file_size = new_size;
self.line_index = new_index;
self.cache.get_mut().invalidate();
Ok(())
}
}
// ─── 3 Variants ───────────────────────────────────────────────────────────────
/// Variant 1: Plain pread, no fadvise.
pub struct PreadReaderPlain {
inner: PreadReaderCore,
}
impl PreadReaderPlain {
pub fn refresh_index(&mut self) -> std::io::Result<()> {
self.inner.refresh_index()
}
}
impl FileReaderBackend for PreadReaderPlain {
fn name(&self) -> &str {
"pread_plain"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
/// Variant 2: pread with POSIX_FADV_RANDOM — disable kernel readahead.
/// Best for random-access line lookup patterns.
pub struct PreadReaderRandom {
inner: PreadReaderCore,
}
impl FileReaderBackend for PreadReaderRandom {
fn name(&self) -> &str {
"pread_random"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
inner.advise(libc::POSIX_FADV_RANDOM)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
/// Variant 3: pread with POSIX_FADV_SEQUENTIAL — aggressive kernel readahead.
/// Best for sequential scan patterns.
pub struct PreadReaderSequential {
inner: PreadReaderCore,
}
impl FileReaderBackend for PreadReaderSequential {
fn name(&self) -> &str {
"pread_sequential"
}
fn open(path: &Path) -> std::io::Result<Self>
where
Self: Sized,
{
let inner = PreadReaderCore::open_raw(path)?;
inner.advise(libc::POSIX_FADV_SEQUENTIAL)?;
Ok(Self { inner })
}
fn file_size(&self) -> u64 {
self.inner.file_size()
}
fn total_lines(&self) -> usize {
self.inner.total_lines()
}
fn get_line(&self, idx: usize) -> Option<String> {
self.inner.get_line_impl(idx)
}
fn read_range(&self, offset: u64, len: usize) -> Option<Vec<u8>> {
self.inner.read_range_impl(offset, len)
}
fn close(self) {}
}
// ─── Tests ────────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write as _;
use tempfile::NamedTempFile;
fn create_temp_file(content: &[u8]) -> NamedTempFile {
let mut f = NamedTempFile::new().unwrap();
f.write_all(content).unwrap();
f.flush().unwrap();
f
}
#[test]
fn test_plain_open_and_read_lines() {
let f = create_temp_file(b"hello\nworld\nfoo\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_plain");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
assert_eq!(reader.get_line(1), Some("world".to_owned()));
assert_eq!(reader.get_line(2), Some("foo".to_owned()));
assert_eq!(reader.get_line(3), None);
reader.close();
}
#[test]
fn test_plain_out_of_bounds_returns_none() {
let f = create_temp_file(b"line1\nline2\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(100), None);
assert_eq!(reader.get_line(usize::MAX), None);
reader.close();
}
#[test]
fn test_plain_read_range() {
let content = b"hello world\nfoo bar\n";
let f = create_temp_file(content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
let range = reader.read_range(0, 5).unwrap();
assert_eq!(&range, b"hello");
let range = reader.read_range(6, 5).unwrap();
assert_eq!(&range, b"world");
assert_eq!(reader.read_range(0, 1000), None);
reader.close();
}
#[test]
fn test_plain_empty_file() {
let f = create_temp_file(b"");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 0);
assert_eq!(reader.file_size(), 0);
assert_eq!(reader.get_line(0), None);
reader.close();
}
#[test]
fn test_random_variant() {
let f = create_temp_file(b"one\ntwo\nthree\n");
let reader = PreadReaderRandom::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_random");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(2), Some("three".to_owned()));
reader.close();
}
#[test]
fn test_sequential_variant() {
let f = create_temp_file(b"alpha\nbeta\ngamma\n");
let reader = PreadReaderSequential::open(f.path()).unwrap();
assert_eq!(reader.name(), "pread_sequential");
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(1), Some("beta".to_owned()));
reader.close();
}
#[test]
fn test_file_size() {
let content = b"hello world";
let f = create_temp_file(content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.file_size(), content.len() as u64);
reader.close();
}
#[test]
fn test_no_trailing_newline() {
let f = create_temp_file(b"line1\nline2");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("line1".to_owned()));
assert_eq!(reader.get_line(1), Some("line2".to_owned()));
reader.close();
}
#[test]
fn test_line_spanning_cache_boundary() {
// Create a line that spans a 4KB boundary
let mut content = vec![b'a'; 4090];
content.push(b'\n');
content.extend_from_slice(b"target\n");
let f = create_temp_file(&content);
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.get_line(0).unwrap().len(), 4090);
assert_eq!(reader.get_line(1), Some("target".to_owned()));
assert_eq!(reader.get_line(2), None);
reader.close();
}
#[test]
fn test_many_lines_random_access() {
// Create 300 lines to test block boundary (256 lines per block)
let mut content = String::new();
for i in 0..300 {
content.push_str(&format!("line_{}\n", i));
}
let f = create_temp_file(content.as_bytes());
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 300);
// Test lines across the 256-line block boundary
assert_eq!(reader.get_line(0), Some("line_0".to_owned()));
assert_eq!(reader.get_line(255), Some("line_255".to_owned()));
assert_eq!(reader.get_line(256), Some("line_256".to_owned()));
assert_eq!(reader.get_line(299), Some("line_299".to_owned()));
assert_eq!(reader.get_line(300), None);
reader.close();
}
#[test]
fn test_refresh_index_sees_appended_lines() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("refresh_test.log");
// Phase 1: write 3 initial lines
{
let mut f = std::fs::File::create(&path).unwrap();
std::io::Write::write_all(&mut f, b"alpha\nbeta\ngamma\n").unwrap();
}
let mut reader = PreadReaderPlain::open(&path).unwrap();
assert_eq!(reader.total_lines(), 3);
assert_eq!(reader.get_line(0), Some("alpha".to_owned()));
assert_eq!(reader.get_line(3), None, "should be out of bounds before append");
// Phase 2: append 2 more lines
{
use std::io::Write as _;
let mut f = std::fs::OpenOptions::new().append(true).open(&path).unwrap();
f.write_all(b"delta\nepsilon\n").unwrap();
}
// Still stale — refresh needed
assert_eq!(reader.get_line(3), None, "stale index before refresh");
reader.refresh_index().unwrap();
assert_eq!(reader.total_lines(), 5);
assert_eq!(reader.get_line(3), Some("delta".to_owned()));
assert_eq!(reader.get_line(4), Some("epsilon".to_owned()));
assert_eq!(reader.get_line(5), None);
reader.close();
}
#[test]
fn test_refresh_index_no_change_is_noop() {
let f = create_temp_file(b"one\ntwo\n");
let mut reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.total_lines(), 2);
reader.refresh_index().unwrap();
assert_eq!(reader.total_lines(), 2);
assert_eq!(reader.get_line(0), Some("one".to_owned()));
assert_eq!(reader.get_line(1), Some("two".to_owned()));
reader.close();
}
#[test]
fn test_truncation_graceful_error() {
// Verify that truncated file access returns None/Err instead of panicking
let f = create_temp_file(b"hello\nworld\n");
let reader = PreadReaderPlain::open(f.path()).unwrap();
assert_eq!(reader.get_line(0), Some("hello".to_owned()));
// Truncate behind the reader — pread may return zeros or error
let _ = reader.get_line(1); // must not panic
reader.close();
}
}
+263
View File
@@ -0,0 +1,263 @@
use crate::metrics;
use crate::types::BenchmarkResult;
fn format_rss_mb(kb: u64) -> String {
format!("{:.1}MB", kb as f64 / 1024.0)
}
fn format_faults(count: u64) -> String {
if count < 1000 {
count.to_string()
} else {
let s = count.to_string();
let mut result = String::new();
for (i, c) in s.chars().rev().enumerate() {
if i > 0 && i % 3 == 0 {
result.push(',');
}
result.push(c);
}
result.chars().rev().collect()
}
}
// Format benchmark results as Markdown, grouped by category.
pub fn format_report(results: &[BenchmarkResult]) -> String {
let mut report = String::new();
report.push_str("# Benchmark: mmap vs pread\n\n");
let mut categories: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
std::collections::BTreeMap::new();
for r in results {
categories.entry(&r.category).or_default().push(r);
}
for (category, category_results) in &categories {
report.push_str(&format!("## {}\n\n", capitalize(category)));
let mut tests: std::collections::BTreeMap<&str, Vec<&BenchmarkResult>> =
std::collections::BTreeMap::new();
for r in category_results {
tests.entry(&r.test_name).or_default().push(r);
}
let mut variants: Vec<(String, String)> = Vec::new();
for r in category_results {
if !variants
.iter()
.any(|(b, v)| b == &r.backend && v == &r.variant)
{
variants.push((r.backend.clone(), r.variant.clone()));
}
}
variants.sort();
report.push_str("### Latency\n\n");
report.push_str("| Test |");
for (backend, variant) in &variants {
report.push_str(&format!(" {} ({}) |", backend, variant));
}
report.push_str(" Winner |\n");
report.push_str("|------|");
for _ in &variants {
report.push_str("------|");
}
report.push_str("--------|\n");
for (test_name, test_results) in &tests {
report.push_str(&format!("| {} |", test_name));
let mut all_means: Vec<f64> = Vec::new();
for (backend, variant) in &variants {
if let Some(r) = test_results
.iter()
.find(|r| r.backend == *backend && r.variant == *variant)
{
let avg = if r.latency_us.is_empty() {
0.0
} else {
metrics::mean(&r.latency_us)
};
all_means.push(avg);
}
}
let use_ms = all_means.iter().any(|&v| v >= 1000.0);
let mut best_backend = String::new();
let mut best_latency = f64::MAX;
for (backend, variant) in &variants {
let matching = test_results
.iter()
.find(|r| r.backend == *backend && r.variant == *variant);
if let Some(r) = matching {
let avg = if r.latency_us.is_empty() {
0.0
} else {
metrics::mean(&r.latency_us)
};
let sd = metrics::stdev(&r.latency_us);
let p95_val = metrics::p95(&r.latency_us) as f64;
let cell = if use_ms {
format!(
"{:.2}\u{00b1}{:.2}ms (p95:{:.2}ms)",
avg / 1000.0,
sd / 1000.0,
p95_val / 1000.0
)
} else {
format!(
"{:.1}\u{00b1}{:.1}\u{b5}s (p95:{:.1}\u{b5}s)",
avg, sd, p95_val
)
};
report.push_str(&format!(" {} |", cell));
if avg > 0.0 && avg < best_latency {
best_latency = avg;
best_backend = format!("{} ({})", backend, variant);
}
} else {
report.push_str(" - |");
}
}
report.push_str(&format!(
" {} |\n",
if best_backend.is_empty() {
"-"
} else {
&best_backend
}
));
}
report.push('\n');
let has_memory = category_results
.iter()
.any(|r| r.rss_kb > 0 || r.rss_peak_kb > 0);
if has_memory {
report.push_str("### Memory\n\n");
report.push_str("| Test | Variant | RSS | Peak RSS | Page Faults |\n");
report.push_str("|------|---------|-----|----------|-------------|\n");
let mut mem_rows: Vec<&BenchmarkResult> = category_results.to_vec();
mem_rows.sort_by(|a, b| {
(&a.test_name, &a.backend, &a.variant)
.cmp(&(&b.test_name, &b.backend, &b.variant))
});
for r in mem_rows {
let variant_label = format!("{} ({})", r.backend, r.variant);
report.push_str(&format!(
"| {} | {} | {} | {} | {} |\n",
r.test_name,
variant_label,
format_rss_mb(r.rss_kb),
format_rss_mb(r.rss_peak_kb),
format_faults(r.page_faults),
));
}
report.push('\n');
}
type ExtraEntry = (String, f64);
type ExtraGroup = (String, String, Vec<ExtraEntry>);
let mut extras: Vec<ExtraGroup> = category_results
.iter()
.filter(|r| !r.extra.is_empty())
.map(|r| {
let mut pairs: Vec<(String, f64)> =
r.extra.iter().map(|(k, &v)| (k.clone(), v)).collect();
pairs.sort_by(|a, b| a.0.cmp(&b.0));
(
r.test_name.clone(),
format!("{} ({})", r.backend, r.variant),
pairs,
)
})
.collect();
extras.sort_by(|a, b| (&a.0, &a.1).cmp(&(&b.0, &b.1)));
if !extras.is_empty() {
report.push_str("### Extra Metrics\n\n");
report.push_str("| Test | Variant | Metric | Value |\n");
report.push_str("|------|---------|--------|-------|\n");
for (test_name, variant_label, pairs) in &extras {
for (key, val) in pairs {
report.push_str(&format!(
"| {} | {} | {} | {:.3} |\n",
test_name, variant_label, key, val
));
}
}
report.push('\n');
}
}
report.push_str("## Summary\n\n");
report.push_str(&format!("- Total benchmarks: {}\n", results.len()));
report.push_str("- Categories: ");
report.push_str(&categories.keys().cloned().collect::<Vec<_>>().join(", "));
report.push('\n');
report
}
fn capitalize(s: &str) -> String {
let mut c = s.chars();
match c.next() {
None => String::new(),
Some(f) => f.to_uppercase().chain(c).collect(),
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::HashMap;
fn make_result(
category: &str,
test_name: &str,
backend: &str,
variant: &str,
latency_us: Vec<u64>,
) -> BenchmarkResult {
BenchmarkResult {
category: category.to_string(),
test_name: test_name.to_string(),
backend: backend.to_string(),
variant: variant.to_string(),
latency_us,
rss_kb: 0,
rss_peak_kb: 0,
page_faults: 0,
extra: HashMap::new(),
}
}
#[test]
fn report_ordering_independent_of_input_order() {
let set_a = vec![
make_result("sequential", "read_1mb", "pread", "default", vec![100, 110, 105]),
make_result("sequential", "read_1mb", "mmap", "default", vec![80, 85, 90]),
make_result("sequential", "read_4kb", "pread", "default", vec![10, 12, 11]),
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
];
let set_b = vec![
make_result("sequential", "read_4kb", "mmap", "default", vec![8, 9, 7]),
make_result("sequential", "read_1mb", "mmap", "default", vec![80, 85, 90]),
make_result("sequential", "read_4kb", "pread", "default", vec![10, 12, 11]),
make_result("sequential", "read_1mb", "pread", "default", vec![100, 110, 105]),
];
let report_a = format_report(&set_a);
let report_b = format_report(&set_b);
assert_eq!(report_a, report_b, "Reports must be identical regardless of input order");
}
}
+221
View File
@@ -0,0 +1,221 @@
use crate::metrics::MetricsCollector;
use crate::types::BenchmarkResult;
use std::path::PathBuf;
use std::sync::atomic::{AtomicBool, Ordering};
/// All recognized benchmark suite identifiers.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Suite {
Startup,
Render,
Jump,
Memory,
Growth,
Rotation,
Concurrent,
}
impl Suite {
/// All valid suite identifiers, in execution order.
pub const ALL: &[Suite] = &[
Suite::Startup,
Suite::Render,
Suite::Jump,
Suite::Memory,
Suite::Growth,
Suite::Rotation,
Suite::Concurrent,
];
#[must_use]
pub const fn as_str(self) -> &'static str {
match self {
Suite::Startup => "startup",
Suite::Render => "render",
Suite::Jump => "jump",
Suite::Memory => "memory",
Suite::Growth => "growth",
Suite::Rotation => "rotation",
Suite::Concurrent => "concurrent",
}
}
}
impl std::str::FromStr for Suite {
type Err = String;
fn from_str(s: &str) -> Result<Self, Self::Err> {
Suite::ALL
.iter()
.find(|suite| suite.as_str() == s)
.copied()
.ok_or_else(|| {
let valid = Suite::ALL
.iter()
.map(|s| s.as_str())
.collect::<Vec<_>>()
.join(", ");
format!("invalid value '{s}' for '--suites': valid values are {valid}")
})
}
}
pub struct BenchConfig {
pub test_file: PathBuf,
pub quick_mode: bool,
pub suites: Option<Vec<Suite>>,
}
/// Track whether we have already warned about VmHWM reset failure.
/// Prevents duplicate warnings across multiple suite runs.
static HWM_WARNED: AtomicBool = AtomicBool::new(false);
pub fn warn_reset_hwm() {
if let Err(e) = MetricsCollector::reset_vm_hwm() {
if HWM_WARNED.swap(true, Ordering::Relaxed) {
return;
}
if e.kind() == std::io::ErrorKind::PermissionDenied {
eprintln!(
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
Memory peak values may be contaminated across benchmark suites. \
Try running as root."
);
} else {
eprintln!(
"WARNING: Failed to reset VmHWM via /proc/self/clear_refs: {e}. \
Memory peak values may be contaminated across benchmark suites."
);
}
}
}
#[cfg(test)]
fn reset_hwm_warned() {
HWM_WARNED.store(false, Ordering::Relaxed);
}
pub fn run_all(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let should_run = |suite: Suite| -> bool {
match &config.suites {
Some(suites) => suites.contains(&suite),
None => true,
}
};
if should_run(Suite::Startup) {
warn_reset_hwm();
results.extend(crate::suites::startup::run(config));
}
if should_run(Suite::Render) {
warn_reset_hwm();
results.extend(crate::suites::render::run(config));
}
if should_run(Suite::Jump) {
warn_reset_hwm();
results.extend(crate::suites::jump::run(config));
}
if should_run(Suite::Memory) {
warn_reset_hwm();
results.extend(crate::suites::memory::run(config));
}
if should_run(Suite::Growth) {
warn_reset_hwm();
results.extend(crate::suites::growth::run(config));
}
if should_run(Suite::Rotation) {
warn_reset_hwm();
results.extend(crate::suites::rotation::run(config));
}
if should_run(Suite::Concurrent) {
warn_reset_hwm();
results.extend(crate::suites::concurrent::run(config));
}
results
}
#[cfg(test)]
mod tests {
use super::Suite;
use std::str::FromStr;
#[test]
fn parse_all_valid_suites() {
let expected = [
("startup", Suite::Startup),
("render", Suite::Render),
("jump", Suite::Jump),
("memory", Suite::Memory),
("growth", Suite::Growth),
("rotation", Suite::Rotation),
("concurrent", Suite::Concurrent),
];
for (s, expected_suite) in expected {
assert_eq!(Suite::from_str(s).unwrap(), expected_suite, "failed to parse '{s}'");
}
}
#[test]
fn misspelled_suite_returns_error() {
let err = Suite::from_str("startp").unwrap_err();
assert!(
err.contains("invalid value 'startp'"),
"error should mention the bad value: {err}"
);
}
#[test]
fn error_message_lists_all_valid_values() {
let err = Suite::from_str("bogus").unwrap_err();
for name in Suite::ALL.iter().map(|s| s.as_str()) {
assert!(
err.contains(name),
"error should list valid suite '{name}': {err}"
);
}
}
#[test]
fn mixed_valid_invalid_stops_at_first_error() {
// "startup" is valid, "zzz" is not — collect hits the first Err
let names = ["startup".to_string(), "zzz".to_string()];
let result: Result<Vec<Suite>, _> = names.iter().map(|s| s.parse()).collect();
assert!(result.is_err());
}
#[test]
fn hwm_warned_flag_prevents_reentry() {
use super::reset_hwm_warned;
use std::sync::atomic::Ordering;
reset_hwm_warned();
assert!(
!super::HWM_WARNED.load(Ordering::Relaxed),
"flag should be false after reset"
);
// Simulate the flag being set (as warn_reset_hwm would do on error)
super::HWM_WARNED.store(true, Ordering::Relaxed);
// swap should now return true (old value), indicating already warned
assert!(
super::HWM_WARNED.swap(true, Ordering::Relaxed),
"swap should return the previous value (true)"
);
}
#[test]
fn warn_reset_hwm_does_not_panic() {
use super::reset_hwm_warned;
reset_hwm_warned();
// Whether reset_vm_hwm succeeds or fails, warn_reset_hwm must not panic.
// Multiple calls must also be safe.
super::warn_reset_hwm();
super::warn_reset_hwm();
super::warn_reset_hwm();
}
}
+117
View File
@@ -0,0 +1,117 @@
use std::collections::HashMap;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_parallel_reads::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_parallel_reads::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_parallel_reads::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_parallel_reads::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_parallel_reads::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_parallel_reads::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_parallel_reads::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_parallel_reads::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results
}
fn bench_parallel_reads<B: FileReaderBackend + Send + 'static>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = config.test_file.clone();
let iterations = if config.quick_mode { 250 } else { 1000 };
let total_lines = {
let reader = B::open(&path).expect("Failed to open file for line count");
let count = reader.total_lines();
reader.close();
count
};
let overall_start = std::time::Instant::now();
let num_threads = 4usize;
let handles: Vec<_> = (0..num_threads)
.map(|thread_id| {
let path = path.clone();
std::thread::spawn(move || {
let reader = B::open(&path).expect("Failed to open file in thread");
let mut latencies = Vec::with_capacity(iterations);
for i in 0..iterations {
let line_idx = (thread_id * iterations + i) % total_lines.max(1);
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
reader.close();
latencies
})
})
.collect();
let thread_latencies: Vec<Vec<u64>> = handles
.into_iter()
.map(|h| h.join().expect("Thread panicked"))
.collect();
let total_elapsed = overall_start.elapsed();
let all_latencies: Vec<u64> = thread_latencies.into_iter().flatten().collect();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("num_threads".into(), num_threads as f64);
extra.insert("iterations_per_thread".into(), iterations as f64);
extra.insert("total_time_us".into(), total_elapsed.as_micros() as f64);
extra.insert("total_lines".into(), total_lines as f64);
vec![BenchmarkResult {
category: "concurrent".into(),
test_name: "parallel_reads".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: all_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+295
View File
@@ -0,0 +1,295 @@
use std::collections::HashMap;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::MmapReaderPlain;
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let dir = tempfile::tempdir().expect("Failed to create temp dir");
results.extend(bench_append_visibility_mmap(config, dir.path()));
results.extend(bench_append_visibility_pread(config, dir.path()));
results.extend(bench_remap_cost(config, dir.path()));
results.extend(bench_scroll_during_append(config, dir.path()));
results.extend(bench_high_frequency_append(config, dir.path()));
results
}
fn bench_append_visibility_mmap(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let path = data_gen::generate_growable_file(dir).expect("Failed to create growable file");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open growable file");
let original_lines = reader.total_lines();
let original_size = reader.file_size();
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
let new_metadata = std::fs::metadata(&path).expect("Failed to read metadata");
let new_size = new_metadata.len() as usize;
let remap_start = std::time::Instant::now();
reader.remap(new_size).expect("Failed to remap");
let remap_elapsed = remap_start.elapsed();
let new_line_bytes =
reader.read_range(original_size, (new_size - original_size as usize).min(256));
let visible = new_line_bytes.is_some();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("original_lines".into(), original_lines as f64);
extra.insert("appended_lines".into(), append_count as f64);
extra.insert("new_bytes_visible".into(), visible as u64 as f64);
extra.insert("original_size".into(), original_size as f64);
extra.insert("new_size".into(), new_size as f64);
extra.insert("remap_us".into(), remap_elapsed.as_micros() as f64);
reader.close();
vec![BenchmarkResult {
category: "growth".into(),
test_name: "append_visibility_mmap".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: vec![remap_elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_append_visibility_pread(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("pread_growth");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let reader = PreadReaderPlain::open(&path).expect("Failed to open growable file");
let original_lines = reader.total_lines();
reader.close();
data_gen::append_lines(&path, append_count).expect("Failed to append lines");
let reopen_start = std::time::Instant::now();
let new_reader = PreadReaderPlain::open(&path).expect("Failed to reopen file");
let reopen_elapsed = reopen_start.elapsed();
let new_lines = new_reader.total_lines();
let can_read_new = new_lines > original_lines;
if can_read_new {
let last_line = new_lines.saturating_sub(1);
let _ = new_reader.get_line(last_line);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("original_lines".into(), original_lines as f64);
extra.insert("appended_lines".into(), append_count as f64);
extra.insert("new_total_lines".into(), new_lines as f64);
extra.insert("reopen_us".into(), reopen_elapsed.as_micros() as f64);
new_reader.close();
vec![BenchmarkResult {
category: "growth".into(),
test_name: "append_visibility_pread".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: vec![reopen_elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_remap_cost(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("remap_cost");
let append_count: usize = if config.quick_mode { 100 } else { 1000 };
let iterations: usize = if config.quick_mode { 5 } else { 20 };
let mut latencies = Vec::with_capacity(iterations);
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let mut reader = MmapReaderPlain::open(&path).expect("Failed to open file");
data_gen::append_lines(&path, append_count).expect("Failed to append");
let new_size = std::fs::metadata(&path).expect("metadata").len() as usize;
let t = std::time::Instant::now();
reader.remap(new_size).expect("remap failed");
latencies.push(t.elapsed().as_micros() as u64);
reader.close();
let _ = std::fs::remove_file(&path);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("appended_per_iter".into(), append_count as f64);
extra.insert("iterations".into(), iterations as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "remap_cost".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_scroll_during_append(config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("scroll_append");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let duration_secs: u64 = if config.quick_mode { 2 } else { 10 };
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
let bg_path = path.clone();
let bg_handle = std::thread::spawn(move || {
let batch_size = 100;
let batch_interval =
std::time::Duration::from_micros(1_000_000 / (append_rate / batch_size).max(1) as u64);
let start = std::time::Instant::now();
while start.elapsed().as_secs() < duration_secs {
data_gen::append_lines(&bg_path, batch_size).ok();
std::thread::sleep(batch_interval);
}
});
let mut reader = PreadReaderPlain::open(&path).expect("Failed to open file");
let initial_lines = reader.total_lines();
let mut frame_latencies = Vec::new();
let mut current_line = 0usize;
let mut refresh_count: u64 = 0;
let mut none_count: u64 = 0;
let scroll_start = std::time::Instant::now();
let mut last_refresh = std::time::Instant::now();
let refresh_interval = std::time::Duration::from_millis(250);
while scroll_start.elapsed().as_secs() < duration_secs {
if last_refresh.elapsed() >= refresh_interval {
reader.refresh_index().ok();
refresh_count += 1;
last_refresh = std::time::Instant::now();
continue;
}
if let Some(_line) = reader.get_line(current_line) {
let t = std::time::Instant::now();
frame_latencies.push(t.elapsed().as_micros() as u64);
current_line += 1;
} else {
none_count += 1;
std::thread::sleep(std::time::Duration::from_millis(1));
}
}
let max_line = current_line;
assert!(
max_line > initial_lines,
"benchmark never read past initial {initial_lines} lines (max={max_line})"
);
reader.close();
bg_handle.join().ok();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("duration_secs".into(), duration_secs as f64);
extra.insert("append_rate_per_sec".into(), append_rate as f64);
extra.insert("frames_rendered".into(), frame_latencies.len() as f64);
extra.insert("refresh_count".into(), refresh_count as f64);
extra.insert("none_count".into(), none_count as f64);
extra.insert("initial_lines".into(), initial_lines as f64);
extra.insert("max_line_seen".into(), max_line as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "scroll_during_append".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: frame_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_high_frequency_append(
config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("high_freq_append");
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create growable file");
let duration_secs: u64 = if config.quick_mode { 3 } else { 30 };
let append_rate: usize = if config.quick_mode { 1000 } else { 10000 };
let batch_size: usize = 100;
let batches_per_sec = append_rate / batch_size;
let total_batches = (duration_secs as usize * batches_per_sec).max(1);
let mut detect_latencies = Vec::with_capacity(total_batches);
for _ in 0..total_batches {
data_gen::append_lines(&path, batch_size).expect("Failed to append");
let t = std::time::Instant::now();
if let Ok(reader) = PreadReaderPlain::open(&path) {
let total = reader.total_lines();
let _ = reader.get_line(total.saturating_sub(1));
reader.close();
}
detect_latencies.push(t.elapsed().as_micros() as u64);
std::thread::sleep(std::time::Duration::from_micros(
1_000_000 / batches_per_sec as u64,
));
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("duration_secs".into(), duration_secs as f64);
extra.insert("append_rate_per_sec".into(), append_rate as f64);
extra.insert("batch_size".into(), batch_size as f64);
extra.insert("total_batches".into(), total_batches as f64);
vec![BenchmarkResult {
category: "growth".into(),
test_name: "high_frequency_append".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: detect_latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+251
View File
@@ -0,0 +1,251 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_near_jump::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_near_jump::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_near_jump::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_near_jump::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_near_jump::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_near_jump::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_near_jump::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_near_jump::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_far_jump::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_far_jump::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_far_jump::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_far_jump::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_far_jump::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_far_jump::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_far_jump::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_far_jump::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_jump_end::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_jump_end::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_reverse_scan::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_reverse_scan::<PreadReaderPlain>(
"pread", "plain", config,
));
results
}
fn bench_near_jump<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations: usize = if config.quick_mode { 10 } else { 100 };
let mut latencies = Vec::with_capacity(iterations);
let mut current = 0usize;
for _ in 0..iterations {
let target = (current + 15).min(total.saturating_sub(1));
let t = std::time::Instant::now();
let _ = reader.get_line(target);
latencies.push(t.elapsed().as_micros() as u64);
current = target;
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "near_jump".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: HashMap::new(),
}]
}
fn bench_far_jump<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let repetitions: usize = if config.quick_mode { 3 } else { 10 };
let fractions = [0.25, 0.50, 0.75];
let mut latencies = Vec::new();
let mut extra = HashMap::new();
for &frac in &fractions {
let target = ((total as f64 * frac) as usize).min(total.saturating_sub(1));
for _ in 0..repetitions {
let t = std::time::Instant::now();
let _ = reader.get_line(target);
latencies.push(t.elapsed().as_micros() as u64);
}
}
extra.insert("jump_positions".into(), 3.0);
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "far_jump".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_jump_end<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations: usize = if config.quick_mode { 5 } else { 10 };
let last_line = total.saturating_sub(1);
let mut latencies = Vec::with_capacity(iterations);
for _ in 0..iterations {
let t = std::time::Instant::now();
let _ = reader.get_line(last_line);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("last_line_idx".into(), last_line as f64);
reader.close();
vec![BenchmarkResult {
category: "jump".into(),
test_name: "jump_end".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_reverse_scan<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
if total <= FRAME_LINES {
reader.close();
return vec![];
}
let iterations: usize = if config.quick_mode { 5 } else { 10 };
let mut latencies = Vec::with_capacity(iterations * FRAME_LINES);
for _ in 0..iterations {
let start = total - FRAME_LINES;
for i in (start..total).rev() {
let t = std::time::Instant::now();
let _ = reader.get_line(i);
latencies.push(t.elapsed().as_micros() as u64);
}
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
let mut extra = HashMap::new();
extra.insert("lines_per_scan".into(), FRAME_LINES as f64);
extra.insert("iterations".into(), iterations as f64);
vec![BenchmarkResult {
category: "jump".into(),
test_name: "reverse_scan".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+237
View File
@@ -0,0 +1,237 @@
use std::collections::HashMap;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_idle_rss::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_idle_rss::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_idle_rss::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_idle_rss::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_idle_rss::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_idle_rss::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_idle_rss::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_idle_rss::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_scroll_rss::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_scroll_rss::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_jump_end_rss::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_jump_end_rss::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_rss_reclaim::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_rss_reclaim::<PreadReaderPlain>(
"pread", "plain", config,
));
results
}
fn bench_idle_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("total_lines".into(), reader.total_lines() as f64);
extra.insert(
"file_size_mb".into(),
reader.file_size() as f64 / (1024.0 * 1024.0),
);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "idle_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_scroll_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let sample_interval = 100_000;
let max_lines = if config.quick_mode { 100_000 } else { total };
let upper = max_lines.min(total);
let mut rss_samples = Vec::new();
let mut hwm_samples = Vec::new();
let mut lines_read = 0usize;
for i in (0..upper).step_by(sample_interval) {
if reader.get_line(i).is_some() {
lines_read += 1;
}
let rss = MetricsCollector::read_rss();
rss_samples.push(rss.vm_rss_kb);
hwm_samples.push(rss.vm_hwm_kb);
}
let final_rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("rss_samples_count".into(), rss_samples.len() as f64);
extra.insert(
"max_rss_kb".into(),
rss_samples.iter().copied().fold(0u64, u64::max) as f64,
);
extra.insert(
"max_hwm_kb".into(),
hwm_samples.iter().copied().fold(0u64, u64::max) as f64,
);
extra.insert("lines_read".into(), lines_read as f64);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "scroll_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: final_rss.vm_rss_kb,
rss_peak_kb: final_rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_jump_end_rss<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let last_line = total.saturating_sub(1);
let _ = reader.get_line(last_line);
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("last_line_idx".into(), last_line as f64);
extra.insert("total_lines".into(), total as f64);
reader.close();
vec![BenchmarkResult {
category: "memory".into(),
test_name: "jump_end_rss".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_rss_reclaim<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let reader = B::open(&config.test_file).expect("Failed to open file");
let total = reader.total_lines();
let last_line = total.saturating_sub(1);
let _ = reader.get_line(last_line);
let wait_secs: u64 = if config.quick_mode { 5 } else { 30 };
let sample_interval: u64 = 5;
let num_samples = (wait_secs / sample_interval) as usize;
let mut rss_samples = Vec::with_capacity(num_samples);
let mut hwm_samples = Vec::with_capacity(num_samples);
for _ in 0..num_samples {
std::thread::sleep(std::time::Duration::from_secs(sample_interval));
let rss = MetricsCollector::read_rss();
rss_samples.push(rss.vm_rss_kb);
hwm_samples.push(rss.vm_hwm_kb);
}
let final_rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
reader.close();
let mut extra = HashMap::new();
extra.insert("wait_total_secs".into(), wait_secs as f64);
extra.insert("rss_samples".into(), rss_samples.len() as f64);
if let (Some(&first), Some(&last)) = (rss_samples.first(), rss_samples.last()) {
extra.insert("rss_first_kb".into(), first as f64);
extra.insert("rss_last_kb".into(), last as f64);
extra.insert(
"rss_change_pct".into(),
if first > 0 {
((last as f64 - first as f64) / first as f64) * 100.0
} else {
0.0
},
);
}
vec![BenchmarkResult {
category: "memory".into(),
test_name: "rss_reclaim".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![],
rss_kb: final_rss.vm_rss_kb,
rss_peak_kb: final_rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+12
View File
@@ -0,0 +1,12 @@
pub mod concurrent;
pub mod growth;
pub mod jump;
pub mod memory;
pub mod render;
pub mod rotation;
pub mod startup;
/// Number of lines read per single-frame render benchmark.
/// Also used as the scan window size for reverse-scan benchmarks.
/// Shared across suites to ensure consistent workload sizing.
pub(crate) const FRAME_LINES: usize = 35;
+315
View File
@@ -0,0 +1,315 @@
use std::collections::HashMap;
use super::FRAME_LINES;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_single_frame::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_single_frame::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_single_frame::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_single_frame::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_single_frame::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_single_frame::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_single_frame::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_single_frame::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results.extend(bench_continuous_scroll::<MmapReaderPlain>(
"mmap", "plain", config,
));
results.extend(bench_continuous_scroll::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_continuous_scroll::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_continuous_scroll::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_continuous_scroll::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_continuous_scroll::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_continuous_scroll::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_continuous_scroll::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
results
}
fn bench_single_frame<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let mut results = Vec::new();
for (pos_name, start_line) in select_frame_positions(total) {
let mut latencies = Vec::with_capacity(FRAME_LINES);
for i in 0..FRAME_LINES {
let line_idx = start_line + i;
if line_idx >= total {
break;
}
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
results.push(BenchmarkResult {
category: "render".into(),
test_name: format!("single_frame_{pos_name}"),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: HashMap::new(),
});
}
reader.close();
results
}
/// Select non-overlapping benchmark positions for single-frame tests.
///
/// Returns `(label, start_line)` pairs. Positions are chosen so that no
/// two frames overlap, skipping any position that cannot fit without
/// colliding with a previously selected range.
pub(crate) fn select_frame_positions(total: usize) -> Vec<(&'static str, usize)> {
let frame = FRAME_LINES;
if total == 0 {
return vec![];
}
let mut positions: Vec<(&'static str, usize)> = vec![("head", 0)];
// Need at least 3 full frames to place head, middle, and tail without overlap.
if total >= 3 * frame {
let head_end = frame;
let tail_start = total - frame;
let gap = tail_start - head_end;
// Center the middle frame within the gap, ensuring it fits entirely
let middle_start = head_end + gap.saturating_sub(frame) / 2;
positions.push(("middle", middle_start));
positions.push(("tail", tail_start));
}
positions
}
fn bench_continuous_scroll<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
let path = &config.test_file;
let reader = B::open(path).expect("Failed to open file");
let total = reader.total_lines();
let iterations = if config.quick_mode { 100 } else { 1000 };
let timeout = if config.quick_mode {
std::time::Duration::from_secs(1)
} else {
std::time::Duration::from_secs(10)
};
let mut latencies = Vec::with_capacity(iterations);
let start = std::time::Instant::now();
for i in 0..iterations {
if start.elapsed() > timeout {
break;
}
let line_idx = i % total.max(1);
let t = std::time::Instant::now();
let _ = reader.get_line(line_idx);
latencies.push(t.elapsed().as_micros() as u64);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("total_lines_scrolled".into(), latencies.len() as f64);
reader.close();
vec![BenchmarkResult {
category: "render".into(),
test_name: "continuous_scroll".into(),
backend: backend.into(),
variant: variant.into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
#[cfg(test)]
mod tests {
use super::*;
fn starts(pos: &[(&str, usize)]) -> Vec<usize> {
pos.iter().map(|&(_, s)| s).collect()
}
fn labels<'a>(pos: &[(&'a str, usize)]) -> Vec<&'a str> {
pos.iter().map(|&(l, _)| l).collect()
}
fn ranges_overlap(a_start: usize, b_start: usize) -> bool {
let a_end = a_start + FRAME_LINES;
let b_end = b_start + FRAME_LINES;
a_start < b_end && b_start < a_end
}
#[test]
fn zero_lines_no_positions() {
assert!(select_frame_positions(0).is_empty());
}
#[test]
fn one_line_head_only() {
let pos = select_frame_positions(1);
assert_eq!(labels(&pos), vec!["head"]);
assert_eq!(starts(&pos), vec![0]);
}
#[test]
fn at_frame_lines_head_only() {
let pos = select_frame_positions(FRAME_LINES);
assert_eq!(labels(&pos), vec!["head"]);
assert_eq!(starts(&pos), vec![0]);
}
#[test]
fn just_above_threshold_head_middle_tail() {
let total = 3 * FRAME_LINES;
let pos = select_frame_positions(total);
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
assert_eq!(starts(&pos)[0], 0);
assert_eq!(*starts(&pos).last().unwrap(), total - FRAME_LINES);
// Verify no overlap
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap: {:?} @ {} vs {:?} @ {} (total={})",
pos[i].0, pos[i].1, pos[j].0, pos[j].1, total
);
}
}
}
#[test]
fn no_overlap_at_total_70() {
// total=70 < 3*35=105, so only head is returned
let pos = select_frame_positions(70);
assert_eq!(labels(&pos), vec!["head"]);
}
#[test]
fn no_overlap_at_total_104() {
let pos = select_frame_positions(104);
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=104: {:?} @ {} vs {:?} @ {}",
pos[i].0, pos[i].1, pos[j].0, pos[j].1
);
}
}
}
#[test]
fn no_overlap_at_total_105() {
let pos = select_frame_positions(105);
for i in 0..pos.len() {
for j in (i + 1)..pos.len() {
assert!(
!ranges_overlap(pos[i].1, pos[j].1),
"overlap at total=105: {:?} @ {} vs {:?} @ {}",
pos[i].0, pos[i].1, pos[j].0, pos[j].1
);
}
}
}
#[test]
fn middle_is_centered_between_head_and_tail() {
let total = 1000;
let pos = select_frame_positions(total);
assert_eq!(pos.len(), 3);
let (_, head_start) = pos[0];
let (_, middle_start) = pos[1];
let (_, tail_start) = pos[2];
assert_eq!(head_start, 0);
assert_eq!(tail_start, total - FRAME_LINES);
let head_end = head_start + FRAME_LINES;
let gap = tail_start - head_end;
assert_eq!(middle_start, head_end + gap.saturating_sub(FRAME_LINES) / 2);
}
#[test]
fn large_file_all_three_positions() {
let pos = select_frame_positions(1_000_000);
assert_eq!(labels(&pos), vec!["head", "middle", "tail"]);
}
}
+170
View File
@@ -0,0 +1,170 @@
use std::collections::HashMap;
use crate::data_gen;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{self, MmapReaderPlain};
use crate::pread_reader::PreadReaderPlain;
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
let dir = tempfile::tempdir().expect("Failed to create temp dir");
results.extend(bench_truncate_safety_mmap(config, dir.path()));
results.extend(bench_truncate_safety_pread(config, dir.path()));
results.extend(bench_rotation_detection(config, dir.path()));
results
}
fn bench_truncate_safety_mmap(
_config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_mmap");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut sigbus_detected = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
mmap_reader::reset_sigbus_flag();
let reader = MmapReaderPlain::open(&path).expect("Failed to open file");
let original_size = reader.file_size();
let truncate_size = original_size / 2;
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
let t = std::time::Instant::now();
let mid_offset = original_size as u64 / 2;
let _ = reader.read_range(mid_offset, 64);
latencies.push(t.elapsed().as_micros() as u64);
if mmap_reader::sigbus_flag() {
sigbus_detected += 1;
}
reader.close();
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("sigbus_detected".into(), sigbus_detected as f64);
extra.insert("crashed".into(), 0.0);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "truncate_safety_mmap".into(),
backend: "mmap".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_truncate_safety_pread(
_config: &BenchConfig,
dir: &std::path::Path,
) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("trunc_pread");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut error_count = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let reader = PreadReaderPlain::open(&path).expect("Failed to open file");
let original_size = reader.file_size();
let truncate_size = original_size / 2;
data_gen::truncate_file(&path, truncate_size).expect("Failed to truncate");
let t = std::time::Instant::now();
let mid_offset = original_size as u64 / 2;
let result = reader.read_range(mid_offset, 64);
latencies.push(t.elapsed().as_micros() as u64);
if result.is_none() {
error_count += 1;
}
reader.close();
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("errors_returned".into(), error_count as f64);
extra.insert("crashed".into(), 0.0);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "truncate_safety_pread".into(),
backend: "pread".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
fn bench_rotation_detection(_config: &BenchConfig, dir: &std::path::Path) -> Vec<BenchmarkResult> {
let sub_dir = dir.join("rotation_detect");
let iterations: usize = if _config.quick_mode { 3 } else { 10 };
let mut latencies = Vec::with_capacity(iterations);
let mut detected_count = 0usize;
for _ in 0..iterations {
let path = data_gen::generate_growable_file(&sub_dir).expect("Failed to create file");
let original_inode = MetricsCollector::get_inode(&path).expect("Failed to get inode");
let _rotated = data_gen::rotate_file(&path).expect("Failed to rotate file");
let t = std::time::Instant::now();
let detected = MetricsCollector::detect_rotation(original_inode, &path);
latencies.push(t.elapsed().as_micros() as u64);
if detected {
detected_count += 1;
}
let _ = std::fs::remove_file(&path);
let rotated = sub_dir.join("growable.log.1");
let _ = std::fs::remove_file(&rotated);
}
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let mut extra = HashMap::new();
extra.insert("iterations".into(), iterations as f64);
extra.insert("rotations_detected".into(), detected_count as f64);
vec![BenchmarkResult {
category: "rotation".into(),
test_name: "rotation_detection".into(),
backend: "both".into(),
variant: "plain".into(),
latency_us: latencies,
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra,
}]
}
+143
View File
@@ -0,0 +1,143 @@
use std::collections::HashMap;
use std::path::Path;
use crate::metrics::MetricsCollector;
use crate::mmap_reader::{
MmapReaderPhaseAware, MmapReaderPlain, MmapReaderPopulate, MmapReaderRandom,
MmapReaderSequential,
};
use crate::pread_reader::{PreadReaderPlain, PreadReaderRandom, PreadReaderSequential};
use crate::runner::BenchConfig;
use crate::types::BenchmarkResult;
use crate::FileReaderBackend;
pub fn run(config: &BenchConfig) -> Vec<BenchmarkResult> {
let mut results = Vec::new();
results.extend(bench_hot_open::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_hot_open::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_hot_open::<MmapReaderRandom>("mmap", "random", config));
results.extend(bench_hot_open::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_hot_open::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_hot_open::<PreadReaderPlain>("pread", "plain", config));
results.extend(bench_hot_open::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_hot_open::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
if !config.quick_mode {
match MetricsCollector::clear_file_cache(&config.test_file) {
Ok(()) => {
results.extend(bench_cold_open::<MmapReaderPlain>("mmap", "plain", config));
results.extend(bench_cold_open::<MmapReaderSequential>(
"mmap",
"sequential",
config,
));
results.extend(bench_cold_open::<MmapReaderRandom>(
"mmap", "random", config,
));
results.extend(bench_cold_open::<MmapReaderPopulate>(
"mmap", "populate", config,
));
results.extend(bench_cold_open::<MmapReaderPhaseAware>(
"mmap",
"phase_aware",
config,
));
results.extend(bench_cold_open::<PreadReaderPlain>(
"pread", "plain", config,
));
results.extend(bench_cold_open::<PreadReaderRandom>(
"pread", "random", config,
));
results.extend(bench_cold_open::<PreadReaderSequential>(
"pread",
"sequential",
config,
));
}
Err(e) => {
eprintln!(
"WARNING: Failed to clear file cache; skipping cold startup benchmarks because results would not be cold: {e}"
);
}
}
}
results
}
fn bench_hot_open<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
open_and_measure::<B>(backend, variant, &config.test_file, "hot_open")
}
fn bench_cold_open<B: FileReaderBackend>(
backend: &str,
variant: &str,
config: &BenchConfig,
) -> Vec<BenchmarkResult> {
match MetricsCollector::clear_file_cache(&config.test_file) {
Ok(()) => open_and_measure::<B>(backend, variant, &config.test_file, "cold_open"),
Err(e) => {
eprintln!(
"WARNING: Failed to clear file cache for {backend}/{variant}; skipping cold benchmark: {e}"
);
Vec::new()
}
}
}
fn open_and_measure<B: FileReaderBackend>(
backend: &str,
variant: &str,
path: &Path,
test_name: &str,
) -> Vec<BenchmarkResult> {
let start = std::time::Instant::now();
let reader = B::open(path).expect("Failed to open file");
let elapsed = start.elapsed();
let rss = MetricsCollector::read_rss();
let faults = MetricsCollector::read_page_faults();
let result = BenchmarkResult {
category: "startup".into(),
test_name: test_name.into(),
backend: backend.into(),
variant: variant.into(),
latency_us: vec![elapsed.as_micros() as u64],
rss_kb: rss.vm_rss_kb,
rss_peak_kb: rss.vm_hwm_kb,
page_faults: faults.minor_faults + faults.major_faults,
extra: {
let mut m = HashMap::new();
m.insert("total_lines".into(), reader.total_lines() as f64);
m.insert(
"file_size_mb".into(),
reader.file_size() as f64 / (1024.0 * 1024.0),
);
m
},
};
reader.close();
vec![result]
}
+13
View File
@@ -0,0 +1,13 @@
use std::collections::HashMap;
pub struct BenchmarkResult {
pub category: String,
pub test_name: String,
pub backend: String,
pub variant: String,
pub latency_us: Vec<u64>,
pub rss_kb: u64,
pub rss_peak_kb: u64,
pub page_faults: u64,
pub extra: HashMap<String, f64>,
}
+217 -32
View File
@@ -9,6 +9,14 @@ use crate::io::index_cache::IndexCache;
use crate::io::line_index::LineIndex;
use std::path::{Path, PathBuf};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AppendStatus {
Unchanged,
Appended(u64),
/// File shrank; mmap and index rebuilt via reload().
Reloaded,
}
pub struct FileReader {
path: PathBuf,
mmap: Option<memmap2::Mmap>,
@@ -25,20 +33,23 @@ impl FileReader {
} else {
// SAFETY: 使用只读 Mmap(非 MmapMut),文件以只读方式打开。
// memmap2 内部持有文件描述符,确保 mmap 期间文件不会被关闭。
//
// ⚠️ Known limitation (Phase 5): 如果文件在 mmap 期间被外部进程截断,
// 访问截断区域的内存会触发 SIGBUS(致命信号,无法恢复)。
// FileWatcher Phase 将添加文件修改检测和 re-mmap 机制来处理此情况。
// 在 Phase 5 中,假设打开的文件不会被外部修改。
Some(unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?)
let m = unsafe { memmap2::Mmap::map(&file) }
.map_err(|e| CoreError::Mmap(e.to_string()))?;
// Layer 3: mmap 后立即 stat 同一 fd,检测截断(TOCTOU 缓解,非安全证明)
let current_size = file.metadata()?.len();
if current_size < m.len() as u64 {
None
} else {
Some(m)
}
};
let line_index = {
let mut reader = std::io::BufReader::new(&file);
LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io {
source: e,
context: "building line index".into(),
})?
// 直接从 mmap 快照构建行索引,确保索引与数据来自同一内存映射,
// 消除 mmap + BufReader 双读之间的 TOCTOU 竞态窗口。
let line_index = match &mmap {
Some(m) => LineIndex::from_bytes(m.as_ref()),
None => LineIndex::from_bytes(&[]),
};
Ok(Self {
@@ -76,9 +87,8 @@ impl FileReader {
}
}
/// Save the line index cache to disk.
pub fn save_cache(&self) -> std::io::Result<()> {
IndexCache::save(&self.path, &self.line_index)?;
IndexCache::save_with_hash(&self.path, &self.line_index, self.data())?;
Ok(())
}
@@ -94,27 +104,35 @@ impl FileReader {
self.mmap = None;
if file_size > 0 {
self.mmap = Some(
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?,
);
let m =
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
let current_size = file.metadata()?.len();
self.mmap = if current_size < m.len() as u64 {
None
} else {
Some(m)
};
}
let mut reader = std::io::BufReader::new(&file);
self.line_index = LineIndex::from_reader(&mut reader).map_err(|e| CoreError::Io {
source: e,
context: "rebuilding line index on reload".into(),
})?;
self.line_index = match &self.mmap {
Some(m) => LineIndex::from_bytes(m.as_ref()),
None => LineIndex::from_bytes(&[]),
};
Ok(())
}
pub fn update_for_append(&mut self) -> Result<u64> {
pub fn update_for_append(&mut self) -> Result<AppendStatus> {
let file = std::fs::File::open(&self.path)?;
let new_size = file.metadata()?.len();
let old_size = self.mmap.as_ref().map_or(0u64, |m| m.len() as u64);
if new_size <= old_size {
return Ok(0);
if new_size < old_size {
self.reload()?;
return Ok(AppendStatus::Reloaded);
}
if new_size == old_size {
return Ok(AppendStatus::Unchanged);
}
let old_lines = self.line_index.line_count() as u64;
@@ -123,11 +141,19 @@ impl FileReader {
let mmap =
unsafe { memmap2::Mmap::map(&file) }.map_err(|e| CoreError::Mmap(e.to_string()))?;
let current_size = file.metadata()?.len();
if current_size < mmap.len() as u64 {
self.line_index = LineIndex::from_bytes(&[]);
return Ok(AppendStatus::Reloaded);
}
self.line_index
.extend_from_bytes(&mmap[old_size as usize..], old_size);
self.mmap = Some(mmap);
Ok(self.line_index.line_count() as u64 - old_lines)
Ok(AppendStatus::Appended(
self.line_index.line_count() as u64 - old_lines,
))
}
}
@@ -305,8 +331,8 @@ mod tests {
file.write_all(b"ccc\nddd\n").unwrap();
}
let new_lines = reader.update_for_append().unwrap();
assert_eq!(new_lines, 2);
let status = reader.update_for_append().unwrap();
assert_eq!(status, AppendStatus::Appended(2));
assert_eq!(reader.line_count(), 4);
assert_eq!(reader.get_line(0), Some("aaa"));
assert_eq!(reader.get_line(1), Some("bbb"));
@@ -320,8 +346,8 @@ mod tests {
let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 1);
let new_lines = reader.update_for_append().unwrap();
assert_eq!(new_lines, 0);
let status = reader.update_for_append().unwrap();
assert_eq!(status, AppendStatus::Unchanged);
assert_eq!(reader.line_count(), 1);
}
@@ -341,8 +367,11 @@ mod tests {
file.write_all(b"x\n").unwrap();
}
let new_lines = reader.update_for_append().unwrap();
assert_eq!(new_lines, 0);
let status = reader.update_for_append().unwrap();
assert_eq!(status, AppendStatus::Reloaded);
assert_eq!(reader.line_count(), 1);
assert_eq!(reader.file_size(), 2);
assert_eq!(reader.get_line(0), Some("x"));
}
#[test]
@@ -368,4 +397,160 @@ mod tests {
let idx = reader.line_index();
assert_eq!(idx.line_count(), 2);
}
// ─── TOCTOU 修复验证 ─────────────────────────────────────────────────
// open() 和 reload() 现在从 mmap 快照构建索引(from_bytes),
// 而非从独立 BufReader 读取(from_reader)。以下测试验证:
// 1. from_bytes 与 from_reader 产出完全一致的索引
// 2. 修改文件后 reload 仍返回正确内容(无残留旧状态)
// 3. 极端情况下(空文件、单行、跨块)open+reload 一致性
#[test]
fn test_open_from_bytes_matches_from_reader() {
let cases: Vec<&[u8]> = vec![
b"",
b"hello",
b"hello\n",
b"aaa\nbbb\nccc",
b"\n",
b"a\n\nb",
];
let mut large_cases = Vec::new();
for &n in &[256usize, 300, 512] {
let mut buf = Vec::new();
for i in 0..n {
buf.extend_from_slice(format!("line{}\n", i).as_bytes());
}
large_cases.push(buf);
}
let all_contents: Vec<&[u8]> = cases
.iter()
.copied()
.chain(large_cases.iter().map(Vec::as_slice))
.collect();
for content in all_contents {
let f = create_temp_file(content);
let reader = FileReader::open(f.path()).unwrap();
let from_bytes_idx = LineIndex::from_bytes(content);
let mut buf_reader = std::io::BufReader::new(content);
let from_reader_idx = LineIndex::from_reader(&mut buf_reader).unwrap();
assert_eq!(
reader.line_index().total_lines(),
from_bytes_idx.total_lines(),
"open() line_count should match from_bytes for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.total_lines(),
from_reader_idx.total_lines(),
"from_bytes should match from_reader total_lines for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.sampled_offsets(),
from_reader_idx.sampled_offsets(),
"from_bytes should match from_reader sampled_offsets for {:?}B file",
content.len()
);
assert_eq!(
from_bytes_idx.has_trailing_newline(),
from_reader_idx.has_trailing_newline(),
"from_bytes should match from_reader has_trailing_newline for {:?}B file",
content.len()
);
}
}
#[test]
fn test_reload_after_external_modify_returns_correct_content() {
let f = create_temp_file(b"aaa\nbbb\n");
let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.get_line(0), Some("aaa"));
assert_eq!(reader.get_line(1), Some("bbb"));
assert_eq!(reader.line_count(), 2);
// 外部修改:覆盖写入新内容
{
use std::io::Write;
std::fs::write(f.path(), b"xxx\nyyy\nzzz\n").unwrap();
}
reader.reload().unwrap();
assert_eq!(reader.line_count(), 3);
assert_eq!(reader.get_line(0), Some("xxx"));
assert_eq!(reader.get_line(1), Some("yyy"));
assert_eq!(reader.get_line(2), Some("zzz"));
}
#[test]
fn test_reload_after_truncate_then_rewrite_no_stale_data() {
let f = create_temp_file(b"line0\nline1\nline2\nline3\n");
let mut reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 4);
// 截断后写入更短内容
{
use std::io::Write;
let mut file = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
file.write_all(b"new\n").unwrap();
}
reader.reload().unwrap();
assert_eq!(reader.line_count(), 1);
assert_eq!(reader.get_line(0), Some("new"));
assert_eq!(reader.get_line(1), None, "old line should not be accessible");
}
#[test]
fn test_open_reload_idempotent_cross_block() {
let mut content = Vec::new();
for i in 0..600u32 {
content.extend_from_slice(format!("line{}\n", i).as_bytes());
}
let f = create_temp_file(&content);
let reader = FileReader::open(f.path()).unwrap();
let mut reloaded = FileReader::open(f.path()).unwrap();
reloaded.reload().unwrap();
assert_eq!(reader.line_count(), reloaded.line_count());
for i in 0..600 {
assert_eq!(
reader.get_line(i),
reloaded.get_line(i),
"line {i} mismatch between open and reload"
);
}
}
#[test]
fn test_open_stat_after_mmap_detects_truncation() {
let content = b"line0\nline1\nline2\nline3\n";
let f = create_temp_file(content);
let reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 4);
{
use std::io::Write;
let _ = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
}
let reader = FileReader::open(f.path()).unwrap();
assert_eq!(reader.line_count(), 0);
assert_eq!(reader.file_size(), 0);
}
}
+68 -5
View File
@@ -8,6 +8,42 @@ pub struct IndexCache;
impl IndexCache {
/// Save a `LineIndex` to disk using atomic write (write to .tmp, then rename).
///
/// The file hash is derived from `data` (the same byte slice used to build the index),
/// avoiding TOCTOU issues from re-reading the file from disk.
pub fn save_with_hash(
file_path: &Path,
index: &LineIndex,
data: &[u8],
) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
})?;
let file_hash = compute_data_hash(data);
let index_bytes = bincode::serialize(index)
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e.to_string()))?;
let mut buf = Vec::with_capacity(1 + 8 + index_bytes.len());
buf.push(CACHE_VERSION);
buf.extend_from_slice(&file_hash.to_le_bytes());
buf.extend_from_slice(&index_bytes);
let tmp_path = dest.with_extension("index.tmp");
{
let mut f = std::fs::File::create(&tmp_path)?;
f.write_all(&buf)?;
f.sync_all()?;
}
std::fs::rename(&tmp_path, &dest)?;
Ok(())
}
/// Save a `LineIndex` to disk, computing the hash by re-reading the file.
///
/// Prefer [`save_with_hash`] when the source data is already in memory
/// to avoid a TOCTOU race between indexing and hash computation.
pub fn save(file_path: &Path, index: &LineIndex) -> std::io::Result<()> {
let dest = cache_path(file_path).ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::NotFound, "cannot determine cache path")
@@ -60,6 +96,32 @@ impl IndexCache {
}
}
/// Compute a fingerprint from in-memory data, using the same algorithm as `compute_file_hash`.
/// Returns 0 for empty data.
pub fn compute_data_hash(data: &[u8]) -> u64 {
let len = data.len();
if len == 0 {
return 0;
}
let head_size = 4096.min(len);
let tail_size = 4096.min(len);
let mut hasher_state = xxhash_rust::xxh3::Xxh3::new();
hasher_state.update(&data[..head_size]);
if len > head_size + tail_size {
hasher_state.update(&data[len - tail_size..]);
} else {
// For small files the tail overlaps with the head; hash from the real tail start.
let tail_start = len.saturating_sub(tail_size);
hasher_state.update(&data[tail_start..]);
}
hasher_state.update(&(len as u64).to_le_bytes());
hasher_state.digest()
}
/// Compute a fast fingerprint of the file: xxhash of (head 4KB + tail 4KB + file size).
/// Returns 0 for empty files.
fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> {
@@ -86,11 +148,12 @@ fn compute_file_hash(file_path: &Path) -> std::io::Result<u64> {
let mut tf = std::io::BufReader::new(file);
tf.read_exact(&mut tail)?;
} else {
// File is small enough that head already covers everything;
// tail overlaps with head — just take the last tail_size bytes
let start = head.len().saturating_sub(tail_size);
tail = head[start..].to_vec();
tail.resize(tail_size, 0);
// File fits within head+tail window. Seek to read the real tail
// for correctness — approximating from head misses bytes beyond head_size.
let tail_start = file_size.saturating_sub(tail_size as u64);
let mut file = std::fs::File::open(file_path)?;
std::io::Seek::seek(&mut file, std::io::SeekFrom::Start(tail_start))?;
file.read_exact(&mut tail)?;
}
let mut hasher_state = xxhash_rust::xxh3::Xxh3::new();
+4 -4
View File
@@ -20,14 +20,14 @@ const BLOCK_SIZE: usize = 256;
pub struct LineIndex {
// 采样偏移量:每 BLOCK_SIZE 行记录一个起始字节偏移。
// sampled_offsets[i] 存储第 (i * BLOCK_SIZE) 行的字节起始位置。
sampled_offsets: Vec<u64>,
pub(crate) sampled_offsets: Vec<u64>,
// 文件总行数。
total_lines: u64,
pub(crate) total_lines: u64,
// 文件最后一个字节是否是换行符 \n。
#[allow(dead_code)]
has_trailing_newline: bool,
pub(crate) has_trailing_newline: bool,
}
impl LineIndex {
@@ -233,7 +233,7 @@ impl LineIndex {
}
let block = idx / BLOCK_SIZE;
let offset_in_block = idx % BLOCK_SIZE;
let mut pos = self.sampled_offsets[block] as usize;
let mut pos = (*self.sampled_offsets.get(block)?) as usize;
for _ in 0..offset_in_block {
match memchr::memchr(b'\n', &data[pos..]) {
Some(rel) => pos = pos + rel + 1,
+212 -71
View File
@@ -1,9 +1,10 @@
use std::cell::RefCell;
use std::fmt;
use std::io::BufRead;
use std::path::{Path, PathBuf};
use crate::error::{CoreError, Result};
use crate::io::file_reader::FileReader;
use crate::io::file_reader::{AppendStatus, FileReader};
use crate::io::index_cache::IndexCache;
use crate::io::line_index::LineIndex;
use crate::io::line_sampler::sample_line_count;
@@ -251,7 +252,7 @@ pub fn spawn_indexer(
return;
}
};
let file_size = match file.metadata() {
let target_len = match file.metadata() {
Ok(m) => m.len(),
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
@@ -262,11 +263,17 @@ pub fn spawn_indexer(
}
};
let mmap = if file_size == 0 {
None
} else {
match unsafe { memmap2::Mmap::map(&file) } {
Ok(m) => Some(m),
let mut buf_reader = std::io::BufReader::with_capacity(64 * 1024, file);
let mut sampled_offsets: Vec<u64> = vec![0];
let mut next_line_idx: usize = 1;
let mut newline_count: usize = 0;
let mut chunk_offset: u64 = 0;
let mut last_byte: Option<u8> = None;
let mut bytes_since_check: usize = 0;
loop {
let buf = match buf_reader.fill_buf() {
Ok(b) => b,
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
generation,
@@ -274,31 +281,35 @@ pub fn spawn_indexer(
});
return;
}
};
if buf.is_empty() {
break;
}
};
let data = mmap.as_deref().unwrap_or(&[]);
if !data.is_empty() {
let mut newline_count: usize = 0;
let mut chars_since_check: usize = 0;
let mut prev_pos: usize = 0;
for pos in memchr::memchr_iter(b'\n', data) {
chars_since_check += pos - prev_pos;
prev_pos = pos;
if chars_since_check >= 1_000_000 {
chars_since_check = 0;
if cancel_rx.try_recv().is_ok() {
return;
}
}
if let Some(&b) = buf.last() {
last_byte = Some(b);
}
for pos in memchr::memchr_iter(b'\n', buf) {
newline_count += 1;
if next_line_idx.is_multiple_of(256) {
sampled_offsets.push(chunk_offset + pos as u64 + 1);
}
next_line_idx += 1;
}
if newline_count % 256_000 == 0 {
let percent = (pos as f64 / file_size as f64) * 100.0;
let consumed = buf.len();
bytes_since_check += consumed;
chunk_offset += consumed as u64;
buf_reader.consume(consumed);
if bytes_since_check >= 1_000_000 {
bytes_since_check = 0;
if cancel_rx.try_recv().is_ok() {
return;
}
if target_len > 0 {
let percent = (chunk_offset as f64 / target_len as f64) * 100.0;
let _ = tx.send(IndexerMessage::Progress {
generation,
percent,
@@ -312,9 +323,64 @@ pub fn spawn_indexer(
return;
}
let line_index = LineIndex::from_bytes(data);
let line_index = if chunk_offset == 0 {
LineIndex {
sampled_offsets: vec![],
total_lines: 0,
has_trailing_newline: false,
}
} else {
let has_trailing_newline = last_byte == Some(b'\n') && newline_count > 0;
let total_lines = if has_trailing_newline && newline_count > 0 {
newline_count as u64
} else {
(1 + newline_count) as u64
};
let _ = IndexCache::save(&path, &line_index);
if has_trailing_newline && newline_count > 0 {
let trailing_line_idx = newline_count;
if trailing_line_idx.is_multiple_of(256) {
sampled_offsets.pop();
}
}
LineIndex {
sampled_offsets,
total_lines,
has_trailing_newline,
}
};
let mmap = if target_len == 0 {
None
} else {
match std::fs::File::open(&path) {
Ok(mmap_file) => match unsafe { memmap2::Mmap::map(&mmap_file) } {
Ok(m) => match mmap_file.metadata() {
Ok(metadata) if metadata.len() >= m.len() as u64 => Some(m),
Ok(_) | Err(_) => None,
},
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
generation,
message: e.to_string(),
});
return;
}
},
Err(e) => {
let _ = tx.send(IndexerMessage::Error {
generation,
message: e.to_string(),
});
return;
}
}
};
if let Some(data) = mmap.as_deref() {
let _ = IndexCache::save_with_hash(&path, &line_index, data);
}
let reader = FileReader::from_parts(path, mmap, line_index);
@@ -355,26 +421,37 @@ pub fn spawn_visual_height_rebuild(
Err(_) => return,
};
let file_size = match file.metadata() {
Ok(m) => m.len(),
Err(_) => return,
};
let mut reader = std::io::BufReader::with_capacity(64 * 1024, file);
let mut visual_heights = Vec::with_capacity(line_index.line_count());
let mut line_buf = Vec::new();
let mmap = if file_size == 0 {
None
} else {
match unsafe { memmap2::Mmap::map(&file) } {
Ok(m) => Some(m),
loop {
if cancel_rx.try_recv().is_ok() {
return;
}
line_buf.clear();
match std::io::BufRead::read_until(&mut reader, b'\n', &mut line_buf) {
Ok(0) => break,
Ok(_) => {
let line_text = std::str::from_utf8(&line_buf)
.ok()
.map(|s| s.trim_end_matches(['\r', '\n']))
.unwrap_or("");
visual_heights.push(compute_line_visual_height(
line_text,
terminal_width,
json_format,
));
}
Err(_) => return,
}
};
}
if cancel_rx.try_recv().is_ok() {
if visual_heights.len() != line_index.line_count() {
return;
}
let reader = FileReader::from_parts(path, mmap, line_index);
let visual_heights = compute_visual_heights(&reader, terminal_width, json_format);
let index =
VisualHeightIndex::build(&visual_heights).with_params(json_format, terminal_width);
@@ -389,6 +466,11 @@ pub fn spawn_visual_height_rebuild(
/// Maximum bytes to scan during initial open for the Sampling state.
const INITIAL_SCAN_BYTES: usize = 64 * 1024;
/// Maximum number of additional lines to scan beyond what's already cached
/// in a single `get_line()` call. Prevents O(N) blocking when the user
/// jumps far ahead (e.g. `G` to end-of-file) during the Loading state.
const SCAN_AHEAD_LIMIT: usize = 10_000;
pub struct ProgressiveFileReader {
path: PathBuf,
pub state: ReaderState,
@@ -512,32 +594,23 @@ impl ProgressiveFileReader {
let mut newlines = scanned_newlines.borrow_mut();
let mut up_to = scanned_up_to.borrow_mut();
// We need `idx + 1` newlines to have `idx` lines available.
// Line i starts after the i-th newline (or at byte 0 for line 0)
// and ends at the (i+1)-th newline (or end of file).
// So to return line `idx`, we need at least `idx + 1` newline positions
// (the idx-th newline marks end of line idx-1/start of line idx,
// and the (idx+1)-th newline marks end of line idx).
// Actually: line 0 starts at byte 0, ends at newline[0].
// line 1 starts at newline[0]+1, ends at newline[1].
// line i starts at newline[i-1]+1, ends at newline[i].
// So to serve line idx, we need newline positions up to index idx.
let scan_limit = newlines.len() + SCAN_AHEAD_LIMIT;
// Extend scan if needed
while newlines.len() <= idx && *up_to < mmap_data.len() {
// Extend scan if needed, but stop at scan_limit to avoid O(N) blocking
while newlines.len() <= idx
&& newlines.len() < scan_limit
&& *up_to < mmap_data.len()
{
let remaining = &mmap_data[*up_to..];
if let Some(rel_pos) = memchr::memchr(b'\n', remaining) {
newlines.push(*up_to + rel_pos);
*up_to += rel_pos + 1;
} else {
// No more newlines; rest of file is the last line
*up_to = mmap_data.len();
break;
}
}
// Check if line idx is beyond what's available
// If idx == newlines.len(), it's the last line (after last newline, no trailing \n)
if idx > newlines.len() {
return None;
}
@@ -660,10 +733,10 @@ impl ProgressiveFileReader {
}
}
pub fn update_for_append(&mut self) -> Result<u64> {
pub fn update_for_append(&mut self) -> Result<AppendStatus> {
match &mut self.state {
ReaderState::Ready { reader, .. } => reader.update_for_append(),
_ => Ok(0),
_ => Ok(AppendStatus::Unchanged),
}
}
@@ -1197,18 +1270,20 @@ mod tests {
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(temp.path().to_path_buf(), 1, 80, false, cancel_rx);
let msg = rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap();
match msg {
IndexerMessage::Complete {
visual_height_index,
..
} => {
let idx = visual_height_index.expect("should have visual height index");
assert_eq!(idx.visual_height_of_line(0), 1);
assert_eq!(idx.visual_height_of_line(1), 1);
loop {
match rx.recv_timeout(std::time::Duration::from_secs(10)).unwrap() {
IndexerMessage::Progress { .. } => continue,
IndexerMessage::Complete {
visual_height_index,
..
} => {
let idx = visual_height_index.expect("should have visual height index");
assert_eq!(idx.visual_height_of_line(0), 1);
assert_eq!(idx.visual_height_of_line(1), 1);
break;
}
other => panic!("expected Complete, got {:?}", other),
}
other => panic!("expected Complete, got {:?}", other),
}
}
@@ -1262,6 +1337,72 @@ mod tests {
idx.extend_from_heights(&[1, 2, 3]);
assert_eq!(idx.total_visual_rows(), 6);
assert_eq!(idx.line_count(), 3);
}
#[test]
fn test_spawn_indexer_file_truncated_during_scan() {
let mut content = Vec::new();
for i in 0..100_000 {
writeln!(content, "line number {:08}", i).unwrap();
}
let f = create_temp_file(&content);
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_indexer(f.path().to_path_buf(), 1, 80, false, cancel_rx);
{
use std::io::Write;
let _ = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
}
let result = rx.recv_timeout(std::time::Duration::from_secs(10));
match result {
Ok(IndexerMessage::Complete { reader, .. }) => {
assert!(reader.line_count() <= 100_000);
}
Ok(IndexerMessage::Error { .. }) => {}
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
other => panic!("unexpected: {:?}", other),
}
}
#[test]
fn test_spawn_visual_height_rebuild_line_count_mismatch_discards() {
let content = b"line0\nline1\nline2\n";
let f = create_temp_file(content);
let data = std::fs::read(f.path()).unwrap();
let index = LineIndex::from_bytes(&data);
IndexCache::save(f.path(), &index).unwrap();
{
use std::io::Write;
let mut file = std::fs::OpenOptions::new()
.write(true)
.truncate(true)
.open(f.path())
.unwrap();
file.write_all(b"only_one_line\n").unwrap();
}
let (_cancel_tx, cancel_rx) = crossbeam_channel::bounded(1);
let rx = spawn_visual_height_rebuild(
f.path().to_path_buf(),
1,
80,
false,
cancel_rx,
);
let result = rx.recv_timeout(std::time::Duration::from_secs(5));
match result {
Err(crossbeam_channel::RecvTimeoutError::Disconnected) => {}
Err(crossbeam_channel::RecvTimeoutError::Timeout) => {}
Ok(_) => panic!("should have been discarded due to line count mismatch"),
}
}
}
+435
View File
@@ -0,0 +1,435 @@
// NOTE: This module is implemented and tested but not yet integrated into
// the production read path (FileReader uses mmap for line access).
// It is kept as a building block for a future pread-based reader that
// would avoid the SIGBUS risk of mmap.
// ─── read_cache.rs ─────────────────────────────────────────────────────────
// 16-slot LRU read cache with 4KB page-aligned keys.
// Reduces syscalls by caching recently-read 4KB blocks.
// Cross-block reads are handled via a spill buffer (not cached).
// ──────────────────────────────────────────────────────────────────────────
use std::fs::File;
use std::io;
use std::os::unix::fs::FileExt;
const LRU_SLOTS: usize = 16;
pub const BLOCK_ALIGN: usize = 4096;
struct CacheSlot {
buf: Vec<u8>,
block_offset: u64,
len: usize,
last_access: u64,
}
pub struct LruReadCache {
slots: [CacheSlot; LRU_SLOTS],
spill_buf: Vec<u8>,
spill_len: usize,
tick: u64,
}
pub type ReadCache = LruReadCache;
impl Default for LruReadCache {
fn default() -> Self {
Self {
slots: std::array::from_fn(|_| CacheSlot {
buf: vec![0u8; BLOCK_ALIGN],
block_offset: 0,
len: 0,
last_access: 0,
}),
spill_buf: Vec::new(),
spill_len: 0,
tick: 0,
}
}
}
impl LruReadCache {
pub fn new() -> Self {
Self::default()
}
/// Read `len` bytes starting at `offset`. Returns a slice into the cache
/// on a hit, or fills a cache slot on a miss. Cross-block reads go through
/// the spill buffer and are not cached.
pub fn get(&mut self, file: &File, offset: u64, len: usize) -> io::Result<&[u8]> {
let aligned_key = offset & !(BLOCK_ALIGN as u64 - 1);
let request_end = offset.saturating_add(len as u64);
let block_end = aligned_key + BLOCK_ALIGN as u64;
if request_end > block_end {
self.spill_buf.resize(len, 0);
let bytes_read = file.read_at(&mut self.spill_buf[..len], offset)?;
if bytes_read == 0 {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
}
if bytes_read < len {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
}
self.spill_len = len;
return Ok(&self.spill_buf[..len]);
}
let hit_idx = self.slots.iter().position(|slot| {
slot.block_offset == aligned_key && request_end <= slot.block_offset + slot.len as u64
});
if let Some(idx) = hit_idx {
self.slots[idx].last_access = self.tick;
self.tick += 1;
let start = (offset - self.slots[idx].block_offset) as usize;
return Ok(&self.slots[idx].buf[start..start + len]);
}
let mut evict_idx = 0;
let mut min_access = self.slots[0].last_access;
for (i, slot) in self.slots.iter().enumerate() {
if slot.last_access < min_access {
min_access = slot.last_access;
evict_idx = i;
}
}
let slot = &mut self.slots[evict_idx];
let bytes_read = file.read_at(&mut slot.buf, aligned_key)?;
// Note: get(file, 0, 0) on an empty file now returns Err (old code returned Ok(&[])).
// No callers pass len == 0, so this is a safe semantic change.
if bytes_read == 0 {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "read 0 bytes"));
}
slot.block_offset = aligned_key;
slot.len = bytes_read;
slot.last_access = self.tick;
self.tick += 1;
if request_end > aligned_key + bytes_read as u64 {
return Err(io::Error::new(io::ErrorKind::UnexpectedEof, "short read"));
}
let start = (offset - slot.block_offset) as usize;
Ok(&slot.buf[start..start + len])
}
/// Invalidate all cache slots and the spill buffer.
pub fn clear(&mut self) {
for slot in &mut self.slots {
slot.len = 0;
}
self.spill_len = 0;
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use tempfile::NamedTempFile;
fn make_file(data: &[u8]) -> NamedTempFile {
let mut f = NamedTempFile::new().unwrap();
f.write_all(data).unwrap();
f.flush().unwrap();
f
}
#[test]
fn cache_hit_returns_same_data() {
// Read the same range twice — second read should be a cache hit.
let f = make_file(b"Hello, World! This is a test of the cache.");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let first = cache.get(&file, 0, 13).unwrap().to_vec();
let second = cache.get(&file, 0, 13).unwrap().to_vec();
assert_eq!(first, second);
assert_eq!(&first, b"Hello, World!");
}
#[test]
fn cache_miss_reads_correct_data() {
// Two non-overlapping ranges — both must be misses but return correct data.
let data = b"0123456789ABCDEFGHIJ";
let f = make_file(data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let a = cache.get(&file, 0, 10).unwrap().to_vec();
assert_eq!(&a, b"0123456789");
let b = cache.get(&file, 10, 10).unwrap().to_vec();
assert_eq!(&b, b"ABCDEFGHIJ");
}
#[test]
fn cross_block_read_uses_spill_buffer() {
// Read spanning a 4KB block boundary uses spill buffer, not cache.
let data = vec![0xABu8; 8192];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// First, cache block 0.
let a = cache.get(&file, 0, 100).unwrap().to_vec();
assert_eq!(a, vec![0xABu8; 100]);
// Now read spanning the boundary: offset=4000, len=200 spans [0,4096) and [4096,8192).
let b = cache.get(&file, 4000, 200).unwrap().to_vec();
assert_eq!(b, vec![0xABu8; 200]);
}
#[test]
fn empty_file_read_fails() {
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Reading 1 byte from empty file should fail.
let result = cache.get(&file, 0, 1);
assert!(result.is_err());
}
#[test]
fn clear_invalidates_cache() {
let data = b"original data here";
let f = make_file(data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Populate cache.
let first = cache.get(&file, 0, 10).unwrap().to_vec();
assert_eq!(&first, b"original d");
// Invalidate.
cache.clear();
// After clear, reading same range should still work (re-reads from file).
let after = cache.get(&file, 0, 10).unwrap().to_vec();
assert_eq!(&after, b"original d");
}
// ─── New LRU-specific tests ───────────────────────────────────────────
#[test]
fn lru_multi_block_hit() {
// Read 3 different aligned blocks, verify re-reading each hits cache.
let data = vec![0u8; BLOCK_ALIGN * 4];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Write distinct patterns to each block.
drop(file);
{
use std::io::Seek;
let mut f2 = std::fs::OpenOptions::new()
.write(true)
.open(f.path())
.unwrap();
f2.seek(std::io::SeekFrom::Start(0)).unwrap();
f2.write_all(&[1u8; BLOCK_ALIGN]).unwrap();
f2.seek(std::io::SeekFrom::Start(BLOCK_ALIGN as u64))
.unwrap();
f2.write_all(&[2u8; BLOCK_ALIGN]).unwrap();
f2.seek(std::io::SeekFrom::Start((BLOCK_ALIGN * 2) as u64))
.unwrap();
f2.write_all(&[3u8; BLOCK_ALIGN]).unwrap();
}
let file = File::open(f.path()).unwrap();
let block0 = cache.get(&file, 0, 16).unwrap().to_vec();
let block1 = cache.get(&file, BLOCK_ALIGN as u64, 16).unwrap().to_vec();
let block2 = cache
.get(&file, (BLOCK_ALIGN * 2) as u64, 16)
.unwrap()
.to_vec();
assert_eq!(block0, vec![1u8; 16]);
assert_eq!(block1, vec![2u8; 16]);
assert_eq!(block2, vec![3u8; 16]);
// Re-read — should hit cache and return same data.
let block0_again = cache.get(&file, 0, 16).unwrap().to_vec();
let block1_again = cache.get(&file, BLOCK_ALIGN as u64, 16).unwrap().to_vec();
let block2_again = cache
.get(&file, (BLOCK_ALIGN * 2) as u64, 16)
.unwrap()
.to_vec();
assert_eq!(block0_again, block0);
assert_eq!(block1_again, block1);
assert_eq!(block2_again, block2);
}
#[test]
fn lru_eviction_order() {
// Fill all 16 slots, re-access slot 0, add 17th block,
// verify slot at offset 4096 (slot 1) evicted, not slot 0.
let data = vec![0u8; BLOCK_ALIGN * 20];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Fill all 16 slots with blocks 0..16.
for i in 0..16u64 {
cache.get(&file, i * BLOCK_ALIGN as u64, 1).unwrap();
}
// Re-access block 0 so it's not the LRU.
cache.get(&file, 0, 1).unwrap();
// Add 17th block — should evict block 1 (offset 4096), which is the oldest
// since block 0 was re-accessed.
cache.get(&file, 16 * BLOCK_ALIGN as u64, 1).unwrap();
// Reading block 1 (offset 4096) should be a miss (evicted).
// We verify by checking the cache slots: block 1 should not be cached.
let has_block1 = cache
.slots
.iter()
.any(|s| s.block_offset == BLOCK_ALIGN as u64 && s.len > 0);
assert!(!has_block1, "block 1 should have been evicted");
// Block 0 should still be cached.
let has_block0 = cache.slots.iter().any(|s| s.block_offset == 0 && s.len > 0);
assert!(has_block0, "block 0 should still be cached");
}
#[test]
fn lru_clear_all_slots() {
// Fill 3+ slots, call clear(), verify subsequent reads all miss.
let data = vec![0x42u8; BLOCK_ALIGN * 4];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Fill 3 slots.
cache.get(&file, 0, 1).unwrap();
cache.get(&file, BLOCK_ALIGN as u64, 1).unwrap();
cache.get(&file, (BLOCK_ALIGN * 2) as u64, 1).unwrap();
cache.clear();
// All slots should have len == 0.
for slot in &cache.slots {
assert_eq!(slot.len, 0);
}
assert_eq!(cache.spill_len, 0);
// Re-read should still work (reads from file).
let val = cache.get(&file, 0, 1).unwrap();
assert_eq!(val[0], 0x42);
}
#[test]
fn lru_aligned_keys() {
// offset=100 and offset=200 both align to block 0 — should hit same slot.
let data = vec![0xEEu8; BLOCK_ALIGN];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let a = cache.get(&file, 100, 10).unwrap().to_vec();
assert_eq!(a, vec![0xEEu8; 10]);
let b = cache.get(&file, 200, 10).unwrap().to_vec();
assert_eq!(b, vec![0xEEu8; 10]);
// Both should be served from the same cache slot (block 0).
let slot_count = cache
.slots
.iter()
.filter(|s| s.block_offset == 0 && s.len > 0)
.count();
assert_eq!(slot_count, 1, "only one slot should hold block 0");
}
#[test]
fn lru_cross_block_uses_spill_buffer() {
// File [0xAA×4096, 0xBB×4096], get(file, 4090, 20) → [0xAA×6, 0xBB×14].
let mut data = vec![0xAAu8; BLOCK_ALIGN];
data.extend_from_slice(&vec![0xBBu8; BLOCK_ALIGN]);
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, 4090, 20).unwrap().to_vec();
assert_eq!(&result[..6], &[0xAAu8; 6]);
assert_eq!(&result[6..], &[0xBBu8; 14]);
// Verify no cache slot holds block 0 or block 1.
for slot in &cache.slots {
assert!(
slot.len == 0,
"cross-block data should not be cached in slots"
);
}
}
#[test]
fn lru_partial_last_block() {
// 5000 byte file, get(file, 4096, 904) reads last 904 bytes,
// then get(file, 4096, 100) hits cache.
let data = vec![0x77u8; 5000];
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let first = cache.get(&file, 4096, 904).unwrap().to_vec();
assert_eq!(first.len(), 904);
assert_eq!(first, vec![0x77u8; 904]);
// Second read of 100 bytes from same block should hit cache.
let second = cache.get(&file, 4096, 100).unwrap().to_vec();
assert_eq!(second.len(), 100);
assert_eq!(second, vec![0x77u8; 100]);
}
#[test]
fn lru_short_file_overread() {
// 1 byte file, get(file, 0, 100) → Err (not panic).
let f = make_file(b"X");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, 0, 100);
assert!(result.is_err());
}
#[test]
fn lru_empty_file_returns_error() {
// Empty file, get(file, 0, 1) → Err.
let f = make_file(b"");
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
let result = cache.get(&file, 0, 1);
assert!(result.is_err());
}
#[test]
fn long_line_spans_multiple_blocks() {
// Create file with line >4KB: b'A'×4090 + "\n" + b'B'×4090 + "\n"
let mut data = vec![b'A'; 4090];
data.push(b'\n');
data.extend_from_slice(&vec![b'B'; 4090]);
data.push(b'\n');
let f = make_file(&data);
let file = File::open(f.path()).unwrap();
let mut cache = ReadCache::new();
// Read the first line: offset 0, len 4091 (fits in block 0 since 4091 <= 4096).
let line1 = cache.get(&file, 0, 4091).unwrap().to_vec();
assert_eq!(&line1[..4090], &[b'A'; 4090]);
assert_eq!(line1[4090], b'\n');
// Read the second line: offset 4091, len 4091 (starts in block 0, ends in block 1 — cross-block).
let line2 = cache.get(&file, 4091, 4091).unwrap().to_vec();
assert_eq!(&line2[..4090], &[b'B'; 4090]);
assert_eq!(line2[4090], b'\n');
}
}
+235 -1
View File
@@ -1,2 +1,236 @@
pub struct FileWatcher {/* TODO */}
use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex};
use crossbeam_channel::{bounded, Receiver, Sender};
use notify::{Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher};
use crate::error::Result;
// ─── FileEvent ──────────────────────────────────────────────────────────────
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum FileEvent {
Appended { new_size: u64 },
Truncated { new_size: u64 },
Rotated { new_inode: u64 },
Removed,
}
// ─── get_inode ──────────────────────────────────────────────────────────────
#[cfg(unix)]
fn get_inode(path: &Path) -> std::io::Result<u64> {
use std::os::unix::fs::MetadataExt;
Ok(std::fs::metadata(path)?.ino())
}
#[cfg(not(unix))]
fn get_inode(_path: &Path) -> std::io::Result<u64> {
Ok(0) // rotation detection not supported on non-Unix
}
// ─── WatchState ─────────────────────────────────────────────────────────────
struct WatchState {
last_size: u64,
last_inode: u64,
}
// ─── FileWatcher ────────────────────────────────────────────────────────────
pub struct FileWatcher {
rx: Receiver<FileEvent>,
_watcher: RecommendedWatcher,
}
impl FileWatcher {
pub fn watch(path: &Path) -> Result<Self> {
let (tx, rx): (Sender<FileEvent>, Receiver<FileEvent>) = bounded(100);
let initial_size = std::fs::metadata(path)?.len();
let initial_inode = get_inode(path).unwrap_or(0);
let state = Arc::new(Mutex::new(WatchState {
last_size: initial_size,
last_inode: initial_inode,
}));
let watch_path: PathBuf = path.to_path_buf();
let mut watcher =
notify::recommended_watcher(move |res: std::result::Result<Event, notify::Error>| {
let event = match res {
Ok(e) => e,
Err(_) => return,
};
match event.kind {
EventKind::Modify(_) | EventKind::Create(_) | EventKind::Any => {}
EventKind::Remove(_) => {
let _ = tx.try_send(FileEvent::Removed);
return;
}
_ => return,
}
if !event.paths.iter().any(|p| p == &watch_path) {
return;
}
let current_inode = get_inode(&watch_path).unwrap_or(0);
let current_size = std::fs::metadata(&watch_path).map(|m| m.len()).unwrap_or(0);
let mut st = state.lock().unwrap_or_else(|poison| {
// Recover from poisoned mutex — state only tracks last_size
// and last_inode for event dedup. Stale values at worst
// cause a duplicate event, which is harmless.
poison.into_inner()
});
if current_inode != 0 && st.last_inode != 0 && current_inode != st.last_inode {
let _ = tx.try_send(FileEvent::Rotated {
new_inode: current_inode,
});
st.last_inode = current_inode;
st.last_size = current_size;
} else if current_size > st.last_size {
let _ = tx.try_send(FileEvent::Appended {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
} else if current_size < st.last_size {
let _ = tx.try_send(FileEvent::Truncated {
new_size: current_size,
});
st.last_size = current_size;
st.last_inode = current_inode;
}
})?;
watcher.watch(path, RecursiveMode::NonRecursive)?;
Ok(Self {
rx,
_watcher: watcher,
})
}
pub fn try_recv(&self) -> Option<FileEvent> {
self.rx.try_recv().ok()
}
}
// ─── SmartFollow ────────────────────────────────────────────────────────────
pub struct SmartFollow {/* TODO */}
// ─── Tests ──────────────────────────────────────────────────────────────────
#[cfg(test)]
mod tests {
use super::*;
use std::io::Write;
use std::thread;
use std::time::Duration;
fn collect_events(watcher: &FileWatcher, timeout_ms: u64) -> Vec<FileEvent> {
let mut events = Vec::new();
let deadline = std::time::Instant::now() + Duration::from_millis(timeout_ms);
while std::time::Instant::now() < deadline {
if let Some(e) = watcher.try_recv() {
events.push(e);
}
thread::sleep(Duration::from_millis(50));
}
// final drain
while let Some(e) = watcher.try_recv() {
events.push(e);
}
events
}
#[test]
fn test_watcher_append() {
let dir = tempfile::tempdir().expect("create temp dir");
let path = dir.path().join("test.log");
std::fs::write(&path, b"hello\n").expect("write initial");
let watcher = FileWatcher::watch(&path).expect("start watcher");
{
let mut f = std::fs::OpenOptions::new()
.append(true)
.open(&path)
.expect("open for append");
f.write_all(b"world\n").expect("append data");
}
let events = collect_events(&watcher, 1000);
let appended: Vec<&FileEvent> = events
.iter()
.filter(|e| matches!(e, FileEvent::Appended { .. }))
.collect();
assert!(
!appended.is_empty(),
"should detect append event, got: {events:?}"
);
}
#[test]
fn test_watcher_truncate() {
let dir = tempfile::tempdir().expect("create temp dir");
let path = dir.path().join("test.log");
std::fs::write(&path, b"hello world this is a long line\n").expect("write initial");
let watcher = FileWatcher::watch(&path).expect("start watcher");
// File::create truncates to 0 bytes
let _ = std::fs::File::create(&path).expect("truncate file");
let events = collect_events(&watcher, 1000);
let truncated: Vec<&FileEvent> = events
.iter()
.filter(|e| matches!(e, FileEvent::Truncated { .. }))
.collect();
assert!(
!truncated.is_empty(),
"should detect truncate event, got: {events:?}"
);
}
#[test]
fn test_watcher_idle() {
let dir = tempfile::tempdir().expect("create temp dir");
let path = dir.path().join("idle.log");
std::fs::write(&path, b"static content\n").expect("write");
let watcher = FileWatcher::watch(&path).expect("start watcher");
thread::sleep(Duration::from_millis(300));
assert_eq!(watcher.try_recv(), None, "no events on idle file");
}
#[test]
fn test_get_inode() {
let dir = tempfile::tempdir().expect("create temp dir");
let path = dir.path().join("inode_test.txt");
std::fs::write(&path, b"test").expect("write file");
let inode = get_inode(&path).expect("get inode");
#[cfg(unix)]
assert!(inode > 0, "inode should be positive on Unix");
#[cfg(not(unix))]
assert_eq!(inode, 0, "inode should be 0 on non-Unix");
}
#[test]
fn test_file_event_equality() {
let a = FileEvent::Appended { new_size: 100 };
let b = FileEvent::Appended { new_size: 100 };
assert_eq!(a, b);
let c = FileEvent::Truncated { new_size: 0 };
assert_ne!(a, c);
let d = FileEvent::Rotated { new_inode: 42 };
assert_ne!(a, d);
}
}
+1
View File
@@ -10,6 +10,7 @@ clap.workspace = true
anyhow.workspace = true
log-viewer-core.workspace = true
serde_json.workspace = true
crossbeam-channel.workspace = true
[dev-dependencies]
tempfile = { workspace = true }
+1688 -218
View File
File diff suppressed because it is too large Load Diff
+75 -21
View File
@@ -1,3 +1,6 @@
use std::time::Duration;
use anyhow::Context;
use clap::Parser;
mod app;
@@ -11,42 +14,93 @@ struct Cli {
files: Vec<String>,
}
// ── RAII terminal guard ────────────────────────────────────────────
//
// Holds the ratatui Terminal and guarantees terminal state restoration
// (disable raw mode, leave alternate screen, show cursor) on **every**
// exit path: normal return, `?` error propagation, and panic unwind.
//
// IMPORTANT: `Drop` does **not** run on `std::process::exit` or
// `panic = "abort"`. Therefore we must never call `process::exit`
// inside the guarded scope — use `?` to return `Err` instead.
type Backend = ratatui::backend::CrosstermBackend<std::io::Stdout>;
struct TerminalGuard {
terminal: ratatui::Terminal<Backend>,
}
impl TerminalGuard {
/// Enable raw mode, enter alternate screen, and create the terminal.
/// On partial failure, rolls back any steps that already succeeded.
fn enter() -> anyhow::Result<Self> {
crossterm::terminal::enable_raw_mode()
.map_err(|e| anyhow::anyhow!("enable_raw_mode: {e}"))?;
let mut stdout = std::io::stdout();
if let Err(e) = crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen) {
let _ = crossterm::terminal::disable_raw_mode();
return Err(anyhow::anyhow!("EnterAlternateScreen: {e}"));
}
let backend = ratatui::backend::CrosstermBackend::new(stdout);
match ratatui::Terminal::new(backend) {
Ok(terminal) => Ok(Self { terminal }),
Err(e) => {
// Roll back: leave alternate screen + disable raw mode
let _ = crossterm::execute!(
std::io::stdout(),
crossterm::terminal::LeaveAlternateScreen
);
let _ = crossterm::terminal::disable_raw_mode();
Err(anyhow::anyhow!("Terminal::new: {e}"))
}
}
}
fn terminal(&mut self) -> &mut ratatui::Terminal<Backend> {
&mut self.terminal
}
}
impl Drop for TerminalGuard {
fn drop(&mut self) {
// Best-effort cleanup; suppress errors (Drop must not panic).
let _ = crossterm::terminal::disable_raw_mode();
let _ = crossterm::execute!(
self.terminal.backend_mut(),
crossterm::terminal::LeaveAlternateScreen
);
let _ = self.terminal.show_cursor();
}
}
// ── main ───────────────────────────────────────────────────────────
fn main() -> anyhow::Result<()> {
let cli = Cli::parse();
crossterm::terminal::enable_raw_mode()?;
let mut stdout = std::io::stdout();
crossterm::execute!(stdout, crossterm::terminal::EnterAlternateScreen)?;
let backend = ratatui::backend::CrosstermBackend::new(stdout);
let mut terminal = ratatui::Terminal::new(backend)?;
let mut guard = TerminalGuard::enter()?;
let mut app = app::App::new();
app.color_config = log_viewer_core::config::ColorConfig::load();
if let Some(file) = cli.files.first()
&& let Err(e) = app.load_file(file)
{
eprintln!("Error loading file: {e}");
std::process::exit(1);
if let Some(file) = cli.files.first() {
app.load_file(file)
.with_context(|| format!("loading file {file}"))?;
}
while !app.should_quit {
terminal.draw(|frame| ui::render(frame, &mut app))?;
if crossterm::event::poll(std::time::Duration::from_millis(100))? {
app.poll_background_indexer();
app.poll_file_watcher();
guard.terminal().draw(|frame| ui::render(frame, &mut app))?;
if crossterm::event::poll(Duration::from_millis(100))? {
match crossterm::event::read()? {
crossterm::event::Event::Key(key) => app.handle_key(key),
crossterm::event::Event::Resize(_w, _h) => {}
crossterm::event::Event::Resize(_, _) => {}
_ => {}
}
}
}
crossterm::terminal::disable_raw_mode()?;
crossterm::execute!(
terminal.backend_mut(),
crossterm::terminal::LeaveAlternateScreen
)?;
terminal.show_cursor()?;
Ok(())
}
+225 -36
View File
@@ -1,4 +1,4 @@
use ratatui::style::{Color, Style};
use ratatui::style::{Color, Modifier, Style};
use ratatui::text::{Line, Span};
use crate::app::{App, AppMode};
@@ -42,6 +42,10 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
// ── Title bar ──────────────────────────────────────────────────
let title_text = if app.mode == AppMode::Settings {
" Color Settings".to_string()
} else if app.is_loading() {
let name = app.file_name().unwrap_or("unknown");
let pct = app.loading_progress().map_or(0, |p| p as usize);
format!(" {} [Loading... {}%]", name, pct)
} else if app.is_loaded() {
let name = app.file_name().unwrap_or("unknown");
let cursor_display = if app.total_lines() == 0 {
@@ -61,6 +65,22 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
// ── Content area ───────────────────────────────────────────────
if app.mode == AppMode::Settings {
render_settings(frame, app, outer[1]);
} else if app.is_error() {
let msg = app.error_message().unwrap_or_default();
let error_lines = vec![
Line::from(""),
Line::from(""),
Line::styled(
format!(" Error: {}", msg),
Style::default().fg(Color::Red).add_modifier(Modifier::BOLD),
),
Line::from(""),
Line::styled(" Press q to quit", Style::default().fg(Color::DarkGray)),
];
frame.render_widget(Paragraph::new(error_lines).centered(), outer[1]);
} else if app.is_loading() {
// Show content from sampling during loading
render_content(frame, app, outer[1]);
} else if !app.is_loaded() {
frame.render_widget(Paragraph::new(" No file loaded"), outer[1]);
} else {
@@ -70,6 +90,30 @@ pub fn render(frame: &mut ratatui::Frame, app: &mut App) {
// ── Status bar ─────────────────────────────────────────────────
let status_text = if app.mode == AppMode::Settings {
" j/k:navigate ←/→:change 1-8:jump Enter:save Esc:cancel"
} else if app.is_error() {
" Press q to quit"
} else if app.is_loading() {
let pct = app.loading_progress().map_or(0, |p| p as usize);
let est = app
.estimated_lines()
.map_or("?".to_string(), |e| format!("~{}", e));
let name = app.file_name().unwrap_or("unknown");
let status = format!(
" Indexing... {}% | {} lines | {} | j/k:scroll q:quit",
pct, est, name
);
frame.render_widget(
Paragraph::new(status).style(Style::default().fg(Color::Yellow)),
outer[2],
);
return;
} else if app.is_loaded() {
let name = app.file_name().unwrap_or("unknown");
let total = app.total_lines();
let cursor_display = if total == 0 { 0 } else { app.cursor_line + 1 };
let status = format!(" {} [{}/{}] | j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit", name, cursor_display, total);
frame.render_widget(Paragraph::new(status), outer[2]);
return;
} else {
" j/k:scroll d/u:half-page f/b:page G/gg:jump Tab:format S:settings q:quit"
};
@@ -165,8 +209,11 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
} else {
0
};
let is_loading = app.is_loading();
let gutter_prefix_extra = if is_loading { 1 } else { 0 };
let gutter_width = if total_lines > 0 {
line_num_width + 1 + 1
line_num_width + gutter_prefix_extra + 1 + 1
} else {
0
};
@@ -177,62 +224,73 @@ fn render_content(frame: &mut ratatui::Frame, app: &mut App, area: ratatui::layo
return;
}
app.recompute_wrap_cache(actual_content_width);
let mut visual_acc: usize = 0;
let mut start_logical: usize = 0;
let mut offset_in_line: usize = 0;
let v_offset = app.v_offset;
for (i, &h) in app.visual_heights.iter().enumerate() {
if visual_acc.saturating_add(h) > v_offset {
start_logical = i;
offset_in_line = v_offset.saturating_sub(visual_acc);
break;
}
visual_acc += h;
if i == app.visual_heights.len() - 1 {
start_logical = i;
offset_in_line = 0;
}
}
let (start_logical, offset_in_line) = app.ensure_viewport_cache(actual_content_width);
let mut lines: Vec<Line> = Vec::new();
let mut current_visual_offset: usize = 0;
let available_rows = content_height;
for logical_line in start_logical..total_lines {
let wrapped = &app.wrap_cache[logical_line];
let gutter_style = if is_loading {
Style::default()
.fg(Color::DarkGray)
.add_modifier(Modifier::DIM)
} else {
Style::default().fg(Color::DarkGray)
};
for (entry_idx, entry) in app.viewport_cache.entries.iter().enumerate() {
let logical_line = app.viewport_cache.logical_start + entry_idx;
let start_row = if logical_line == start_logical {
offset_in_line
} else {
0
};
for (visual_row, text) in wrapped.iter().enumerate().skip(start_row) {
for (visual_row, text) in entry.wrapped_rows.iter().enumerate().skip(start_row) {
if current_visual_offset >= available_rows {
break;
}
let is_cursor = logical_line == app.cursor_line;
let level = app.level_cache.get(logical_line).and_then(|l| l.as_ref());
let level = entry.level.as_ref();
let bg_color = if is_cursor {
Color::DarkGray
} else {
Color::Reset
};
let level_fg = level_fg(level, &app.color_config).unwrap_or(Color::White);
let gutter_text = if visual_row == 0 {
format!(
"{:>width$} \u{2502}",
logical_line + 1,
width = line_num_width
)
if is_loading {
format!(
"~{:>width$} \u{2502}",
logical_line + 1,
width = line_num_width
)
} else {
format!(
"{:>width$} \u{2502}",
logical_line + 1,
width = line_num_width
)
}
} else if is_loading {
format!(" {:width$} \u{2502}", "", width = line_num_width)
} else {
format!("{:width$} \u{2502}", "", width = line_num_width)
};
lines.push(build_line_spans(
gutter_text,
text.clone(),
is_cursor,
level,
&app.color_config,
));
let effective_gutter_style = if is_cursor {
gutter_style.bg(bg_color)
} else {
gutter_style
};
lines.push(Line::from(vec![
Span::styled(gutter_text, effective_gutter_style),
Span::styled(text.clone(), Style::default().fg(level_fg).bg(bg_color)),
]));
current_visual_offset += 1;
}
@@ -304,4 +362,135 @@ mod tests {
assert_eq!(line.spans[0].style.bg, Some(Color::Reset));
assert_eq!(line.spans[1].style.bg, Some(Color::Reset));
}
fn render_to_buffer(app: &mut App, width: u16, height: u16) -> ratatui::buffer::Buffer {
let backend = ratatui::backend::TestBackend::new(width, height);
let mut terminal = ratatui::Terminal::new(backend).unwrap();
terminal.draw(|frame| render(frame, app)).unwrap();
terminal.backend().buffer().clone()
}
fn make_temp_file(content: &str) -> std::path::PathBuf {
use std::sync::atomic::{AtomicU64, Ordering};
static COUNTER: AtomicU64 = AtomicU64::new(0);
let id = COUNTER.fetch_add(1, Ordering::Relaxed);
let dir = std::env::temp_dir();
let name = format!("log_viewer_ui_test_{}_{}", std::process::id(), id);
let path = dir.join(name);
std::fs::write(&path, content).unwrap();
path
}
#[test]
fn test_render_error_state() {
let mut app = App::new();
app.set_error_state("file not found");
let buf = render_to_buffer(&mut app, 80, 24);
let mut found_error = false;
let mut found_quit = false;
for row in 0..23 {
let content: String = (0..80)
.map(|c| buf.cell((c, row)).unwrap().symbol().to_string())
.collect();
if content.contains("Error") && content.contains("file not found") {
found_error = true;
}
if content.contains("Press q to quit") {
found_quit = true;
}
}
assert!(found_error, "content area should show error message");
assert!(found_quit, "content area should show quit hint");
}
#[test]
fn test_render_loading_state_status_bar() {
let path = make_temp_file("line1\nline2\nline3\nline4\nline5\n");
let result = std::panic::catch_unwind(|| {
let mut app = App::new();
app.load_file(path.to_str().unwrap()).unwrap();
if app.is_loading() {
let buf = render_to_buffer(&mut app, 80, 24);
let status: String = (0..80)
.map(|c| buf.cell((c, 23)).unwrap().symbol().to_string())
.collect();
assert!(
status.contains("Indexing"),
"status bar should show 'Indexing', got: {}",
status
);
assert!(
status.contains("%"),
"status bar should show percentage, got: {}",
status
);
assert!(
status.contains("~"),
"status bar should show ~ for estimated lines, got: {}",
status
);
}
});
let _ = std::fs::remove_file(&path);
assert!(result.is_ok());
}
#[test]
fn test_render_loading_gutter_tilde_prefix() {
let path = make_temp_file("line1\nline2\nline3\n");
let result = std::panic::catch_unwind(|| {
let mut app = App::new();
app.load_file(path.to_str().unwrap()).unwrap();
if app.is_loading() {
let buf = render_to_buffer(&mut app, 80, 24);
let first_line: String = (0..10)
.map(|c| buf.cell((c, 1)).unwrap().symbol().to_string())
.collect();
assert!(
first_line.contains("~"),
"loading state gutter should have ~ prefix, got: {}",
first_line
);
}
});
let _ = std::fs::remove_file(&path);
assert!(result.is_ok());
}
#[test]
fn test_render_ready_state_status_bar() {
let path = make_temp_file("alpha\nbeta\ngamma\n");
let result = std::panic::catch_unwind(|| {
let data = std::fs::read(&path).unwrap();
let index = log_viewer_core::io::line_index::LineIndex::from_bytes(&data);
let _ = log_viewer_core::io::index_cache::IndexCache::save_with_hash(&path, &index, &data);
let mut app = App::new();
app.load_file(path.to_str().unwrap()).unwrap();
assert!(
app.is_loaded() && !app.is_loading(),
"should be in Ready state with cache hit"
);
let buf = render_to_buffer(&mut app, 80, 24);
let status: String = (0..80)
.map(|c| buf.cell((c, 23)).unwrap().symbol().to_string())
.collect();
assert!(
status.contains("1/") || status.contains("1/3"),
"status bar should show cursor position, got: {}",
status
);
});
let _ = std::fs::remove_file(&path);
assert!(result.is_ok());
}
}