fix(webrtc): propagate real capture PTS through WebRTC channel (closes #24)

Root cause (3-stage bug found via Oracle round 1+2 review):

Browser WebRTC clients accumulated 2-3 seconds jitter buffer under
damage-driven variable frame rate (KWin Portal/PipeWire). User moved
mouse, saw action 2-3 seconds later on client.

Three coordinated bugs formed a chain that defeated any single-point fix:

1. state_portal.rs:455 used sequential frame counter as PTS instead of
   real capture time. (Portal path only — wlr-screencopy already correct.)

2. avhw.rs Channel output sent only Vec<u8>, DISCARDING AVPacket PTS.
   Even with correct encoder PTS, timing metadata was thrown away.

3. webrtc.rs:695 computed RTP timestamp as 'frame_number * 90000 / fps'
   from a counter, ignoring any real PTS. Browser saw uniform 33ms RTP
   spacing regardless of actual 1.6-57fps variable delivery, growing
   jitter buffer to compensate for perceived 'network jitter'.

Fix (7-step ordered implementation per Oracle round 2):

1. EncodedH264Frame struct in avhw.rs carries data + pts_ticks
2. FrameOutput::Channel type changed from Sender<Vec<u8>> to
   Sender<EncodedH264Frame>; both new_webrtc signatures updated
3. Channel drain in avhw.rs extracts pkt.pts(), normalizes via
   'p - start_ts' (mirrors existing Muxer branch logic). Drops
   packets with missing PTS instead of silently emitting zero.
4. write_h264_frame signature: frame_number:u64 -> pts_ticks:i64
   (both WebRtcState and WebRtcInner layers). Extracted pure function
   rtp_timestamp_from_pts_ticks(pts_ticks, fps) with 5 unit tests
   covering zero, one-frame, one-second, negative clamp, fps=0.
5. state_portal.rs receiver loop consumes EncodedH264Frame, passes
   .data and .pts_ticks to write_h264_frame.
6. state.rs (wlr-screencopy) receiver loop updated for compile
   compatibility — its existing real-time PTS computation at state.rs:606
   was already correct, now properly propagates through new channel type.
7. state_portal.rs:467 PTS computation GATED on output mode:
   - WebRTC branch: compute_capture_pts() uses PipeWire's ns timestamp
     (or Instant fallback), normalizes to first-frame-origin, converts
     to encoder time_base units with i128 intermediate math, enforces
     monotonicity via safe checked_add pattern.
   - MP4 branch: KEEPS self.frames_encoded as i64 (sequential counter).
     File output does not need real-time PTS; changing it would alter
     file playback speed during static periods.

Oracle round 2 critical revisions incorporated:

- Single-point PTS normalization (only in avhw Channel drain), NOT at
  source. Avoids double-subtraction with existing Muxer logic.
- MP4 path explicitly preserved — real PTS only applied to WebRTC branch.
- Safe Rust monotonicity guard (no unsafe pointer tricks).
- i64::try_from(ticks_i128) instead of broken i128::try_from(...).unwrap_or(i64::MAX).
- pkt.pts() missing -> log + drop, not silent unwrap_or(0).
- Updates span 4 files (avhw.rs, webrtc.rs, state_portal.rs, state.rs)
  because channel type change ripples through both Portal and wlr paths.

Verification expectations:

- Browser jitter buffer should stabilize at 100-500ms (typical) instead
  of growing to 2-3 seconds under damage-driven delivery.
- chrome://webrtc-internals: jitterBufferDelay / jitterBufferEmittedCount
  ratio should drop significantly.
- Server-side metrics (output_bps, frame rate, IDR size) unchanged.
- MP4 file output (--output mode) behavior unchanged.

Out of scope (deferred):

- frame_age metric fix (Oracle: separate commit to isolate behavioral
  change from observability change)
- VFR encoder redesign (1/fps time_base sufficient for this fix)
- MP4 VFR recording (semantic change, separate decision)
- Existing client jitter buffers may not auto-shrink; reconnect may be
  required for users with already-accumulated latency

Tests:
- cargo build --release: clean, 0 new warnings (19 pre-existing)
- cargo test: 96 lib + 96 bin + 3 integration, 0 failed
- 5 new RTP unit tests covering edge cases
- SAFETY comments preserved verbatim
- 4 files changed, +157/-27 lines
This commit is contained in:
dailz
2026-06-20 21:53:27 +08:00
parent 2f0b858920
commit 079611acfc
4 changed files with 157 additions and 27 deletions
+62 -8
View File
@@ -9,8 +9,8 @@ use anyhow::{bail, Result}; // 错误处理工具
use crate::args::Args; // 命令行参数
use crate::avhw::{
self, BitrateCommand, CpuNv12Frame, EncodeOutcome, ResolutionChange, SwEncEncode, SwEncImport,
SwEncState,
self, BitrateCommand, CpuNv12Frame, EncodeOutcome, EncodedH264Frame, ResolutionChange,
SwEncEncode, SwEncImport, SwEncState,
}; // 软件编码器状态(VAAPI 导入 + H.264 编码)
use crate::cap_portal::{CapPortal, PwCtrlEvent, PwDmaBufFrame}; // PipeWire 屏幕采集端点
use crate::stats::{FrameTimings, PipelineStats}; // 管道统计(帧计时、每秒快照)
@@ -64,6 +64,10 @@ pub struct StatePortal {
last_capture_arrival: Option<Instant>, // timestamp of last real frame arrival
idle_log_start: Option<Instant>, // when current idle period began (one-shot DEBUG log guard)
shutdown_started: bool, // idempotency guard; plain bool because &mut self is exclusive (not AtomicBool)
// Issue #24: real-capture PTS origin/tracking for WebRTC RTP timestamps.
first_pts_ns: Option<i128>,
capture_start: Option<Instant>,
last_pts_emitted: Option<i64>,
}
impl StatePortal {
@@ -107,6 +111,9 @@ impl StatePortal {
last_capture_arrival: None,
idle_log_start: None,
shutdown_started: false,
first_pts_ns: None,
capture_start: None,
last_pts_emitted: None,
})
}
@@ -452,7 +459,16 @@ impl StatePortal {
}
}
let t_import_start = Instant::now();
let pts = self.frames_encoded as i64;
// WebRTC: use real PipeWire capture time so RTP timestamps reflect reality
// (sequential counter caused client jitter buffers to grow to 2-3s under
// damage-driven variable fps — issue #24). MP4: keep sequential counter;
// file output doesn't need real-time PTS and changing it would alter
// playback speed during static periods.
let pts = if self.webrtc.is_some() {
self.compute_capture_pts(frame.pts)
} else {
self.frames_encoded as i64
};
if let Some(enc) = self.enc.as_mut() {
// 将 DMA-BUF 帧零拷贝导入 VAAPI 硬件帧池
@@ -537,6 +553,43 @@ impl StatePortal {
Ok(())
}
/// Compute PTS in encoder time_base units (1/fps) from PipeWire's nanosecond
/// capture timestamp. Falls back to `Instant`-based elapsed time when PipeWire
/// does not provide PTS. Maintains strict monotonicity (encoder requirement).
fn compute_capture_pts(&mut self, pw_pts_ns: i64) -> i64 {
const NS_PER_SEC: i128 = 1_000_000_000;
let raw_ns: i128 = if pw_pts_ns > 0 {
i128::from(pw_pts_ns)
} else {
let start = self.capture_start.get_or_insert_with(Instant::now);
i128::try_from(start.elapsed().as_nanos()).unwrap_or(0)
};
if self.first_pts_ns.is_none() && raw_ns > 0 {
self.first_pts_ns = Some(raw_ns);
}
let origin = self.first_pts_ns.unwrap_or(0);
let relative_ns = if raw_ns >= origin {
raw_ns - origin
} else {
// PipeWire PTS went backwards (stream restart) — reset origin.
self.first_pts_ns = Some(raw_ns);
0
};
let ticks_i128 = (relative_ns.saturating_mul(i128::from(self.args.fps))) / NS_PER_SEC;
let mut pts = i64::try_from(ticks_i128).unwrap_or(i64::MAX);
if let Some(last) = self.last_pts_emitted {
if pts <= last {
pts = last.checked_add(1).unwrap_or(last);
}
}
self.last_pts_emitted = Some(pts);
pts
}
/// 关闭状态:刷新编码器并清理资源(幂等)。
///
/// `shutdown_started` 守卫在清理之前置位——防止 panic 时 `Drop` 重入 unwinding。
@@ -633,7 +686,7 @@ fn encode_thread_loop(
fn webrtc_thread_loop(
mut wrtc: WebRtcState,
webrtc_rx: crossbeam_channel::Receiver<Vec<u8>>,
webrtc_rx: crossbeam_channel::Receiver<EncodedH264Frame>,
fps: u32,
enc_width: u32,
enc_height: u32,
@@ -736,8 +789,8 @@ fn webrtc_thread_loop(
}
if connected {
while let Ok(data) = webrtc_rx.try_recv() {
if let Err(e) = wrtc.write_h264_frame(&data, frames_sent, fps) {
while let Ok(enc_frame) = webrtc_rx.try_recv() {
if let Err(e) = wrtc.write_h264_frame(&enc_frame.data, enc_frame.pts_ticks, fps) {
tracing::debug!("WebRTC write frame error: {e}");
}
frames_sent = frames_sent.saturating_add(1);
@@ -752,9 +805,10 @@ fn webrtc_thread_loop(
}
match webrtc_rx.recv_timeout(timeout) {
Ok(data) => {
Ok(enc_frame) => {
if wrtc.is_connected() {
if let Err(e) = wrtc.write_h264_frame(&data, frames_sent, fps) {
if let Err(e) = wrtc.write_h264_frame(&enc_frame.data, enc_frame.pts_ticks, fps)
{
tracing::debug!("WebRTC write frame error: {e}");
}
frames_sent = frames_sent.saturating_add(1);