fix(webrtc): propagate real capture PTS through WebRTC channel (closes #24)
Root cause (3-stage bug found via Oracle round 1+2 review):
Browser WebRTC clients accumulated 2-3 seconds jitter buffer under
damage-driven variable frame rate (KWin Portal/PipeWire). User moved
mouse, saw action 2-3 seconds later on client.
Three coordinated bugs formed a chain that defeated any single-point fix:
1. state_portal.rs:455 used sequential frame counter as PTS instead of
real capture time. (Portal path only — wlr-screencopy already correct.)
2. avhw.rs Channel output sent only Vec<u8>, DISCARDING AVPacket PTS.
Even with correct encoder PTS, timing metadata was thrown away.
3. webrtc.rs:695 computed RTP timestamp as 'frame_number * 90000 / fps'
from a counter, ignoring any real PTS. Browser saw uniform 33ms RTP
spacing regardless of actual 1.6-57fps variable delivery, growing
jitter buffer to compensate for perceived 'network jitter'.
Fix (7-step ordered implementation per Oracle round 2):
1. EncodedH264Frame struct in avhw.rs carries data + pts_ticks
2. FrameOutput::Channel type changed from Sender<Vec<u8>> to
Sender<EncodedH264Frame>; both new_webrtc signatures updated
3. Channel drain in avhw.rs extracts pkt.pts(), normalizes via
'p - start_ts' (mirrors existing Muxer branch logic). Drops
packets with missing PTS instead of silently emitting zero.
4. write_h264_frame signature: frame_number:u64 -> pts_ticks:i64
(both WebRtcState and WebRtcInner layers). Extracted pure function
rtp_timestamp_from_pts_ticks(pts_ticks, fps) with 5 unit tests
covering zero, one-frame, one-second, negative clamp, fps=0.
5. state_portal.rs receiver loop consumes EncodedH264Frame, passes
.data and .pts_ticks to write_h264_frame.
6. state.rs (wlr-screencopy) receiver loop updated for compile
compatibility — its existing real-time PTS computation at state.rs:606
was already correct, now properly propagates through new channel type.
7. state_portal.rs:467 PTS computation GATED on output mode:
- WebRTC branch: compute_capture_pts() uses PipeWire's ns timestamp
(or Instant fallback), normalizes to first-frame-origin, converts
to encoder time_base units with i128 intermediate math, enforces
monotonicity via safe checked_add pattern.
- MP4 branch: KEEPS self.frames_encoded as i64 (sequential counter).
File output does not need real-time PTS; changing it would alter
file playback speed during static periods.
Oracle round 2 critical revisions incorporated:
- Single-point PTS normalization (only in avhw Channel drain), NOT at
source. Avoids double-subtraction with existing Muxer logic.
- MP4 path explicitly preserved — real PTS only applied to WebRTC branch.
- Safe Rust monotonicity guard (no unsafe pointer tricks).
- i64::try_from(ticks_i128) instead of broken i128::try_from(...).unwrap_or(i64::MAX).
- pkt.pts() missing -> log + drop, not silent unwrap_or(0).
- Updates span 4 files (avhw.rs, webrtc.rs, state_portal.rs, state.rs)
because channel type change ripples through both Portal and wlr paths.
Verification expectations:
- Browser jitter buffer should stabilize at 100-500ms (typical) instead
of growing to 2-3 seconds under damage-driven delivery.
- chrome://webrtc-internals: jitterBufferDelay / jitterBufferEmittedCount
ratio should drop significantly.
- Server-side metrics (output_bps, frame rate, IDR size) unchanged.
- MP4 file output (--output mode) behavior unchanged.
Out of scope (deferred):
- frame_age metric fix (Oracle: separate commit to isolate behavioral
change from observability change)
- VFR encoder redesign (1/fps time_base sufficient for this fix)
- MP4 VFR recording (semantic change, separate decision)
- Existing client jitter buffers may not auto-shrink; reconnect may be
required for users with already-accumulated latency
Tests:
- cargo build --release: clean, 0 new warnings (19 pre-existing)
- cargo test: 96 lib + 96 bin + 3 integration, 0 failed
- 5 new RTP unit tests covering edge cases
- SAFETY comments preserved verbatim
- 4 files changed, +157/-27 lines
This commit is contained in:
dailz
+40
-6
@@ -711,9 +711,25 @@ impl EncState {
|
||||
// SwEncState - VAAPI GPU downscale + software H.264 encode
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Encoded H.264 frame with timing metadata for WebRTC output.
|
||||
///
|
||||
/// MP4 file output (FrameOutput::Muxer) does NOT use this - it writes via
|
||||
/// avformat which preserves PTS internally. WebRTC output (FrameOutput::Channel)
|
||||
/// requires explicit PTS propagation so RTP timestamps reflect real capture time.
|
||||
/// Without this, WebRTC clients' jitter buffers grow to seconds under
|
||||
/// damage-driven variable frame rate. See issue #24.
|
||||
#[derive(Debug)]
|
||||
pub struct EncodedH264Frame {
|
||||
/// H.264 NAL byte stream (Annex B or AVCC depending on encoder configuration)
|
||||
pub data: Vec<u8>,
|
||||
/// PTS in encoder time_base units (1/fps seconds), normalized so first frame = 0.
|
||||
/// Derived from real capture time, NOT frame counter.
|
||||
pub pts_ticks: i64,
|
||||
}
|
||||
|
||||
pub enum FrameOutput {
|
||||
Muxer(ff::format::context::Output),
|
||||
Channel(crossbeam_channel::Sender<Vec<u8>>),
|
||||
Channel(crossbeam_channel::Sender<EncodedH264Frame>),
|
||||
}
|
||||
|
||||
/// Owned CPU NV12 frame data for cross-thread transfer.
|
||||
@@ -1078,7 +1094,7 @@ impl SwEncEncode {
|
||||
fps: u32,
|
||||
bitrate: u64,
|
||||
gop_size: u32,
|
||||
tx: crossbeam_channel::Sender<Vec<u8>>,
|
||||
tx: crossbeam_channel::Sender<EncodedH264Frame>,
|
||||
webrtc_paused: Arc<AtomicBool>,
|
||||
bitrate_rx: crossbeam_channel::Receiver<BitrateCommand>,
|
||||
resolution_rx: crossbeam_channel::Receiver<ResolutionChange>,
|
||||
@@ -1377,18 +1393,36 @@ impl SwEncEncode {
|
||||
// slice is copied into a Vec before the packet is unreffed.
|
||||
let data: &[u8] =
|
||||
unsafe { std::slice::from_raw_parts(raw.data, raw.size as usize) };
|
||||
match tx.try_send(data.to_vec()) {
|
||||
// Normalize PTS: subtract starting_timestamp so first frame = 0.
|
||||
// Mirrors the Muxer branch normalization above. `start_ts` is
|
||||
// self.starting_timestamp.unwrap_or(0) (passed by caller), so
|
||||
// when no origin is recorded yet the subtraction is a no-op.
|
||||
let pts_ticks = match pkt.pts() {
|
||||
Some(p) => p - start_ts,
|
||||
None => {
|
||||
// libx264 should always set PTS; emitting RTP ts=0
|
||||
// here would recreate issue #24. Drop the packet.
|
||||
tracing::warn!(
|
||||
"encoder produced packet without PTS, dropping"
|
||||
);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
match tx.try_send(EncodedH264Frame {
|
||||
data: data.to_vec(),
|
||||
pts_ticks,
|
||||
}) {
|
||||
Ok(()) => {}
|
||||
Err(crossbeam_channel::TrySendError::Full(frame)) => {
|
||||
tracing::warn!(
|
||||
"WebRTC channel full, dropping frame: {} bytes lost",
|
||||
frame.len()
|
||||
frame.data.len()
|
||||
);
|
||||
}
|
||||
Err(crossbeam_channel::TrySendError::Disconnected(frame)) => {
|
||||
tracing::warn!(
|
||||
"WebRTC channel disconnected: {} bytes lost",
|
||||
frame.len()
|
||||
frame.data.len()
|
||||
);
|
||||
self.webrtc_disconnected = true;
|
||||
break;
|
||||
@@ -1458,7 +1492,7 @@ impl SwEncState {
|
||||
fps: u32,
|
||||
bitrate: u64,
|
||||
gop_size: u32,
|
||||
tx: crossbeam_channel::Sender<Vec<u8>>,
|
||||
tx: crossbeam_channel::Sender<EncodedH264Frame>,
|
||||
webrtc_paused: Arc<AtomicBool>,
|
||||
) -> Result<Self> {
|
||||
tracing::info!(
|
||||
|
||||
Reference in New Issue
Block a user