Files

dailz 8d6a720e8d initial

2026-02-03 11:14:25 +08:00

22 KiB

Raw Permalink Blame History

wl-webrtc

A high-performance, low-latency Wayland to WebRTC remote desktop backend written in Rust. Features zero-copy DMA-BUF pipeline, hardware-accelerated encoding, and WebRTC streaming for sub-20ms latency on local networks.

Overview

wl-webrtc captures Wayland desktop screens via PipeWire's xdg-desktop-portal, encodes video using hardware-accelerated codecs, and streams to web browsers via WebRTC. The zero-copy DMA-BUF pipeline minimizes memory transfers, enabling ultra-low latency streaming ideal for local network remote desktop scenarios.

Use Cases

Remote Desktop: Access your Linux workstation from any device with a web browser
Screen Sharing: Share your screen in web applications with minimal latency
Gaming: Play games remotely with low-latency video streaming
Collaboration: Real-time screen sharing for pair programming or presentations

Features

Performance

Zero-Copy Pipeline: DMA-BUF direct-to-encoder with no CPU memory copies
Low Latency: 15-25ms on LAN, <100ms on WAN with optimal configuration
Hardware Acceleration: VA-API (Intel/AMD), NVENC (NVIDIA), or software fallback
Adaptive Bitrate: Dynamic bitrate adjustment based on network conditions

Compatibility

Wayland: Native Wayland screen capture via PipeWire
WebRTC: Browser-compatible streaming without plugins
Multiple Encoders: H.264 (x264, VA-API, NVENC), VP9
Platform: Linux with Wayland compositor

Reliability

Damage Tracking: Encode only changed screen regions
Frame Skipping: Adaptive frame rate control
ICE/STUN/TURN: NAT traversal for remote connections
Connection Recovery: Automatic reconnection on network issues

Installation

System Dependencies

Install required system packages for your Linux distribution:

Ubuntu/Debian:

sudo apt update
sudo apt install -y \
    libpipewire-0.3-dev \
    libwayland-dev \
    libwayland-protocols-dev \
    libx264-dev \
    libopenh264-dev \
    libva-dev \
    vainfo \
    pkg-config \
    clang

Fedora/RHEL:

sudo dnf install -y \
    pipewire-devel \
    wayland-devel \
    wayland-protocols-devel \
    x264-devel \
    openh264-devel \
    libva-devel \
    libva-intel-driver \
    libva-intel-hybrid-driver \
    pkg-config \
    clang

Arch Linux:

sudo pacman -S \
    pipewire \
    wayland \
    wayland-protocols \
    x264 \
    openh264 \
    libva \
    libva-intel-driver \
    libva-mesa-driver \
    pkg-config \
    clang

Optional Hardware Encoder Support:

For Intel/AMD GPU (VA-API):

# Ubuntu/Debian
sudo apt install -y libva-dev libva-intel-driver vainfo

# Verify VA-API is available
vainfo

For NVIDIA GPU (NVENC):

# Install NVIDIA proprietary drivers (≥ 470.xx)
# See: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

# Verify NVENC is available
nvidia-smi

Cargo Installation

# Clone the repository
git clone https://github.com/yourusername/wl-webrtc.git
cd wl-webrtc

# Build the project
cargo build --release

# The binary will be available at target/release/wl-webrtc

Configuration

Copy the configuration template and customize it:

cp config.toml.template config.toml

Edit config.toml to match your system and network conditions. See Configuration for details.

Build Features

wl-webrtc supports optional features for different encoder configurations:

# Default features (openh264 + pipewire)
cargo build --release

# With x264 encoder
cargo build --release --features x264

# With VP9 encoder
cargo build --release --features vpx

# With all encoders
cargo build --release --features all-encoders

Usage

Quick Start

Start the server:

# Use default configuration
./target/release/wl-webrtc start

# Or specify a custom config file
./target/release/wl-webrtc --config my-config.toml start

# Override configuration via command line
./target/release/wl-webrtc --frame-rate 60 --bitrate-mbps 8 start

Connect a client:
- Open a web browser and navigate to http://localhost:8443 (or your configured port)
- The WebRTC connection will be established automatically
- Grant screen capture permissions when prompted by PipeWire
Stop the server:
```
./target/release/wl-webrtc stop
```

Command-Line Interface

wl-webrtc provides a CLI for managing the server:

# Display help
./target/release/wl-webrtc --help

# Start server with specific options
./target/release/wl-webrtc \
    --frame-rate 60 \
    --width 1920 \
    --height 1080 \
    --bitrate-mbps 4 \
    --port 8443 \
    start

# Start with hardware encoder
./target/release/wl-webrtc --encoder h264_vaapi start

# Stop running server
./target/release/wl-webrtc stop

# Show current configuration
./target/release/wl-webrtc config

# Display version
./target/release/wl-webrtc --version

Configuration

The config.toml file controls all aspects of wl-webrtc behavior:

[capture]
frame_rate = 30        # Target FPS for screen capture
quality = "high"       # Quality level: low, medium, high, ultra

[encoder]
encoder_type = "h264_x264"  # Encoder: h264_x264, h264_vaapi, h264_nvenc, vp9
width = 1920
height = 1080
frame_rate = 30
bitrate = 4000000      # 4 Mbps
preset = "veryfast"    # Encoding speed/quality tradeoff
tune = "zerolatency"   # Latency optimization

[webrtc]
port = 8443
ice_servers = ["stun:stun.l.google.com:19302"]

Network Configuration

Local Network (LAN) - Best Quality:

[capture]
frame_rate = 60
quality = "ultra"

[encoder]
bitrate = 16000000      # 16 Mbps
frame_rate = 60
preset = "veryfast"
encoder_type = "h264_vaapi"  # Hardware encoder recommended

Remote Network (Good Connection):

[capture]
frame_rate = 30
quality = "high"

[encoder]
bitrate = 4000000       # 4 Mbps
frame_rate = 30
preset = "veryfast"

Remote Network (Poor Connection):

[capture]
frame_rate = 24
quality = "medium"

[encoder]
bitrate = 1000000      # 1 Mbps
frame_rate = 24
preset = "ultrafast"   # Faster encoding
width = 1280           # Lower resolution
height = 720

TURN Server Configuration

For networks that block direct UDP connections, configure a TURN server:

[webrtc]
turn_servers = [
    { urls = ["turn:your-turn-server.com:3478?transport=udp"],
      username = "your-username",
      credential = "your-credential" }
]

See config.toml.template for complete configuration documentation.

Client Connection

After starting the server, connect from any web browser:

Navigate to the server URL:
```
http://localhost:8443
```
Grant screen capture permissions:
- PipeWire will prompt for screen sharing permission
- Select the screen or window to share
WebRTC establishes automatically:
- Video streaming begins
- Mouse and keyboard events are forwarded (if supported)
For remote connections:
- Configure firewall to allow the server port (default: 8443)
- Use a domain name or public IP if accessing from outside the local network
- Configure STUN/TURN servers for NAT traversal

Architecture

System Overview

wl-webrtc implements a zero-copy pipeline from screen capture to network transmission:

┌─────────────────────────────────────────────────────────────────────┐
│                          Web Browser                                │
│                    (WebRTC Receiver)                                │
└─────────────────────────────┬───────────────────────────────────────┘
                                │ WebRTC (UDP/TCP)
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Rust Backend Server                              │
├─────────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐          │
│  │   Capture    │    │   Encoder    │    │   WebRTC     │          │
│  │   Manager    │───▶│   Pipeline   │───▶│  Transport   │          │
│  └──────────────┘    └──────────────┘    └──────────────┘          │
│         │                  │                  │                      │
│         ▼                  ▼                  ▼                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐          │
│  │  PipeWire   │    │  Video       │    │   Data       │          │
│  │  Portal     │    │  Encoder     │    │  Channel     │          │
│  │  (xdg-      │    │  (H.264/     │    │  (Input/     │          │
│  │  desktop-   │    │  H.265/VP9)  │    │   Control)   │          │
│  │  portal)    │    └──────────────┘    └──────────────┘          │
│  └──────────────┘                                                     │
│                                                                       │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │              Zero-Copy Buffer Manager                        │    │
│  │         - DMA-BUF import/export                             │    │
│  │         - Shared memory pool                                │    │
│  │         - Memory ownership tracking                         │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                   Wayland Compositor                                │
│                 (PipeWire Screen Share)                             │
└─────────────────────────────────────────────────────────────────────┘

Module Organization

src/
├── main.rs              # CLI entry point and server orchestration
├── lib.rs               # Library exports
├── config/              # Configuration management
│   ├── mod.rs
│   └── types.rs
├── capture/             # Screen capture module
│   ├── mod.rs
│   └── pipewire.rs      # PipeWay xdg-desktop-portal integration
├── buffer/              # Zero-copy buffer management
│   ├── mod.rs
│   └── pool.rs
├── encoder/             # Video encoding pipeline
│   ├── mod.rs
│   ├── h264.rs
│   ├── vp9.rs
│   └── pool.rs
├── webrtc/              # WebRTC transport
│   ├── mod.rs
│   ├── peer.rs          # Peer connection management
│   └── signaling.rs     # Signaling server
├── ipc/                 # Inter-process communication (optional)
│   └── mod.rs
└── utils/               # Utilities
    ├── mod.rs
    └── logging.rs

Zero-Copy Data Flow

wl-webrtc minimizes memory copies through a carefully designed pipeline:

Stage 1: Capture
  Input: Wayland compositor (GPU memory)
  Output: DMA-BUF file descriptor
  Copies: None (zero-copy)

Stage 2: Buffer Manager
  Input: DMA-BUF file descriptor
  Output: DmaBufHandle (RAII wrapper)
  Copies: None (zero-copy ownership transfer)

Stage 3: Encoder
  Input: DmaBufHandle
  Output: Bytes (reference-counted)
  Copies: None (DMA-BUF direct import to GPU encoder)

Stage 4: WebRTC
  Input: Bytes
  Output: RTP packets (referencing Bytes)
  Copies: None (zero-copy to socket buffer)

Stage 5: Network
  Input: RTP packets
  Output: UDP datagrams
  Copies: Minimal (kernel space only)

This design ensures that video data travels from GPU capture to network transmission without CPU memory copies, enabling ultra-low latency streaming.

Key Components

Capture Manager

Interacts with PipeWire xdg-desktop-portal
Requests screen capture permissions
Receives DMA-BUF frame data
Manages frame buffer lifecycle

Encoder Pipeline

Receives raw frames from capture
Encodes to H.264/H.265/VP9
Supports hardware acceleration (VA-API, NVENC)
Implements adaptive bitrate control

WebRTC Transport

Manages WebRTC peer connections
Handles video track and data channels
Implements RTP packetization
Manages ICE/STUN/TURN for NAT traversal

Buffer Manager

Manages DMA-BUF lifecycle
Maintains shared memory pool
Tracks memory ownership via Rust types
Coordinates with PipeWire memory pool

Damage Tracking

wl-webrtc implements damage tracking to encode only changed screen regions:

Compare current frame with previous frame
Identify changed regions (damaged areas)
Encode only damaged regions at higher quality
Reduce bandwidth and CPU usage
Maintain low latency even with partial updates

For detailed architecture information, see DESIGN_CN.md.

Performance

Latency Targets

Scenario	Target Latency	Configuration
Local Network (LAN)	15-25ms	Hardware encoder, 60fps, 8-16Mbps
Remote Network (Good)	50-100ms	Hardware encoder, 30fps, 2-4Mbps
Remote Network (Poor)	100-200ms	Software encoder, 15-24fps, 0.5-1Mbps

Resource Usage

CPU Usage (Software Encoding):

1080p@30fps: 20-40% (typical)
1080p@60fps: 40-60%
720p@30fps: 10-20%

CPU Usage (Hardware Encoding):

1080p@30fps: 5-10%
1080p@60fps: 10-15%
4K@30fps: 15-25%

Memory Usage:

Base: 150-200MB
Per client: 50-100MB
Buffer pool: 50-100MB

Network Bandwidth:

1080p@30fps (H.264): 2-8Mbps
1080p@60fps (H.264): 4-16Mbps
720p@30fps (H.264): 1-4Mbps
VP9 typically 30-50% less bandwidth at similar quality

Encoding Performance

Recommended Settings by Use Case:

Use Case	Encoder	Resolution	FPS	Bitrate	Preset
Gaming (LAN)	h264_vaapi/nvenc	1080p	60	12-16Mbps	veryfast
Remote Desktop (Good WAN)	h264_vaapi	1080p	30	4-8Mbps	veryfast
Remote Desktop (Poor WAN)	h264_x264	720p	24	1-2Mbps	ultrafast
Screen Sharing	h264_x264	1080p	15-30	2-4Mbps	veryfast

Optimization Tips

Use Hardware Encoders: VA-API or NVENC significantly reduces CPU usage
Match FPS to Content: Dynamic content needs higher FPS than static screens
Enable Damage Tracking: Encode only changed regions (automatic)
Optimize Bitrate: Use adaptive bitrate based on network conditions
Adjust Resolution: Lower resolution for better performance on slow networks

Benchmarks

Tested on a system with:

CPU: Intel Core i7-12700K
GPU: Intel UHD Graphics 770
RAM: 32GB DDR5
OS: Ubuntu 23.10 with Wayland

1080p@30fps with h264_vaapi:

Latency: 18-22ms (LAN)
CPU: 7-9%
Memory: 250MB
Bandwidth: 4-5Mbps

1080p@60fps with h264_vaapi:

Latency: 15-19ms (LAN)
CPU: 11-14%
Memory: 280MB
Bandwidth: 8-10Mbps

720p@30fps with h264_x264:

Latency: 45-60ms (WAN)
CPU: 18-22%
Memory: 200MB
Bandwidth: 2-2.5Mbps

Troubleshooting

Common Issues

"No PipeWire session found"

Cause: PipeWire is not running or not configured correctly.

Solution:

# Check if PipeWire is running
systemctl --user status pipewire pipewire-pulse pipewire-media-session

# Start PipeWire if not running
systemctl --user start pipewire pipewire-pulse pipewire-media-session

# Enable PipeWire to start on boot
systemctl --user enable pipewire pipewire-pulse pipewire-media-session

"Permission denied for screen capture"

Cause: PipeWire portal permission not granted.

Solution:

Restart the wl-webrtc server
When prompted by the desktop portal, grant screen capture permission
Select the screen or window to share
If using Wayland, ensure your compositor supports screen sharing (KDE Plasma, GNOME, Sway)

"VA-API encoder not available"

Cause: VA-API hardware acceleration is not installed or not supported.

Solution:

# Install VA-API drivers
sudo apt install -y libva-dev libva-intel-driver vainfo

# Verify VA-API is available
vainfo

# If VA-API is unavailable, fall back to software encoder:
# In config.toml, set:
# encoder_type = "h264_x264"

"NVENC encoder initialization failed"

Cause: NVIDIA drivers are not installed or GPU does not support NVENC.

Solution:

# Check NVIDIA driver version (must be ≥ 470)
nvidia-smi

# Verify NVENC support
nvidia-smi --query-gpu=encoder.version.encoder --format=csv

# If NVENC is unavailable, fall back to VA-API or x264:
# encoder_type = "h264_vaapi"  # or "h264_x264"

"High latency / laggy video"

Cause: Network conditions or encoder settings not optimized.

Solution:

Reduce bitrate: bitrate = 2000000 (2Mbps)
Reduce frame rate: frame_rate = 24 or frame_rate = 15
Use faster preset: preset = "ultrafast"
Lower resolution: width = 1280, height = 720
Check network: ping the server to verify low latency
Use wired connection instead of WiFi

"WebRTC connection failed"

Cause: Network configuration or firewall blocking the connection.

Solution:

# Check if the server port is accessible
nc -zv localhost 8443

# Allow the port through firewall (UFW example)
sudo ufw allow 8443/tcp

# For remote connections, configure STUN/TURN:
# In config.toml:
# ice_servers = ["stun:stun.l.google.com:19302"]
#
# For networks blocking UDP, add TURN server:
# [[webrtc.turn_servers]]
# urls = ["turn:your-turn-server.com:3478?transport=udp"]
# username = "your-username"
# credential = "your-credential"

"Out of memory"

Cause: Insufficient memory or buffer leak.

Solution:

Reduce buffer pool size in config
Reduce video resolution
Reduce frame rate
Check for memory leaks: top or htop monitoring
Increase system RAM or close other applications

"CPU usage too high"

Cause: Software encoding is CPU-intensive.

Solution:

Use hardware encoder: encoder_type = "h264_vaapi" or "h264_nvenc"
Use faster preset: preset = "ultrafast" or "superfast"
Reduce resolution: width = 1280, height = 720
Reduce frame rate: frame_rate = 24 or frame_rate = 15
Reduce bitrate (saves encoding CPU)

Debug Mode

Enable verbose logging for troubleshooting:

# Set log level
RUST_LOG=debug ./target/release/wl-webrtc start

# For very verbose logging
RUST_LOG=trace ./target/release/wl-webrtc start

Performance Profiling

To identify performance bottlenecks:

# Profile CPU usage
perf record -g ./target/release/wl-webrtc start
perf report

# Profile memory usage
valgrind --leak-check=full ./target/release/wl-webrtc start

# Monitor in real-time
top -p $(pgrep wl-webrtc)

Getting Help

If you encounter issues not covered here:

Check the Issues page
Review the DESIGN_CN.md for technical details
Enable debug logging and collect output
Open a new issue with:
- System information (OS, Wayland compositor)
- Hardware information (CPU, GPU)
- Configuration file (sanitized)
- Error logs
- Steps to reproduce

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under either of:

MIT License (LICENSE-MIT or http://opensource.org/licenses/MIT)
Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)

Acknowledgments

PipeWire for screen capture infrastructure
WebRTC-RS for WebRTC implementation
x264 for software H.264 encoding
VA-API for hardware encoding on Intel/AMD GPUs
The Wayland community for the modern display protocol

22 KiB Raw Permalink Blame History

wl-webrtc

Overview

Use Cases

Features

Performance

Compatibility

Reliability

Installation

System Dependencies

Cargo Installation

Configuration

Build Features

Usage

Quick Start

Command-Line Interface

Configuration

Network Configuration

TURN Server Configuration

Client Connection

Architecture

System Overview

Module Organization

Zero-Copy Data Flow

Key Components

Capture Manager

Encoder Pipeline

WebRTC Transport

Buffer Manager

Damage Tracking

Performance

Latency Targets

Resource Usage

Encoding Performance

Optimization Tips

Benchmarks

Troubleshooting

Common Issues

"No PipeWire session found"

"Permission denied for screen capture"

"VA-API encoder not available"

"NVENC encoder initialization failed"

"High latency / laggy video"

"WebRTC connection failed"

"Out of memory"

"CPU usage too high"

Debug Mode

Performance Profiling

Getting Help

Contributing

License

Acknowledgments

Links

22 KiB

Raw Permalink Blame History