# wl-webrtc

A high-performance, low-latency Wayland to WebRTC remote desktop backend written in Rust. Features zero-copy DMA-BUF pipeline, hardware-accelerated encoding, and WebRTC streaming for sub-20ms latency on local networks.

## Overview

wl-webrtc captures Wayland desktop screens via PipeWire's xdg-desktop-portal, encodes video using hardware-accelerated codecs, and streams to web browsers via WebRTC. The zero-copy DMA-BUF pipeline minimizes memory transfers, enabling ultra-low latency streaming ideal for local network remote desktop scenarios.

### Use Cases

- **Remote Desktop**: Access your Linux workstation from any device with a web browser
- **Screen Sharing**: Share your screen in web applications with minimal latency
- **Gaming**: Play games remotely with low-latency video streaming
- **Collaboration**: Real-time screen sharing for pair programming or presentations

## Features

### Performance
- **Zero-Copy Pipeline**: DMA-BUF direct-to-encoder with no CPU memory copies
- **Low Latency**: 15-25ms on LAN, <100ms on WAN with optimal configuration
- **Hardware Acceleration**: VA-API (Intel/AMD), NVENC (NVIDIA), or software fallback
- **Adaptive Bitrate**: Dynamic bitrate adjustment based on network conditions

### Compatibility
- **Wayland**: Native Wayland screen capture via PipeWire
- **WebRTC**: Browser-compatible streaming without plugins
- **Multiple Encoders**: H.264 (x264, VA-API, NVENC), VP9
- **Platform**: Linux with Wayland compositor

### Reliability
- **Damage Tracking**: Encode only changed screen regions
- **Frame Skipping**: Adaptive frame rate control
- **ICE/STUN/TURN**: NAT traversal for remote connections
- **Connection Recovery**: Automatic reconnection on network issues

## Installation

### System Dependencies

Install required system packages for your Linux distribution:

**Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install -y \
    libpipewire-0.3-dev \
    libwayland-dev \
    libwayland-protocols-dev \
    libx264-dev \
    libopenh264-dev \
    libva-dev \
    vainfo \
    pkg-config \
    clang
```

**Fedora/RHEL:**
```bash
sudo dnf install -y \
    pipewire-devel \
    wayland-devel \
    wayland-protocols-devel \
    x264-devel \
    openh264-devel \
    libva-devel \
    libva-intel-driver \
    libva-intel-hybrid-driver \
    pkg-config \
    clang
```

**Arch Linux:**
```bash
sudo pacman -S \
    pipewire \
    wayland \
    wayland-protocols \
    x264 \
    openh264 \
    libva \
    libva-intel-driver \
    libva-mesa-driver \
    pkg-config \
    clang
```

**Optional Hardware Encoder Support:**

For **Intel/AMD GPU** (VA-API):
```bash
# Ubuntu/Debian
sudo apt install -y libva-dev libva-intel-driver vainfo

# Verify VA-API is available
vainfo
```

For **NVIDIA GPU** (NVENC):
```bash
# Install NVIDIA proprietary drivers (≥ 470.xx)
# See: https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

# Verify NVENC is available
nvidia-smi
```

### Cargo Installation

```bash
# Clone the repository
git clone https://github.com/yourusername/wl-webrtc.git
cd wl-webrtc

# Build the project
cargo build --release

# The binary will be available at target/release/wl-webrtc
```

### Configuration

Copy the configuration template and customize it:

```bash
cp config.toml.template config.toml
```

Edit `config.toml` to match your system and network conditions. See [Configuration](#configuration) for details.

### Build Features

wl-webrtc supports optional features for different encoder configurations:

```bash
# Default features (openh264 + pipewire)
cargo build --release

# With x264 encoder
cargo build --release --features x264

# With VP9 encoder
cargo build --release --features vpx

# With all encoders
cargo build --release --features all-encoders
```

## Usage

### Quick Start

1. **Start the server:**
   ```bash
   # Use default configuration
   ./target/release/wl-webrtc start

   # Or specify a custom config file
   ./target/release/wl-webrtc --config my-config.toml start

   # Override configuration via command line
   ./target/release/wl-webrtc --frame-rate 60 --bitrate-mbps 8 start
   ```

2. **Connect a client:**
   - Open a web browser and navigate to `http://localhost:8443` (or your configured port)
   - The WebRTC connection will be established automatically
   - Grant screen capture permissions when prompted by PipeWire

3. **Stop the server:**
   ```bash
   ./target/release/wl-webrtc stop
   ```

### Command-Line Interface

wl-webrtc provides a CLI for managing the server:

```bash
# Display help
./target/release/wl-webrtc --help

# Start server with specific options
./target/release/wl-webrtc \
    --frame-rate 60 \
    --width 1920 \
    --height 1080 \
    --bitrate-mbps 4 \
    --port 8443 \
    start

# Start with hardware encoder
./target/release/wl-webrtc --encoder h264_vaapi start

# Stop running server
./target/release/wl-webrtc stop

# Show current configuration
./target/release/wl-webrtc config

# Display version
./target/release/wl-webrtc --version
```

### Configuration

The `config.toml` file controls all aspects of wl-webrtc behavior:

```toml
[capture]
frame_rate = 30        # Target FPS for screen capture
quality = "high"       # Quality level: low, medium, high, ultra

[encoder]
encoder_type = "h264_x264"  # Encoder: h264_x264, h264_vaapi, h264_nvenc, vp9
width = 1920
height = 1080
frame_rate = 30
bitrate = 4000000      # 4 Mbps
preset = "veryfast"    # Encoding speed/quality tradeoff
tune = "zerolatency"   # Latency optimization

[webrtc]
port = 8443
ice_servers = ["stun:stun.l.google.com:19302"]
```

#### Network Configuration

**Local Network (LAN) - Best Quality:**
```toml
[capture]
frame_rate = 60
quality = "ultra"

[encoder]
bitrate = 16000000      # 16 Mbps
frame_rate = 60
preset = "veryfast"
encoder_type = "h264_vaapi"  # Hardware encoder recommended
```

**Remote Network (Good Connection):**
```toml
[capture]
frame_rate = 30
quality = "high"

[encoder]
bitrate = 4000000       # 4 Mbps
frame_rate = 30
preset = "veryfast"
```

**Remote Network (Poor Connection):**
```toml
[capture]
frame_rate = 24
quality = "medium"

[encoder]
bitrate = 1000000      # 1 Mbps
frame_rate = 24
preset = "ultrafast"   # Faster encoding
width = 1280           # Lower resolution
height = 720
```

#### TURN Server Configuration

For networks that block direct UDP connections, configure a TURN server:

```toml
[webrtc]
turn_servers = [
    { urls = ["turn:your-turn-server.com:3478?transport=udp"],
      username = "your-username",
      credential = "your-credential" }
]
```

See [config.toml.template](config.toml.template) for complete configuration documentation.

### Client Connection

After starting the server, connect from any web browser:

1. **Navigate to the server URL:**
   ```
   http://localhost:8443
   ```

2. **Grant screen capture permissions:**
   - PipeWire will prompt for screen sharing permission
   - Select the screen or window to share

3. **WebRTC establishes automatically:**
   - Video streaming begins
   - Mouse and keyboard events are forwarded (if supported)

4. **For remote connections:**
   - Configure firewall to allow the server port (default: 8443)
   - Use a domain name or public IP if accessing from outside the local network
   - Configure STUN/TURN servers for NAT traversal

## Architecture

### System Overview

wl-webrtc implements a zero-copy pipeline from screen capture to network transmission:

```
┌─────────────────────────────────────────────────────────────────────┐
│                          Web Browser                                │
│                    (WebRTC Receiver)                                │
└─────────────────────────────┬───────────────────────────────────────┘
                                │ WebRTC (UDP/TCP)
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Rust Backend Server                              │
├─────────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐          │
│  │   Capture    │    │   Encoder    │    │   WebRTC     │          │
│  │   Manager    │───▶│   Pipeline   │───▶│  Transport   │          │
│  └──────────────┘    └──────────────┘    └──────────────┘          │
│         │                  │                  │                      │
│         ▼                  ▼                  ▼                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐          │
│  │  PipeWire   │    │  Video       │    │   Data       │          │
│  │  Portal     │    │  Encoder     │    │  Channel     │          │
│  │  (xdg-      │    │  (H.264/     │    │  (Input/     │          │
│  │  desktop-   │    │  H.265/VP9)  │    │   Control)   │          │
│  │  portal)    │    └──────────────┘    └──────────────┘          │
│  └──────────────┘                                                     │
│                                                                       │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │              Zero-Copy Buffer Manager                        │    │
│  │         - DMA-BUF import/export                             │    │
│  │         - Shared memory pool                                │    │
│  │         - Memory ownership tracking                         │    │
│  └─────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────┐
│                   Wayland Compositor                                │
│                 (PipeWire Screen Share)                             │
└─────────────────────────────────────────────────────────────────────┘
```

### Module Organization

```
src/
├── main.rs              # CLI entry point and server orchestration
├── lib.rs               # Library exports
├── config/              # Configuration management
│   ├── mod.rs
│   └── types.rs
├── capture/             # Screen capture module
│   ├── mod.rs
│   └── pipewire.rs      # PipeWay xdg-desktop-portal integration
├── buffer/              # Zero-copy buffer management
│   ├── mod.rs
│   └── pool.rs
├── encoder/             # Video encoding pipeline
│   ├── mod.rs
│   ├── h264.rs
│   ├── vp9.rs
│   └── pool.rs
├── webrtc/              # WebRTC transport
│   ├── mod.rs
│   ├── peer.rs          # Peer connection management
│   └── signaling.rs     # Signaling server
├── ipc/                 # Inter-process communication (optional)
│   └── mod.rs
└── utils/               # Utilities
    ├── mod.rs
    └── logging.rs
```

### Zero-Copy Data Flow

wl-webrtc minimizes memory copies through a carefully designed pipeline:

```
Stage 1: Capture
  Input: Wayland compositor (GPU memory)
  Output: DMA-BUF file descriptor
  Copies: None (zero-copy)

Stage 2: Buffer Manager
  Input: DMA-BUF file descriptor
  Output: DmaBufHandle (RAII wrapper)
  Copies: None (zero-copy ownership transfer)

Stage 3: Encoder
  Input: DmaBufHandle
  Output: Bytes (reference-counted)
  Copies: None (DMA-BUF direct import to GPU encoder)

Stage 4: WebRTC
  Input: Bytes
  Output: RTP packets (referencing Bytes)
  Copies: None (zero-copy to socket buffer)

Stage 5: Network
  Input: RTP packets
  Output: UDP datagrams
  Copies: Minimal (kernel space only)
```

This design ensures that video data travels from GPU capture to network transmission without CPU memory copies, enabling ultra-low latency streaming.

### Key Components

#### Capture Manager
- Interacts with PipeWire xdg-desktop-portal
- Requests screen capture permissions
- Receives DMA-BUF frame data
- Manages frame buffer lifecycle

#### Encoder Pipeline
- Receives raw frames from capture
- Encodes to H.264/H.265/VP9
- Supports hardware acceleration (VA-API, NVENC)
- Implements adaptive bitrate control

#### WebRTC Transport
- Manages WebRTC peer connections
- Handles video track and data channels
- Implements RTP packetization
- Manages ICE/STUN/TURN for NAT traversal

#### Buffer Manager
- Manages DMA-BUF lifecycle
- Maintains shared memory pool
- Tracks memory ownership via Rust types
- Coordinates with PipeWire memory pool

### Damage Tracking

wl-webrtc implements damage tracking to encode only changed screen regions:

1. Compare current frame with previous frame
2. Identify changed regions (damaged areas)
3. Encode only damaged regions at higher quality
4. Reduce bandwidth and CPU usage
5. Maintain low latency even with partial updates

For detailed architecture information, see [DESIGN_CN.md](DESIGN_CN.md).

## Performance

### Latency Targets

| Scenario | Target Latency | Configuration |
|----------|----------------|---------------|
| Local Network (LAN) | 15-25ms | Hardware encoder, 60fps, 8-16Mbps |
| Remote Network (Good) | 50-100ms | Hardware encoder, 30fps, 2-4Mbps |
| Remote Network (Poor) | 100-200ms | Software encoder, 15-24fps, 0.5-1Mbps |

### Resource Usage

**CPU Usage (Software Encoding):**
- 1080p@30fps: 20-40% (typical)
- 1080p@60fps: 40-60%
- 720p@30fps: 10-20%

**CPU Usage (Hardware Encoding):**
- 1080p@30fps: 5-10%
- 1080p@60fps: 10-15%
- 4K@30fps: 15-25%

**Memory Usage:**
- Base: 150-200MB
- Per client: 50-100MB
- Buffer pool: 50-100MB

**Network Bandwidth:**
- 1080p@30fps (H.264): 2-8Mbps
- 1080p@60fps (H.264): 4-16Mbps
- 720p@30fps (H.264): 1-4Mbps
- VP9 typically 30-50% less bandwidth at similar quality

### Encoding Performance

**Recommended Settings by Use Case:**

| Use Case | Encoder | Resolution | FPS | Bitrate | Preset |
|----------|---------|------------|-----|---------|--------|
| Gaming (LAN) | h264_vaapi/nvenc | 1080p | 60 | 12-16Mbps | veryfast |
| Remote Desktop (Good WAN) | h264_vaapi | 1080p | 30 | 4-8Mbps | veryfast |
| Remote Desktop (Poor WAN) | h264_x264 | 720p | 24 | 1-2Mbps | ultrafast |
| Screen Sharing | h264_x264 | 1080p | 15-30 | 2-4Mbps | veryfast |

### Optimization Tips

1. **Use Hardware Encoders**: VA-API or NVENC significantly reduces CPU usage
2. **Match FPS to Content**: Dynamic content needs higher FPS than static screens
3. **Enable Damage Tracking**: Encode only changed regions (automatic)
4. **Optimize Bitrate**: Use adaptive bitrate based on network conditions
5. **Adjust Resolution**: Lower resolution for better performance on slow networks

### Benchmarks

Tested on a system with:
- CPU: Intel Core i7-12700K
- GPU: Intel UHD Graphics 770
- RAM: 32GB DDR5
- OS: Ubuntu 23.10 with Wayland

**1080p@30fps with h264_vaapi:**
- Latency: 18-22ms (LAN)
- CPU: 7-9%
- Memory: 250MB
- Bandwidth: 4-5Mbps

**1080p@60fps with h264_vaapi:**
- Latency: 15-19ms (LAN)
- CPU: 11-14%
- Memory: 280MB
- Bandwidth: 8-10Mbps

**720p@30fps with h264_x264:**
- Latency: 45-60ms (WAN)
- CPU: 18-22%
- Memory: 200MB
- Bandwidth: 2-2.5Mbps

## Troubleshooting

### Common Issues

#### "No PipeWire session found"
**Cause:** PipeWire is not running or not configured correctly.

**Solution:**
```bash
# Check if PipeWire is running
systemctl --user status pipewire pipewire-pulse pipewire-media-session

# Start PipeWire if not running
systemctl --user start pipewire pipewire-pulse pipewire-media-session

# Enable PipeWire to start on boot
systemctl --user enable pipewire pipewire-pulse pipewire-media-session
```

#### "Permission denied for screen capture"
**Cause:** PipeWire portal permission not granted.

**Solution:**
1. Restart the wl-webrtc server
2. When prompted by the desktop portal, grant screen capture permission
3. Select the screen or window to share
4. If using Wayland, ensure your compositor supports screen sharing (KDE Plasma, GNOME, Sway)

#### "VA-API encoder not available"
**Cause:** VA-API hardware acceleration is not installed or not supported.

**Solution:**
```bash
# Install VA-API drivers
sudo apt install -y libva-dev libva-intel-driver vainfo

# Verify VA-API is available
vainfo

# If VA-API is unavailable, fall back to software encoder:
# In config.toml, set:
# encoder_type = "h264_x264"
```

#### "NVENC encoder initialization failed"
**Cause:** NVIDIA drivers are not installed or GPU does not support NVENC.

**Solution:**
```bash
# Check NVIDIA driver version (must be ≥ 470)
nvidia-smi

# Verify NVENC support
nvidia-smi --query-gpu=encoder.version.encoder --format=csv

# If NVENC is unavailable, fall back to VA-API or x264:
# encoder_type = "h264_vaapi"  # or "h264_x264"
```

#### "High latency / laggy video"
**Cause:** Network conditions or encoder settings not optimized.

**Solution:**
1. Reduce bitrate: `bitrate = 2000000` (2Mbps)
2. Reduce frame rate: `frame_rate = 24` or `frame_rate = 15`
3. Use faster preset: `preset = "ultrafast"`
4. Lower resolution: `width = 1280`, `height = 720`
5. Check network: `ping` the server to verify low latency
6. Use wired connection instead of WiFi

#### "WebRTC connection failed"
**Cause:** Network configuration or firewall blocking the connection.

**Solution:**
```bash
# Check if the server port is accessible
nc -zv localhost 8443

# Allow the port through firewall (UFW example)
sudo ufw allow 8443/tcp

# For remote connections, configure STUN/TURN:
# In config.toml:
# ice_servers = ["stun:stun.l.google.com:19302"]
#
# For networks blocking UDP, add TURN server:
# [[webrtc.turn_servers]]
# urls = ["turn:your-turn-server.com:3478?transport=udp"]
# username = "your-username"
# credential = "your-credential"
```

#### "Out of memory"
**Cause:** Insufficient memory or buffer leak.

**Solution:**
1. Reduce buffer pool size in config
2. Reduce video resolution
3. Reduce frame rate
4. Check for memory leaks: `top` or `htop` monitoring
5. Increase system RAM or close other applications

#### "CPU usage too high"
**Cause:** Software encoding is CPU-intensive.

**Solution:**
1. Use hardware encoder: `encoder_type = "h264_vaapi"` or `"h264_nvenc"`
2. Use faster preset: `preset = "ultrafast"` or `"superfast"`
3. Reduce resolution: `width = 1280`, `height = 720`
4. Reduce frame rate: `frame_rate = 24` or `frame_rate = 15`
5. Reduce bitrate (saves encoding CPU)

### Debug Mode

Enable verbose logging for troubleshooting:

```bash
# Set log level
RUST_LOG=debug ./target/release/wl-webrtc start

# For very verbose logging
RUST_LOG=trace ./target/release/wl-webrtc start
```

### Performance Profiling

To identify performance bottlenecks:

```bash
# Profile CPU usage
perf record -g ./target/release/wl-webrtc start
perf report

# Profile memory usage
valgrind --leak-check=full ./target/release/wl-webrtc start

# Monitor in real-time
top -p $(pgrep wl-webrtc)
```

### Getting Help

If you encounter issues not covered here:

1. Check the [Issues](https://github.com/yourusername/wl-webrtc/issues) page
2. Review the [DESIGN_CN.md](DESIGN_CN.md) for technical details
3. Enable debug logging and collect output
4. Open a new issue with:
   - System information (OS, Wayland compositor)
   - Hardware information (CPU, GPU)
   - Configuration file (sanitized)
   - Error logs
   - Steps to reproduce

## Contributing

Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.

## License

This project is licensed under either of:
- MIT License ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)

## Acknowledgments

- [PipeWire](https://pipewire.org/) for screen capture infrastructure
- [WebRTC-RS](https://webrtc.rs/) for WebRTC implementation
- [x264](https://www.videolan.org/developers/x264.html) for software H.264 encoding
- [VA-API](https://01.org/vaapi) for hardware encoding on Intel/AMD GPUs
- The Wayland community for the modern display protocol

## Links

- [Documentation](https://docs.rs/wl-webrtc) - API documentation (Rustdoc)
- [DESIGN_CN.md](DESIGN_CN.md) - Technical design document
- [config.toml.template](config.toml.template) - Configuration reference
- [examples/](examples/) - Example client applications