Keeping it real(-time)

Streaming live video from an Android camera

Lloyd Henning @lloydhenning
Peter Sutton @suttonpeter

New year's Eve 2012

The brilliant idea

  • Make a stage show of inventions
  • Involving an audience camera
  • Android camera → laptop → projector
peepers overview: phone camera to laptop to projector

The scope

  • < 1s latency
  • No audio
  • Dedicated same-room WiFi network
    • No bandwidth limit
    • Negligible jitter

Someone must have done this already?

Measuring latency

  • VLC
  • Low-latency playback: disable network caching

    # vlc --network-caching 0 source
  • spydroid latency: ~3s
  • ping to android device on local WiFi: ~300ms

spydroid

  • Too many features
    • Java: 3299 lines
    • Flatulence SFX
  • Stripped down
    • Education
    • Reduce latency
    • Java: 781 lines

How spydroid works

  • No access to camera stream via the Android SDK/NDK

The hack

  1. Use a MediaRecorder to capture the camera stream
  2. Transmit its output over the network

The MediaRecorder

MediaRecorder recorder = new MediaRecorder();

recorder.setVideoSource(MediaRecorder.VideoSource.DEFAULT);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.H263);
recorder.setOutputFile(outputFile);
recorder.setPreviewDisplay(holder.getSurface());

recorder.prepare();
recorder.start();

// ...

recorder.stop();

Video only accessible after stop() returns

Tricking the MediaRecorder

Use a LocalSocket instead of a file

LocalServerSocket server = new LocalServerSocket("some name");

receiver = new LocalSocket();
receiver.connect(server.getLocalSocketAddress());
receiver.setReceiveBufferSize(BUFFER_SIZE);

LocalServerSocket sender = server.accept();
sender.setSendBufferSize(BUFFER_SIZE);

// ...

// Give the MediaRecorder our fake file
recorder.setOutputFile(sender.getFileDescriptor());

Video is accessible as soon as the MediaRecorder writes it

What's this rubbish?

It's a video file, not a stream

mpeg-4 container

From Wikipedia

Specifically, an MPEG-4 file

MPEG-4 Container format

  • A collection of boxes
    1. Meta-data
      • Timestamps
      • Frame rate
      • ...
    2. ...
    3. Video
    4. ...

Extracting video from an MPEG-4 file

Skip over the meta-data to the video box

byte[] mdat = { 'm', 'd', 'a', 't' };
byte[] buffer = new byte[mdat.length];

do
{
  fillBuffer(buffer);
} while (!Arrays.equals(buffer, mdat));

Over the network

TCP

tcp

Reliable channel of communication

Over the network

UDP

udp

Fire and forget, no guarantees, less overhead

Only care about latest frame: dropped packets are OK

Raw H.263 over UDP

Transcoded to OGG for browser playback. Download the original.

Informing the client: RTP

  • Real-time transport protocol
  • Video codec and other meta-data
  • Video stream, inside an RTP packet, inside a UDP packet, ...
rtp stack

Troubles with Timestamps

timestamp
  • RTP header timestamps control playback
  • Can't get the true timestamps from the MediaRecorder
  • You have to guess

H.263 over RTP

Transcoded to OGG for browser playback. Download the original.

Investigating the latency

Latency too high

latency tortoise

Altering bit-rate, quality, packet size, ... had no effect

Latency is from the media encoding

A new hope

Preview surface is real-time ... stream that?

Motion JPEG

camera preview to mjpeg

A stream of JPEGs, each one representing a frame

Streaming the camera preview

Camera.PreviewCallback previewCallback = new Camera.PreviewCallback()
{
  public void onPreviewFrame(byte[] data, Camera camera)
  {
    // Create JPEG
    YuvImage image = new YuvImage(data, format, width, height,
            null /* strides */);
    image.compressToJpeg(crop, quality, outputStream);

    // Send it over the network ...
  }
};

camera.setPreviewCallback(previewCallback);

Use a Camera.PreviewCallback to get the raw camera image

MJPEG over RTP

Unreliable playback on VLC

Transcoded to OGG for browser playback. Download the original.

Non-Android testing

Using wireshark, pycapy, and javacv, created a local version for faster development.

local architecture

Webcam icon by ~kyo-tux. The Android robot is reproduced or modified from work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License.

MJPEG over HTTP

Quickly created a local MJPEG over HTTP streamer

MJPEG over HTTP

Use A/B buffering instead of dropping packets

Questions?


The project is on Github. Try it out thedemo. Slides created using reveal.js.