Keeping it real(-time)

New year's Eve 2012

The brilliant idea

• Make a stage show of inventions
• Involving an audience camera
• Android camera → laptop → projector

The scope

• < 1s latency
• No audio
• Dedicated same-room WiFi network
• No bandwidth limit
• Negligible jitter

Someone must have done this already?

• Open source
• android-eye
• spydroid
• Closed source
• IP Camera

Measuring latency

• VLC

• Low-latency playback: disable network caching

  # vlc --network-caching 0 source
• spydroid latency: ~3s
• ping to android device on local WiFi: ~300ms

spydroid

• Too many features
• Java: 3299 lines
• Flatulence SFX
• Stripped down
• Education
• Reduce latency
• Java: 781 lines

How spydroid works

• No access to camera stream via the Android SDK/NDK

The hack

1. Use a MediaRecorder to capture the camera stream
2. Transmit its output over the network

The MediaRecorder

MediaRecorder recorder = new MediaRecorder();

recorder.setVideoSource(MediaRecorder.VideoSource.DEFAULT);
recorder.setOutputFormat(MediaRecorder.OutputFormat.MPEG_4);
recorder.setVideoEncoder(MediaRecorder.VideoEncoder.H263);
recorder.setOutputFile(outputFile);
recorder.setPreviewDisplay(holder.getSurface());

recorder.prepare();
recorder.start();

// ...

recorder.stop();

Video only accessible after stop() returns

Tricking the MediaRecorder

Use a LocalSocket instead of a file

LocalServerSocket server = new LocalServerSocket("some name");

receiver = new LocalSocket();
receiver.connect(server.getLocalSocketAddress());
receiver.setReceiveBufferSize(BUFFER_SIZE);

LocalServerSocket sender = server.accept();
sender.setSendBufferSize(BUFFER_SIZE);

// ...

// Give the MediaRecorder our fake file
recorder.setOutputFile(sender.getFileDescriptor());

Video is accessible as soon as the MediaRecorder writes it

What's this rubbish?

It's a video file, not a stream

From Wikipedia

Specifically, an MPEG-4 file

MPEG-4 Container format

• A collection of boxes
1. Meta-data
• Timestamps
• Frame rate
• ...
2. ...
3. Video
4. ...

Extracting video from an MPEG-4 file

Skip over the meta-data to the video box

byte[] mdat = { 'm', 'd', 'a', 't' };
byte[] buffer = new byte[mdat.length];

do
{
  fillBuffer(buffer);
} while (!Arrays.equals(buffer, mdat));

Over the network

TCP

Reliable channel of communication

Over the network

UDP

Fire and forget, no guarantees, less overhead

Only care about latest frame: dropped packets are OK

Raw H.263 over UDP

Informing the client: RTP

• Real-time transport protocol
• Video codec and other meta-data
• Video stream, inside an RTP packet, inside a UDP packet, ...

Troubles with Timestamps

• RTP header timestamps control playback
• Can't get the true timestamps from the MediaRecorder
• You have to guess

H.263 over RTP

Investigating the latency

Latency too high

Altering bit-rate, quality, packet size, ... had no effect

Latency is from the media encoding

A new hope

Preview surface is real-time ... stream that?

Motion JPEG

A stream of JPEGs, each one representing a frame

Streaming the camera preview

Camera.PreviewCallback previewCallback = new Camera.PreviewCallback()
{
  public void onPreviewFrame(byte[] data, Camera camera)
  {
    // Create JPEG
    YuvImage image = new YuvImage(data, format, width, height,
            null /* strides */);
    image.compressToJpeg(crop, quality, outputStream);

    // Send it over the network ...
  }
};

camera.setPreviewCallback(previewCallback);

Use a Camera.PreviewCallback to get the raw camera image

MJPEG over RTP

Unreliable playback on VLC

Non-Android testing

Using wireshark, pycapy, and javacv, created a local version for faster development.

Webcam icon by ~kyo-tux. The Android robot is reproduced or modified from work created and shared by Google and used according to terms described in the Creative Commons 3.0 Attribution License.

MJPEG over HTTP

Quickly created a local MJPEG over HTTP streamer

Use A/B buffering instead of dropping packets

Questions?

The project is on Github. Try it out thedemo. Slides created using reveal.js.