Implement computer vision for PlayerUnknown’s Battlegrounds live streams  

Build a web app that uses OCR on live-streaming video platform Twitch

Last updated | By Cullen Taylor, Spencer Krum

Description

PUBG Rotisserie features a video stream of a player in the extremely popular multiplayer video game, PlayerUnknown’s Battlegrounds (PUBG). It performs optical character recognition (OCR) on the live streams found on Twitch to identify which streams have only a handful of players left and are close to the final moments of play. In this developer journey, learn one of the many ways to implement computer vision to derive data from a video.

Overview

On Twitch and other websites, professional and regular computer players live-stream their games. At any given time, thousands of people are streaming on Twitch, and hundreds of thousands of viewers are watching those streams.

PUBG is a first-person or third-person shooter game. One hundred players spawn on an island, searching for loot and killing other players’ characters. There are no re-spawns. Only one player or team can win. Think Hunger Games meets Battlefield. The problem is that part of this game is relatively boring.

Watching a streamer when close to winning (25 people or fewer left alive) is very exciting. But watching a streamer when there are 90 or even 50 people still alive is much less exciting. This developer journey shows you one of the many ways to implement computer vision to derive data from a video. You will learn how to set up this application locally or in containers. You will also dive into deployment in Kubernetes, a container orchestration platform.

PUBG Rotisserie watches all the streams available on Twitch and switches between streams, showing the streams that are the most exciting. This can be left on a second monitor or on a TV in a lobby or sports bar.

Using computers to watch the entire streaming space for a game is a new thing only PUBG Rotisserie does right now. Using algorithms to switch between the streams like a virtual director is another innovative step.

Flow

  1. The main application uses the Twitch API to fetch the list of PUBG streams. Livestreamer, an open source project, takes a short recording of each of those streams.
  2. FFmpeg is used to take a single frame out of that short recording.
  3. That frame is then cropped with ImageMagick so the image only contains the number of players alive.
  4. The final image is pushed to the OCR service that uses Tesseract, an open source OCR engine. This service reads the number on image.
  5. The translated data will be in a JSON block received by the main app. The app fetches another list of streams every 15 seconds.
  6. The user will see the stream with the fewest remaining people alive. The front end is served by the main Node.js app. The user’s client will check every 15 seconds and switch if another stream is found to have even fewer remaining players.

Related Blogs

Leveraging the power of AI at Unite Berlin

Last week, from June 19 – 21, we were at Unity’s premiere in Berlin: Unite 2018. This conference brought together Unity’s video game and development community. Unity touches 770 million gamers all over the world and is the market leader for consumer AR and VR use cases and is also rapidly emerging as the market...

Continue reading Leveraging the power of AI at Unite Berlin

Are You Developers? WeAreDevelopers, 2018

Earlier this year, we attended the WeAreDevelopers World Congress conference in Vienna. Named Europe’s largest playground for developers, founded as recently as 2015, the Congress has truly grown from strength to strength. From May 16th – 18th there were 8,000 participants, 250+ speakers, 100+ sponsors, who all convened in Austria’s capital for 3 days of...

Continue reading Are You Developers? WeAreDevelopers, 2018

Related Links

Tesseract OCR

Learn more about the Tesseract open source OCR engine.

PUBG Rotisserie

The web app made by IBM developer advocates shows the most exciting PUBG live Twitch streams.