Implement computer vision for PlayerUnknown’s Battlegrounds live streams
Build a web app that uses OCR on live-streaming video platform Twitch
PUBG Rotisserie features a video stream of a player in the extremely popular multiplayer video game, PlayerUnknown’s Battlegrounds (PUBG). It performs optical character recognition (OCR) on the live streams found on Twitch to identify which streams have only a handful of players left and are close to the final moments of play. In this developer pattern, learn one of the many ways to implement computer vision to derive data from a video.
On Twitch and other websites, professional and regular computer players live-stream their games. At any given time, thousands of people are streaming on Twitch, and hundreds of thousands of viewers are watching those streams.
PUBG is a first-person or third-person shooter game. One hundred players spawn on an island, searching for loot and killing other player’ characters. There are no re-spawns. Only one player or team can win. Think Hunger Games meets Battlefield. The problem is that part of this game is relatively boring.
Watching a streamer when close to winning (25 people or fewer left alive) is very exciting. But watching a streamer when there are 90 or even 50 people still alive is much less exciting. This developer pattern shows you one of the many ways to implement computer vision to derive data from a video. You will learn how to set up this application locally or in containers. You will also dive into deployment in Kubernetes, a container orchestration platform.
PUBG Rotisserie watches all the streams available on Twitch and switches between streams, showing the streams that are the most exciting. This can be left on a second monitor or on a TV in a lobby or sports bar.
Using computers to watch the entire streaming space for a game is a new thing only PUBG Rotisserie does right now. Using algorithms to switch between the streams like a virtual director is another innovative step.
- The main application uses the Twitch API to fetch the list of PUBG streams. Livestreamer, an open source project, takes a short recording of each of those streams.
- FFmpeg is used to take a single frame out of that short recording.
- That frame is then cropped with ImageMagick so the image only contains the number of players alive.
- The final image is pushed to the OCR service that uses Tesseract, an open source OCR engine. This service reads the number on image.
- The translated data will be in a JSON block received by the main app. The app fetches another list of streams every 15 seconds.
- The user will see the stream with the fewest remaining people alive. The front end is served by the main Node.js app. The user’s client will check every 15 seconds and switch if another stream is found to have even fewer remaining players.
Find the detailed steps for this pattern in the README.