Decoding RTSP to a UTexture2D in Unreal (IPStreamMedia #4)

Where to start ?

It’s been a bit longer than I promised— I’d decided not to write again until I had something tangible on the execution side. After the FFmpeg tests that we discussed in the previous devlog, it was time to launch Unreal Engine, create a new empty plugin and run some tests to confirm the idea’s feasibility…

Milestone 1 - The Spike

Me and Claude planned the project such that we divided things broadly into 3 segments - prove FFmpeg integrates with Unreal Engine, setup a skeleton structure of the plugin modules, and wire the plugin to the FFmpeg libraries. I began with the first segment, the spike, where the idea was to create a simple system inside our plugin which would:

Use the FFmpeg libraries to connect to an RTSP stream ⇒ extract one frame of the video feed as AVPackets ⇒ decode it to AVFrame ⇒ convert it from YUV420P to BGRA8 (GPU format) ⇒ return it as a transient UTexture2D ⇒ slap it on a Plane using a dynamic material instance.

GrabOneFrame()

The simplest way execute this was to create a Blueprint Function Library class with a function that could take in an RTSP URL and return a single video frame as a UTexture2D

GrabOneFrame(FString RtspUrl) starts by creating an AVFormatContext, configured via an AVDictionary that holds the stream specs (rtsp, over tcp, etc). It then opens the stream with avformat_open_input(), checks if a video stream exists, and builds an AVCodecContext matching the stream’s encoding to initialize the decoder. From there it just runs the chain I have explained above and returns UTexture2D.

That’s it. The function finally ends after delivering a single frame of video.

Callbacks - the unsung heroes

If you notice in my explanation above, nowhere have I mentioned multi-threading, async tasks, or more such funny words 😵. That’s because this entire function call happens on the game thread, and yes, it does block the game thread. Now on a healthy stream that block’s for about a second, but what if the URL is valid but the source is not streaming? This may end up blocking the game thread infinitely.

By “block the game thread…” I mean that anything else that needs the game thread (for ex. almost everything that exists in your game! including the editor) will have to wait for this function to return before they can use the game thread.

To get around this issue FFmpeg uses AVIOInterruptCB, it’s a callback that continuously checks the status of blocking I/O operations like av_read_frame(). We setup this callback right at the beginning and connect it to the AVFormatContext, after which it keeps monitoring all the blocking I/O operations and exits out of them in-case of an error after a certain time-limit (which we set, and I had set it to 6 sec.).

Bro, just give me the figures -_-

So I ran two tests, one with a working stream coming from a CCTV on my LAN, the other was a broken stream whose URL I just found on Reddit. These figures were recorded by printing system time in the Output Log using Tick() and checking the gap in time when GrabOneFrame() was called,

Working Stream, first frame arrived at 0.999s. (Yeah I know I could have just said 1 sec 😬)

Broken Stream froze game thread for 6.039s, a 39ms overhead on the callback time of 6sec, not bad.

Last, but the most interesting 🧐

If you are someone who’s thinking about creating your Unreal plugin using FFmpeg, you might want to check out the next devlog before you begin…

Done ?

Yes. The Github repo is a running a bit ahead comparatively, I will try to cover things up here as soon as I can. This has already become a bit long so,

That’s all for this one. Cheers! 🍻