HTML5 + JavaScript + CSS3 RGBA video overlays on top of live GStreamer video pipelines

cover
This blog post was published 6 years ago and may or may not have aged well. While reading please keep in mind that it may no longer be accurate or even relevant.

GStreamer comes with a number of plugins that allow rendering of text and/or graphics overlays on top of video: rsvgoverlay, subtitleoverlay, textoverlay, cairooverlay, gdkpixbufoverlay, opencvtextoverlay, etc. However, some of these plugins often allow only static graphics and text, and often do not approach the flexibility and power of dedicated video post-processing software products.

“noweffects” (a play on the name of a popular video post-processing software) is a proof-of-concept of leveraging the power of a modern HTML5 + JavaScript + CSS3 web browser engine to render high-quality, programmable, alpha-aware, animated, vector- and bitmap based content, which is then rendered into an RGBA raw video stream, which can then be transferred via some kind of IPC method to separate GStreamer processes, where it can be composited with other content via GStreamers regular compositor or videomixer plugins.

Qt was chosen for its ease of integration of modern WebKit (QtWebKit) and GStreamer (qt-gstreamer), and its ability to render widgets to RGBA images. The QMainWindow widget is rendered in regular intervals to QImages in RGBA format, then inserted into a GStreamer pipeline via the appsrc plugin. This pipeline simply uses udpsink to multicast the raw video RTP packets on localhost to allow for multiple ‘subscribers’. A second GStreamer pipleline can then use udpsrc and apply the overlay.

Proof-of-concept code is available at: https://github.com/michaelfranzl/noweffects

The following demonstration video was generated with “noweffects”: A website (showing CSS3 animations), rendered to an RGBA video via QtWebKit, then overlaid on top of a video test pattern in a separate GStreamer process.

If you think you found a mistake in this blog post, or would like to suggest an improvement to this blog post, you can write me an e-mail to the address public dot michael at franzl dot name; as subject please use the prefix "Comment to blog post" and append the post title.
 
Copyright © 2023 Michael Franzl