bigbluebutton-Github/bigbluebutton-html5/imports/ui/services/virtual-background

Virtual Backgrounds for BBB

Inspired from https://ai.googleblog.com/2020/10/background-features-in-google-meet.html, https://github.com/Volcomix/virtual-background.git and https://github.com/jitsi/jitsi-meet/tree/master/react/features/stream-effects/virtual-background/vendor

iOS and macOS

The feature works on macOS, however Safari is not supported. Due to limitations on iOS, it is currently not possible to enable virtual backgrounds feature on iPhones and iPads. The reason behind is documented on Apple Developer Documentation website.

The image object can be an img element, a canvas element, or a video element. Use of the video element is not supported in Safari on iOS, however.

Settings

Virtual backgrounds feature for BBB is turned on by default as long as the version you have comes with the feature included. In case your settings.yml doesn't include the virtual background settings, it will fall back to the default settings, which are identical to having the following parameters in your settings.yml file.

virtualBackgrounds:
 enabled: true
 storedOnBBB: true
 showThumbnails: true
 imagesPath: /resources/images/virtual-backgrounds/
 thumbnailsPath: /resources/images/virtual-backgrounds/thumbnails/
 fileNames:
  - home.jpg
  - coffeeshop.jpg
  - board.jpg

In case you don't have the virtual background settings in your settings.yml, add the above lines under public.app namespace, so it looks like the following.

public:
 app:
  ...
  ...
  virtualBackgrounds:
   (your settings here)

Explanation of settings

Setting	Description	Type
enabled	Enables or disables virtual background feature	boolean
storedOnBBB	Determines where the images and thumbnails are stored. If set to true, virtual backgrounds and thumbnails will be fetched from /resources/images/virtual-backgrounds/.	boolean
showThumbnails	Determines whether to show or hide thumbnails. If set to false, a dropdown of fileNames will be shown.	boolean
imagesPath	Location of virtual background images. If storedOnBBB is set to false, it is possible to give an external location. IMPORTANT: File names must be given explicitly under fileNames if using non-default images.	string
thumbnailsPath	Location of virtual background image thumbnails. If storedOnBBB is set to false, it is possible to give an external location. IMPORTANT: Thumbnail file names and extensions must match their corresponding virtual background images. If showThumbnails is set to false, this can be ignored.	string
fileNames	List of file names that will be used as virtual backgrounds. File extensions must also be given. The same values are used for images and thumbnails.	array

Canvas 2D + CPU

This rendering pipeline is pretty much the same as for BodyPix. It relies on Canvas compositing properties to blend rendering layers according to the segmentation mask.

Interactions with TFLite inference tool are executed on CPU to convert from UInt8 to Float32 for the model input and to apply softmax on the model output.

The framerate is higher and the quality looks better than BodyPix

SIMD and non-SIMD

How to test on SIMD:

1. Go to chrome://flags/
2. Search for SIMD flag
3. Enable WebAssembly SIMD support(Enables support for the WebAssembly SIMD proposal).
4. Reopen Google Chrome

More details:

• WebAssembly
• WebAssembly SIMD
• TFLite

LICENSE

The models vendored here were downloaded early January (they were available as early as the 4th), before Google switched the license away from Apache 2. Thus we understand they are not covered by the new license which according to the model card dates from the 21st of January.

We are not lawyers so do get legal advise if in doubt.

References:

• Model license discussion: https://github.com/tensorflow/tfjs/issues/4177
• Current vendored model is discovered: https://github.com/tensorflow/tfjs/issues/4177#issuecomment-753934631
• License change is noticed: https://github.com/tensorflow/tfjs/issues/4177#issuecomment-771536641