BigBlueButton/bigbluebutton-Github
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(webcams): skip video preview if valid input devices stored (#20696)

Browse Source 
* feat(webcams): skip video preview if valid input devices stored

Additionally:

- refactor: re-use the existing VirtualBackground_* storage info instead
  of creating a new one
- fix: store background choices per deviceId instead of globally
- fix: guarantee background restore attempts are *critical* when
  video-preview is supposed to be skipped. We want the preview to be
  shown if the previous background could not be restored to preserver
  the user's privacy choice
- fix: cameras could not be shared if no previous device info was in
  the user's session
- fix: uploaded background images were not correctly restored
- fix: do not spin up virtual bg workers for brightness if it has not
  been altered by the user
- refactor: remove old video-provider background restore routine,
  centralize it in video-preview

* fix(skip-video-preview): correct storage check and add playwright test and docs

---------

Co-authored-by: prlanzarin <4529051+prlanzarin@users.noreply.github.com>
This commit is contained in:
germanocaumo 2024-07-18 20:24:10 +00:00 committed by GitHub
parent c048a8050f
commit cbe0b4f6ae
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
13 changed files with 399 additions and 263 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

308
bigbluebutton-html5/imports/ui/components/video-preview/component.jsx
View File

@ -17,11 +17,14 @@ import {
EFFECT_TYPES,
setSessionVirtualBackgroundInfo,
getSessionVirtualBackgroundInfo,
removeSessionVirtualBackgroundInfo,
isVirtualBackgroundSupported,
} from '/imports/ui/services/virtual-background/service';
import { getSettingsSingletonInstance } from '/imports/ui/services/settings';
import Checkbox from '/imports/ui/components/common/checkbox/component'
import AppService from '/imports/ui/components/app/service';
import { CustomVirtualBackgroundsContext } from '/imports/ui/components/video-preview/virtual-background/context';
import VBGSelectorService from '/imports/ui/components/video-preview/virtual-background/service';
const VIEW_STATES = {
finding: 'finding',
@ -216,6 +219,8 @@ const intlMessages = defineMessages({
});
class VideoPreview extends Component {
static contextType = CustomVirtualBackgroundsContext;
constructor(props) {
super(props);
@ -247,6 +252,7 @@ class VideoPreview extends Component {
previewError: null,
brightness: 100,
wholeImageBrightness: false,
skipPreviewFailed: false,
};
}
@ -266,6 +272,13 @@ class VideoPreview extends Component {
return this._currentVideoStream;
}
shouldSkipVideoPreview() {
const { skipPreviewFailed } = this.state;
const { forceOpen } = this.props;
return PreviewService.getSkipVideoPreview() && !forceOpen && !skipPreviewFailed;
}
componentDidMount() {
const {
webcamDeviceId,
@ -275,10 +288,31 @@ class VideoPreview extends Component {
this._isMounted = true;
if (deviceInfo.hasMediaDevices) {
navigator.mediaDevices.enumerateDevices().then((devices) => {
navigator.mediaDevices.enumerateDevices().then(async (devices) => {
VideoService.updateNumberOfDevices(devices);
// Video preview skip is activated, short circuit via a simpler procedure
if (PreviewService.getSkipVideoPreview() && !forceOpen) return this.skipVideoPreview();
// Tries to skip video preview - this can happen if:
// 1. skipVideoPreview, skipVideoPreviewOnFirstJoin, or
// skipVideoPreviewIfPreviousDevice flags are enabled and meet their
// own conditions
// 2. forceOpen flag was not specified to this component
//
// This will fail if no skip conditions are met, or if an unexpected
// failure occurs during the process. In that case, the error will be
// handled and the component will display the default video preview UI
if (this.shouldSkipVideoPreview()) {
try {
await this.skipVideoPreview()
return;
} catch (error) {
logger.warn({
logCode: 'video_preview_skip_failure',
extraInfo: {
errorName: error.name,
errorMessage: error.message,
},
}, 'Skipping video preview failed');
}
}
// Late enumerateDevices resolution, stop.
if (!this._isMounted) return;
@ -297,37 +331,35 @@ class VideoPreview extends Component {
}, `Enumerate devices came back. There are ${devices.length} devices and ${webcams.length} are video inputs`);
if (webcams.length > 0) {
this.getInitialCameraStream(webcams[0].deviceId)
.then(async () => {
// Late gUM resolve, stop.
if (!this._isMounted) return;
await this.getInitialCameraStream(webcams[0].deviceId);
// Late gUM resolve, stop.
if (!this._isMounted) return;
if (!areLabelled || !areIdentified) {
// If they aren't labelled or have nullish deviceIds, run
// enumeration again and get their full versions
// Why: fingerprinting countermeasures obfuscate those when
// no permission was granted via gUM
try {
const newDevices = await navigator.mediaDevices.enumerateDevices();
webcams = PreviewService.digestVideoDevices(newDevices, webcamDeviceId).webcams;
} catch (error) {
// Not a critical error because it should only affect UI; log it
// and go ahead
logger.error({
logCode: 'video_preview_enumerate_relabel_failure',
extraInfo: {
errorName: error.name, errorMessage: error.message,
},
}, 'enumerateDevices for relabelling failed');
}
}
if (!areLabelled || !areIdentified) {
// If they aren't labelled or have nullish deviceIds, run
// enumeration again and get their full versions
// Why: fingerprinting countermeasures obfuscate those when
// no permission was granted via gUM
try {
const newDevices = await navigator.mediaDevices.enumerateDevices();
webcams = PreviewService.digestVideoDevices(newDevices, webcamDeviceId).webcams;
} catch (error) {
// Not a critical error beucase it should only affect UI; log it
// and go ahead
logger.error({
logCode: 'video_preview_enumerate_relabel_failure',
extraInfo: {
errorName: error.name, errorMessage: error.message,
},
}, 'enumerateDevices for relabelling failed');
}
}
this.setState({
availableWebcams: webcams,
viewState: VIEW_STATES.found,
});
this.displayPreview();
});
this.setState({
availableWebcams: webcams,
viewState: VIEW_STATES.found,
});
this.displayPreview();
} else {
// There were no webcams coming from enumerateDevices. Throw an error.
const noWebcamsError = new Error('NotFoundError');
@ -369,11 +401,11 @@ class VideoPreview extends Component {
this._isMounted = false;
}
startCameraBrightness() {
async startCameraBrightness() {
const ENABLE_CAMERA_BRIGHTNESS = window.meetingClientSettings.public.app.enableCameraBrightness;
const CAMERA_BRIGHTNESS_AVAILABLE = ENABLE_CAMERA_BRIGHTNESS && isVirtualBackgroundSupported();
if (CAMERA_BRIGHTNESS_AVAILABLE) {
if (CAMERA_BRIGHTNESS_AVAILABLE && this.currentVideoStream) {
const setBrightnessInfo = () => {
const stream = this.currentVideoStream || {};
const service = stream.virtualBgService || {};
@ -382,20 +414,31 @@ class VideoPreview extends Component {
};
if (!this.currentVideoStream.virtualBgService) {
this.startVirtualBackground(
const switched = await this.startVirtualBackground(
this.currentVideoStream,
EFFECT_TYPES.NONE_TYPE,
).then((switched) => {
if (switched) {
setBrightnessInfo();
}
});
);
if (switched) setBrightnessInfo();
} else {
setBrightnessInfo();
}
}
}
async setCameraBrightness(brightness) {
const ENABLE_CAMERA_BRIGHTNESS = window.meetingClientSettings.public.app.enableCameraBrightness;
const CAMERA_BRIGHTNESS_AVAILABLE = ENABLE_CAMERA_BRIGHTNESS && isVirtualBackgroundSupported();
if (CAMERA_BRIGHTNESS_AVAILABLE && this.currentVideoStream) {
if (this.currentVideoStream?.virtualBgService == null) {
await this.startCameraBrightness();
}
this.currentVideoStream.changeCameraBrightness(brightness);
this.setState({ brightness });
}
}
handleSelectWebcam(event) {
const webcamValue = event.target.value;
@ -420,27 +463,29 @@ class VideoPreview extends Component {
}
}
updateVirtualBackgroundInfo = () => {
updateVirtualBackgroundInfo () {
const { webcamDeviceId } = this.state;
// Update this session's virtual camera effect information if it's enabled
setSessionVirtualBackgroundInfo(
this.currentVideoStream.virtualBgType,
this.currentVideoStream.virtualBgName,
webcamDeviceId,
);
if (this.currentVideoStream) {
setSessionVirtualBackgroundInfo(
webcamDeviceId,
this.currentVideoStream.virtualBgType,
this.currentVideoStream.virtualBgName,
this.currentVideoStream.virtualBgUniqueId,
);
}
};
// Resolves into true if the background switch is successful, false otherwise
handleVirtualBgSelected(type, name, customParams) {
const { sharedDevices } = this.props;
const { webcamDeviceId } = this.state;
const { webcamDeviceId, brightness } = this.state;
const shared = this.isAlreadyShared(webcamDeviceId);
const ENABLE_CAMERA_BRIGHTNESS = window.meetingClientSettings.public.app.enableCameraBrightness;
const CAMERA_BRIGHTNESS_AVAILABLE = ENABLE_CAMERA_BRIGHTNESS && isVirtualBackgroundSupported();
if (type !== EFFECT_TYPES.NONE_TYPE || CAMERA_BRIGHTNESS_AVAILABLE) {
if (type !== EFFECT_TYPES.NONE_TYPE || CAMERA_BRIGHTNESS_AVAILABLE && brightness !== 100) {
return this.startVirtualBackground(this.currentVideoStream, type, name, customParams).then((switched) => {
// If it's not shared we don't have to update here because
// it will be updated in the handleStartSharing method.
@ -487,7 +532,7 @@ class VideoPreview extends Component {
});
}
handleStartSharing() {
async handleStartSharing() {
const {
resolve,
startSharing,
@ -515,17 +560,18 @@ class VideoPreview extends Component {
this.stopVirtualBackground(this.currentVideoStream);
}
this.updateVirtualBackgroundInfo();
this.cleanupStreamAndVideo();
PreviewService.changeProfile(selectedProfile);
PreviewService.changeWebcam(webcamDeviceId);
if (cameraAsContent) {
startSharingCameraAsContent(webcamDeviceId);
} else {
if (!cameraAsContent) {
// Store selected profile, camera ID and virtual background in the storage
// for future use
PreviewService.changeProfile(selectedProfile);
PreviewService.changeWebcam(webcamDeviceId);
this.updateVirtualBackgroundInfo();
this.cleanupStreamAndVideo();
startSharing(webcamDeviceId);
} else {
this.cleanupStreamAndVideo();
startSharingCameraAsContent(webcamDeviceId);
}
if (resolve) resolve();
}
handleStopSharing() {
@ -658,13 +704,67 @@ class VideoPreview extends Component {
const { cameraAsContent } = this.props;
const defaultProfile = !cameraAsContent ? PreviewService.getDefaultProfile() : PreviewService.getCameraAsContentProfile();
return this.getCameraStream(deviceId, defaultProfile).then(() => {
this.updateDeviceId(deviceId);
return this.getCameraStream(deviceId, defaultProfile);
}
applyStoredVirtualBg(deviceId = null) {
const webcamDeviceId = deviceId || this.state.webcamDeviceId;
// Apply the virtual background stored in Local/Session Storage, if any
// If it fails, remove the stored background.
return new Promise((resolve, reject) => {
let customParams;
const virtualBackground = getSessionVirtualBackgroundInfo(webcamDeviceId);
if (virtualBackground) {
const { type, name, uniqueId } = virtualBackground;
const handleFailure = (error) => {
this.handleVirtualBgError(error, type, name);
removeSessionVirtualBackgroundInfo(webcamDeviceId);
reject(error);
};
const applyCustomVirtualBg = (backgrounds) => {
const background = backgrounds[uniqueId]
|| Object.values(backgrounds).find(bg => bg.uniqueId === uniqueId);
if (background && background.data) {
customParams = {
uniqueId,
file: background?.data,
};
} else {
handleFailure(new Error('Missing virtual background data'));
return;
}
this.handleVirtualBgSelected(type, name, customParams).then(resolve, handleFailure);
};
// If uniqueId is defined, this is a custom background. Fetch the custom
// params from the context and apply them
if (uniqueId) {
if (!this.context.loaded) {
// Virtual BG context might not be loaded yet (in case this is
// skipping the video preview). Load it manually.
VBGSelectorService.load(handleFailure, applyCustomVirtualBg);
} else {
applyCustomVirtualBg(this.context.backgrounds);
}
return;
}
// Built-in background, just apply it.
this.handleVirtualBgSelected(type, name, customParams).then(resolve, handleFailure);
} else {
resolve();
}
});
}
getCameraStream(deviceId, profile) {
async getCameraStream(deviceId, profile) {
const { webcamDeviceId } = this.state;
const { cameraAsContent, forceOpen } = this.props;
this.setState({
selectedProfile: profile.id,
@ -675,25 +775,42 @@ class VideoPreview extends Component {
this.terminateCameraStream(this.currentVideoStream, webcamDeviceId);
this.cleanupStreamAndVideo();
// The return of doGUM is an instance of BBBVideoStream (a thin wrapper over a MediaStream)
return PreviewService.doGUM(deviceId, profile).then((bbbVideoStream) => {
// Late GUM resolve, clean up tracks, stop.
if (!this._isMounted) return this.terminateCameraStream(bbbVideoStream, deviceId);
try {
// The return of doGUM is an instance of BBBVideoStream (a thin wrapper over a MediaStream)
const bbbVideoStream = await PreviewService.doGUM(deviceId, profile);
this.currentVideoStream = bbbVideoStream;
this.startCameraBrightness();
this.setState({
isStartSharingDisabled: false,
});
}).catch((error) => {
this.updateDeviceId(deviceId);
} catch(error) {
// When video preview is set to skip, we need some way to bubble errors
// up to users; so re-throw the error
if (!PreviewService.getSkipVideoPreview()) {
if (!this.shouldSkipVideoPreview()) {
this.handlePreviewError('do_gum_preview', error, 'displaying final selection');
} else {
throw error;
}
});
}
// Restore virtual background if it was stored in Local/Session Storage
try {
if (!cameraAsContent) await this.applyStoredVirtualBg(deviceId);
} catch (error) {
// Only bubble up errors in this case if we're skipping the video preview
// This is because virtual background failures are deemed critical when
// skipping the video preview, but not otherwise
if (this.shouldSkipVideoPreview()) {
throw error;
}
} finally {
// Late VBG resolve, clean up tracks, stop.
if (!this._isMounted) {
this.terminateCameraStream(bbbVideoStream, deviceId);
this.cleanupStreamAndVideo();
return;
}
this.setState({
isStartSharingDisabled: false,
});
}
}
displayPreview() {
@ -703,11 +820,20 @@ class VideoPreview extends Component {
}
skipVideoPreview() {
this.getInitialCameraStream().then(() => {
const { webcamDeviceId } = this.state;
const { forceOpen } = this.props;
return this.getInitialCameraStream(webcamDeviceId).then(() => {
this.handleStartSharing();
}).catch(error => {
PreviewService.clearWebcamDeviceId();
PreviewService.clearWebcamProfileId();
removeSessionVirtualBackgroundInfo(webcamDeviceId);
this.cleanupStreamAndVideo();
notify(this.handleGUMError(error), 'error', 'video');
// Mark the skip as failed so that the component will override any option
// to skip the video preview and display the default UI
if (this._isMounted) this.setState({ skipPreviewFailed: true });
throw error;
});
}
@ -849,10 +975,19 @@ class VideoPreview extends Component {
);
}
handleBrightnessAreaChange() {
const { wholeImageBrightness } = this.state;
this.currentVideoStream.toggleCameraBrightnessArea(!wholeImageBrightness);
this.setState({ wholeImageBrightness: !wholeImageBrightness });
async handleBrightnessAreaChange() {
const ENABLE_CAMERA_BRIGHTNESS = window.meetingClientSettings.public.app.enableCameraBrightness;
const CAMERA_BRIGHTNESS_AVAILABLE = ENABLE_CAMERA_BRIGHTNESS && isVirtualBackgroundSupported();
if (CAMERA_BRIGHTNESS_AVAILABLE && this.currentVideoStream) {
if (this.currentVideoStream?.virtualBgService == null) {
await this.startCameraBrightness();
}
const { wholeImageBrightness } = this.state;
this.currentVideoStream.toggleCameraBrightnessArea(!wholeImageBrightness);
this.setState({ wholeImageBrightness: !wholeImageBrightness });
}
}
renderBrightnessInput() {
@ -904,8 +1039,7 @@ class VideoPreview extends Component {
aria-describedby={'brightness-slider-desc'}
onChange={(e) => {
const brightness = e.target.valueAsNumber;
this.currentVideoStream.changeCameraBrightness(brightness);
this.setState({ brightness });
this.setCameraBrightness(brightness);
}}
disabled={!isVirtualBackgroundSupported() || isStartSharingDisabled}
/>
@ -937,7 +1071,8 @@ class VideoPreview extends Component {
const { isStartSharingDisabled, webcamDeviceId } = this.state;
const initialVirtualBgState = this.currentVideoStream ? {
type: this.currentVideoStream.virtualBgType,
name: this.currentVideoStream.virtualBgName
name: this.currentVideoStream.virtualBgName,
uniqueId: this.currentVideoStream.virtualBgUniqueId,
} : getSessionVirtualBackgroundInfo(webcamDeviceId);
const {
@ -1070,9 +1205,8 @@ class VideoPreview extends Component {
deviceError,
previewError,
} = this.state;
const shouldDisableButtons = PreviewService.getSkipVideoPreview()
&& !forceOpen
&& !(deviceError || previewError);
const shouldDisableButtons = this.shouldSkipVideoPreview()
&& !(deviceError || previewError);
const shared = this.isAlreadyShared(webcamDeviceId);
@ -1166,7 +1300,7 @@ class VideoPreview extends Component {
return null;
}
if (PreviewService.getSkipVideoPreview() && !forceOpen) {
if (this.shouldSkipVideoPreview()) {
return null;
}

44
bigbluebutton-html5/imports/ui/components/video-preview/service.js
View File

@ -16,9 +16,9 @@ const getDefaultProfile = () => {
// Unfiltered, includes hidden profiles
const CAMERA_PROFILES = window.meetingClientSettings.public.kurento.cameraProfiles || [];
return CAMERA_PROFILES.find(profile => profile.id === BBBStorage.getItem('WebcamProfileId'))
|| CAMERA_PROFILES.find(profile => profile.id === VideoService.getUserParameterProfile())
|| CAMERA_PROFILES.find(profile => profile.default)
return CAMERA_PROFILES.find((profile) => profile.id === BBBStorage.getItem('WebcamProfileId'))
|| CAMERA_PROFILES.find((profile) => profile.id === VideoService.getUserParameterProfile())
|| CAMERA_PROFILES.find((profile) => profile.default)
|| CAMERA_PROFILES[0];
};
@ -26,14 +26,14 @@ const getCameraAsContentProfile = () => {
// Unfiltered, includes hidden profiles
const CAMERA_PROFILES = window.meetingClientSettings.public.kurento.cameraProfiles || [];
return CAMERA_PROFILES.find(profile => profile.id == CAMERA_AS_CONTENT_PROFILE_ID)
|| CAMERA_PROFILES.find(profile => profile.default)
return CAMERA_PROFILES.find((profile) => profile.id == CAMERA_AS_CONTENT_PROFILE_ID)
|| CAMERA_PROFILES.find((profile) => profile.default);
};
const getCameraProfile = (id) => {
// Unfiltered, includes hidden profiles
const CAMERA_PROFILES = window.meetingClientSettings.public.kurento.cameraProfiles || [];
return CAMERA_PROFILES.find(profile => profile.id === id);
return CAMERA_PROFILES.find((profile) => profile.id === id);
};
// VIDEO_STREAM_STORAGE: Map<deviceId, MediaStream>. Registers WEBCAM streams.
@ -61,22 +61,18 @@ const storeStream = (deviceId, stream) => {
});
return true;
}
};
const getStream = (deviceId) => {
return VIDEO_STREAM_STORAGE.get(deviceId);
}
const getStream = (deviceId) => VIDEO_STREAM_STORAGE.get(deviceId);
const hasStream = (deviceId) => {
return VIDEO_STREAM_STORAGE.has(deviceId);
}
const hasStream = (deviceId) => VIDEO_STREAM_STORAGE.has(deviceId);
const deleteStream = (deviceId) => {
const stream = getStream(deviceId);
if (stream == null) return false;
MediaStreamUtils.stopMediaStreamTracks(stream);
return VIDEO_STREAM_STORAGE.delete(deviceId);
}
};
const promiseTimeout = (ms, promise) => {
const timeout = new Promise((resolve, reject) => {
@ -100,6 +96,7 @@ const promiseTimeout = (ms, promise) => {
const getSkipVideoPreview = () => {
const KURENTO_CONFIG = window.meetingClientSettings.public.kurento;
const BBBStorage = getStorageSingletonInstance();
const skipVideoPreviewOnFirstJoin = getFromUserSettings(
'bbb_skip_video_preview_on_first_join',
@ -110,8 +107,14 @@ const getSkipVideoPreview = () => {
KURENTO_CONFIG.skipVideoPreview,
);
const skipVideoPreviewIfPreviousDevice = getFromUserSettings(
'bbb_skip_video_preview_if_previous_device',
KURENTO_CONFIG.skipVideoPreviewIfPreviousDevice,
);
return (
(Storage.getItem('isFirstJoin') !== false && skipVideoPreviewOnFirstJoin)
|| (BBBStorage.getItem('WebcamDeviceId') && skipVideoPreviewIfPreviousDevice)
|| skipVideoPreview
);
};
@ -129,7 +132,7 @@ const digestVideoDevices = (devices, priorityDevice) => {
devices.forEach((device) => {
if (device.kind === 'videoinput') {
if (!webcams.some(d => d.deviceId === device.deviceId)) {
if (!webcams.some((d) => d.deviceId === device.deviceId)) {
// We found a priority device. Push it to the beginning of the array so we
// can use it as the "initial device"
if (priorityDevice && priorityDevice === device.deviceId) {
@ -138,8 +141,8 @@ const digestVideoDevices = (devices, priorityDevice) => {
webcams.push(device);
}
if (!device.label) { areLabelled = false }
if (!device.deviceId) { areIdentified = false }
if (!device.label) { areLabelled = false; }
if (!device.deviceId) { areIdentified = false; }
}
}
});
@ -226,9 +229,9 @@ const doGUM = (deviceId, profile) => {
const terminateCameraStream = (bbbVideoStream, deviceId) => {
// Cleanup current stream if it wasn't shared/stored
if (bbbVideoStream && !hasStream(deviceId)) {
bbbVideoStream.stop()
bbbVideoStream.stop();
}
}
};
export default {
promiseTimeout,
@ -236,9 +239,12 @@ export default {
getStorageSingletonInstance().setItem('WebcamDeviceId', deviceId);
},
webcamDeviceId: () => getStorageSingletonInstance().getItem('WebcamDeviceId'),
clearWebcamDeviceId: () => getStorageSingletonInstance().removeItem('WebcamDeviceId'),
changeProfile: (profileId) => {
getStorageSingletonInstance().setItem('WebcamProfileId', profileId);
},
webcamProfileId: () => getStorageSingletonInstance().getItem('WebcamProfileId'),
clearWebcamProfileId: () => getStorageSingletonInstance().removeItem('WebcamProfileId'),
getSkipVideoPreview,
storeStream,
getStream,

174
bigbluebutton-html5/imports/ui/components/video-preview/virtual-background/component.jsx
View File

@ -1,4 +1,6 @@
import React, { useState, useRef, useContext, useEffect } from 'react';
import React, {
useState, useRef, useContext, useEffect,
} from 'react';
import { findDOMNode } from 'react-dom';
import { defineMessages, injectIntl } from 'react-intl';
import PropTypes from 'prop-types';
@ -17,6 +19,7 @@ import withFileReader from '/imports/ui/components/common/file-reader/component'
import Skeleton, { SkeletonTheme } from 'react-loading-skeleton';
import 'react-loading-skeleton/dist/skeleton.css';
import { getSettingsSingletonInstance } from '/imports/ui/services/settings';
import { getStorageSingletonInstance } from '/imports/ui/services/storage';
const { MIME_TYPES_ALLOWED, MAX_FILE_SIZE } = VirtualBgService;
@ -144,47 +147,58 @@ const VirtualBgSelector = ({
}
}, [isCustomVirtualBackgroundsEnabled]);
const _virtualBgSelected = (type, name, index, customParams) =>
handleVirtualBgSelected(type, name, customParams)
.then(switched => {
// Reset to the base NONE_TYPE effect if it failed because the expected
// behaviour from upstream's method is to actually stop/reset the effect
// service if it fails
if (!switched) {
return setCurrentVirtualBg({ type: EFFECT_TYPES.NONE_TYPE });
}
useEffect(() => {
const virtualBgData = {
name: currentVirtualBg.name,
type: currentVirtualBg.type,
};
setCurrentVirtualBg({ type, name });
const virtualBgArray = [virtualBgData];
if (!index || index < 0) return;
const BBBStorage = getStorageSingletonInstance();
BBBStorage.setItem('WebcamBackground', virtualBgArray);
}, [currentVirtualBg]);
if (!shouldEnableBackgroundUpload(isCustomVirtualBackgroundsEnabled)) {
findDOMNode(inputElementsRef.current[index]).focus();
const _virtualBgSelected = (type, name, index, customParams) => handleVirtualBgSelected(type, name, customParams)
.then((switched) => {
// Reset to the base NONE_TYPE effect if it failed because the expected
// behaviour from upstream's method is to actually stop/reset the effect
// service if it fails
if (!switched) {
return setCurrentVirtualBg({ type: EFFECT_TYPES.NONE_TYPE });
}
setCurrentVirtualBg({ type, name, uniqueId: customParams?.uniqueId });
if (!index || index < 0) return;
if (!shouldEnableBackgroundUpload(isCustomVirtualBackgroundsEnabled)) {
findDOMNode(inputElementsRef.current[index]).focus();
} else {
if (customParams) {
dispatch({
type: 'update',
background: {
filename: name,
uniqueId: customParams.uniqueId,
data: customParams.file,
custom: true,
lastActivityDate: Date.now(),
},
});
} else {
if (customParams) {
dispatch({
type: 'update',
background: {
filename: name,
uniqueId: customParams.uniqueId,
data: customParams.file,
custom: true,
lastActivityDate: Date.now(),
},
});
} else {
dispatch({
type: 'update',
background: {
uniqueId: name,
custom: false,
lastActivityDate: Date.now(),
},
});
}
findDOMNode(inputElementsRef.current[0]).focus();
dispatch({
type: 'update',
background: {
uniqueId: name,
custom: false,
lastActivityDate: Date.now(),
},
});
}
});
findDOMNode(inputElementsRef.current[0]).focus();
}
});
const renderDropdownSelector = () => {
const disabled = locked || !isVirtualBackgroundSupported();
@ -195,7 +209,7 @@ const VirtualBgSelector = ({
<Styled.Select
value={JSON.stringify(currentVirtualBg)}
disabled={disabled}
onChange={event => {
onChange={(event) => {
const { type, name } = JSON.parse(event.target.value);
_virtualBgSelected(type, name);
}}
@ -211,18 +225,21 @@ const VirtualBgSelector = ({
{IMAGE_NAMES.map((imageName, i) => {
const k = `${imageName}-${i}`;
return (
<option key={k} value={JSON.stringify({
type: EFFECT_TYPES.IMAGE_TYPE,
name: imageName,
})}>
{imageName.split(".")[0]}
<option
key={k}
value={JSON.stringify({
type: EFFECT_TYPES.IMAGE_TYPE,
name: imageName,
})}
>
{imageName.split('.')[0]}
</option>
);
})}
</Styled.Select>
</div>
);
}
};
const handleCustomBgChange = (event) => {
const file = event.target.files[0];
@ -261,32 +278,30 @@ const VirtualBgSelector = ({
const renderThumbnailSelector = () => {
const disabled = locked || !isVirtualBackgroundSupported();
const Settings = getSettingsSingletonInstance();
const isRTL = Settings.application.isRTL;
const { isRTL } = Settings.application;
const IMAGE_NAMES = getImageNames();
const renderBlurButton = (index) => {
return (
<Styled.ThumbnailButtonWrapper
key={`blur-${index}`}
>
<Styled.ThumbnailButton
background={getVirtualBackgroundThumbnail(BLUR_FILENAME)}
aria-label={intl.formatMessage(intlMessages.blurLabel)}
label={intl.formatMessage(intlMessages.blurLabel)}
aria-describedby={`vr-cam-btn-blur`}
tabIndex={disabled ? -1 : 0}
hideLabel
aria-pressed={currentVirtualBg?.name?.includes('blur') || currentVirtualBg?.name?.includes('Blur')}
disabled={disabled}
ref={ref => { inputElementsRef.current[index] = ref; }}
onClick={() => _virtualBgSelected(EFFECT_TYPES.BLUR_TYPE, 'Blur', index)}
/>
<div aria-hidden className="sr-only" id={`vr-cam-btn-blur`}>
{intl.formatMessage(intlMessages.camBgAriaDesc, { 0: EFFECT_TYPES.BLUR_TYPE })}
</div>
</Styled.ThumbnailButtonWrapper>
);
};
const renderBlurButton = (index) => (
<Styled.ThumbnailButtonWrapper
key={`blur-${index}`}
>
<Styled.ThumbnailButton
background={getVirtualBackgroundThumbnail(BLUR_FILENAME)}
aria-label={intl.formatMessage(intlMessages.blurLabel)}
label={intl.formatMessage(intlMessages.blurLabel)}
aria-describedby="vr-cam-btn-blur"
tabIndex={disabled ? -1 : 0}
hideLabel
aria-pressed={currentVirtualBg?.name?.includes('blur') || currentVirtualBg?.name?.includes('Blur')}
disabled={disabled}
ref={(ref) => { inputElementsRef.current[index] = ref; }}
onClick={() => _virtualBgSelected(EFFECT_TYPES.BLUR_TYPE, 'Blur', index)}
/>
<div aria-hidden className="sr-only" id="vr-cam-btn-blur">
{intl.formatMessage(intlMessages.camBgAriaDesc, { 0: EFFECT_TYPES.BLUR_TYPE })}
</div>
</Styled.ThumbnailButtonWrapper>
);
const renderDefaultButton = (imageName, index) => {
const label = intl.formatMessage(intlMessages[imageName.split('.').shift()], {
@ -307,7 +322,7 @@ const VirtualBgSelector = ({
aria-describedby={`vr-cam-btn-${index + 1}`}
aria-pressed={currentVirtualBg?.name?.includes(imageName.split('.').shift())}
hideLabel
ref={ref => inputElementsRef.current[index] = ref}
ref={(ref) => inputElementsRef.current[index] = ref}
onClick={() => _virtualBgSelected(EFFECT_TYPES.IMAGE_TYPE, imageName, index)}
disabled={disabled}
background={getVirtualBackgroundThumbnail(imageName)}
@ -317,7 +332,7 @@ const VirtualBgSelector = ({
{intl.formatMessage(intlMessages.camBgAriaDesc, { 0: label })}
</div>
</Styled.ThumbnailButtonWrapper>
)
);
};
const renderCustomButton = (background, index) => {
@ -339,7 +354,7 @@ const VirtualBgSelector = ({
aria-describedby={`vr-cam-btn-${index + 1}`}
aria-pressed={currentVirtualBg?.name?.includes(filename)}
hideLabel
ref={ref => inputElementsRef.current[index] = ref}
ref={(ref) => inputElementsRef.current[index] = ref}
onClick={() => _virtualBgSelected(
EFFECT_TYPES.IMAGE_TYPE,
filename,
@ -381,9 +396,9 @@ const VirtualBgSelector = ({
const renderInputButton = () => (
<>
<Styled.BgCustomButton
icon='plus'
icon="plus"
label={intl.formatMessage(intlMessages.customLabel)}
aria-describedby={`vr-cam-btn-custom`}
aria-describedby="vr-cam-btn-custom"
hideLabel
tabIndex={disabled ? -1 : 0}
disabled={disabled}
@ -402,7 +417,7 @@ const VirtualBgSelector = ({
style={{ display: 'none' }}
accept={MIME_TYPES_ALLOWED.join(', ')}
/>
<div aria-hidden className="sr-only" id={`vr-cam-btn-custom`}>
<div aria-hidden className="sr-only" id="vr-cam-btn-custom">
{intl.formatMessage(intlMessages.customDesc)}
</div>
</>
@ -411,17 +426,17 @@ const VirtualBgSelector = ({
const renderNoneButton = () => (
<>
<Styled.BgNoneButton
icon='close'
icon="close"
label={intl.formatMessage(intlMessages.noneLabel)}
aria-pressed={currentVirtualBg?.name === undefined}
aria-describedby={`vr-cam-btn-none`}
aria-describedby="vr-cam-btn-none"
hideLabel
tabIndex={disabled ? -1 : 0}
disabled={disabled}
onClick={() => _virtualBgSelected(EFFECT_TYPES.NONE_TYPE)}
data-test="noneBackgroundButton"
/>
<div aria-hidden className="sr-only" id={`vr-cam-btn-none`}>
<div aria-hidden className="sr-only" id="vr-cam-btn-none">
{intl.formatMessage(intlMessages.camBgAriaDesc, { 0: EFFECT_TYPES.NONE_TYPE })}
</div>
</>
@ -461,10 +476,9 @@ const VirtualBgSelector = ({
.map((background, index) => {
if (background.custom !== false) {
return renderCustomButton(background, index);
} else {
const isBlur = background.uniqueId.includes('Blur');
return isBlur ? renderBlurButton(index) : renderDefaultButton(background.uniqueId, index);
}
const isBlur = background.uniqueId.includes('Blur');
return isBlur ? renderBlurButton(index) : renderDefaultButton(background.uniqueId, index);
})}
{renderInputButton()}

45
bigbluebutton-html5/imports/ui/components/video-provider/component.tsx
View File

@ -1124,23 +1124,6 @@ class VideoProvider extends Component<VideoProviderProps, VideoProviderState> {
// hidden/shown when the stream is attached.
notifyStreamStateChange(stream, pc.connectionState);
VideoProvider.attach(peer, videoElement);
if (isLocal) {
if (peer.bbbVideoStream == null) {
this.handleVirtualBgError(new TypeError('Undefined media stream'));
return;
}
const deviceId = MediaStreamUtils.extractDeviceIdFromStream(
peer.bbbVideoStream.mediaStream,
'video',
);
const { type, name } = getSessionVirtualBackgroundInfo(deviceId);
VideoProvider.restoreVirtualBackground(peer.bbbVideoStream, type, name).catch((error) => {
this.handleVirtualBgError(error, type, name);
});
}
}
}
@ -1172,34 +1155,6 @@ class VideoProvider extends Component<VideoProviderProps, VideoProviderState> {
}, `Failed to start virtual background by dropping image: ${error.message}`);
}
static restoreVirtualBackground(stream: BBBVideoStream, type: string, name: string) {
return new Promise((resolve, reject) => {
if (type !== EFFECT_TYPES.NONE_TYPE) {
stream.startVirtualBackground(type, name).then(() => {
resolve(null);
}).catch((error: Error) => {
reject(error);
});
}
resolve(null);
});
}
handleVirtualBgError(error: Error, type?: string, name?: string) {
const { intl } = this.props;
logger.error({
logCode: 'video_provider_virtualbg_error',
extraInfo: {
errorName: error.name,
errorMessage: error.message,
virtualBgType: type,
virtualBgName: name,
},
}, `Failed to restore virtual background after reentering the room: ${error.message}`);
notify(intl.formatMessage(intlClientErrors.virtualBgGenericError), 'error', 'video');
}
createVideoTag(stream: string, video: HTMLVideoElement) {
const peer = this.webRtcPeers[stream];
this.videoTags[stream] = video;

64
bigbluebutton-html5/imports/ui/services/virtual-background/service.js
View File

@ -8,29 +8,29 @@ const EFFECT_TYPES = {
BLUR_TYPE: 'blur',
IMAGE_TYPE: 'image',
NONE_TYPE: 'none',
}
};
const MODELS = {
model96: {
path: '/resources/tfmodels/segm_lite_v681.tflite',
segmentationDimensions: {
height: 96,
width: 160
}
height: 96,
width: 160,
},
},
model144: {
path: '/resources/tfmodels/segm_full_v679.tflite',
segmentationDimensions: {
height: 144,
width: 256
}
height: 144,
width: 256,
},
},
};
const getBasePath = () => {
const BASE_PATH = window.meetingClientSettings.public.app.cdn
+ window.meetingClientSettings.public.app.basename
+ window.meetingClientSettings.public.app.instanceId;
+ window.meetingClientSettings.public.app.basename
+ window.meetingClientSettings.public.app.instanceId;
return BASE_PATH;
};
@ -65,47 +65,46 @@ const createVirtualBackgroundStream = (type, name, isVirtualBackground, stream,
backgroundFilename: name,
isVirtualBackground,
customParams,
}
};
return createVirtualBackgroundService(buildParams).then((service) => {
const effect = service.startEffect(stream)
const effect = service.startEffect(stream);
return { service, effect };
});
}
};
const getVirtualBackgroundThumbnail = (name) => {
if (name === BLUR_FILENAME) {
return getBasePath() + '/resources/images/virtual-backgrounds/thumbnails/' + name;
return `${getBasePath()}/resources/images/virtual-backgrounds/thumbnails/${name}`;
}
return (getIsStoredOnBBB() ? getBasePath() : '') + getThumbnailsPath() + name;
}
};
// Stores the last chosen camera effect into the session storage in the following format:
// {
// type: <EFFECT_TYPES>,
// name: effect filename, if any
// }
const setSessionVirtualBackgroundInfo = (type, name, deviceId) => (
Session.setItem(`VirtualBackgroundInfo_${deviceId}`, { type, name })
);
const setSessionVirtualBackgroundInfo = (deviceId, type, name, uniqueId = null) => {
Session.setItem(`VirtualBackgroundInfo_${deviceId}`, { type, name, uniqueId });
};
const getSessionVirtualBackgroundInfo = (deviceId) => (
Session.getItem(`VirtualBackgroundInfo_${deviceId}`) || {
type: EFFECT_TYPES.NONE_TYPE,
}
);
const getSessionVirtualBackgroundInfo = (deviceId) => Session
.getItem(`VirtualBackgroundInfo_${deviceId}`) || {
type: EFFECT_TYPES.NONE_TYPE,
};
const getSessionVirtualBackgroundInfoWithDefault = (deviceId) => (
Session.getItem(`VirtualBackgroundInfo_${deviceId}`) || {
type: EFFECT_TYPES.BLUR_TYPE,
name: BLUR_FILENAME,
}
);
const getSessionVirtualBackgroundInfoWithDefault = (deviceId) => Session
.getItem(`VirtualBackgroundInfo_${deviceId}`) || {
type: EFFECT_TYPES.BLUR_TYPE,
name: BLUR_FILENAME,
};
const isVirtualBackgroundSupported = () => {
return !(deviceInfo.isIos || browserInfo.isSafari);
}
const removeSessionVirtualBackgroundInfo = (deviceId) => Session
.removeItem(`VirtualBackgroundInfo_${deviceId}`);
const isVirtualBackgroundSupported = () => !(deviceInfo.isIos || browserInfo.isSafari);
const getVirtualBgImagePath = () => {
const {
@ -113,7 +112,7 @@ const getVirtualBgImagePath = () => {
} = window.meetingClientSettings.public.virtualBackgrounds;
return (getIsStoredOnBBB() ? getBasePath() : '') + IMAGES_PATH;
}
};
export {
getBasePath,
@ -124,6 +123,7 @@ export {
setSessionVirtualBackgroundInfo,
getSessionVirtualBackgroundInfo,
getSessionVirtualBackgroundInfoWithDefault,
removeSessionVirtualBackgroundInfo,
isVirtualBackgroundSupported,
createVirtualBackgroundStream,
getVirtualBackgroundThumbnail,

4
bigbluebutton-html5/imports/ui/services/webrtc-base/bbb-video-stream.js
View File

@ -44,6 +44,7 @@ export default class BBBVideoStream extends EventEmitter2 {
this.virtualBgService = null;
this.virtualBgType = EFFECT_TYPES.NONE_TYPE;
this.virtualBgName = BLUR_FILENAME;
this.virtualBgUniqueId = null;
this._trackOriginalStreamTermination();
}
@ -90,6 +91,7 @@ export default class BBBVideoStream extends EventEmitter2 {
});
this.virtualBgType = type;
this.virtualBgName = name;
this.virtualBgUniqueId = customParams?.uniqueId;
return Promise.resolve();
} catch (error) {
return Promise.reject(error);
@ -109,6 +111,7 @@ export default class BBBVideoStream extends EventEmitter2 {
this.virtualBgService = service;
this.virtualBgType = type;
this.virtualBgName = name;
this.virtualBgUniqueId = customParams?.uniqueId;
this.originalStream = this.mediaStream;
this.mediaStream = effect;
this.isVirtualBackgroundEnabled = true;
@ -135,6 +138,7 @@ export default class BBBVideoStream extends EventEmitter2 {
this.virtualBgType = EFFECT_TYPES.NONE_TYPE;
this.virtualBgName = undefined;
this.virtualBgUniqueId = null;
this.mediaStream = this.originalStream;
this.isVirtualBackgroundEnabled = false;
}

1
bigbluebutton-html5/private/config/settings.yml
View File

@ -447,6 +447,7 @@ public:
autoShareWebcam: false
skipVideoPreview: false
skipVideoPreviewOnFirstJoin: false
skipVideoPreviewIfPreviousDevice: false
# cameraSortingModes.paginationSorting: sorting mode to be applied when pagination is active
# cameraSortingModes.defaultSorting: sorting mode when pagination is not active (full mesh)
# Current implemented modes are:

1
bigbluebutton-tests/playwright/core/settings.js
View File

@ -42,6 +42,7 @@ async function generateSettingsData(page) {
webcamSharingEnabled: settingsData.kurento.enableVideo,
skipVideoPreview: settingsData.kurento.skipVideoPreview,
skipVideoPreviewOnFirstJoin: settingsData.kurento.skipVideoPreviewOnFirstJoin,
skipVideoPreviewIfPreviousDevice: settingsData.kurento.skipVideoPreviewIfPreviousDevice,
}
return settings;

1
bigbluebutton-tests/playwright/parameters/constants.js
View File

@ -49,6 +49,7 @@ exports.forceRestorePresentationOnNewEvents = 'userdata-bbb_force_restore_presen
exports.recordMeeting = 'record=true';
exports.skipVideoPreview = 'userdata-bbb_skip_video_preview=true';
exports.skipVideoPreviewOnFirstJoin = 'userdata-bbb_skip_video_preview_on_first_join=true';
exports.skipVideoPreviewIfPreviousDevice = 'userdata-bbb_skip_video_preview_if_previous_device=true';
exports.mirrorOwnWebcam = 'userdata-bbb_mirror_own_webcam=true';
exports.showParticipantsOnLogin = 'userdata-bbb_show_participants_on_login=false';
exports.hideActionsBar = 'userdata-bbb_hide_actions_bar=true';

8
bigbluebutton-tests/playwright/parameters/customparameters.js
View File

@ -171,6 +171,14 @@ class CustomParameters extends MultiUsers {
await this.modPage.shareWebcam(true, videoPreviewTimeout);
}
async skipVideoPreviewIfPreviousDevice() {
await this.modPage.waitForSelector(e.joinVideo);
const { videoPreviewTimeout } = this.modPage.settings;
await this.modPage.shareWebcam(true, videoPreviewTimeout);
await this.modPage.waitAndClick(e.leaveVideo, VIDEO_LOADING_WAIT_TIME);
await this.modPage.shareWebcam(false);
}
async mirrorOwnWebcam() {
await this.modPage.waitAndClick(e.joinVideo);
await this.modPage.waitForSelector(e.webcamMirroredVideoPreview);

6
bigbluebutton-tests/playwright/parameters/parameters.spec.js
View File

@ -505,6 +505,12 @@ test.describe.parallel('Custom Parameters', () => {
await customParam.skipVideoPreviewOnFirstJoin();
});
test('Skip Video Preview if Previous Device', async ({ browser, context, page }) => {
const customParam = new CustomParameters(browser, context);
await customParam.initModPage(page, true, { joinParameter: c.skipVideoPreviewIfPreviousDevice });
await customParam.skipVideoPreviewIfPreviousDevice();
});
test('Mirror Own Webcam', async ({ browser, context, page }) => {
const customParam = new CustomParameters(browser, context);
await customParam.initModPage(page, true, { joinParameter: c.mirrorOwnWebcam });

1
docs/docs/administration/customize.md
View File

@ -1422,6 +1422,7 @@ Useful tools for development:
| `userdata-bbb_record_video=` | If set to `false`, the user won't have her/his video stream recorded | `true` |
| `userdata-bbb_skip_video_preview=` | If set to `true`, the user will not see a preview of their webcam before sharing it | `false` |
| `userdata-bbb_skip_video_preview_on_first_join=` | (Introduced in BigBlueButton 2.3) If set to `true`, the user will not see a preview of their webcam before sharing it when sharing for the first time in the session. If the user stops sharing, next time they try to share webcam the video preview will be displayed, allowing for configuration changes to be made prior to sharing | `false` |
| `userdata-bbb_skip_video_preview_if_previous_device=` | (Introduced in BigBlueButton 3.0) If set to `true`, the user will not see a preview of their webcam before sharing it if session has a valid input device stored previously | `false` |
| `userdata-bbb_mirror_own_webcam=` | If set to `true`, the client will see a mirrored version of their webcam. Doesn't affect the incoming video stream for other users. | `false` |
| `userdata-bbb_fullaudio_bridge=` | Specifies the audio bridge to be used in the client. Supported values: `sipjs`, `fullaudio`. | `fullaudio` |
| `userdata-bbb_transparent_listen_only=` | If set to `true`, the experimental "transparent listen only" audio mode will be used | `false` |

5
docs/docs/new-features.md
View File

@ -147,6 +147,11 @@ In BigBlueButton 3.0.0-alpha.5 we replaced the JOIN parameter `defaultLayout` wi
In BigBlueButton 2.7.5/3.0.0-alpha.5 we stopped propagating the events.xml event TranscriptUpdatedRecordEvent due to some issues with providing too much and too repetitive data.
#### Added new setting and userdata to allow skipping video preview if session has valid input devices stored
- Client settings.yml: `skipVideoPreviewIfPreviousDevice`. Defaults to `false`
- Can be overrided on join Custom Parameter with: `userdata-bbb_skip_video_preview_if_previous_device=`
### Changes to events.xml
Retired events