There are still a bunch of edge cases and issues with reconnection
scenarios for video:
- Signaling socket refuses to reconnect once maxRetries expire
- Race conditions on local stream attachment: local camera wouldn't be
correctly rendered _if_ the attached stream existed _without_ video
tracks yet
- Video tracks leak on local streams when replacing them (virtual bgs)
- Completely ignoring Meteor state when trying to reconnect cameras
- Streams aren't proactively stopped when the signaling socket dies
- Outbound request queues aren't isolated by stream nor are they
flushed when a newer peer with the same ID is created
- Server originated negotiation errors won't trigger a local peer
cleanup - thus leaving dangling peers that take way too long to
reconnect
This commit fixes or improves all of the aforementioned issues, +:
- Remove unused arguments in the peer (client->SFU) 'start' request
- Prevent crashes when trying to render video-list-items without user
data (which might happen on re-connections)
video-provider's current ping-pong is as good as nothing in 2.5+. We
were counting on Meteor's (and consequently the component's mount state)
before 2.5 to act as a "heartbeat" as far as the socket is concerned.
The ping-pong served only to sustain traffic for finnicky,
traffic-dependant firewall.
Since 2.5, the component's state is _kind of_ detached from Meteor's -
which means it won't unmount when Meteor disconnects. That causes the
video-provider websocket to lose its borrowed heartbeat and leads to a
bunch of reconnectiong inconsistencies, the worst of them being a stuck,
useless signaling socket that will cause cameras not to work until a
client refresh.
This commit does the following:
- Implements actual heartbeat checks to trigger signaling socket
reconnects when necessary, all within the scope of video-provider
- Remove borked, eons old 'offline'/'online' event handlers: they were
causing unnecessary camera drops AND causing video-provider to
generate a stuck signaling socket
- Properly catch WebSockets.send errors
The stream state change handler in video-list-item is using a component
state reference inside a DOM event callback - which means it is always
presuming `isStreamHealthy` is false (initial value). That prevents the
health state from actually transitioning when necessary (and
consequently rendering the reconnecting view in video-list item).
This commit removes the state-based transition check in the state change
handler and unifies the reconnecting view to use the username
placeholde (replacing the loading spinners).
Firefox has a buggy ICE implementation and needs WebRTC media traffic to
be routed through a turn server to work reliably with mediasoup.
Use the information fetched by the STUN API to determine if the operator
has configured a turn server. If there is one force firefox to use it.
Closes#16164
Under some specific scenarios, the virtual background restore code might
kick in _before_ the preloaded MediaStream is actually assigned to the
peer instance. That causes a crash because the peer attachment code
(where the vbg restore is triggered) tries to access the device ID of
said MediaStream - and it is undefined in the first place because it was
only being set in after the initial offer is generated which is an async
procedure.
This commit guarantees a preloaded stream is set before the peer is
flagged as created - so it's accessible by the attachment code.
It also checks whether there's a MediaStream available when trying to
restore VBGs to prevent undefined behaviors.
RTCRTPSender exposes DSCP marking via `networkPriority` in the encodings
configuration dictionaries. That should allow us to control
QoS priorities for different media streams, eg audio with higher network
priority than video. The only browser that implements that right
now is Chromium.
To use this, the public.app.media.networkPriorities configuration in
settings.yml. Audio, camera and screenshare priorities can be controlled
separately. For further info on the possible values, see:
- https://www.w3.org/TR/webrtc-priority/
- https://datatracker.ietf.org/doc/html/rfc8837#section-5
The 'inactive' event is fired whenever the stream gets inactive (ie it
cannot be used anymore), and there are scenarios where that is
unexpected behavior and must be handled accordingly.
The main example of that is when gUM permissions are revoked by the user
via the browser's permission management panel.
Since MediaStream/Track inactive events aren't being handled in such
scenarios, what actually happens is that the camera just freezes without
further indication why.
This commit handles those scenarios in both video-preview and
video-provider by:
- 1) correctly stopping the camera (provider)
- 2) surfacing a toast (provider) or error indication (preview)
Add a new avatar component to video list item
Change the design of the components, following the new video list idea
Add icons related to the state of the user
New features:
- A simplified echo test mode that only does a local loopback (instead of
going to FS and back)
- A volume meter for microphone streams to the AudioSettings view
Those two features are experimental and disabled by default; see
public.app.media.simplifiedEchoTest and public.app.media.showVolumeMeter configs
Collateral changes:
- fix: localize fallback device strings in AudioSettings/DeviceSelector
- Refactor on some media stream utils to be re-usable across components
- Refactor in AudioSettings to keep gUM #uses stable.
* TODO: need to pass streams through AudioManager to avoid the surplus gUM.
- fix(audio): drop ScriptProcessorNode usage (deprecated)
* Used in volume meter for tracking - use hark instead
Tries to mitigate too-rapidly-switching camera profiles causing video freezes
due to encoder resets. Excluding constraints might not help a lot since
the thing that actually restarts the encoder is the bitrate change, but
they're not really important in the context of dynamic profiles.
We can't get rid of bitrate changes, though, since it's what does the actual
quality constraining.
The camera profile change debounce timer is 2.5s by default (which is
the same timer used for floor changes).
Also fixed an issue with camera profile backfiring due to badly defined peers
Include `meetingCameraCap` API param on create and enforce both server and
client to control the number of simultaneous webcams a meeting can have.
Disabled by default.
Include `userCameraCap` API param on create and enforce both server and
client to control the number of simultaneous webcams an user can share.
Default set to 3.
video-provider is a race-condition prone mess and those callbacks were missing try-catches so eventual failures would bubble up as uncaught errors and not be logged properly
It`s worth mentioning that due to a number of tangential design failures in that component, 90% of the errors generate are, to the end user, invisible false positives. Thus: expect an increase in false-positive errors logs with this
Makes clicking the camera button always open the video preview modal
when a camera is already shared on a mobile endpoint even when
skip_video_preview is true.
This allows mobile users to change virtual backgrounds.
Video preview modal opens when user clicks on the webcam selector
button/chevron even if the client is set to skip video preview
(userdata-bbb_skip_video_preview=true).
Added extra proptypes validation to video-list-item
Made sure to add extra ?. checks to voiceUsers as well because... who knows?
Removed unused user role prop from video-list-item
The base peer object reference was moved from the component to service for _reasons_
That caused an issue where the component lifecycle would mess up that
centralized reference dictionary on certain conditions. That could cause viewing
sessions to fail intermittently
This reverts the location of the base dictionary reference back to
video-provider/component
Fix a problem that username in video container isn't wrapped if it's too long.
Fix a problem that can cause the audio indicator to disappear.
Fix some problems with the overflow property.
Refactor in css for the talking indicator be more centralized
- forceRelayOnFirefox: whether TURN/relay usage should be forced to work
around Firefox's lack of support for regular nomination when dealing with
ICE-litee peers (e.g.: mediasoup).
* See: https://bugzilla.mozilla.org/show_bug.cgi?id=1034964
- iOS endpoints are ignored from the trigger because _all_ iOS browsers
are either native WebKit or WKWebView based (so they shouldn't be affected)
Add the <ButtonEmoji/> to the webcam button. Now it
is possible to change the webcam settings and change
your settings without having to turn off sharing.
Modifies the <ButtonEmoji/> component so that it can
receive an onClick per props.
It is possible to disable this feat in the settings.yml
ICE lite servers (eg mediasoup) dont need candidates signaled out-of-band; neither does KMS in certain scenarios
Disable their signaling saves us some ticks in bbb-webrtc-sfu and some bandwidth all around
There could be a race condition where a peer creation (async) would resolve after the provider was unmounted
That would lead to a state inconsistency which could generate all sorts of cryptic problems
Applies to video, listen only and screen sharing
New metadata values: media-server-video, media-server-listenonly, media-server-screenshare; parameter is a String
Added support for getStats in screenshare's service. This works similar
to the getStats for video provider, and the information retrieved from
screenshare is added to the video information for cameras.
We now retrieve update information about active video peers, and calculates
download and upload rates. These rates are the sum of data transfered in
all video peers.
Screenshare stats is not being added to the sum, yet.
Let the server generate subscriber offers so that it becomes easier for us to
do payload type normalization between publishers and subscribers
Also split peer creation for pubs and subs into separate methods for clarity
Should fix an issue with the recent Chrome 92 intervention that limits
the number of concurrent WebMediaPlayers (an inner element of
HTMLMediaElements) to 75/40.
Webcam video elements were being left dangling in paused state despite
the elements themselves being cleaned up from the component. That
generated a skewed accounting of WebMediaPlayers in the session.
Changes (maybe not a complete list):
- Disable virtualbgs by default
- Move the virtualbg selector in video-preview to the side below the
profile selection
- Restore old video-preview sizes
- Add a wrapper class for MediaStreams (BBBVideoStream)
- Centralize virtualbg services and business logic code into BBBVideoStream
- Refactor and centralize virtualbg constant fetching
- Refactor and centralize virtualbg config fetching
- Organize virtualbg type definitions
- Remove added states in video-provider to prevent further bloat
- Remove added states in video-preview to prevent further bloat
- Lock virtual bg switching while video-preview itself is locked
- Add proper virtualbg error surfacing via toasts
- Refactor iOS availability detection to use centralized UA checker
- Avoid calling gUM when toggling virtualbgs on/off
- Make virtualbg video-list-item action a toggle instead of a
state-aware action
- Make virtualbg switching work in video-preview for cameras that are
already shared. Especially useful when there are multiple source
cameras, and will be important in the near future
- Add Derivative Work notices in files that are partially copied from
jitsi-meet
- Simplify track replacing in video-provider
- Split video-preview UI code for virtualbgs into a separate functional component
Shave off the number of calls in video-preview and video-provider by
using a stream storage
We don´t call an upfront gUM in video-preview anymore to lift the
fingerprinting barrier on device labels and IDs. Flow has been reversed:
upfront enumerate, load first preview, then check if previous
enumeration was obfuscated.
Add a stream storage in video-preview`s service to avoid re-fetching
them in video-providerj
Remove some unneeded video-preview container props
Improve some of video-preview`s error locales
Remove parts of a previous connection monitor.
To add some context (as far as my memory goes) to the multiple connection
monitor features the product has, `stats` (currently named `connection status`)
was introduced at the Flash client back in ~2016. @fcecagno and I did it
as a BigBlueButton's Summit activity. Our work was squashed into a single
commit in 92554f8b3e :).
I'm not sure about the whole story behind `network information` (the late
connection monitor added to the HTML5 client) but I assume it should work
as a collector for a bunch of different connectivity monitors. I remember
when it was introduced but I don't know why it wasn't adopted. My best guess
would be because of some performance issues the `user list` had back then.
To follow on why `connection status` replaced `network information` at the
HTML5 client, when I did the `multiple webcams` feature I had to refactor
a big chunk of the `video provider` (#8374). Something that wasn't really
helping there was the adaptation of `stats` that was made to show local
feedback for each webcam connection. Although this feature wasn't being
used anymore, `network information` did rely on that to build up data. With
this monitor gone I assumed it was my responsibility to provide an alternative
so I promoted Mconf's port of the Flash `stats` monitor to BigBlueButton's
HTML5 client (#8579).
Well, that's my perspective on how things went for those features. If
anyone would like to correct me on something or add something else on
that history I would appreciate to know.
Add the `camera` icon in the user list for whoever is sharing,
in order to improve the understanding of who is sharing the webcam.
It is possible to enable/disable this indication in the settings.yml
Video streams can be sorted by voice floor activity in the client according to FreeSWITCH´s floor events. The feature works together with pagination, essentially giving an Last-N like experience while not disrupting too much
Made video stream sorting extensible in a way. The sorting modes for pagination and unbounded can be configured in settings.yml and new sorting modes can be added to the stream sorting util under video-provider. Inline docs explain how to do that
Changed how the stream ID attribute from video-streams collection was passed to downstream components; we had an array map that was executed every change just to map stream to cameraId, which is bizarre. So I changed the cameraId usage in downstream components to be conformat with the collection attributes and shaved off the map where it wasnt needed
Add better selectors to video-list-item container´s VoiceUser fetch
New config called paginationThreshold defines classes of page sizes that depend on the number of participants of a meeting
The rationale is pretty much the same as the cameraQualityThresholds, but the thresholds are users here and the ceilings are the page sizes
Problem: setReconnectionTimeout was being called in the first candidate generation to set the negotiation/reconnection timeout up. That caused some browsers or specific scenarios (mainly envs without STUN) to establish the negotiation (playStart) before generating any useful out-of-band candidates (relay). That would cause the timeout to be set AFTER it is supposed to be cleared due to success (playStart), making the webcam drop after a while
So I moved the setReconnectionTimeout call to a safer spot: right after the first negotiation requisition goes out to bbb-webrtc-sfu
There's some scenarios that video errors triggers multiple toast notifications
that don't have any mapped defined message feedback so they all drop to the default
permission error. This leaves the impression that something is broken at the toast
container.
Since those messages don't bring up much information about the problem we can avoid sending
them until we don't have a more informative one to notify the user.
