This is an initial, experimental implementation of the feature proposed in
https://github.com/bigbluebutton/bigbluebutton/issues/14021.
The intention is to phase out the explicit listen only mode with two
overarching goals:
- Reduce UX friction and increase familiarity: the existence of a separate
listen only mode is a source of confusion for the majority of users
Reduce average server-side CPU usage while also making it possible for
having full audio-only meetings.
The proof-of-concept works based on the assumption that a "many
concurrent active talkers" scenario is both rare and not useful. With
that in mind, this including two server-side triggers:
- On microphone inactivity (currently mute action that is sustained for
4 seconds, configurable): FreeSWITCH channels are held (which translates
to much lower CPU usage, virtually 0%). Receiving channels are switched,
server side, to a listening mode (SFU, mediasoup).
* This required an extension to mediasoup two allow re-assigning producers
to already established consumers. No re-negotiation is done.
- On microphone activity (currently unmute action, immediate):
FreeSWITCH channels are unheld, listening mode is deactivated and the
mute state is updated accordingly (in this order).
This is *off by default*. It needs to be enabled in two places:
- `/etc/bigbluebutton/bbb-webrtc-sfu/production.yml` ->
`transparentListenOnly: true`
- End users:
* Server wide: `/etc/bigbluebutton/bbb-html5.yml` ->
`public.media.transparentListenOnly: true`
* Per user: `userdata-bbb_transparent_listen_only=true`
Add secondsToActivateAudio, inputDeviceId, outputDeviceId and isListenOnly
to audio_joined.extraInfo
Add inputDeviceId, outputDeviceId and isListenOnly to
audio_failure.extraInfo
Add a try-catch to the device enforcement procedure triggered by
onAudioJoin - it may throw and block the modal.
In peer.js offer generation method, the provided constraitns are only
being type checked. It needs a value check when the constraint is a
boolean.
This commit should prevent useless transceivers from being added when
explictly specifying audio/video as `false`
There's an edge case in finnicky networks where ALG-like firewalls
tamper with USE-CANDIDATE STUN packets and, consequently, bork ICE-lite
connectivity establishment. The odd part is that client-side gathering
seems to complete if intermediate STUN bindings work (before the final
USE-CANDIDATE), which may cause the peer not to generate relay
candidates == connectivity fails.
This adds the `public.kurento.gatheringTimeout` option to forcefully extend
the candidate gathering window in peers that act as offerers. The
behavior is as follows: if the flag is set (ms), the peer will wait
either the gathering completed stage or, _at most_,
public.kurento.gatheringTimeout ms before proceeding with calls chained
to setLocalDescription.
This option is disabled by default and intentionally ommited from the
base settings.yml file as to not encourage its use. Don't use it unless
you know what you're doing :).
There are still a bunch of edge cases and issues with reconnection
scenarios for video:
- Signaling socket refuses to reconnect once maxRetries expire
- Race conditions on local stream attachment: local camera wouldn't be
correctly rendered _if_ the attached stream existed _without_ video
tracks yet
- Video tracks leak on local streams when replacing them (virtual bgs)
- Completely ignoring Meteor state when trying to reconnect cameras
- Streams aren't proactively stopped when the signaling socket dies
- Outbound request queues aren't isolated by stream nor are they
flushed when a newer peer with the same ID is created
- Server originated negotiation errors won't trigger a local peer
cleanup - thus leaving dangling peers that take way too long to
reconnect
This commit fixes or improves all of the aforementioned issues, +:
- Remove unused arguments in the peer (client->SFU) 'start' request
- Prevent crashes when trying to render video-list-items without user
data (which might happen on re-connections)
Same rationale as in video-provider's commit
(34fa37ae4f092af4a5aef0cf01d96c033d97473c).
This commit does the following:
- Implement actual heartbeat checks to trigger reconnects when
necessary
- Properly catch and log WebSocket.send errors
There may be other bridges may not need to force relay traffic on
firefox as @prlanzarin pointed out. So set the default to a
configuration that works out of the box but leave other choices for the
operator.
The option is moved from kurento namespace to media next to the general
forceRelay option.
Firefox has a buggy ICE implementation and needs WebRTC media traffic to
be routed through a turn server to work reliably with mediasoup.
Use the information fetched by the STUN API to determine if the operator
has configured a turn server. If there is one force firefox to use it.
Closes#16164
When BBB is run as a single node or in a scaleout setup with a cluster
proxy in front (see https://docs.bigbluebutton.org/admin/clusterproxy.html) it
is useful to store client settings in browser localStorage instead of
sessionStorage. If localStorage is configured then the client will keep seetings
like notifications for user joining, chat etc across meetings.
It is not advisable to set the setting to `local` in a setup of multiple BBB
nodes without a cluster proxy in front of it because this would lead to
unexpected behaviour at users. The browser would store settings for each server
and for users it would look like BBB is sometimes store the settings and
sometimes not.
It adds the new setting
```yaml
public:
app:
userSettingsStorage: (session|local)
```
Currently, collection cleanup code is only run when an added event
is received from the server. Where that fails is in scenarios where
a server-side collection turns empty while an affected users is
disconnected - and then reconnects. There's no removed (or updated)
event so no cleanup code is run and you have stale data.
This commit guarantees a stale data check is run whenever a subscription
is established again. The `added` check was also maintained, although
I'm not too sure anymore it's is still needed. That may need to be
revisited.
There are some situations where previously set deviceIds (
local/session storage) may become stale. This causes an unexpected
behavior where audio is temporarily borked until the user clears their
local storage.
This issue has been seen more recently on Safari endpoints when switching
back-and-forth breakout rooms in environments running under iframes.
Also seen randomly on endpoints with virtual input devices.
This centralizes audio gUM calling into a single method that retries the
gUM procedure without pre-set deviceIds only if the initial call fails
due with an OverconstrainedError - hopefully circumventing the issue.
BBBVideoStream 'inactive' event currently does not send the
MediaStream id as the payload for handlers. This can cause theoretical
race conditions due to media stream mismatches in places where
'inactive' handling is done globally (eg video-preview).
This adds the MediaStream id to BBBVideoStream's 'inactive' event
and uses it in video-preview to avoid such race conditions.
Extract the deviceId again from the stream to guarantee consistency
between stream DID vs chosen DID. That's necessary in scenarios where,
eg, there's no default/pre-set deviceId ('') and the browser's
default device has been altered by the user (browser default != system's
default).
There's no rollback procedure in case a device switch fails right now,
nor does the code entrypoints that call the switching procedures wait
for resolution or failure before marking the new device as chosen. That
may cause inconsistent states in a couple of ways:
- No rollback: switch fails, audio is still on but no actual
microphone input is being transmitted
- Not waiting for resolutions: inconsistent chosen devices on failures
Device switching errors are also not surfaced to the end user
This commit:
- Adds device rollback and proper resolution/failure response
awaits to try and make the state a bit more consistent.
- Centralizes the input device switching code to be reused between
different bridges
- Centralizes device ID state management in audio-manager to try and
mantain them a bit more consistent across the board
- Surface device switching failures to the end user
- Guarantee device IDs are set to the session storage on all
appropriate scenarios
RTCRTPSender exposes DSCP marking via `networkPriority` in the encodings
configuration dictionaries. That should allow us to control
QoS priorities for different media streams, eg audio with higher network
priority than video. The only browser that implements that right
now is Chromium.
To use this, the public.app.media.networkPriorities configuration in
settings.yml. Audio, camera and screenshare priorities can be controlled
separately. For further info on the possible values, see:
- https://www.w3.org/TR/webrtc-priority/
- https://datatracker.ietf.org/doc/html/rfc8837#section-5
Firefox doesn't fire the ended evt/onended callback for live
MediaStreamTrack(s). We rely on that event.
Manually emit the ended event which works with the onended callback
when a track is stopped
kurento-utils is unmaintained. It's served us well, but its age
shows. We need to transition to something else if we want to
have better maintainability and include simulcast, multistream, ...
This introduces a simplified/leaner wrapper kit that's almost
API-compatible with what we use right now - so widespread changes
are minimal). It's easier to maintain/read/transition from. This
can be read as an intermediate step to transitioning to
something definitive (ie mediasoup-client).
Under some scenarios, cameras are freezing when the virtual background
code is running due to runPostProcessing(_renderMask) throwing
NS_ERROR_FAILURE - mainly on Firefox - consequently preventing
subsequent TimerWorker ticks from being scheduled.
Cases where I've seen that happen are:
- conferences running under an iframe where the iframe is briefly
stalled for some reason
Address the issue with a try-catch and a log for debugability (it's high
frequency, hence why not error level). We should probably remove the log
entirely once we figure out why the post-processing method is failing.
Use the built-in getLocalStream from the peer wrapper instead (which
relies on getSenders - the proper, spec-compliant way).
Two different issues are addressed:
- RTCPeerConnection.getLocalStreams is a pre-1.0 WebRTC spec which is
deprecated and not recommended.
- Fixed an issue where a switch from full audio to listen only could
cause the latter to be rejected with an error 1004 in rare scenarios.
There could be a race condition where the local getDisplayMedia stream ends
(eg via Chrome`s stop sharing toast) while the broker hasn't finished starting.
That would lead to a scenario where the broker wouldn't emit an end event,
causing screen sharing to be flagged as started with a blank/invalid stream.
- Remove the old listen only bridge (kurento.js), superseded by the equivalent
and equally stable (AS FAR AS LISTEN ONLY IS CONCERNED) sfu-audio-bridge
- Rename FullAudioBridge.js -> sfu-audio-bridge.js
* A more generic name that better represents the capabilities and
the nature of the bridge
* The bridge name identifier in configuration is still the same
('fullaudio')
- Remove the FreeSWITCH listen only fallback
- Temporarily disable the "trickle ICE" pair gathering feature used
in SIP.js (which was always experimental, nonstandard and disabled
by default)
- Updates to settings.yml keys in places where relevant
"default" is not an universally valid default value for deviceIds which was causing issues with Firefox and Safari in some specific scenarios where exact deviceId constraints were being used
