There are some situations where previously set deviceIds (
local/session storage) may become stale. This causes an unexpected
behavior where audio is temporarily borked until the user clears their
local storage.
This issue has been seen more recently on Safari endpoints when switching
back-and-forth breakout rooms in environments running under iframes.
Also seen randomly on endpoints with virtual input devices.
This centralizes audio gUM calling into a single method that retries the
gUM procedure without pre-set deviceIds only if the initial call fails
due with an OverconstrainedError - hopefully circumventing the issue.
Extract the deviceId again from the stream to guarantee consistency
between stream DID vs chosen DID. That's necessary in scenarios where,
eg, there's no default/pre-set deviceId ('') and the browser's
default device has been altered by the user (browser default != system's
default).
There's no rollback procedure in case a device switch fails right now,
nor does the code entrypoints that call the switching procedures wait
for resolution or failure before marking the new device as chosen. That
may cause inconsistent states in a couple of ways:
- No rollback: switch fails, audio is still on but no actual
microphone input is being transmitted
- Not waiting for resolutions: inconsistent chosen devices on failures
Device switching errors are also not surfaced to the end user
This commit:
- Adds device rollback and proper resolution/failure response
awaits to try and make the state a bit more consistent.
- Centralizes the input device switching code to be reused between
different bridges
- Centralizes device ID state management in audio-manager to try and
mantain them a bit more consistent across the board
- Surface device switching failures to the end user
- Guarantee device IDs are set to the session storage on all
appropriate scenarios
Mobile endpoints are flaky with the WebSpeechAPI:
- iOS versions that support it are borking our outbound audio when it's
enabled
- Android speech recognition has flaky locale detection and speech
transcription
Additionally: the support check is not checking the WebSpeechAPI
availability properly, so older devices (eg iOS 12) are flagged as
supported even though they aren't.
This commit adds a configuration flag (public.audioCaptions.mobile) to
control transcription availability on mobile. False by default.
Also extends the setSpeechVoices support check and
hasSpeechRecognitionSupport method to prevent false positives.
Adds two new flags to the settings file which change the way the locale
flag is used:
- forceLocale: (true/false) => If true, enforces the transcription
language to be the locale content field and jumps the language
selector
in audio modal.
- defaultSelectLocale: (true/false) => If true, the default selected
value in the dropdown language selector in audio modal will be defined
by the locale content field.
In any case, if the locale flag holds an invalid value, it defaults to
disabled.
Move the language collection to the HTML settings file. This data defines
the available languages available for the speech API.
These language tags are used to filter SpeechSynthesis' API `getVoices`
result. Tags must use BCP 47 format.
https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesisVoice/lang
Avoid enable audio transcription if the browser's vendor does not provide
voices data.
This should prevent false positives for browsers such as Chromium and
Brave.
Parse the audio transcript before broadcasting it's content back to the
client and the recording actor. Limiting by 8 words per line and max of
2 lines to avoid CPU intensive operations over this recurring event.
Replace Calibri font family with Verdana to improve character spacing,
add relative sizing to the text content and a background padding.
Add a server-side app for the audio captions feature and record proto-events
for this data.
As it is, only behaves as a pass-through module. The idea is to include all
the business intelligence in this app.
The new local echo view doesn't block the "Join audio" button while
awaiting for getUserMedia permission to be granted/denied. That may
cause unexpected behavior when unattentive users just click "Join audio"
without granting or denying gUM.
This commit accounts for gUM resolution when deciding whether to block
the "Join audio" button. It also includes an extra "isConnecting" check
to it to avoid spam-clicking issues.
removing border and implementing box-shadow
adding transparent border
passing styles to common buttons
adding secundary color to component
updating color components
The talking-indicator, emoji-button and input-live-stream-selector
components are passing props downstream that weren't omitted or handled
by inherited components (Button, Icon). That causes a handful of error
logs to be spammed in the console of dev environments, which is
annoying.
This commit addresses the issue by:
- Making the talking, spoke, muted and isViewer props transient
(styled-components) - which means they won't reach the DOM (as
expected since they're style-only)
- Omitting the EmojiButton `rotate` prop in the Icon component itself
* Made that instead of transient because might be useful to migrate
the rotate code to the Icon component?
Move device selection dropdown to microphone button
Move option 'leave audio' to an option within the dropdown
Remove the audio exit button when device !== mobile
The initial selected output device in AudioSettings could be the wrong one if
the user's session had an output device ID already stored, but is joining on a
new session. That would cause the remote-media tag not to be updated with the
correct output device ID when it should (the service.js change)
The issue is tackled by guaranteeing the output device ID is set on all ends
when AudioSettings/AudioModal mounts.
The initial selected input device in AudioSettings could be the wrong one if
- 1) gUM outputs an user-selected device rather than the default
- 2) no previous device was selected for that domain and the enumeration
list order caused the default not to be the first
The issue is tackled re-extracting the deviceId from an input stream if it
exists and making the DeviceSelector value follow what is defined in the client
(audio-manager) via a trackable prop
For scenarios where streams are produced in AudioSettings (local echo,
volume meter), force gUM resolution before devices are enumerated.
This effectively guarantees that all devices are present, labelled and
with deviceIds.
public.media.showVolumeMeterInSettings => public.media.showVolumeMeter
public.media.simplifiedEchoTest => public.media.localEchoTest.enabled
Initial hearing state can be configured in public.media.localEchoTest.initialHearingState
New features:
- A simplified echo test mode that only does a local loopback (instead of
going to FS and back)
- A volume meter for microphone streams to the AudioSettings view
Those two features are experimental and disabled by default; see
public.app.media.simplifiedEchoTest and public.app.media.showVolumeMeter configs
Collateral changes:
- fix: localize fallback device strings in AudioSettings/DeviceSelector
- Refactor on some media stream utils to be re-usable across components
- Refactor in AudioSettings to keep gUM #uses stable.
* TODO: need to pass streams through AudioManager to avoid the surplus gUM.
- fix(audio): drop ScriptProcessorNode usage (deprecated)
* Used in volume meter for tracking - use hark instead
When joining breakouts, we now wait for the bridge to be loaded before
automatically start user's audio.
This problems happens only on fullaudio bridge
We are now leaving the check for the minBrowserVersions object in settings.yml
If the settings enables chrome iOS, audio should allow users to be joining
with audio.
This is related to recent Chrome update (iOS 14.3+) that now allows
camera/microphone to be captured. We are looking for enabling this for
Chrome 93 in iOS (chromeMobileIOS version in settings.yml)
Removed trailing spaces in audio-controls/component.jsx
Fixed browser warning about required BBBMenu's onClick prop in
input-stream-live-selector/component.jsx
Fixed eslint warning "react/button-has-type" in ButtonEmoji.jsx
Fixed browser warning about not recognized hideLabel prop in ButtonEmoji.jsx
When mic is locked, user is not able to talk so it doesn't make sense
to alert the user about unmuting (mute button is also disabled when mic
is locked).
Closes#12048
This commit adds a new small button over the "Phone/audio" button. Currently
this button pops up the device audio selector, which contains the "Leave audio"
option. This commit brings back the "Leave audio" behavior to the phone
button, using a new smaller button to switch between audio devices.
This issue address the problem reported by #12320 and complements the work done
for #9723.
Some technical details:
Added a new component : ButtonEmoji. This allow us to add an emoji inside
another button.
Modified dropdown trigger a bit: if the trigger contains a ButtonEmoji,
then we will use the emoji to trigger the dropdown (instead of the button
itself). This commit doens't change the default behavior of dropdown (when
the trigger doesn't have a ButtonEmoji component), to avoid regressions with
current dropdowns; this will only change it's behavior when used with
the new ButtonEmoji component.
* adds unability to see screenshare button on mobile devices test specs
* simplify code in testMobileDevice()
* userlist and chat panels should not appear at page load in mobile devices
* lint
* updates outdated audio specs due to leaveAudio changes
* correct clicks on disconnectAudio elements
* whiteboard not visible on userlistPanel or on chatPanel
* reworks mobile devices/usersagents
* fixes screenshare mobile/tablet specs
* adds whiteboardNotAppearOnMobile spec
* adds Chat Panel specification to Mobile-Tablet specs
* simplify getArgs() functions for all devices
When listenOnlyMode=false, skipCheck=true and skipCheckOnJoin=true, the
audio tries to start a session more than one time, causing it to fail
at the first one (and reconnect after that).
Now we check if user is already connecting before trying to start a new
audio session.
Added some info in settings.yml for the options related to this commit
Closes#12190
When setting skipCheckOnJoin to true, an extra audio channel is created in
FreeSWITCH, after user accepts the echo test. The extra channel is removed
when user leaves the room, but this still may affect performance.
When joining/returning breakouts, audio would always connect
with full audio. This can lead to a performance problem, once
all listenonly users would join full audio, increasing the
number of streams in FreeSWITCH.
We now have a consistent behavior, which is:
1 - The choice made by the user in the main room is predominant:
if mic is active in main room, user will automatically
join mic in breakout room. When returning from breakout
room, user will also join with mic again.
2 - Changes made in breakout room won't have effect when
returning to the main room. This means if user, for example,
change from listenonly to mic in breakout room, the returning
will consider the option choosen previously (listenonly) and
listenonly will be active again in the main room.
3 - If user didn't join audio in the main room, the audio modal
will be prompted when joining the breakout room (this is
a special case of (1))
The following is some technicall information:
InputStreamLiveSelector (component.jsx) now calls
'handleLeaveAudio' function, which is the default
function when user leaves audio (also used when
dynamic devices are inactive).
We now store information about user's choice (mic or listenonly)
using local storage, instead of the previous cookie method (this
was triggering some warnings in browser's console).
Also did a small refactoring to match eslint rules.
Fixes#11662.
Firefox doesn't create a device called 'default' and we were trying
to set this when user is joining the room. We don't do this anymore, letting
devices to be changed when there's some user request.
Moved outputDeviceId inputDeviceId information to be managed in bridge
(just like we do with inputDeviceId), we don't store this duplicated
information in audio container anymore.
Fixed the eslint warning in "playAlertSound(url) { ..."
We are safe to let users try to change input/output devices because the
device list is retrieved from enumerateDevices.
When listen only fallbacks from Kurento to FreeSWITCH, we must guarantee
the muted alert won't be created, speciallly because listen-only's fallback
uses a flow similar to microphone's. Client currently crashes when this
happens: this commit fixes this peoblem.
Allow listenonly users to change output devices
Fixed dynamic audio device change for firefox
Fixed shortcuts for audio join/leave
Show (with a bold font) the current selected device
[performance] Prevent calling mediaDevices.enumerateDevices every time we render
the selector. This adds a delay (~200ms, on my chrome setup) to render this component
[performance] Do not call enumerateDevices to search for new devices, instead we listen on mediaDevices.deviceChange event
Small refactoring and fixed a few errors that were being throw in browser's console
Fixed device selection when this is done in audio-settings modal
Fallback to default device when current device is removed
Truncate device name length
Renamed "Input","Output" labels to "Microphone","Speakers", respectively
Update eslint rule for accessKey
Currently this information is lost everytime breakout-room component is
unmounted, causing the panel to shows wrong information during next renders
Fixes#11333
After ending the notification playback, we set the ".src" property to null, which immediately stop the internal player of mobile browser (tested on Chrome for Android - device list is on #11458).
For the specific list of devices, this prevents the internal buffer error "-61" described in #11458.
Fixes#11458.
* updating old tests + collecting more snapshots [WIP]
* updates old test suites and collects more visual regressions screenshots
* remove snapshots and their collection temporary
* run tests from packages.json
* update test execution command/export constants from .env to core/constants.js
* update tests/puppeteer/README.md file
* update LOOP_INTERVAL variable call and assign timeouts to the webcam share spec
* redefine waitForSelector func in page.js, update chat test suite specs and add poll chat message test spec
* Merge remote-tracking branch 'upstream/develop' into updating-old-tests-visual-with-visual-regressions
* update webcam test specs collecting videoPreviewTimeout and use it to wait for videoPreview selector
* update custom parameters test suite
* update breakout test suite
* update webcam layout test suite
* update multiusers test suite
* update notifications test suite
* update presentation test suite
* whiteboard test suite
* screenshare test suite
* update sharednotes test suite
* user ELEMENT_WAIT_TIME variable from timeouts constants.js
* list TEST CONSTANTS by category
* add poll test suite and assigns the right unassigned timeouts
* set test pages to headless
After audio reconnection, a muted user would have it's microphone unmuted by default, unless muteOnStart is set to true. This fix this problem.
Fixes#9016
* add param to force echo test when user joins audio after init
* fix UI stuck on connecting when userdata-bbb_auto_join_audio=false
* fix conditions for joinFullAudioImmediately and joinFullAudioEchoTest | remove old format
* remove extra param in getItem
* recover audioLocked | only set getEchoTest if doesnt exist
As explained in #11143, disabling audio filters is desired in some scenarios.
This basically adds an option for user to disable default constraints.
When user doesn't change this value in Settings > Application, the default
value for each audio constraints is retrieved from settings.yml.
When user changes this value in Settings > Application, audio
filters (AGC, Noise Supression and Echo Cancellation) are all set to
true/false, according to the value selected in the Settings GUI.
To start it simple, we decided to not to add a different setting in the GUI for
each audio contraint. This may be added in the future, though (perhaps in a
dedicated Audio Settings tab)
This is related to #4873
As explained in #11143, disabling audio filters is desired in some scenarios.
This basically adds an option for user to disable default constraints.
When user doesn't change this value in Settings > Application, the default
value for each audio constraints is retrieved from settings.yml.
When user changes this value in Settings > Application, audio
filters (AGC, Noise Supression and Echo Cancellation) are all set to
true/false, according to the value selected in the Settings GUI.
To start it simple, we decided to not to add a different setting in the GUI for
each audio contraint. This may be added in the future, though (perhaps in a
dedicated Audio Settings tab)
This is related to #4873
When user joins audio and for some reason an error (such as 1001, 1002,...), happens, the user is not able to click "Mic" and "Listen Only Buttons"; except if the audio window is closed and oppened again.
When getting disconnected with 1001 ("websocket closed unexpectedly" error) we were creating a new SIP session, therefore a new FreeSWITCH channel.
While reconnecting the socket, instead of closing the SIP session, we keep it alive during reconnection (audio should keep working in the meantime). When reconnected we keep using this same session (avoiding the creation of an extra one).
We also better handle WebSocket error codes from SIP.js.
FF immediately closes websocket when unloading page, so we now to stop user agent when 'beforeunload' event is triggered, to avoid leaving open sessions in FreeSWITCH when user leaves page.
When refusing ("thumbs down" button) echo test, user is able to select a different input device. This should work fine for chrome, firefox and safari (once user grants permission when asked by html5client).
For output devices, we depend on setSinkId function, which is enabled by default on current chrome release (2020) but not in Firefox (user needs to enable "setSinkId in about:config page). This implementation is listed as (?) in MDN.
In other words, output device selection should work out of the box for chrome, only.
When selecting an outputDevice, all alert sounds (hangup, screenshare , polling, etc) also goes to the same output device.
This solves #10592
This considerably changes the way we process audio signaling and start audio elements in user's browser.
We now avoid using AudioContext element for both microphone and listenonly calls, once it is unstable for some iOS devices (cracky audio, user stops hearing audio after a while).
Increased default value for listenOnlyCallTimeout: this avoids activating FreeSWITCH's fallback when ICE negotiation takes longer than 15sec (tested on DO).
Increased listenonly logs.
This fixes#8133#10388
