Instead of being executed every 30s by systemd, it's now a service that's
running all the time and will wait for .done files to start the processing
of recordings.
Found another case where the html5 client was passing through control
characters, in the original presentation name field.
Rather than play whack-a-mole with different fields which may eventually
get poorly sanitized user data, apply the control character filtering
to all properties.
Adjust the character range to do the following:
* Allow horizontal tab (0x09), it's not problematic.
* Disallow control characters in the range 0x1A-0x1F. Probably missed by accident.
It used to print:
Failed to download file: undefined local variable or method `respose' for BigBlueButton:Module
Did you mean? response
because the incorrect variable name was used in the error message.
There was no effect other than the message in the log, since the shared notes
couldn't be archived anyways, and the only thing the exception did was ...
prevent the shared notes from being archived.
FFmpeg has pretty good format autodetection even if the filename has the
'.txt' extension, so just rely on that. It'll even pull subtitles out of
video files - although I expect we'll have size limits so that doesn't
happen.
Rather than running the tool in a loop, I'm using inotify to watch for
new files being created (ideally, the other rap workers will be migrated
to this style in the future). The trigger for processing is the creation
of the .json index file for the uploaded track.
Sometimes when text is pasted into the whiteboard text tool from
external apps, it'll include control characters that mess up later
recording processing scripts.
Run the same cleanup as already used for chat messages on the whiteboard
text as well.
The cleanup has been adjusted to allow newline and tab characters. They
won't really cause issues in chat, and newlines (at a minimum) are
required for the whiteboard.
This is a workaround for #7356
This was added to workaround for red5 taking a while to rewrite the
serialized (.ser) data that it streams to disk back to the .flv format.
The workaround is no longer needed, for two reasons:
* The sanity scripts run the red5 code to generate the .flv from the .ser
if needed, and
* We're expecting more people to be using WebRTC media in the future anyways
This makes recordings available up to 2 minutes earlier than they would have
been otherwise.
BigBlueButton can sometimes write events out of order - this particularly
seems to affect the final RecordStatusEvent in a meeting which was ended
while recording was still running. This breaks the recording processing
scripts.
As a workaround, sort the events as they're being written into the events.xml
file. We have the following properties:
* The input data is already mostly sorted
* Items in the wrong position will be no more than a couple spots off from where
they should be
* We should not change the relative order of events with the same timestamp.
The best algorithm to use here is a simple insertion sort. When adding each new
event to the XML structure, it scans backwards through existing events until it
finds the correct position.
For #6035
At some point, BigBlueButton switched from using `"` to `'` in the link tags in
chat messages, which caused the regular expression that was supposed to remove
the `event:` prefix to not match.
I've replaced the error-prone regular expression with an actual HTML parser,
using the "Loofah" HTML transformation/sanitization library based on Nokogiri.
I've removed the code that detected unlinked URLs, since it was broken - and
not needed: current BigBlueButton versions do the link detection in the client.
If someone reprocesses a really old BBB recording with these scripts, URLs in
chat might not be linked in the result. But they wouldn't have been linked in the
client in the original meeting either, so I figure that's ok.
Fixes#6475
In some cases, ffmpeg will be able to read the file, but the video itself
can't be decoded (missing/corrupt stream headers, for example). In this case,
some of the properties on the stream object will be nil.
Make sure that pix_fmt is present in the probed info, since that's a required
property.
Issue #6338
It looks like there was a logic error in the code that was causing it
to break out of the event deletion loop early when deleting events for
the last (or only) segment in a recording. (In this case, last_index
is -1, so i >= last_index is always true).
The trim_events_for call was always succeeding, so the events were
being removed from the event list (meeting:{ID}:recordings key) even
though the events themselves hadn't been deleted in the loop.
I've moved the trim_events_for call to below the event deletion loop
to ensure that if the archive script is interrupted, the events list
will contain all not-yet-deleted events.
The archive_dir can by the raw recording directory in some recording
formats - including, hopefully, presentation at some point (to avoid the
extra copy)
The previous calculation used the video size in pixels, which might lead
to stretched/squashed videos in rare cases where the video has non-square
pixels. The new calculation is correct for all video sizes.
The ffprobe command in ffmpeg 4.0 now omits the aspect ratio fields in the
json output when indeterminate, instead of returning an invalid value with
0 in the numerator or denominator. Handle this correctly.
- some conflicts have been fixed.
The following needs working on:
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/services/ChatCopy.as
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/services/ChatSaver.as
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/views/ChatTab.mxml
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/views/ChatWindowEventHandler.as
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/users/services/MessageReceiver.as
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/users/services/MessageSender.as
both modified: ../bigbluebutton-client/src/org/bigbluebutton/modules/users/views/MediaItemRenderer.mxml
Parking as need to work on something else.
Since the update to the newer red5, seeking in flv files (webcams in
particular are noticable) has been broken, resulting in cameras
appearing to "hang" any time there is a cut in the generated video -
which happens when start/stop button is pushed, or when cameras are
added or removed.
We can detect the problematic video files because the timestamp of the
first frame is large (old red5 versions always set first frame
timestamp to 0.001 seconds). If we see a file like this, having ffmpeg
remux the file - rewriting the timestamps and index - works around the
problem.
Due to improvements in the recording scripts, most of the stuff the sanity
script was checking for is no longer needed (missing/corrupt video files
are handled by the processing scripts). The version of this script in
master has been cleaned up so that the only things it does are:
- Check that the events.xml exists and is properly formatted xml
- Rebuild flv files from red5 .flv.ser/.flv.info files
The script from master is compatible with the 2.0 code, so just use it
as-is.
This fixes a problem where following the recent red5 upgrade in 2.0 branch,
an flv file is never written for webcam streams where no frames were
received, despite there being recording events.
This improves the quality of portrait documents, before they were
1200px when landscape documents got 1600px.
Switching to scaling to a square means that we can use the "-scale-to"
option on pdftocaio, which means that it generates images directly
at the desired size. This can save quite a bit of time (and memory)
if a document was uploaded with extremely large page size.
There's some cases where you can get 0-duration recordings due to
recording event placement (e.g. a single recording event is the last
event in the events.xml). Detect these cases, and treat them like
no recording marks in the archive script (it will stop the recording
from being automatically processed).
I've also adjusted the sanity script to detect these cases and error
out. The recording processing scripts cannot handle 0-length recordings,
you have to manually edit the events. I've added a message to the
sanity log about this.
In some cases (due to network issues), the webcam video can be shorter
than the time between the start/stop events. Pad the input video with a
blank video to make sure that there's input to the video tiling filters,
to fix a problem where the video won't render correctly with ffmpeg v3.4
and later.
In some cases (due to network issues), the webcam video can be shorter
than the time between the start/stop events. Pad the input video with a
blank video to make sure that there's input to the video tiling filters,
to fix a problem where the video won't render correctly with ffmpeg v3.4
and later.
In some cases when there is a slight mismatch between audio file
duration and event timestamp difference, and we have a record
status or chapter break event in a certain location, it could
trigger a seek past the end of an audio file. Detect this
condition and just render silence instead.
Also adjust the thresholds for the audio length scaling - they
were being triggered on short recordings that should be correct.
In some cases when there is a slight mismatch between audio file
duration and event timestamp difference, and we have a record
status or chapter break event in a certain location, it could
trigger a seek past the end of an audio file. Detect this
condition and just render silence instead.
Also adjust the thresholds for the audio length scaling - they
were being triggered on short recordings that should be correct.
With the current segment processing, we might be processing one segment
while archiving a different segment from the same recording. To avoid
that the processing scripts see an incomplete events.xml file, write to
a temp file then rename.
Red5 sometimes writes webcam video files with a large offset in the
video frame offsets, sometimes up to 30 or even 60 seconds. However,
the start event in the events.xml file corresponds to the time at
which red5 received the first keyframe (recorded frame) in the video.
The end result is that the video will sometimes appear to be
delayed (out of sync) in the processed recording.
The correction is simple: We're already reading video metadata,
including the timestamp of the first frame, so we just have to apply
a correction during video processing to undo the frame timestamp
offsets in the video file.
Red5 sometimes writes webcam video files with a large offset in the
video frame offsets, sometimes up to 30 or even 60 seconds. However,
the start event in the events.xml file corresponds to the time at
which red5 received the first keyframe (recorded frame) in the video.
The end result is that the video will sometimes appear to be
delayed (out of sync) in the processed recording.
The correction is simple: We're already reading video metadata,
including the timestamp of the first frame, so we just have to apply
a correction during video processing to undo the frame timestamp
offsets in the video file.
When working with the segmented recording format, the events file might
end with a mismatched start video event for an incomplete file. The
sanity script was removing this event, meaning the video didn't show up
in future segments.
Simply drop the code that tries to find invalid video files and removes
them from the events file. The new video processing code is already
robust against missing or corrupt files.
If no audio files were found, it was running rsync with one argument,
which is a bit unexpected. It just printed a file list in this case, but
we can provide a cleaner error message instead.
It previously checked whether any part of the entire meeting was recorded.
Helper functions are added to look up the time of segment start and end
(which handle non-segmented recordings correctly too).
Part of the events handling code was rewritten to reduce the number of times
that the events.xml file gets parsed.
If you're inserting at position 0 (and there was no previous deleted text
from that position), you can't use the timestamp from the previous character
position, since there's no previous character. Use the timestamp of the
following character instead.
In some unusual cases, the recording start can be the last event in the
events file, or at least have the same timestamp as such.
Add some code to check the array bounds and break if needed, so we
don't check the timestamp on the (non-existant) event after the last
event.
The previous code looked for stop events and tried to find their
associated start event. This obviously doesn't work if there was
no stop event. But if there was a start event, we need to show the
deskshare… so rework to code to try to find the matching stop to each
start instead, and use the end of the meeting if no matching stop was
found.
This is just a bundle of a few things I've been fixing up in the past
while.
= Workaround for BBB 1.1 beta deskshare timestamp bug
This is unlikely to be used, but I have the code for it, might as
well merge it in.
= Rework video tiling code for ffmpeg
Render video using the 'hstack' and 'vstack' filters rather than the
'overlay' filter. This is somewhat faster, particularly with lots of
videos.
= Etc.
- Remove usage of the streamio-ffmpeg gem.
The video rendering code has some stuff to directly read 'ffprobe'
output, so re-use that instead of this gem (which is kind of old and
has issues with newer ffmpeg versions).
- Don't hardcode the deskshare video area size, pull it from the
properties file
- Remove some code that worked around missing video end events.
In some cases this could cause flickering or strange video issues.
It's no longer strictly needed, the new tiling code doesn't break if
the seekpoint is after the end of the video.
Removed from the code, now it can be specified in the properties file.
Makes it easier to include new processing scripts and control dependencies
between them.
With resque workers we don't need to use status files to trigger each
step of the recording process. So instead of using status files as triggers
they are not used only for information, so that the workers can know if
a e.g. processing script failed or succeeded.
For now the differences are that the archive worker will run inside
resque (it is a resque worker now) and there is a "rap-trigger" file that
is executed by systemd to detect recordings that need to be archived and
add a job on resque to archive them.
The process os scheduling jobs still needs to be reviewed, but for now will
be done by rap-trigger.
Current BBB client is generating invalid indexes when characters are
inserted with an IME - the edit indexes are as if the preedit text was
being removed, but the preedit text was never sent in the first place.
For now, just don't crash if there's an edit that would remove text
which is past the end of known text. The result might be broken, but
it won't prevent the rest of the recording from working.
I've had a chance to see how this script behaves with actual live
caption tracks now, and there's room for improvement. In particular,
it often generates cues that overlap - the next one appears before the
previous one disappears. The browser player position handles this
really poorly, and it's nearly unreadable.
The solution is to, if two cues would overlap, merge them into a
single multiline cue that displays for the full time. This is a lot
easier to read.
Some extra code is added to de-overlap any remaining cues (e.g. if
there's a third cue that would also overlap). This will reduce the
time that the earlier cue gets shown below my preferred minimum, but
not really much we can do about that if people are talking/typing
quickly.
The code can easily be tweaked to set a different number of maximum
lines per cue if desired.
If processing didn't succeed, the directory where it is trying to write
the file might not exist, causing an exception in the worker which leads
to it getting stuck in a retry loop.
The purpose of this is to ensure that the recording processing with make
visible forwards progress, by ensuring that different steps in the
recording pipeline don't block each-other.
This fixes:
- recordings being processed prevent archiving new recordings from running
- recordings can't be published until all pending recording processing
completes
It does allow recording archive, sanity, process, publish steps to run in
parallel with each other, but at most one recording will be in each step at
a time. (I.e. while one recording is being processed, a different recording
can be published). This may potentially increase CPU usage for some users.
If you expect this to be a problem, you can set resource controls (see
`man systemd.resource-control`) on the bbb_record_core.slice systemd unit.
This is a demo. Some issues:
- When sharing desktop, cursor is still relative to slides, and not to deskshare video;
- Need to show whiteboard canvas on top of deskshare video as well;
- Need more testing (audio/video sync, playback perfomance, multiple presenters and multiple deskshare events).
Instead of assuming that wav files have 44 byte headers (they usually
don't - freeswitch adds some metadata tags), add a super-simple wave
file parser that just finds the offset where the audio data actually
starts.
It's no longer used anywhere in the recording processing, and the one
remaining use in the sanity script can be replaced with a (less buggy!)
function from the video EDL code.
Main change is to rewrite the start/stop event handling to do fewer
passes and use simpler logic (it no longer has to track durations).
The temporary work audio codec is switched to flac; slightly lower hard
drive use and it avoids the 4-hour length limit of wav.
Some improvements to the video seeking when processing deskshare and
webcam videos, to avoid video dropouts.
Tweaks to the audio encoding quality settings.
Remove an unused function.
When the next line break after a wrap induced due to line length was a
hard line break, the hard line break would override the soft line break,
resulting in over-long lines, and the last word would be repeated on the
last line.
Move the hard break code into an else branch so it's only applied if we
haven't already done a word wrap on this line.
