In cases of extremely short (single frame) input videos, the fps filter
can sometimes generate 0-frame output videos, resulting in the tpad
filter having no input (this breaks it, causing a busy loop).
Move the tpad filter to before the fps filter to solve this problem.
This isn't perfect, since the tpad filter doesn't work well on variable-
framerate video (it generates extremely high framerate video with a lot
of frames that will be discarded), but this only happens between the
tpad and fps filters, and only at the end of an input video (usually
right before a cut) so this seems acceptable.
Since the tpad and fps filter are in the same process, these duplicate
frames don't actually require copying any data (the frame is
reference-counted), and still process reasonably quickly.
Fixes#16407
The tpad filter is problematic on the variable-framerate webcam files,
and the result can end up being hangs (or, at least, very slow
processing) in the compositing.
Move the tpad filter to the compositing process where it can run after
the fps filter has converted the video to constant framerate. It still
needs to run before the start trimming, so switch to using the trim
filter rather than the fps filter's start_pts feature.
The tpad filter is problematic on the variable-framerate webcam files,
and the result can end up being hangs (or, at least, very slow
processing) in the compositing.
Move the tpad filter to the compositing process where it can run after
the fps filter has converted the video to constant framerate. It still
needs to run before the start trimming, so switch to using the trim
filter rather than the fps filter's start_pts feature.
Even with the filter changes made, there's still some cases where
filter chain hangs can result from filter reconfigurations. To solve the
issue completely, I have split out pre-processing video files to
separate ffmpeg processes, so that the filter chain for compositing will
not ever be reconfigured.
Each input video now has a separate ffmpeg process run for it which
does the scaling, padding, and video extending steps. To avoid issues
with disk space usage and extra cpu usage or quality loss, the output
from these separate processes is sent to the compositing ffmpeg process
as uncompressed video in a pipe. To simplify the setup, named pipes
(special files) are used rather than setting up pipes in the ruby code
programmatically.
The extra ffmpeg processes are configured to log to files, and when
complete their log output is copied to the recording processing log.
Processes are joined to ensure zombie processes are not created, and
the return codes of all the processes are checked so errors can be
detected.
Due to the overhead of transferring video through pipes, this might
be a bit slower than the 2.4 recording processing - but on the other
hand, some of the video decoding and scaling happens in parallel, so it
might balance out.
Even with the filter changes made, there's still some cases where
filter chain hangs can result from filter reconfigurations. To solve the
issue completely, I have split out pre-processing video files to
separate ffmpeg processes, so that the filter chain for compositing will
not ever be reconfigured.
Each input video now has a separate ffmpeg process run for it which
does the scaling, padding, and video extending steps. To avoid issues
with disk space usage and extra cpu usage or quality loss, the output
from these separate processes is sent to the compositing ffmpeg process
as uncompressed video in a pipe. To simplify the setup, named pipes
(special files) are used rather than setting up pipes in the ruby code
programmatically.
The extra ffmpeg processes are configured to log to files, and when
complete their log output is copied to the recording processing log.
Processes are joined to ensure zombie processes are not created, and
the return codes of all the processes are checked so errors can be
detected.
Due to the overhead of transferring video through pipes, this might
be a bit slower than the 2.4 recording processing - but on the other
hand, some of the video decoding and scaling happens in parallel, so it
might balance out.
Because the input videos for BigBlueButton recording processing switch
resolution and aspect ratio, the filter chain gets re-initialized, and
any state in the filters is lost. This causes problems with the
following filters:
`color`: Timestamps restart from 0, rather than continuing at the point
where they left off.
`fps=start_time=12.345`: After reset, the fps filter thinks it's at the
start of the file again, so the next frame it sees gets duplicated
output for timestamps from the `start_time` until it catches back up.
`setpts=PTS-STARTPTS`: The 'STARTPTS' is re-read as the first pts the
filter sees after reinitialization, so timestamp of the next frame is
reset to 0. (In practise, this didn't cause any problems because the
duplicate frames created by the fps filter had the original start time.)
The end result of all of these issues is that a lot of duplicate frames
were created with invalid timestamps, which eventually get discarded
by ffmpeg. But a lot of time is wasted, causing recordings to sometimes
take hours to process when they should be ready in minutes.
The fixes are as follows:
* The `color` filters are used to generate the background and
substitutes for missing videos. Move them out to separate filter
chains by using the 'lavfi' input format, which lets you use a filter
as if it was an input file.
* Rather than use the `fps` filter's `start_time` feature, use the
`trim` filter to remove early frames.
* The actual start pts is already known by the script, so replace
`setpts=PTS-STARTPTS` with `setpts=PTS-12.345/TB`, substituting in the
absolute time.
Because the input videos for BigBlueButton recording processing switch
resolution and aspect ratio, the filter chain gets re-initialized, and
any state in the filters is lost. This causes problems with the
following filters:
`color`: Timestamps restart from 0, rather than continuing at the point
where they left off.
`fps=start_time=12.345`: After reset, the fps filter thinks it's at the
start of the file again, so the next frame it sees gets duplicated
output for timestamps from the `start_time` until it catches back up.
`setpts=PTS-STARTPTS`: The 'STARTPTS' is re-read as the first pts the
filter sees after reinitialization, so timestamp of the next frame is
reset to 0. (In practise, this didn't cause any problems because the
duplicate frames created by the fps filter had the original start time.)
The end result of all of these issues is that a lot of duplicate frames
were created with invalid timestamps, which eventually get discarded
by ffmpeg. But a lot of time is wasted, causing recordings to sometimes
take hours to process when they should be ready in minutes.
The fixes are as follows:
* The `color` filters are used to generate the background and
substitutes for missing videos. Move them out to separate filter
chains by using the 'lavfi' input format, which lets you use a filter
as if it was an input file.
* Rather than use the `fps` filter's `start_time` feature, use the
`trim` filter to remove early frames.
* The actual start pts is already known by the script, so replace
`setpts=PTS-STARTPTS` with `setpts=PTS-12.345/TB`, substituting in the
absolute time.
Several improvements to tldraw whiteboard:
- Only send the shape diff on shape updates (reduce a lot the message traffic)
- Shape permissions (don't allow others to select/edit unless you are presenter/moderator)
- This required some changes in akka model
- Tldraw state patch changes to improve stability with fast updates (fix several crashes)
When converting from using the 'movie' source filter to using separate
ffmpeg command line inputs, it wasn't taken into account that the
'movie' filter passes through timestamps from the source file, while the
ffmpeg input adjusts timestamps so that '0' is the selected seek point.
The easiest fix is to add an option to the ffmpeg command to disable the
timestamp adjustments.
Fixes#15644 (This needs to go into BigBlueButton 2.5 and 2.6)
- Return to the ResizeAndMoveSlide event to do pan&zoom, respecting the viewed width and height ratio
- Defaults zoom in toolbar to 100% like before to be more consistent
- Fit to width and Reset Zoom is back (fit tho width still has some sync problems)
- Fix to not change to first page when presenter reloads page
- Update the version to 2.6.0 to ease the detection of old/new whiteboard events
- Fix recording cursor when there is no pan/zoom and annotations in tldraw
- Don't generate slides pngs for 2.6.0, they are not used anymore in playback (svgs instead).
Changed the names of tldraw record events to differentiate from before.
Publish tldraw.json file with all shape information during the meeting to be used in playback.
Adapted cursor.xml and panzoom.xml to store tldraw data.
Publish slides svgs to be used by playback's tldraw component (otherwise we have different image sizes in pngs and thus messing the coordinates).
Retro-compatible with old recordings.
Based on work done by Tiago Jacobs and Guilherme Pereira Leme to
investigate the behaviour of ffmpeg on videos with changing input size.
Rather than having the EDL code set fixed sizes for the scale and pad
filters, instead use ffmpeg's built in features to calculate the scale
and pad automatically, dynamically updating if the input video size
changes.
In the version of ffmpeg in Ubuntu 20.04, the 'movie' input filter is
buggy and doesn't correctly handle the video size changing. Instead of
using the movie filter, use separate inputs to the ffmpeg command (the
design used here is based on the code in audio.rb). The filter chain is
now stored into a file (using -filter_complex_script) to reduce problems
with the command line getting too long.
We are using a version of ffmpeg that's new enough to have the tpad
filter now too, so use it to extend the video if needed instead of an
extra concat.
In BBB 2.5, we switched the recording system to use bundled gems
included privately in the recording package, rather than installed
system-wide. The rap-enqueue.rb script needs to be updated to load the
bundler gems.
According to bundler devs, setting the BUNDLE_GEMFILE environement
variable is the supported way to tell bundler where to find it
(otherwise bundler will search starting at the current working directory
- which in the case of rap-enqueue.rb is probably nowhere near the
Gemfile).
Use a relative path from the directory where the script is located so it
can be run both when installed and from a development environment.
Switch the script interpreter to use /usr/bin/env to load ruby from the
path. Doesn't make a difference in the installed package, but it makes
testing on development systems with multiple ruby environments easier.
Fixes#15085
* fix unit name: the unit name on Ubuntu is `redis-server.service`
* services which need a working redis require both After= and Wants=
See the description in the `systemd.unit` man page.
In some cases with incomplete/partially corrupt files, the input video
file can be shorter than the displayed time. If there is a badly timed
cut, this can result in a seek being generated to a point where ffmpeg
is unable to start at.
Add a detection for this situation, and replace with a blank video.
