Commit Graph

293 Commits

Author SHA1 Message Date
Pedro Beschorner Marin
beda54fb09 Included Etherpad's data as events to be stored 2020-04-27 10:48:52 -03:00
Calvin Walton
a5a46b2c0a Have the recording archive delete originals of archived media
After the last segment of a recording has been archived, it is safe to
delete the original files that were recorded by the various media
handling components. This patch deletes the freeswitch audio files and
kurento webcam/screenshare files after they have been archived.

It needs to be used in combination with some changes to users/groups and
directory permissions to allow the `bigbluebutton` user to delete files
that it could previously only read.
2020-03-27 15:22:53 -04:00
Leonardo Crauss Daronco
5bc3062d58 Fix the paths to the ruby scripts on the rap workers
Was pointing to the wrong directory when loading the ruby scripts. Also
made the directory easier to configure, it's all in a single place now.
2020-02-06 10:43:32 -03:00
Calvin Walton
c52a67186e RaP: Clean up resque worker code, add worker for keep_events feature
I've moved the workers code into the `lib` subdirectory with other library-ish
code; this puts it into the ruby load path used by most scripts so referencing
files is easier.

I've applied various style cleanups based on the rubocop config present.

The `events` processing step has been integrated as a new worker `EventsWorker`,
there is no longer a separate `events/events.rb` script. I've reworked the
`rap-starter.rb` script to check for the done files in both the events and
recorded status directories.
2020-01-21 17:14:39 -05:00
Calvin Walton
aee7baa76b Merge branch 'v2.0-rap-on-resque' of github.com:daronco/bigbluebutton 2020-01-17 17:40:08 -05:00
Calvin Walton
9907588b5e presentation recording: generate poll svg images without gnuplot
Write a tool that generates the poll svg images directly from the
BBB poll description. This avoids the issues with special characters
in the gnuplot labels, and gives us a lot more flexibility in how
the polls are formatted and styled.
2020-01-13 14:34:30 -05:00
Leonardo Crauss Daronco
95573b2467 Improve the yml for rec steps and run captions always before process 2019-12-10 16:28:13 -03:00
Leonardo Crauss Daronco
ac801cfce7 Use journald as the default logger for rap and rap services
Changed only in the main class so journald is the default and in the
scripts related to the processes in resque. Internal scripts might still
be logging to files.
2019-12-06 17:06:32 -03:00
Leonardo Crauss Daronco
f1d37e8a08 Gracefully stop bbb-rap-starter when the service is stopped 2019-12-06 16:36:25 -03:00
Leonardo Crauss Daronco
835449551d Raise an exception if a rap step fails so resque will list it
Resque captures the exceptions and marks the jobs as failed.
2019-12-06 15:51:47 -03:00
Leonardo Crauss Daronco
29cae33636 Add the format parameter to the examples of post scripts
They are optional but it's better to have them than not. If they are passed
and the script doesn't expect it they break.
2019-12-06 15:23:19 -03:00
Leonardo Crauss Daronco
5e05b7a8e8 Fixes for the rap scripts after processing a real recording 2019-12-06 12:26:24 -03:00
Leonardo Crauss Daronco
d74be7962a Process all .done files when bbb-rap-starter is started
Instead of just running when a .done file is added, it now also goes
through the files already there when the process starts.
2019-12-06 11:50:08 -03:00
Leonardo Crauss Daronco
53c3ed0d7d Change rap-starter to run persistently and watch for file changes
Instead of being executed every 30s by systemd, it's now a service that's
running all the time and will wait for .done files to start the processing
of recordings.
2019-12-05 11:41:13 -03:00
Leonardo Crauss Daronco
d769849ad6 Add resque worker to process the captions of a recording
Moved the code that was after the sanity step to its own worker.
2019-12-04 18:32:14 -03:00
Leonardo Crauss Daronco
62d62a2a1e Merge remote-tracking branch 'upstream/master' into v2.0-rap-on-resque
# Conflicts:
#	bigbluebutton-config/bin/bbb-conf
#	record-and-playback/core/Gemfile.lock
#	record-and-playback/core/lib/recordandplayback/events_archiver.rb
#	record-and-playback/core/scripts/archive/archive.rb
#	record-and-playback/core/scripts/bigbluebutton.yml
#	record-and-playback/core/scripts/rap-process-worker.rb
#	record-and-playback/core/scripts/rap-publish-worker.rb
#	record-and-playback/core/scripts/rap-sanity-worker.rb
#	record-and-playback/core/scripts/rap-starter.rb
#	record-and-playback/core/scripts/sanity/sanity.rb
#	record-and-playback/core/systemd/bbb-rap-archive-worker.service
#	record-and-playback/core/systemd/bbb-rap-process-worker.service
#	record-and-playback/core/systemd/bbb-rap-publish-worker.service
#	record-and-playback/core/systemd/bbb-rap-sanity-worker.service
#	record-and-playback/core/systemd/bbb-record-core.target
2019-12-04 17:33:44 -03:00
Leonardo Crauss Daronco
9e5c386e6f Add option to rap-process-worker to accept a filtering pattern
It will only process recordings which ID matches the pattern. By default
the pattern is nil and it will process all recordings.
2019-12-04 12:05:18 -03:00
Fred Dixon
dd2ae19ca0
Merge pull request #8277 from SilentFlameCR/master
edited rap-caption-inbox.rb to get appropriate files in presentation
2019-11-22 23:11:18 +01:00
SilentFlameCR
b88b70d945 edited rap-caption-inbox.rb to get appropriate files in presentation 2019-10-30 13:10:24 -04:00
Calvin Walton
8283bc16f0 recording archive: Archive freeswitch audio in formats other than wav 2019-10-28 13:47:33 -04:00
Calvin Walton
8bdcea424d Fix typos in archive script 2019-10-23 10:16:29 -04:00
Calvin Walton
cad8995858 recording archive: Compress audio files
The wav files from the freeswitch recordings are huge, uncompressed.
There's no reason to keep them as wav, we can compress them in the
archive step.
2019-10-22 17:27:23 -04:00
Richard Alam
e409190114 - convert tabs to spaces 2019-07-10 13:09:38 -07:00
Richard Alam
ec1343931c - add comments to explain how rap-archive-worker and rap-events-worker work together 2019-07-05 08:57:51 -07:00
Richard Alam
40c6dd8827 - run post events scripts 2019-07-05 08:57:22 -07:00
Richard Alam
61d9223a70 - format spaces to tabs 2019-07-04 11:07:57 -07:00
Richard Alam
82bd5e4a6f - run keep-events worker separate from archive-worker 2019-07-04 11:00:59 -07:00
Calvin Walton
4c1e2ed477 RaP: Reduce ffmpeg output spam in logs of caption inbox 2019-06-06 10:49:27 -04:00
Calvin Walton
7f92b58140 RaP: Dir.glob accepts a block directly, no need for a temp variable 2019-06-06 10:44:47 -04:00
Calvin Walton
620ed367be RaP: glob is a method on Dir, not File 2019-06-06 10:43:40 -04:00
Calvin Walton
94ae7c330b RaP: Use correct variable to get info for new caption track 2019-06-06 10:41:14 -04:00
Calvin Walton
aecb66e28d RaP: Create the recording captions dir before moving the converted vtt file 2019-06-06 10:34:08 -04:00
Calvin Walton
07688593ac caption-inbox: clean up track file if language tag is invalid 2019-06-04 14:28:58 -04:00
Calvin Walton
f693915e99 RaP captions: Fix usage of JSON.parse, use a proc to capture local vars 2019-06-04 14:01:51 -04:00
Calvin Walton
f04bf290af Fix requires for journald-logger 2019-06-04 12:18:46 -04:00
Calvin Walton
983751c9c0 RaP: Have the caption inbox run the caption integration scripts. 2019-06-04 11:03:22 -04:00
Calvin Walton
676fa41f06 Add caption integration script for presentation format. 2019-06-04 11:03:22 -04:00
Calvin Walton
b22c905194 rap-caption-inbox: improve formatting, initial pass at conversion
It now attempts to convert the uploaded caption file to WebVTT and
updates the index file.
2019-06-04 11:03:22 -04:00
Calvin Walton
c6e09f52a1 RaP Caption inbox: read uploaded caption json file
Some additional validation/normalization is added for the language tag,
and some cleanup has been done for logging.
2019-06-04 11:03:22 -04:00
Calvin Walton
23da507fa8 Initial outline of the rap worker tool for handling the caption inbox
Rather than running the tool in a loop, I'm using inotify to watch for
new files being created (ideally, the other rap workers will be migrated
to this style in the future). The trigger for processing is the creation
of the .json index file for the uploaded track.
2019-06-04 11:03:22 -04:00
Ghazi Triki
c9194d087e Correctly process captions. 2019-05-13 21:38:51 +01:00
Ghazi Triki
7c5727750f Merge remote-tracking branch 'bigbluebutton/master' into text-tracks 2019-05-13 14:46:20 +01:00
Ghazi Triki
b578aeb243 Move captions.rb to utils directory. 2019-05-10 16:56:29 +01:00
Ghazi Triki
dbd86eb096 Code improve. 2019-05-10 15:51:05 +01:00
Fred Dixon
056ef6ee8c
Merge pull request #7327 from ritzalam/add-redis-password-on-master
Add redis password on master
2019-05-09 16:35:42 -04:00
Ghazi Triki
17a3af0487 Write playback caption files from the captions file generated for API. 2019-05-07 18:13:09 +01:00
Ghazi Triki
35506cdc44 Generate captions.json for API. 2019-05-01 16:01:15 +01:00
Ghazi Triki
f820446965 Process caption files before processing meeting files. 2019-04-30 19:54:02 +01:00
Ghazi Triki
925f5a2894 Put captions files in the new directory. 2019-04-22 20:28:54 +01:00
Richard Alam
f2098bc376 Merge branch 'master' of github.com:bigbluebutton/bigbluebutton into ritzalam-add-redis-password-on-v2.2-dev 2019-04-02 10:43:46 -07:00
Felipe Cecagno
f70331cbba update notes formats archived 2019-03-18 14:33:29 -03:00
Pedro Beschorner Marin
7e5c5c29ed Avoid publishing empty notes 2019-03-15 17:19:54 -03:00
Felipe Cecagno
51b9b91dae rename note to notes 2019-03-11 23:56:29 -03:00
Felipe Cecagno
3467d912c7 fix typo 2019-03-05 13:22:10 -03:00
Felipe Cecagno
657a3ef013 use etherpad api to archive exported note 2019-03-05 13:17:48 -03:00
Calvin Walton
010153cbb9 Recording: Remove the red5 race workaround that delays processing
This was added to workaround for red5 taking a while to rewrite the
serialized (.ser) data that it streams to disk back to the .flv format.

The workaround is no longer needed, for two reasons:
* The sanity scripts run the red5 code to generate the .flv from the .ser
  if needed, and
* We're expecting more people to be using WebRTC media in the future anyways

This makes recordings available up to 2 minutes earlier than they would have
been otherwise.
2019-02-28 11:13:12 -05:00
Ghazi Triki
99aa2b5fcc Use the right bigbluebutton user for recording directories. 2019-02-08 17:53:57 +01:00
Richard Alam
05c7b5f7e7 Merge branch 'v2.2-dev' into merge-with-v2.2-dev 2018-12-14 15:08:47 -08:00
Richard Alam
229d537f91 - add ability to set redis password on RnP 2018-12-14 14:16:12 -08:00
Richard Alam
1b3895e3aa - add redis password to record and playback scripts 2018-12-14 13:37:21 -08:00
Pedro Beschorner Marin
e365bedafb Copying events.xml from archived recordings 2018-12-13 15:21:10 -02:00
Pedro Beschorner Marin
ebffcb0848 Created property keepEvents at bbb-web to make redis record the events for all meetings 2018-12-11 18:37:31 -02:00
jfederico
bb078aab20 Updates for backward compatibility to recording-ready-callback script 2018-10-12 21:41:37 +00:00
Calvin Walton
c5810ae953
Merge pull request #6113 from jfederico/record-and-playback
Added script for sending recording ready notifications
2018-10-02 16:17:22 -04:00
jfederico
90e3c6e360 Added script for recording ready notification 2018-10-02 18:57:15 +00:00
jfederico
b9f52a2438 Script for fixing meeting tag in recordings migrated from BBB 1.0 and before 2018-09-20 14:23:29 +00:00
Calvin Walton
c43863eff6 Merge branch 'playback-async-requests' of https://github.com/fcecagno/bigbluebutton into v2.2-dev 2018-07-24 09:56:50 -04:00
Felipe Cecagno
dc0fbc9bd1 cleanup 2018-07-12 00:06:14 -03:00
Richard Alam
a2eaa9babd Merge branch 'master' of github.com:bigbluebutton/bigbluebutton into 21-and-20-merge-working-branch
- some conflicts have been fixed.

The following needs working on:

       both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/services/ChatCopy.as
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/services/ChatSaver.as
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/views/ChatTab.mxml
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/chat/views/ChatWindowEventHandler.as
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/users/services/MessageReceiver.as
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/users/services/MessageSender.as
        both modified:   ../bigbluebutton-client/src/org/bigbluebutton/modules/users/views/MediaItemRenderer.mxml

Parking as need to work on something else.
2018-06-06 08:23:13 -07:00
Calvin Walton
7d643f9bf8 RaP sanity: Don't delete .ser file unless repair succeeded 2018-05-23 12:03:20 -04:00
Calvin Walton
9cdafbdfe7 RaP: Merge the sanity script from master branch
Due to improvements in the recording scripts, most of the stuff the sanity
script was checking for is no longer needed (missing/corrupt video files
are handled by the processing scripts). The version of this script in
master has been cleaned up so that the only things it does are:

- Check that the events.xml exists and is properly formatted xml
- Rebuild flv files from red5 .flv.ser/.flv.info files

The script from master is compatible with the 2.0 code, so just use it
as-is.

This fixes a problem where following the recent red5 upgrade in 2.0 branch,
an flv file is never written for webcam streams where no frames were
received, despite there being recording events.
2018-05-23 11:18:04 -04:00
Calvin Walton
18d472e3a3 RaP: Improve check for record events in archive, sanity scripts
There's some cases where you can get 0-duration recordings due to
recording event placement (e.g. a single recording event is the last
event in the events.xml). Detect these cases, and treat them like
no recording marks in the archive script (it will stop the recording
from being automatically processed).

I've also adjusted the sanity script to detect these cases and error
out. The recording processing scripts cannot handle 0-length recordings,
you have to manually edit the events. I've added a message to the
sanity log about this.
2018-05-16 11:32:29 -04:00
Calvin Walton
cd3e28a08c Fix some errors in screenshare archive/sanity 2018-04-25 14:28:37 -04:00
Calvin Walton
9f528e37fe Archive kurento screenshare streams 2018-04-25 11:35:36 -04:00
Calvin Walton
c45fa74da0 Archive the webcam files from kurento webcam sharing 2018-04-24 15:18:02 -04:00
Calvin Walton
d8c0f7a67d Don't delete video events in the sanity script.
When working with the segmented recording format, the events file might
end with a mismatched start video event for an incomplete file. The
sanity script was removing this event, meaning the video didn't show up
in future segments.

Simply drop the code that tries to find invalid video files and removes
them from the events file. The new video processing code is already
robust against missing or corrupt files.
2017-12-14 14:43:31 -05:00
Calvin Walton
feee34c144 Fix parsing meeting id for segments in publish step 2017-11-10 13:10:58 -05:00
Calvin Walton
abdb639e01 Bump bbb version to 2.1 so we can identify the new events
BBB 2.1 has multiple presentation pods and multiple chats, so the messages
for those have been altered.
2017-11-08 17:06:00 -05:00
Calvin Walton
e2b40cc5bd Rework a bunch of the code to avoid repeatedly reparsing the events.xml 2017-11-08 10:12:25 -05:00
Calvin Walton
ab1019bf14 Set up rap publish/process scripts for segments 2017-11-07 11:50:28 -05:00
Calvin Walton
4e243536c7 Handle the no audio files case a bit better
If no audio files were found, it was running rsync with one argument,
which is a bit unexpected. It just printed a file list in this case, but
we can provide a cleaner error message instead.
2017-11-07 10:03:20 -05:00
Calvin Walton
50f2c1b90b Fix syntax issues in sanity scripts 2017-11-07 09:53:08 -05:00
Calvin Walton
9add60d81b Fix a bunch of bugs found when testing the archive code 2017-11-06 15:22:43 -05:00
Calvin Walton
5bb5cbcd0e Have archive script check whether a particular segment is recorded
It previously checked whether any part of the entire meeting was recorded.
Helper functions are added to look up the time of segment start and end
(which handle non-segmented recordings correctly too).
Part of the events handling code was rewritten to reduce the number of times
that the events.xml file gets parsed.
2017-11-03 16:12:05 -04:00
Calvin Walton
f9c4843f28 Cleanups, initial work on checking for segment recorded status. 2017-11-03 15:17:04 -04:00
Calvin Walton
0e2785205a Update sanity script for recording segments, fix archive done files 2017-11-03 12:02:36 -04:00
Calvin Walton
037014db94 Adjust sanity rap worker to support segment breaks 2017-11-03 11:27:00 -04:00
Calvin Walton
f5cd1628e0 Trim events list after archive, make archive fail block future segments 2017-11-03 11:16:44 -04:00
Calvin Walton
a78997d1a1 Work in progress on segment archiving 2017-11-01 12:31:07 -04:00
Calvin Walton
0affb6a36e Make progressive archiving of media files work.
Switch to using rsync to archive video directories, with the -t option
so it'll only update modified/new files.
2017-10-30 15:04:30 -04:00
Calvin Walton
0acb3b5750 Adapt rap-archive-worker to segmented recording done file format 2017-10-30 13:40:53 -04:00
Calvin Walton
b2f8c80202 Handle out of order inserts at the start of a caption stream.
If you're inserting at position 0 (and there was no previous deleted text
from that position), you can't use the timestamp from the previous character
position, since there's no previous character. Use the timestamp of the
following character instead.
2017-10-12 14:16:34 -04:00
Leonardo Crauss Daronco
58fe40dadd Fix reference to the format name in rap-publish-worker 2017-10-09 19:06:06 -03:00
Leonardo Crauss Daronco
af87bc2d3f Merge remote-tracking branch 'upstream/master' into v1.1.0-rap-on-resque 2017-10-09 16:37:28 -03:00
Calvin Walton
dba121e6f4 Merge pull request #3998 from kepstin/caption-fixes
Update caption track generation based on feedback
2017-08-25 12:04:59 -04:00
Calvin Walton
33a511d800 Move the custom_hash require to publish rap worker where it's used 2017-08-18 17:45:15 -04:00
Leonardo Crauss Daronco
0b662c5342 Fix call to resque's "prune_dead_workers" on rap workers 2017-08-15 12:10:34 -03:00
Leonardo Crauss Daronco
aa8a7818a9 Remove stale workers when starting a rap worker
A simple but effective way of garbage collection.
See:

  http://www.rubydoc.info/github/resque/resque/Resque%2FWorker:prune_dead_workers
2017-08-11 17:32:42 -03:00
Leonardo Crauss Daronco
2a253cf3bd Schedule the next rec step in the base worker 2017-08-09 11:52:52 -03:00
Leonardo Crauss Daronco
50e02ad7dc Specify the sequence of rec steps in the props file
Removed from the code, now it can be specified in the properties file.
Makes it easier to include new processing scripts and control dependencies
between them.
2017-08-07 18:08:17 -03:00