gns3 virtual network for testing (#15673)

* add scripts to create gns3 virtual network for testing

* gns3 virtual testing network: introduce declarative syntax, i.e,
declare the objects we want and only create them if they don't exist

* gns3 virtual testing network: improve node autostart logic

* gns3 virtual testing network: because of declarative syntax,
we no longer need a special --create-test-server option

* gns3 virtual testing network: allow for servers based on arbitrary git commits

* gns3 virtual testing network: avoid browsers complaining about duplicate certificate serial numbers

* gns3 virtual testing network: allocate space for two more servers

* gns3 virtual testing network: add subnets to NAT devices, so they will boot properly

* gns3 virtual testing network: allow number of ports on an Ethernet switch to be configured

* gns3 virtual testing network: move everything 100 units to the left to center PublicIP switch

* gns3 virtual testing network: restart the server by stop'ing and start'ing it
because bbb-conf --restart doesn't start nginx if it isn't running

* gns3 virtual testing network: -d switch now deletes one server;
--delete-everything does what -d used to do

* gns3 virtual testing network: new GUI image name (it's different for every user)

* gns3 virtual testing network: start node logic now marks
previously running nodes as started

* gns3 virtual testing network: add a Bash function to automate ssh logins

* gns3 virtual testing network: use "$@" instead of $* when relaying arguments to a bash command

* gns3 virtual testing network: allow ssh connections to arbitrary hosts behind NAT4

* gns3 virtual testing network: improve start node logic to start new servers when NAT1 already running

* gns3 virtual testing network: add scp command to bigbluebutton-ssh.sh

* gns3 virtual testing network: gns3-bbb.py script now accepts a list of
servers to install, or nothing to just install the infrastructure

* gns3 virtual testing network: can now copy SSL CA key and certificate
from local directory to virtual machine instead of always having the
VM create a new CA

* gns3 virtual testing network: get the quoting right in scp shell function for filenames with spaces

* gns3 virtual testing network: update README

* gns3 virtual testing network: update README

* gns3 virtual testing network: add NPDC submodule and use its gns3 import in gns3-bbb.py

* gns3 virtual testing network: use 3 network interfaces for testclient

* gns3 virtual testing network: renumber NAT devices to match interface numbering on testclient

* gns3 testing network: fix dnsmasq configuration on NAT1 so that it works for devices on the PublicIP subnet

* gns3 testing network: fix dnsmasq configuration on NAT4/5/6, but still not quite right

* gns3 testing network: "fix" dnsmasq configuration so that proxyjumped ssh works,
along with access to the servers from the testclients

* gns3 virtual testing network: label subnet switches with their CIDR prefices

* gns3 virtual testing network: move the clients and their switches a bit to the right

* gns3 virtual testing network: rename NAT1 to be the project name (default BigBlueButton)
and get rid of InternetSwitch, that I never use

* gns3 virtual testing network: allow multiple clients, all names starting with 'testclient'

* gns3 virtual testing network: factor out server NAT code, to allow for recreating deleted server NAT nodes

* gns3 virtual testing network: refactor client code (move things inside the function)
and set "dhcp-identifier: mac" on all client interfaces

* gns3 virtual testing network: make NAT1 advertise itself as ca.test into DNS domain test

* gns3 virtual testing network: refactor master gateway code into its own subroutine

* gns3 virtual testing network: better use of the gns3 library support for declarative nodes
(nodes that are declared but only created if they don't already exist)

* code cleanups; make notification_url consistently a global var;
change label on cloud from "Internet" to interface name

* typos from 39c5d6

* gns3 virtual testing network: move certificate authority from /ca to /opt/ca

* gns3 virtual testing network: move testserver.sh from / to /root

* gns3 virtual testing network: master gateway can now web proxy for servers, allowing
a web browser on the bare metal machine to access those servers

* gns3 virtual testing network: use a file() function instead of a predetermined list of files we need

* gns3 virtual testing network: add a minimal bird configuration to use OSPF to make proxy work

* gns3 virtual testing network: new-dhcp-lease.sh now picks mac addresses
based on server name, allowing multiple servers, and some bug fixes

* gns3 virtual testing network: get rid of generate NAT per-boot script (do everything at boot with iptables-persistent)

* gns3 virtual testing network: move client NAT boxes 50 coordinates to the right

* gns3 virtual testing network: autostart the nodes that were created

* gns3 virtual testing network: use 2 minute DHCP leases to make
it quicker to recover from update that replace the NAT devices

* gns3 virtual testing network: change NAT rules around because server needs to connect to itself during bbb-install

* gns3 virtual testing network: use version of NPDC that has 2 minutes timeouts
on the bare metal machine's DHCP leases

* gns3 virtual testing network: Improve the DNS configuration by configuring
the gateway to not use the bare metal machine for DNS lookups on the ".test"
domain, on either the real interface or the dummy interfaces used to
inject DNS names into the bare metal machine's DNS table.  This removes
the need to put entries into /etc/hosts (which was causing problems
when things changed IP addresses) because the gateway now consistently
uses its own DNS server (dnsmasq) to resolve the ".test" domain.

* gns3 virtual testing network: set "noninteractive" flag when updating
persistent iptables, otherwise the system tries to prompt the user
during an automatically cloud-init run (the prompts show up in
/var/log/cloud-init-output.log and stall the cloud-init run)

* gns3 virtual testing network: configure proxy to redirect http->https for the BigBlueButton servers

* gns3 virtual testing network: put some extra checks on the SSL certificates
generated for the proxy server, because if they're screwed up the apache
web server won't start at all

* gns3 virtual testing network: simplify getportrange.cgi by just having it accept GETs

* gns3 virtual testing network: two bug fixes with punching UDP ports through the proxy server
   1. if the server changed public IP addresses, a new iptables rule was not created
   2. the math was wrong; it's supposed to be a range of 100 ports, not 1000

* gns3 virtual testing network: iptables rules still weren't right on
the server's NAT gateways; connections to the gateway (128.8.8.254)
need to go through (they weren't), while only connections to the server get hairpined back

* gns3 virtual testing network: make NAT gateways listen for ssh on port 2222,
so you can connect to them easily (with a suitable ssh config file) by
just doing "ssh -p 2222 focal-260"

* gns3 virtual testing network: now that DNS is working right (read: better),
we don't need the funky shell functions for ssh and scp

* gns3 virtual testing network: updated README

* gns3 virtual testing network: switching to ISC bind/dhcpd for dynamic DNS
and dropping the proxy server and the restricted port ranges
This version works, but still has issues

* gns3 virtual testing network: don't need new-dhcp-lease.sh anymore
(it's now done by ISC dhcpd server, which has built-in support for DDNS)

* gns3 virtual testing network: DDNS server no longer requires crypto key

* gns3 virtual testing network: rearrange some comments, remove an unused script

* gns3 virtual testing network: no longer need to set a port range on the servers,
since they are directly reachable from the bare metal machine using OSPF and dynamic DNS

* gns3 virtual testing network: drop final remnants of old "NAT1"
name; it's now consistently "master gateway"

* gns3 virtual testing network: reduce DHCP lease times to ten seconds

* gns3 testing network: move testserver.sh from /root to /, because ubuntu needs permission to exec it

* gns3 virtual testing network: move client nodes a bit to the right in the GUI

* gns3 virtual testing network: simplify bird (OSPF) configuration a bit

* gns3 virtual testing network: add --public-subnet and --domain options

* gns3 virtual testing network: use most recent ubuntu-open-desktop image, if several are available

* gns3 virtual testing network: pick up the server's domain name from
the testing network now that we have a --domain option (commit f35f21)

* gns3 virtual testing network: introduce --server-subnet option

* gns3 virtual testing network: label server subnets with subnet prefix (if possible)

* gns3 virtual testing network: all calls to dpkg-reconfigure are noninteractive

* gns3 virtual testing network: turn off DHCP/DDNS conflict detection

* gns3 virtual testing network: use 120 second DHCP leases; 10 second leases were a problem

* gns3 virtual testing network: fix commands to save iptables (wasn't saving NAT table)

* gns3 virtual testing network: add a 'certbot' node running Let's Encrypt's boulder server

* gns3 virtual testing network: switch 'certbot' to use smallstep's step-ca server

* gns3 virtual testing network: first attempt to dummy letsencrypt's acme service (doesn't work)

* gns3 virtual testing network: a dummy ACME server works; server certbots use it to get SSL certificates

* gns3 virtual testing network: switch testservers to use certbot to get SSL certificates

* gns3 virtual testing network: put generateCA.sh back in (in case CA key and cert don't already exist), and remove getcert.cgi

* gns3 virtual testing network: add a standard BigBlueButton turn server

* gns3 virtual testing network: can now use 400 permission for CA key file

* gns3 virtual testing network: dnsmasq cname doesn't work with IP addresses (why should it?); use interface_name instead

* gns3 virtual testing network: install server certificate from cloud-init, rather than doing it after boot

* gns3 virtual testing network: fix certbot on master gateway so that it can update certificates as they expire

* gns3 virtual testing network: only build and/or start NAT4/5/6 if testclient is requested

* gns3 virtual testing network: generate root CA certificate on bare metal machine and install it in all VMs from there

* gns3 virtual testing network: use certbot nginx plugin instead of standalone mode

interacts better with nginx (don't have to stop and restart nginx everytime you do a certbot authentication)

* gns3 virtual testing network: add --no-nat switch to build virtual servers without NAT gateways

* gns3 virtual testing network: need to not only start step-ca, but enable it (start on every boot)

* gns3 virtual testing network: save DNS domain name in GNS3 project variables,
so we don't have to specify it on the command line every time we run this script

* gns3 virtual testing network: add --no-install switch

* gns3 virtual testing network: fix bogus help text

* gns3 virtual testing network: --delete now deletes just a specific server and its associated nodes

Old behavior (deleting nodes whose name contains a substring) is now done with --delete-substring

The old behavior could delete too much (focal-260 would match focal-260-2) and too little
(subnet switches are named by their CIDR prefix if possible)

* gns3 virtual testing network: when creating a new server, check for existing nodes in all GUI locations to be used

* gns3 virtual testing network: update comments

* gns3 virtual testing network: add return code checks when calling openssl

* gns3 virtual testing network: README update

* gns3 virtual testing network: add 'natturn' TURN server behind a NAT gateway

* gns3 virtual testing network: make TURN server observe --no-install option

* gns3 virtual testing network: only try to install 'natturn' if it doesn't already exist,
because even though it's got "declarative" procedures (you declare what you want but it
only creates it if needed), natturn's subnet will conflict with server subnets

* gns3 virtual testing network: allow Ubuntu release (18 or 20) to be set for a new server

* gns3 virtual testing network: add --repository and --install-script options
and move computation of the install script from testserver.sh to the main script

* gns3 virtual testing network: add --proxy-server option; we pass this option to bbb-install script

* add --release option to specify which BigBlueButton server release should be installed,
and improve --install-script option to accept local files or arbitrary URLs

* gns3 virtual testing network: guess version of install script instead of defaulting to 2.6

* gns3 virtual testing network: enable NAT in both directions on gateway device

* gns3 virtual testing network: check server release for validity before attempting install

* updated README and newest version of NPDC

* gns3 virtual testing network: add option to install greenlight

* gns3 virtual testing network: drop --domain switch and instead use an initialization server instead to figure out DNS domain

* gns3 virtual testing network: don't create turn or natturn devices unless specifically requested

* gns3 virtual testing network: updated README

* gns3 virtual testing network: use a newer version of NPDC that monitors console logs, so we watch the servers boot

* gns3 virtual testing network: updated README

* gns3 virtual testing network: update NPDC for backwards compatibility with Ubuntu 18

* gns3 virtual testing network: update instructions for git submodules

* gns3 virtual testing network: save public IP subnet in GNS3 project variables,
so we don't have to specify it every time the script runs

* gns3 virtual testing network: name NAT gateways more consistently

* gns3 virtual testing network: add --quiet switch

* gns3 virtual testing network: compute veth domain using initsrv and use it to label the cloud icon

* gns3 virtual testing network: improve handling of nginx hash bucket errors on install

* gns3 virtual testing network: bump NPDC version; avoid trying to print console messages on VNC consoles (it won't work)

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "bigbluebutton-tests/gns3/NPDC"]
+	path = bigbluebutton-tests/gns3/NPDC
+	url = https://github.com/BrentBaccala/NPDC

bigbluebutton-tests/gns3/NPDC

@@ -0,0 +1 @@
+Subproject commit af7d93c08759e50cf3a8eca12fade498cee33ecb

bigbluebutton-tests/gns3/README.md

@@ -0,0 +1,122 @@
+
+# GNS3-BBB
+
+Scripts to build a virtual BigBlueButton network in a gns3 project (for testing purposes).
+
+**Prerequisites:** a Ubuntu server with KVM (Kernel Virtual Machine) support and enough CPU and RAM to support the virtual machines in the virtual network
+
+**Note**: gns3 is very picky about matching GUI client and server versions.  I typically put dpkg holds on the gns3 packages, since otherwise an apt upgrade on my laptop requires both an apt upgrade on my gns3 server *and* restarting the gns3 server, which implies stopping and restarting all of the running VMs.
+
+**Note**: gns3 uses qemu, which	can not	run concurrently with VirtualBox.  If VirtualBox virtual machines are running. gns3 virtual machines will not start, and vice versa.
+
+**Note:** Once the script has been used to build the virtual network (takes about an hour), the virtual network can be stopped and restarted without having to re-run the script.
+
+## Design
+
+The script will build a gns3 project that looks like this:
+
+![network diagram](README.png)
+
+The network "highjacks" the 128.8.8.0/24 subnet, so it simulates public IP address space.  You can set a different public subnet using the `--public-subnet` option to the script.
+
+The DNS domain name is configured to match the bare metal hostname.  If the bare metal machine is called `osito`, for example, the virtual machines will be given names like `BigBlueButton.osito` and `focal-260.osito`.
+
+The `BigBlueButton` virtual machine (called `master_gateway` in the script) is named to match the gns3 project name, which is `BigBlueButton` by default.  The project name (and the name of the master gateway) can be changed using the `--project` option.
+
+The master gateway, in addition to providing DNS and DHCP service for the 128.8.8.0/24 subnet, also operates a STUN server that presents itself in DNS as `stun.l.google.com`, so that STUN operations, on both the BigBlueButton clients and servers, yield the 128.8.8.0/24 addresses as public addresses.  `BigBlueButton` also operates an ACME CA signing service (so that `certbot` works), and mimics `resolver1.opendns.com` (used by `bbb-install` to check that the server can reach itself).
+
+The master gateway also announces the 128.8.8.0/24 subnet to the bare metal machine using OSPF, and implements NAT, so that the bare metal machine can connect to the virtual servers.
+
+The `focal-260-NAT` device announces itself into DHCP/DNS as `focal-260.DOMAIN` and forwards ports 80 and 443 (along with UDP ports) through to `focal-260` itself.  Clients can therefore connect to `focal-260.DOMAIN`, just as they would to a typical BBB server.  The NAT device itself listens for ssh on port 2222.  The `--no-nat` option can be specified to create a server without an associated NAT gateway.
+
+Default operation of the script is to install a server whose name is passed into the script and is used both as the hostname of the server and as the release name to install.  Obvious server names include `focal-250`, `focal-25-dev`, and `focal-260`.  You can specify the `-r`/`--repository` option to use a repository other than `ubuntu.bigbluebutton.org` (just like the install script).  The `--install-script` option allows an alternate install script to be used.
+
+Some special names are defined.  Requesting a device name starting with `testclient` creates a test client that connects to NAT4 (overlapping server address space), NAT5 (private address not overlapping server address space), and NAT6 (carrier grade NAT).  Likewise, `turn` and `natturn` devices can also be created, just by requesting them by name.
+
+## Usage
+
+1. You'll need several tools from Brent Baccala's NPDC repository on github, which is a submodule in the NPDC directory, so run this command to fetch it:
+
+   ```
+   git submodule init
+   git submodule update
+   ```
+
+1. Read, understand, and run the `install-gns3.sh` script in `NPDC/GNS3`
+
+1. Upload a current Ubuntu 20 cloud image to the gns3 server using NPDC's `GNS3/upload-image.py`:
+
+   `./upload-image.py https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img`
+
+   The most uncommon Python3 package that this script uses is `python3-requests-toolbelt`.  `python3-clint` is also recommended, to get a progress bar.
+
+   If this step works, then you have REST API access to the GNS3 server.
+
+1. You should now be able to boot an Ubuntu instance with this `NPDC/GNS3` script:
+
+   `./ubuntu.py -r 20 -m 1024 --debug`
+
+   Double-click on the icon that appears in the GUI to access the instance's console.  You should also be able to login using `ssh ubuntu`.
+
+   The `--debug` option adds a login with username `ubuntu` and password `ubuntu`.
+
+   Login and verify, in particular, that networking is working properly.  You should have Internet access.
+
+1. Finally, build the BigBlueButton project in gns3 with `./gns3-bbb.py`
+
+1. Install a server with `./gns3-bbb.py --wait-all focal-260`
+
+   The `--wait-all` option will cause the script to wait for BigBlueButton to install while you watch.  Without this option, the script will pause to wait for the NAT device to boot before starting the BigBlueButton server, then terminate once the BigBlueButton server has begun its install sequence.
+
+1. You can run tests directly from the bare metal machine.  The script created an SSL certificate in its own directory called `bbb-dev-ca.crt` which can be installed and trusted on your web browser.
+1. Add another server with `./gns3-bbb.py focal-250`
+1. Remove a server and its associated NAT gateway and switch with `./gns3-bbb.py --delete focal-250`
+
+1. `ssh` into the server devices directly.
+
+1. You can `ssh` into a server's NAT gateway with `ssh -p 2222 focal-260`.
+
+1. Since test servers come and go fairly frequently, I find the following stanza useful in my `.ssh/config`:
+
+   ```
+   Host BigBlueButton NAT? testclient* focal-*
+     User ubuntu
+     UserKnownHostsFile /dev/null
+     StrictHostKeyChecking no
+   ```
+
+   This stops `ssh` from complaining about server host keys changing, which happens every time you delete and rebuild a server.
+
+### Installing test clients
+
+1. Build a GUI image using NPDC's `GNS3/ubuntu.py`:
+
+   `./ubuntu.py -r 20 -s $((1024*1024)) -m 1024 --boot-script opendesktop.sh --gns3-appliance`
+
+   This step adds the GUI packages to the Ubuntu 20 cloud image and creates a new cloud image used for the test clients. It takes about half an hour.
+
+1. Upload the resulting GUI image to the gns3 server using NPDC's `GNS3/upload-image.py`
+
+1. Add a test client with `./gns3-bbb.py testclient`
+
+1. You can access a testclient's GUI by double-clicking on its icon in the GNS3 GUI.
+
+1. You can `ssh` into a testclient by specifying its NAT gateway as a jump host (`-J`) option to ssh: `ssh -J NAT4 testclient`
+
+### Possible Test Environments
+
+1. UDP ports can be blocked, forcing use of TURN, like this (blocks all servers from bare metal clients):
+
+   ```
+   $ ssh BigBlueButton
+   ubuntu@BigBlueButton:~$ sudo iptables -A FORWARD -p udp -j REJECT
+   ```
+
+   or like this (blocks one server from all clients):
+
+   ```
+   $ ssh -p 2222 focal-260
+   ubuntu@focal-260-NAT:~$ sudo iptables -A FORWARD -p udp -j REJECT
+   ```
+
+2. Install a proxy on BigBlueButton and force its use by blocking TCP traffic?  (bbb-install issue #583)

bigbluebutton-tests/gns3/bird.conf

@@ -0,0 +1,25 @@
+# This is a minimalist bird configuration that causes the psuedo-Internet
+# subnet on ens5 to be announced via OSPF on ens4.  This allows the bare
+# metal system to pick up a route to the psuedo-Internet subnet.  In
+# particular, RTP audio and video to proxied servers won't work without
+# this, because the server and client are far enough removed (two NAT
+# gateways between them) that they don't have any shared address space.
+# By advertising the psuedo-Internet 128.8.8.0/24 to the client, that
+# ensures that the client can reach one of the addresses advertised
+# by the server.
+
+# The Device protocol is not a real routing protocol. It doesn't generate any
+# routes and it only serves as a module for getting information about network
+# interfaces from the kernel. 
+
+protocol device { }
+
+protocol ospf OSPF {
+        area 0 {
+                interface "ens*" {
+                        cost 10;
+                };
+        };
+        import none;
+        export all;
+}

bigbluebutton-tests/gns3/step-ca.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=step-ca
+After=syslog.target network.target
+
+[Service]
+ExecStart=/usr/bin/step-ca /opt/ca/ca.json
+Type=simple
+
+[Install]
+WantedBy=multi-user.target

bigbluebutton-tests/gns3/testclient.sh

@@ -0,0 +1,95 @@
+#!/bin/bash
+#
+# Install Big Blue Button testing client
+#
+# This script runs once as 'ubuntu'
+
+# Make some changes to .bashrc, but they won't affect the shell that is already
+# running in the GUI, so the user will need to '. ~/.bashrc' there.
+#
+# We need NODE_EXTRA_CA_CERTS so that the playwright tests can make API calls
+# without getting certificate errors.
+
+cat >> ~/.bashrc <<EOF
+export NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/bbb-dev/bbb-dev-ca.crt
+export ACTIONS_RUNNER_DEBUG=true
+export BBB_URL=https://bbb-ci.test/bigbluebutton/api
+export BBB_SECRET=bbbci
+EOF
+
+# Which version of the repository should we use for the client test cases
+
+BRANCH=v2.5.x-release
+
+# if these are running, our apt operations may error out unable to get a lock
+sudo systemctl stop unattended-upgrades.service
+echo Waiting for apt-daily.service and apt-daily-upgrade.service
+sudo systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true
+
+sudo apt update
+sudo DEBIAN_FRONTEND=noninteractive apt -y upgrade
+
+# git, since we're about to use it
+# linux-image-extra-virtual to get snd-aloop module for dummy audio
+# v4l2loopback-dkms to get video loopback for dummy webcam
+# sudo apt -y install git-core ant ant-contrib openjdk-8-jdk-headless zip unzip linux-image-extra-virtual
+sudo apt -y install git-core linux-image-extra-virtual v4l2loopback-dkms
+
+# We don't need the whole git history, like this command would do:
+#    git clone https://github.com/bigbluebutton/bigbluebutton.git
+# so instead we do this to pick up a single revision:
+cd
+mkdir bigbluebutton-$BRANCH
+cd bigbluebutton-$BRANCH
+git init
+git remote add origin https://github.com/bigbluebutton/bigbluebutton.git
+git fetch --depth 1 origin $BRANCH
+git checkout FETCH_HEAD
+
+# Focal distributes nodejs 10, which is too old for our playwright test suite.  Use nodejs 16.
+curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash -
+sudo apt install -y nodejs
+
+cd ~/bigbluebutton-$BRANCH/bigbluebutton-tests/playwright
+npm install
+npx --yes playwright install
+
+# patch playwright's firefox so that it uses the system's root certificate authority
+find /home/ubuntu/.cache/ms-playwright -name libnssckbi.so -exec mv {} {}.distrib \; -exec ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so {} \;
+
+# playwright webkit doesn't have a fake audio device, but Linux does
+#   no point in enabling this since playwright can't grant permissions to use microphone on webkit (playwright issue #2973)
+# sudo modprobe snd-aloop
+# echo snd-aloop | sudo tee -a /etc/modules
+
+# this is required to run webkit tests, but conflicts with BBB server dependencies,
+# so can't be installed on the same machine as a BBB server
+sudo npx playwright install-deps
+
+# still need to either install an .env file or set environment variables in ~/.bashrc
+
+# In addition to the system root CA store in /usr/local/share/ca-certificates (used by curl and others),
+# we need to deal with two common browsers that don't use the system store.
+
+# Get Firefox (already installed) to use system's root certificate authority
+# Method suggested by https://askubuntu.com/a/1036637/71866
+# Earlier this this script, we did something similar to modify playwright's version of firefox.
+# This handles the standard system firefox.
+
+sudo mv /usr/lib/firefox/libnssckbi.so /usr/lib/firefox/libnssckbi.so.distrib
+sudo dpkg-divert --no-rename --add /usr/lib/firefox/libnssckbi.so
+sudo ln -s /usr/lib/x86_64-linux-gnu/pkcs11/p11-kit-trust.so /usr/lib/firefox/libnssckbi.so
+
+# Install chromium and the tools we need to install our certificate into Chromium's private store
+sudo DEBIAN_FRONTEND=noninteractive apt -y install chromium-browser libnss3-tools jq
+
+# chromium snap - we now need to install nssdb in ~/snap/chromium/2051/.pki instead of ~/.pki
+# NSSDB=/home/ubuntu/.pki/nssdb
+for CHROMIUM_SNAP in $(find /home/ubuntu/snap/chromium/ -mindepth 1 -maxdepth 1 -type d); do
+    NSSDB=$CHROMIUM_SNAP/.pki/nssdb
+    if [ ! -r $NSSDB ]; then
+	mkdir --parents $NSSDB
+	certutil -d sql:$NSSDB -N --empty-password
+    fi
+    certutil -d sql:$NSSDB -A -t 'C,,' -n bbb-dev-ca -i /usr/local/share/ca-certificates/bbb-dev/bbb-dev-ca.crt
+done

bigbluebutton-tests/gns3/testcreds.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+SECRET=$(grep sharedSecret /etc/bigbluebutton/bbb-apps-akka.conf | sed 's/^.*=//')
+URL=$(grep bigbluebutton.web.serverURL= /etc/bigbluebutton/bbb-web.properties | sed 's/^.*=//')
+
+echo BBB_URL="$URL/bigbluebutton/api"
+echo BBB_SECRET=$SECRET

bigbluebutton-tests/gns3/testserver.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+#
+# Install a Big Blue Button testing server on a VM
+
+# if these are running, our apt operations may error out unable to get a lock
+sudo systemctl stop unattended-upgrades.service
+echo Waiting for apt-daily.service and apt-daily-upgrade.service
+sudo systemd-run --property="After=apt-daily.service apt-daily-upgrade.service" --wait /bin/true
+
+sudo apt update
+sudo DEBIAN_FRONTEND=noninteractive apt -y upgrade
+
+DOMAIN=$(hostname --domain)
+FQDN=$(hostname --fqdn)
+
+EMAIL="root@$FQDN"
+
+# /bbb-install.sh (the proper version; either 2.4, 2.5 or 2.6) is created by gns3-bbb.py
+# INSTALL_OPTIONS and RELEASE get passed in the environment from gns3-bbb.py
+#
+# INSTALL_OPTIONS can include -w (firewall) -a (api demos; deprecated in 2.6) -r (repository)
+
+sudo /bbb-install.sh -v $RELEASE -s $FQDN -e $EMAIL $INSTALL_OPTIONS
+
+sudo bbb-conf --salt bbbci
+echo "NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/bbb-dev/bbb-dev-ca.crt" | sudo tee -a /usr/share/meteor/bundle/bbb-html5-with-roles.conf
+
+# bbb-conf --salt doesn't set the shared secret on the web demo
+if [ -r /var/lib/tomcat9/webapps/demo/bbb_api_conf.jsp ]; then
+   sudo sed -i '/salt/s/"[^"]*"/"bbbci"/'  /var/lib/tomcat9/webapps/demo/bbb_api_conf.jsp
+fi
+
+# if nginx didn't start because of a hash bucket size issue,
+# certbot didn't work properly and we need to re-run the entire install script
+if systemctl -q is-failed nginx; then
+    sudo sed -i '/server_names_hash_bucket_size/s/^\(\s*\)# /\1/' /etc/nginx/nginx.conf
+    sudo /bbb-install.sh -v $RELEASE -s $FQDN -e $EMAIL $INSTALL_OPTIONS
+fi
+
+# We can't restart if nginx isn't running.  It'll just complain "nginx.service is not active, cannot reload"
+# sudo bbb-conf --restart
+sudo bbb-conf --stop
+sudo bbb-conf --start