VPS Docker Configuration with Load Balancing SSH Tunnels

VPS configuration with load balancing SSH tunnels and Docker

The Problem

Recently, I've had the itch, or even the need, to run various projects on a remote server. That is, more than I typically run. I quickly realized that it was a little difficult to keep things organized and secure. So I needed a standardized architecture for how all my services are configured and talk to each other.

I had a few goals for this system:

  • to be able to deploy my services quickly on any linux machine
  • for communication between services to be secure
  • for communication to be easy even when services are on different machines (to enable load balancing)
  • a strong security policy all around

The Solution

I came up with a system that I will describe here before explaining my choices.

  • Services will use docker containers.
  • Most internal, service-to-service communication will occur on localhost.
  • Remote interservice communication will occur through SSH tunnels.
  • Except for SSH, all public facing services (ie websites) will go through an nginx proxy.
  • SSH will use the typical hardening techniques and two factor.

The Why

Now that you see what I came up with, allow me to elaborate and expand on my choices.

Why Docker

Docker has exploded in popularity over the last few years. It's easy to see why. Having services in easy-to-deploy images fulfills the requirement of fast, easy deployment. Containers also contribute to security; minimizing impact if one service becomes compromised, without the overhead of a VM. VM overhead would be a concern in a typical VPS with only 2GB of RAM to spare. So I decided it was time to learn how to use docker, as I have no previous experience.

Localhost? Why not network bridges with Docker?

Simplicity. Realistically, I don't need to isolate groups of containers in their own little networks. This is a tradeoff in security which lessens my point about docker security but that I am comfortable with for its benefit: this means it's easier to use SSH tunnels. If I standardize ports for each type of service, I need to do minimal configuration; I just point the service to localhost:whatever_port_the_required_service_should_be_on. Whether or not there is an SSH tunnel there or an exposed port on a docker container, the service doesn't know and doesn't care. All I need is a daemon to run the necessary tunnels at startup, which you will see later is trivial.

Why SSH tunnels

Simplicity, again. I don't have to rely on maintaining each service to encrypt its own datastream to another remote service. All that needs to be done is a few lines in a file somewhere and suddenly there is a secure connection that can be trusted more than an exposed port. This is because the connections can be configured to use SSH keys, so not just any machine can connect to my services. Don't get me wrong, I will still have some username-password authentication on things \like databases. However, the added layer of authentication is nice because it is divorced from the services themselves (which again, could of course become compromised), and it comes with encrypted communication.

Why Nginx

I chose nginx because I am already familiar with it. Having the proxy handle all external, untrusted connections is of course a single point of failure (like my other solutions, you will notice), but nginx has proven itself fairly secure over the years.

Having a proxy is necessary for a few reasons:

  • Nginx opens up easy load-balancing options.
  • Configuration of external services (especially TLS policy) is easy because it takes place in one location.
  • Multiplexing of websites (is that the right word?) is possible; multiple websites can be hosted on the same machine and point to different containers.

SSH configuration

You may notice, that the VPS will be pretty secure as long as NO ONE can EVER get a remote shell on our machine. Not even the SSH tunneling services. Even if someone can get a shell, their damage could be pretty limited if they cannot escalate privileges. However, SSH is still a very desirable target and we want to be sure of its security. Since I will be the only one accessing the machine, this shouldn't be very difficult.

With that in mind, here is how SSH will be setup:

  • move the default port from 22
  • Honeypot on port 22. This might not fool most scripts these days, but it might trick any unsophisticated script into not bothering to do a port scan. I have future plans for a real juicy honeypot that has the potential to actually waste someone's time, but for now we will use endlessh.
  • use the whitelisting feature of SSH to only allow certain users to login
  • disable the root account outright
  • disable password authentication
  • use a yubikey for a strong two-factor experience
  • the services account that will be used for the SSH tunnels must have no shell access
  • fail2ban

The How

Now we get into implementation. To organize my thoughts, I decided to create a script as if I was about to set up a new server with this structure in mind. I will not provide the script in full, because it does have a couple problems. The script was just for a proof-of-concept. I don't want people copy-pasting it and trying it on their own servers and breaking things. I also want to encourage readers to think for themselves and not just blindly copy-paste.

You will notice a function chngParam, which is basically an alias for a sed command that makes it easy to change standard-looking parameters is configuration files. There are also some echo functions that print things in colors to make things easier to read while the script is running (Advice: don't actually create a script for this, run the commands line by line yourself.)

chngParam() {    # #Parameter, possible states, desired state, file
    sed -i "/^(?:#(\s+)?|)${1}\s*(?:${2})/c ${1} ${3}" $4
}

echop() {
    BLUE='\033[0;35m'
    NC='\033[0m'
    echo -e "${PURPLE}$@${NC}"
}

echog() {
    GREEN='\033[0;32m'
    NC='\033[0m'
    echo -e "${GREEN}$@${NC}"
}

First, I took user input. Going forward, if you see a bash variable you don't recognize, it was probably from this part.

Now create a new sudo user.

useradd -m -s /bin/bash -G sudo $adminUser # <-here's that input I was talking about

At this point in the script the user copies there SSH keys to the new $adminUser. I will describe this more later.

Install packages

Here's most of the packages to install.

apt-get update
apt-get install -y ufw fail2ban libc6-dev \
    nginx net-tools python3-pip certbot
pip3 install bpytop --upgrade

Securing the server

Disable the root account

passwd -l root

Set basic firewall rules with ufw. If you have not seen ufw, here is a great article and here is a Hackersploit video about ufw.

ufw disable
echo "y" | ufw reset
chngParam "IPV6=" "yes|no" "yes" /etc/default/ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow $sshPort

Harden SSH daemon

ssh_config=/etc/ssh/sshd_config

echog Change SSH port
chngParam Port 22 $sshPort $ssh_config
ufw allow $sshPort

echog Turn off SSH password authentication
chngParam PasswordAuthentication "yes|no" no $ssh_config

echog Disallow root SSH login
chngParam PermitRootLogin "yes|no" no $ssh_config

echog Whitelisting users
echo AllowUsers $adminUser $servicesUser >> $ssh_config # servicesUser will be created later

Fail2ban should be configured on whatever port you set your SSH to. If you are not familiar with fail2ban, here is a basic how-to. Fail2ban is a system that will ban ip addresses who fail a login process a certain number of times.

I'm skeptical if these lines actually work, so I won't show them. To do it manually, create a new jail file with cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local and edit the [sshd] jail near the bottom by adding the line enabled=true and changing the port= from ssh or 22 to your ssh port.

Now, if someone tries to brute force enter the server, they will just be blocked by firewall rules.

Next up is the SSH honeypot with endlessh. This is why the libc6-dev package was installed earlier, so this code can be compiled.

echog SSH Honeypot
git clone https://github.com/skeeto/endlessh
cd endlessh
make
mv endlessh /usr/local/bin/
sudo cp util/endlessh.service /etc/systemd/system/
systemctl enable endlessh
mkdir -p /etc/endlessh
echo Port 22 >> /etc/endlessh/config
ufw allow 22/tcp
cd $centralDir
rm -rf endlessh

SSH Tunnels

Setting up the services account is the next order of business. There are a few considerations for this part. We want to lock the services user password to ensure they can only login with their SSH key. Additionally, we set the user shell to nologin so that our services can only create tunnels and cannot have shell or filesystem access, further improving security in the event one of our machines gets pwnd. I also make the assumption that every machine in this network has a default config_user key in the services user authorized_keys file. This means we keep this key safe and copy it to the machine for the purposes of running this script and adding it to our own authorized_keys, then we delete it from the machine. Now any other new machine we set up can take that config_user key so they can copy their own key to this machine. That way we can easily revoke certain machines' access to the SSH tunnels without having to generate a new key and propagating it across all our machines.

echop Setting up service account
services_user_home=/home/$services_user
config_user_key=config_user
services_user_key=$services_user_home/.ssh/id_rsa

useradd -m -s /usr/sbin/nologin -d $services_user_home $services_user # do not configure password as we just want an SSH tunnel endpoint
passwd -l $services_user

mkdir -p $services_user_home/.ssh
touch $services_user_home/.ssh/authorized_keys
cat $config_user_key.pub >> $services_user_home/.ssh/authorized_keys
ssh-keygen -b 4096 -t rsa -f $services_user_key -q -N ""

Take a look at this gist to see how to setup an SSH tunneling service that starts on boot. Provide the necessary template (formatted like secure-tunnel@.service) in the same directory and all your tunnels (formatted like secure-tunnel@host) in a tunnels folder. This part will copy our newly generated key to all the hosts using the config_user key, as well as copy all our services and enable them with a for loop. You should probably use hosts instead of ip addresses. So copy your hosts file to /etc/hosts

echop Creating SSH tunnel services

cp secure-tunnel@.service /etc/systemd/system/
cd tunnels
cp * /etc/default/

echog Copy keys and enable services
for f in *.service; do
service_target=$(grep -oP "(?<=target=)[a-z]+" f)
echo "yes" | ssh-copy-id -i $services_user_key -o "IdentityFile ${config_user_key}" -p 69 $services_user@$service_target
systemctl enable secure-tunnel@$service_target.service
done

Docker

The next section involves installing docker and setting up our containers

Take a look at docker docs to see how to install docker (I'm using debian.)

For the next part we have a bunch of files for each container that look like <service>_build_and_run.sh that take as input the services user home directory and the desired port to expose. We retrieve the services we want to install by specifying a line-delimited list in the file docker_submodules.

Then we have a services_ports_file which can be any file name you choose. This format of this file is each service gets it's own line and is associated with a port. An example would be postgresql 6903 or kestrel_server 666.

If the service name matches what's seen in the docker_submodules file, the port will be passed to the build and run script. The build and run script can do with that information what it wants. For me, some of them clone a git repository or pull from docker, build the image and then run it (with restart=always so the service starts on boot).

Here's an example suite of files like I have just described

# docker_submodules (don't actually include this line if you use my code)
kestrel_server
postgresql
abc123
mysql

# services_ports_file
postgresql 6903
kestrel_server 666
mysql 2555
minecraft_server 25565
abc123 9999


#!/bin/bash
# postgresql_build_and_run.sh
docker volume create pgdata
docker pull postgres
docker run -d \
    --name postgresql \
    -p $2:5432 \
    -e POSTGRES_PASSWORD=hahayouthought \
    -v pgdata:/var/lib/postgresql/data \
    --restart=always \
    postgres:latest

You can see that docker_submodules doesn't have all the services that are listed in services_ports_file. The ports file is just there to pass ports to the scripts that will be run. The reason I didn't just put all the ports in docker_submodules is because I have future plans for services_ports_file that will make maintenance and deployment of new services even easier.

With that being said, here is the actual code that does what I have described:

echog Download and install requested docker submodules
while read p; do
    echog p
    service_port=$(grep -E '(?<=$p\s)\d+' $services_ports_file)
    ./$p_build_and_run.sh $services_user_home $service_port
done <docker_submodules

Nginx

We're at the home stretch. The next step is to configure nginx. Since my method is not unique and I really don't want to be responsible for your nginx configuration, I will not provide this code. I will give an overview of how I set this up manually, though. I create an /etc/nginx/conf.d/ssl.conf file with all my SSL configuration which you can find good pointers about here. For each website, I create a new sites-available/<domain name> file and fill out the server {} object as necessary. Then I create symbolic links for each website I want to enable in the sites-enabled directory as seen here. I test the configuration with nginx -t and then reload with nginx -s reload. If you don't yet know how nginx works check out a youtube video and take a look at the nginx wiki.

The final step before we cleanup is to configure certbot for nginx with sudo certbot --nginx -d example.com -d www.example.com where you specify a new -d for each domain/subdomain. If you didn't know, certbot is a Let's Encrypt tool that makes it simple to get HTTPS certificates for your websites. Without a proper signed certificate, you would not be able to access your site without going through a wall of menus warning you away in your browser.

Cleanup and restart

Our cleanup is pretty simple:

echop Cleaning up

rm $config_user_key

echog Enable firewall
echo "y" | ufw enable

read -p "Ready for reboot..."
reboot

SSH into your VPS with your admin user and run an apt upgrade for good measure.

Even Better Security

I am here to suggest another solution to logging in that just an SSH key, even if password protected. I'm going to be making a long-term home on this server, so I want to be very confident in its security. Good security is something you know, something you have or something you are. And combining these three factors makes for the best security. Something you are is a little difficult, and I have an idea for a project regarding this. For now we will use "something I know" and "something I have."

I could password protect an SSH key, but I'm not confident my machine will never get pwnd. I am planning a switch to Linux but for now I am on a windows installation with a lot of random programs installed. And also, my computer is open-air and not something I couldn't see someone getting a hold of my hard drive or plugging in something malicious. I do have a yubikey, which could provide our two factors: something I know (the passphrase for my yubikey) and something I have (the yubikey itself).

There are two options available. There is a FIDO2 implementation for yubikey, and there is also an OpenPGP keystore. The FIDO2 option is tempting, but there are a few issues with it. Windows itself doesn't have the edcsa-sk or ed25519-sk options in OpenSSH yet, so I would have to always use WSL and hack in a passthrough for my yubikey. Plus, if I loose my yubikey, which seems very easy to do given its size, I will have no way to get into my server. Though I suppose I could encrypt a backup key to a flash drive and hide it somewhere.

The best option seems to me to use an OpenPGP auth key stored on my yubikey. It's something that I back up already anyway onto a flash drive. It was very difficult and frustrating to set this up on windows. There are guides that make setting up the gpg agent deceptively simple, but be warned that any well-used windows installation with a bunch of random unix-compatibility crap installed in random places makes it a huge time sink. A lot of the blogs I took a look at were also pretty disappointed with how unsupported things seemed to be.

Conclusion

I haven't had a chance to test SSH tunnels a lot, but so far my containers are holding up and communicating with one another on localhost. All public-facing services except SSH are exposed by nginx. I am confident in the security of my SSH setup, especially compared to a typical setup.

I enjoy writing about my projects. I enjoy coming across other personal programming or sysadmin blogs. I have found some high quality content and learned a lot from these blogs. I hope for this website to become that same source of information and cool projects that I admire, and to improve my writing skills. For this reason I am committing to releasing one post per week. Come back next week for a post about minecraft! See you then.