Deploy Phoenix app in bare metal

Disclaimer

Almost all of what I wrote below are taken from previous blogposts with the only exception: this is not an umbrella app, but a single Phoenix app.
Again, not to be used in production (staging environment). For learning purposes only.
Wrote this for my personal documentation.
CI/CD via Github Actions.

Free credits on VMs

Init and user runtime scripts

In this post, I used Vultr.

Github workflow files

Make sure the workflow script (ci.yml, cd.yml) is already setup in the repo. If you want to see an example of it, check my previous post (located right before the end of post).

Choosing server

Before deploying, make sure to add the ssh key and "limit access user".

Managed DB (`Postgres`)

Before importing the sql, make sure the username of the exported sql file matches vultr (e.g. vultradmin)

Note: copy the database url, as you'll need this in the post-install script later.

Format for database url that will be used for our Phoenix app: ecto://username:password@domain.co:port/database_name

psql postgres://username:password@domain.co:port/defaultdb

CREATE DATABASE database_name;
GRANT ALL PRIVILEGES ON DATABASE database_name TO username;

psql -U username -h staging.co -p 5432 database_name < path/to/postgre.sql

Create init script

#!/usr/bin/env bash
# NOTE: this init script is tested using Debian 11
ssh_tarpit=52698
sudo apt update -y

# allocate swap
fallocate -l 1G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | tee -a /etc/fstab
sysctl vm.swappiness=10
echo vm.swappiness=10 >> /etc/sysctl.conf
echo vm.vfs_cache_pressure=50 >> /etc/sysctl.conf

# limit journalctl
sed -i -e 's/#Compress=yes/Compress=yes/g' /etc/systemd/journald.conf
sed -i -e 's/#SystemMaxUse=/SystemMaxUse=100M/g' /etc/systemd/journald.conf
sed -i -e 's/#SystemMaxFileSize=/SystemMaxFileSize=50M/g' /etc/systemd/journald.conf
systemctl restart systemd-journald

# backup sshd config
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak

# enable port 80 and 443
ufw allow https
ufw allow http
ufw allow $ssh_tarpit/tcp

# install authbind (using ports below 1024 for non-root users)
# install packages
apt install authbind snapd fail2ban build-essential libc6-dev git -y

# fail2ban
cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sed -i "s/#Port 22/Port $ssh_tarpit/g" /etc/ssh/sshd_config

systemctl restart sshd
ufw --force enable

# use endlessh tarpit
git clone https://github.com/do-community/endlessh
cd endlessh
make

# persist endlessh tarpit after session ends
mv ./endlessh /usr/local/bin/
cp util/endlessh.service /etc/systemd/system/
sed -i "s/#AmbientCapabilities=CAP_NET_BIND_SERVICE/AmbientCapabilities=CAP_NET_BIND_SERVICE/g" /etc/systemd/system/endlessh.service
sed -i "s/PrivateUsers=true/#PrivateUsers=true/g" /etc/systemd/system/endlessh.service
sed -i "s/InaccessiblePaths=\/run \/var/#InaccessiblePaths=\/run \/var/g" /etc/systemd/system/endlessh.service
setcap 'cap_net_bind_service=+ep' /usr/local/bin/endlessh
mkdir /etc/endlessh
echo "Port 22" >> /etc/endlessh/config
sudo systemctl daemon-reload
systemctl --now enable endlessh

# to check endlessh tarpit status
# journalctl -u endlessh.service -b

# install certbot
snap install core
snap install --classic certbot
ln -s /snap/bin/certbot /usr/bin/certbot

Waiting game

Assign the vm instance's ip to A record then wait for dns propagation.

Certbot

sudo certbot certonly --noninteractive --standalone --domain staging.co --register-unsafely-without-email --agree-tos

Create post-install script

Name the script ~/postinstall.sh:

#!/usr/bin/env bash
domain="staging.co"
app_name="staging_app"
certbot_email="support@staging.co"
database_url=INSERT_DATABASE_URL
secret_key_base="paste here using mix phx.gen.secret"

ssh-keygen -t ed25519 -C $certbot_email -q -f "$HOME/.ssh/id_ed25519" -N ""
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys

# restricted usage of ports below 1024 for non-root users
# so we're using authbind for a specific user running the app
sudo touch /etc/authbind/byport/80
sudo touch /etc/authbind/byport/443
sudo chmod 500 /etc/authbind/byport/80
sudo chmod 500 /etc/authbind/byport/443
sudo chown $USER /etc/authbind/byport/80
sudo chown $USER /etc/authbind/byport/443

# create environment file
sudo bash -c "cat >> $HOME/$app_name.env" << EOL
HOME="$HOME/$app_name"
RELX_REPLACE_OS_VARS=true
SECRET_KEY_BASE=$secret_key_base
SSL_CERT_PATH="$HOME/ssl_cert/$domain/cert.pem"
SSL_KEY_PATH="$HOME/ssl_cert/$domain/privkey.pem"
SSL_CACERT_PATH="$HOME/ssl_cert/$domain/chain.pem"
DATABASE_URL=$database_url
PHX_SERVER=true
MIX_ENV=prod
PHX_HOST=$domain
PORT=80
ELIXIR_ERL_OPTIONS="+fnu"
EOL

sudo chmod 644 "$HOME/$app_name.env"
sudo chown $USER:$USER "$HOME/$app_name.env"

# create systemd service for the app
sudo bash -c "cat >> /etc/systemd/system/$app_name.service" << EOL
[Unit]
Description=App for $app_name
After=network.target
Requires=network.target

[Service]
RemainAfterExit=yes
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=$app_name
WorkingDirectory=$HOME/$app_name
User=$USER
Group=$USER
Restart=always
RestartSec=5
EnvironmentFile=$HOME/$app_name.env
ExecStart=authbind --deep $HOME/$app_name/bin/start
ExecStop=authbind --deep $HOME/$app_name/bin/$app_name stop

[Install]
WantedBy=multi-user.target
EOL
sudo chmod 644 "/etc/systemd/system/$app_name.service"
sudo systemctl daemon-reload
sudo systemctl enable $app_name
sudo systemctl stop $app_name

# create renewal hooks (deploy)
sudo bash -c "cat >> /etc/letsencrypt/renewal-hooks/deploy/$domain.sh" << EOL
#!/bin/bash
domain="$domain"
archive_cert_dir="/etc/letsencrypt/archive/\$domain"
user_cert_dir="\$HOME/ssl_cert/\$domain"

mkdir -p \$archive_cert_dir
mkdir -p \$user_cert_dir

for file in \$(sudo ls -t \$archive_cert_dir | head -n 4); do
  new_file=\$(echo \$file | sed 's/[0-9]*//g')
  sudo cp "\$archive_cert_dir/$file" "\$user_cert_dir/\$new_file"
  unset \$file
  unset \$new_file
done

sudo chown -R \$USER:\$USER "\$HOME/ssl_cert"
sudo chmod -R 755 "\$HOME/ssl_cert"
EOL
sudo chmod 750 "/etc/letsencrypt/renewal-hooks/deploy/$domain.sh"

# create renewal pre-hooks
sudo bash -c "cat >> /etc/letsencrypt/renewal-hooks/pre/$domain.sh" << EOL
#!/bin/bash
webapp="$app_name"
sudo systemctl stop \$webapp
EOL
sudo chmod 750 "/etc/letsencrypt/renewal-hooks/pre/$domain.sh"

# create renewal post-hooks
sudo bash -c "cat >> /etc/letsencrypt/renewal-hooks/post/$domain.sh" << EOL
#!/bin/bash
webapp="$app_name"
sudo systemctl start \$webapp
EOL
sudo chmod 750 "/etc/letsencrypt/renewal-hooks/post/$domain.sh"

run the script:

chmod +x ~/postinstall.sh
./postinstall.sh

Once the commands ran successfully, remove the script:

rm ~/postinstall.sh

Then:

domain="staging.co"
archive_cert_dir="/etc/letsencrypt/archive/$domain"
user_cert_dir="$HOME/ssl_cert/$domain"
mkdir -p $user_cert_dir

for file in $(sudo ls -t $archive_cert_dir | head -n 4); do
  new_file=$(echo $file | sed 's/[0-9]*//g')
  sudo cp "$archive_cert_dir/$file" "$user_cert_dir/$new_file"
  unset $file
  unset $new_file
done

sudo chown -R $USER:$USER "$HOME/ssl_cert"
sudo chmod -R 755 "$HOME/ssl_cert"

Github-specific CI/CD workflow

Add the following secrets:

source: https://github.com/<ORG_OR_USERNAME>/<REPO>/settings/secrets/actions

name: STAGING_SSHKEY value: PASTE THE RESULT from cat ~/.ssh/id_ed25519 (linuxuser's ssh key)

name: CODECOV_TOKEN value: api token taken from https://app.codecov.io/account/gh/ORG_OR_USERNAME/access


Commit and push changes to repo, then check logs.

## Misc

### Deleting certbot domain certificate:

```bash
certbot delete --cert-name domain.co

Running `mix ecto.create` in `iex`

Running mix ecto.create inside shell (source):

alias StagingApp.Repo
Repo.__adapter__().storage_up(Repo.config())