Docs Install

The preferred way to run Owlkins is in an Ubuntu virtual machine that has at least 2 cores and 4GB of RAM. You won't utilize these resources all the time so the resources can be shared. Running the web server and framework will take anywhere from 350MB to 600MB depending on how many photos you have.

When a user uploads a video, Owlkins will convert it via an ffmpeg command which will utilize max(N-1, 1) available processors. During video conversion, RAM usage will depend on the properties of the video such as its length. If your system has a lower amount of RAM, make sure you have enough swap space to facilitate converting videos, and a bit of patience as well for the website during times of heavy swapping!

Download, install and configure git on your machine.

Navigate to the location on your machine that you want a new folder to be made with the contents of the repo. We recommend creating a new user whose sole purpose is to run owlkins, and then for example you can install here in that user's home directory.

Once you are ready, you can clone the repo:

git clone https://gitlab.com/owlkins/owlkins

You will now have a directory owlkins on your local machine which you can change directory (cd) into.

Even if the only thing you will run on this machine is Owlkins (for example if you rented a VM in the cloud or installed a new one on your machine), you'll want to make sure Owlkins runs in a virtual environment.

A Python virtual environment allows for the specific package version to be installed without risking conflicts with the python that comes with your operating system.

sudo apt-get install nginx postgresql postgresql-contrib libpq-dev libjpeg-dev libffi-dev ffmpeg zlib1g-dev libmemcached-dev libimage-exiftool-perl gcc python3-dev build-essential memcached libmemcached-tools python3-venv

Next, create the virtual environment

python3 -m venv .venv

Now you can enter the virtual environment

source .venv/bin/activate

Owlkins recommends using PostgreSQL, though Django has other alternatives, which I imagine would work "ok" if you are willing to put up with a non- or less-tested configuration.

Enter the PostgreSQL console by switching to the postgres user and executing the command psql:

sudo -u postgres psql

Now you can create the database and the Django user. Come up with a user and password, substitute in the below steps (without the curly braces), and execute all of these lines in the psql console.

You will have to save the user and passwordin the secrets.py file noted in step 4 below.

CREATE DATABASE {user};
CREATE USER {user} WITH PASSWORD '{password}';
ALTER ROLE {user} SET client_encoding TO 'utf8';
ALTER ROLE {user} SET default_transaction_isolation TO 'read committed';
ALTER ROLE {user} SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE {user} TO {user};

To exit from the PostgreSQL prompt and return to your terminal, type \q and press enter.

You should now be in the virtual environment named ".venv", check this by observing your current terminal line has (.venv) at the start on the far left, perhaps looking like:

(.venv) you@your_machine:~/owlkins

Now you can use the python package installer "pip" to install the current version of all requirements as identified in the requirements.txt file you pulled from the git repo.

pip install -r requirements.txt

There are two Owlkins text files you will have to edit to complete setup. To create the default version of the files copy the config_defaults.py and secrets_defaults.py to config.py and secrets.py, respectively:

cp owlkins/config_defaults.py owlkins/config.py

cp owlkins/secrets_defaults.py owlkins/secrets.py

This will create two files: owlkins/secrets.py and owlkins/config.py.

• The secrets file should be used to store variables like API keys and the secret key for Django.
• The config file will need to be customized to include things like the name of your child that you want to display to users, and your own email that users should reply to on the daily digests.

Next we will fill in a few variables in the secrets.py file. Generate a new Django secret key with a command like the below and store the result in the secrets.py file.

python -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())'

Now, we have some things which we can add to our secrets.py file:

• Save this secret key as the value of the SECRET_KEY variable in your secrets.py file.
• At the same time, save the database user and password as the DB_USER and DB_PASSWORD variables in the secrets.py file.
• If you followed the PostgreSQL install steps exactly, then your database will also be called the same as your user, and you can store this value in the DB_NAME variable in secrets.py
• Finally, you will have to customize the DB_SERVER variable in secrets.py to point to your PostgreSQL server. If you are running PostgreSQL on the same machine, you can leave this setting blank.

Now you are ready to make the initial database migrations via the following command:

python manage.py migrate

If you see any issues or errors, check that the secrets.py file contains the correct values.

Create "you" as a superuser

Now that you created all the tables in the database that will allow Owlkins to store relational data, such as information about photos and a link to their url, user information, and everything else other than files such as the actual photos and videos themselves.

Your user table at this point exists but doesn't have anyone in it. To log into Owlkins and be able to access the prompt to create other users, you'll have to create a user that has administrative privileges. You can do this via another Django management command:

python manage.py createsuperuser

You should set your email as the "username" and the Email Address field, since with Owlkins the frontend login flow is email and password based and subsequent normal users won't be able to make a "username" that is different from their email address. The workflow should look something like the following

(.venv) user@your-machine:~/owlkins$ python manage.py createsuperuser
Username (leave blank to use 'user'): YOUR_EMAIL
Email address: YOUR_EMAIL
Password:
Password (again):
Superuser created successfully.

This is the email and password you will then use to log in to your Owlkins installation once up and running. If you forget your password, you can change it with another management command:

manage.py changepassword *email* 

The preferred way to reset your own password and that of any other user though is the Forgot Password? workflow on the website itself, though this will require email to be set up first (see step 8. below).

config.py file changes

You're required to put BASE_URL to get the server working. If you have your domain now you can add it, else this will also be discussed in section 6.

Owlkins recommends using Memcached for the caching layer, but there are other Django caching solutions if you so choose. If you followed the instructions thus far, you will have called sudo apt-get install with libmemcached-dev memcached libmemcached-tools, meaning now you have Memcached installed and configured on your system. You should configure it further such that memcached listens on a socket instead of via TCP on a local port. First, open your memcache config file

sudo nano /etc/memcached.conf

Comment these lines

#-p 11211

#-l 127.0.0.1

add these lines

# Enablinb SASL
# -S

# Set unix socket which we put in the folder /var/run/memcached and made memcache user the owner
-s /var/run/memcached/memcached.sock

# set permissions for the memcached socket so memcache user and www-data group can execute
-a 0666

# The default maximum object size is 1MB. In memcached 1.4.2 and later, you can change the maximum si>
-I 10m

To allow these settings to take effect you can restart memcached

sudo systemctl restart memcached

Now your Django installation can use memcached.

Owlkins is meant to be installed on a machine you own or have paid to operate, which hosts a website that users can visit. Part of this then is the domain name that you can share with your users which they can type into their web browsers or get links to from your email digest.

If you are not familiar with buying a domain, there are several different providers such as GoDaddy, Namecheap, or others. Personally, as someone with a Gmail account, I've always found Google Domains the easiest and most straightforward to use.

If you choose Google Domains, the process looks like this:

1. Navigate to the Get a new domain section on the left navigation bar (on desktop) or navbar at the top (on mobile>
2. Search for your perfect domain name! Hopefully you lucky and are able to find something that is straightfoward like your-baby-name.com, but there are endless combinations and other Top Level Domains (TLDs) such that you don't have to use a .com address. Just note that some TLDs cost more per year (like .io) than others.
3. After you find the domain you love, follow through the checkout process to purchase the domain.
4. Once purchased, you can now see the domain listed in the My domains section.
5. Click your domain from the list to go into the settings for this domain.
6. The menu bar will now change to be specific to your domain, and an option DNS will appear.
7. You will have to use the DNS section to configure your mail settings below as well as pointing your domain to your server.
8. If you know your server's IP address, you can add it now on the DNS dashboard for your domain, where you can see a button Manage custom records.
   1. Add a new row by clicking Create new record if needed.
   2. Leave the first input field Host Name blank if you want the site to be run at the root of the domain.
   3. The second input Type should be A for an IPv4 address (likely) or AAAA if your server is at an IPv6 address (less likely).
   4. Click Save
9. Take the domain name and store it in the BASE_URL variable in your config.py file.

If you are behind a residential connection, your IP address may change at a rate that it will be annoying for you to come back to this form and manually update it after you notice your site goes "offline". Nowadays, it may seem like your IP address changes that much, but the "right" answer in this scenario is to setup DDNS instead. One reason I like to use Google Domains is a popular DDNS client DDCLIENT supports Google Domains.

1. To access DDNS on Google Domains, scroll to the very bottom of the DNS page and click Show advance settings to reveal the Dynamic DNS section
2. Click the Manage dynamic DNS button.
3. If needed, make a new row with Create new record and then similar to above leave Hostname blank for the root of the domain or fill in a subdomain, Type will be pre-filled as A for you, and Data will initially be essentially blank with a value such as 0.0.0.0
4. Click Save
5. Now reveal your dynamic domains with Your domain has Dynamic DNS set up button
6. Find the entry you just added and click View Credentials
7. Copy these credentials into your DDNS client, such as DDClient, which you can install on the same machine you are running Owlkins on.

Now that we have Owlkins set up and a domain, we can set up the web server.

Setup the gunicorn service file

sudo nano /etc/systemd/system/gunicorn.service

... with the following contents

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User={YOUR_UBUNTU_USER}
Group=www-data
WorkingDirectory=/home/{YOUR_UBUNTU_USER}/owlkins
ExecStart=/home/{YOUR_UBUNTU_USER}/owlkins/.venv/bin/gunicorn \
          --access-logfile - \
          --workers 3 \
          --bind unix:/run/gunicorn.sock \
          --timeout 240 \
          owlkins.wsgi:application

[Install]
WantedBy=multi-user.target

Now create the socket file

sudo nano /etc/systemd/system/gunicorn.socket

...with the following content

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

Now, you can enable Gunicorn

sudo systemctl start gunicorn.service
sudo systemctl enable gunicorn.service
sudo systemctl status gunicorn.service

You should see that gunicorn is Running. Press q to exit out of the status of gunicorn.

Next we can set up Nginx. First, create a site "owlkins"

sudo nano /etc/nginx/sites-available/owlkins

... and populate it with information for your domain that you want to host the site at. If you don't already have experience with Nginx or a working setup for your environment, you could check out the NGINXConfig tool hosted from DigitalOcean. In addition or in place, you can base your installation off the below example, your config may look like the following (replace YOUR_DOMAIN_HERE with your fully qualified domain name):

server {
    listen [::]:443 ssl http2;
    listen 443 ssl http2;

    server_name YOUR_DOMAIN_HERE;

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;

    # Detect bad actors
    if ($http_host != $server_name){
        return 444;
    }
    if ($host != $server_name){
        return 444;
    }
    fastcgi_buffers 8 4K;
    fastcgi_ignore_headers X-Accel-Buffering;

    # For certbot acquisition of LetsEncrypt HTTPS certificate
    location ~ /.well-known {
        allow all;
    }
    location = /favicon.ico {
        return 301 https://static.owlkins.com/favicon.ico;
    }


    # IF YOU HAVE TO SERVE STATIC CONTENT FROM THIS MACHINE:
    # Remove the below "#static#" comments (effectively uncommenting the line) and
    # then you will be able to serve static media (images / videos) from this machine.
    # Note that:
    #    - The preferred way to serve media (images / videos) is S3 -> Cloudfront on AWS
    #    - You should not adjust Django's settings to force Django to serve static content
    # See the Architecture page for more info.
    #media# location /media/ {
    #media#    alias /YOUR/OWLKINS_INSTALLATION/PATH/media/;
    #media#}


    location / {
        # NOTES: On NOTE 1 and NOTE 2 below, see the architecture section for more info

        # NOTE 1: The below two lines should be used if you are running a public internet facing machine
        # that will receive HTTPS requests and pass it to Gunicorn.  If this is you, uncomment the two lines
        # below (delete "#" and "1" and "#", all three)
        #
        # Example would be, renting a VM from AWS or DigitalOcean.  If you are running on your home residential
        # connection behind your router, you will have to go to "NOTE 2" below and comment out these two lines.

        #1# include proxy_params;
        #1# proxy_pass http://unix:/run/gunicorn.sock;

        # NOTE 2: If you need to set up a reverse proxy to handle all incoming traffic,
        #       say, on a residential connection all HTTP and HTTPS traffic that hits
        #       your router sent to a central server running Nginx that then passes the request
        #       on to another server that actually is running your Owlkins software,
        #       comment the above two lines and uncomment the following "#2#" lines (delete "#" and "2" and "#", all three)
        #       REPLACE -> "XXX" with your subnet and "XX" with the IP of the machine that
        #                        will run Nginx/Gunicorn/Django

        #2# proxy_pass_header Authorization;
        #2# proxy_pass http://192.168.XXX.XX$request_uri;
        #2# proxy_set_header Connection "";
        #2# proxy_buffering off;
        #2# proxy_read_timeout 36000s;

    }

    # NOTE 3: AFTER YOU RUN `certbot` THERE SHOULD BE LINES FOR THESE THREE PARAMETERS HERE
    # You don't have to edit these comments, as certbot will modify this file itself.
    # ssl_dhparam
    # ssl_certificate
    # ssl_certificate_key
    #
    # The two "snakeoil" below will allow you to allow nginx to start listening for your domain without fully
    # setting up HTTPS, which is important for you to run `certbot`

    ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
    ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
}

server {
    listen 80;
    listen [::]:80;

    server_name YOUR_DOMAIN_HERE;

    root /var/www/html;
    index index.nginx-debian.html;

    location ~ /.well-known {
        allow all;
    }

    if ($host = server_name) {
        return 301 https://$host$request_uri;
    }

}

Next you can enable the site by linking the file you just created to the "sites-enabled" directory. Note that you need to do the absolute path here, don't do a relative path based on your current location.

sudo ln -s /etc/nginx/sites-available/owlkins /etc/nginx/sites-enabled/

Before moving forward, it is worthwhile to check your system wide nginx.conf file to potentially change the max upload size. This should be one of the first areas to look if a large file fails to upload. Invoke the following command to enter the nginx.conf file:

sudo nano /etc/nginx/nginx.conf

client_max_body_size 5000M;

Next you can test your configuration to be sure that all is OK. If not, look over the last couple steps and see what needs to be fixed.

sudo nginx -t

Now you are ready to restart nginx before applying for a, SSL certificate

sudo systemctl restart nginx

Now apply for an SSL certificate from Lets Encrypt!

sudo certbot --nginx -d {YOUR_DOMAIN}

Follow the directions to setup the certificate, and optionally allow redirection from HTTP to HTTPS. If you use your default setup then you'll already have redirection. Now, test again and restart nginx and you'll get your first glimpse of Owlkins running on your domain!

sudo nginx -t
sudo systemctl restart nginx

Email is a lot more complex than people think, and can be quite a rabbit hole to go down. Very likely you need an SMTP relay to send messages if you want the emails you send to have a higher chance of skipping a user's spam filter. You absolutely need an SMTP relay if you decide to self-host on a machine at your house behind a residential IP address as any modern spam list should and will include a blanket block (not necessarily even a spam label) on email originating from residential IP addresses. In terms of inbound mail, most of the major providers like XFINITY will block all traffic on the ports that mail servers send and receive mail on so if you were to go so far as to setting up a mailserver "at home" for inbound mail on your domain, you'd have to setup some iptables MASQUERADE rule on some external server before sending to your IP address on some non-standard port. Fortunately, for the scope of Owlkins, inbound mail is not needed at all so you can focus solely on sending high quality mail.

You can use any SMTP relay such as Mailgun, Sendgrid, or Mailjet, but we will discuss AWS Simple Email Service below as I find it nice that you can consolidate both Email and distribution of photos and videos using AWS's CloudFront Content Distribution Network (CDN) into one bill. The downside of AWS SES is they may deny your use case if you are not already approved to send email on your account, especially if you just made an account (see more below on Step 18).

The goal of any SMTP relay is to populate the following values:

in config.py, populate

• EMAIL_HOST with the smtp relay. For the above example with us-east-2, the value would be email-smtp.us-east-2.amazonaws.com
• EMAIL_DOMAIN with the domain that you add to SES as a "Verified Identity". You have to prove that you have the right to send mail from the domain by adding DNS records that SES will generate for you.

To get started with SES, sign into your AWS account and navigate to the dashboard for the region you wish to send email from. For example, here is the dashboard for us-east-2 Verified Identites page. Here you setup the domain:

1. Click Create Identity
2. Click Domain for the Identity type
3. Type your domain into the text field
4. Leave "Assign a default configuration set" unchecked
5. Check "Use a custom MAIL FROM domain"
6. Fill in the "mail from" domain (a subdomain), typically mail
7. Leave Behavior on MX failure as Use default MAIL FROM domain
8. Under the next section, Verifying your domain, click Advanced DKIM settings to reveal more options
9. Select Easy DKIM as the Identity Type
10. Select RSA_2048_BIT as the DKIM signing key length
11. If you don't manage your domain on AWS, through Route53, then uncheck the box "Enabled" for Publish DNS records to Route53
12. Click Create Identity
13. Now the AWS Dashboard will redirect to a new view showing your new identity. Next you will have to modify your DNS records with several values that were just created for your verified domain.
    1. There are three CNAME values that you will have to add to your domain. If you are using a site like Google Domains for your DNS settings, make sure you appropriately copy the key over, as a direct COPY + PASTE will result in copying the value from AWS's dashboard like 1234._domainkey.yourdomain.com which when pasted directly into Google's DNS UI will result in a setting like 1234._domainkey.yourdomain.com.yourdomain.com which is not correct.
    2. There is one MX value you will have to add for the mail from domain you selected in the earlier step, where if you follow the above with mail and us-east-2, then add a MX record for subdomain mail with value 10 feedback-smtp.us-east-2.amazonses.com
    3. There is one TXT value you will have to set for the mail from domain which is typically something like "v=spf1 include:amazonses.com ~all" (double quotes included).
14. Once your domain is verified, it is now time to get your credentials. Click the Account dashboard button on the left navigation bar. Scroll down to the Simple Mail Transfer Protocol (SMTP) settings section, where there is a button Create SMTP credentials
15. A new window will load now in the IAM dashboard view where you can set a username or accept the default which is usually something like ses-smtp-user.20220130-210943. This username is for you to easily identify the credentials in your IAM settings and does not relate to any external usage.
16. Click the Create button in the bottom right, and reveal the credentials in the next view by clicking Show User SMTP Security Credentials and/or selecting the Download Credentials button in the bottom right. Note: After you navigate away from this page the credentials will no longer be retrievable.
17. These two values should now be saved in your secrets.py file:
    • SMTP Username: should be stored in the EMAIL_HOST_USER variable
    • SMTP Password: should be stored in the EMAIL_HOST_PASSWORD variable
18. Permission: Now you are all set up to send email, except, you may not be authorized to send! Go back to the Acount Dashboard and see if it says Your Amazon SES account is in the sandbox for your chosen region. In this section there will be a button Request production access which will then take you to a new form.
    1. Mail type: Transactional (your Owlkins server will send emails allowing users to set their password and daily email digests announcing new photos and videos)
    2. Website URL: You can either use your Owlkins distribution URL, or if needed you can try to send this link to an explainer on our site about Owlkins email activity.
    3. Use case description: AWS will need to ensure that you are only sending emails that are high quality and won't ruin the reputation of Amazon's servers. Try to write a personal message and note these key points quoted from a denial email:

       tell us how often you send email, how you maintain your recipient lists, and how you manage bounces, complaints, and unsubscribe requests. It is also helpful to provide examples of the email you plan to send so we can ensure that you are sending high-quality content.

19. Check the Acknowledges checkbox and then Click Submit request

Now you are all finished with setting up email! If you find yourself repeatedly rejected by Amazon, try to use one of the other providers, where the steps to authorize a domain via DNS and getting credentials will be pretty similar.

As discussed in the Architecture page, one of the biggest impacts you can have on your user's performance is to serve your photos and videos from a Content Distribution Network (CDN). While completely necessary if hosting Owlkins behind a residential connection, it is still important even if you purchase the biggest, fastest VM with the most I/O that you can find from some provider. Why? You just can't beat a CDN at its own game: getting your content to your users as fast and reliably as possible.

We will also discuss Amazon S3 here, which as you may have guessed is the main driver of the "S3-compatible" software ecosystem that you can now find at many competing vendors. Some S3-compatible solutions also combine the "S3" and "CDN" aspects into one product like DigitalOcean Spaces, however I think the AWS S3 + AWS CloudFront combination is especially important for Owlkins users who want to maintain privacy while also reaping the benefits of a CDN. With S3, you can lock down all content such that only your CloudFront distribution can access it, and then on CloudFront you can setup trusted signers to enforce access to your photos and videos. This feature does take some extra time to setup, but we will go into the steps below.

Create the S3 bucket

First we must create the S3 bucket that you will store your photos and videos in. From your AWS console:

1. Navigate to the S3 dashboard
2. Click the Create bucket button to create a new bucket
3. Give the bucket a name in the Bucket name section.
   • Note that this should not be visible to your users once you finish setting up CloudFront below, but at some level the name will be publicly visible so name appropriately
4. Choose the AWS region or leave it default. The region will have a minor impact in uploading files, choose a region geographically closest to your server running Owlkins.
5. Object Ownership you can leave with the default ACLs disabled (recommended)
6. Block Public Access settings for this bucket you will want to leave set as the default Block all public access, as users will solely receive content through CloudFront.
7. Bucket Versioning you can leave disabled, but this is not a hard requirement and do as you see fit.
   • There is a sample backup script you can follow if you want to keep all files uploaded to S3 in sync with local storage for safekeeping.
8. Default encryption you can generally as the default (disabled), but if you know what you're doing you can optionally set this up.
9. Advanced Settings you can leave with the defaults
10. Now, click Create bucket to proceed
11. Important: now, take your bucket name and store this in the AWS_STORAGE_BUCKET_NAME variable in your config.py file.

CloudFront

Next we must create the CloudFront distribution that will transmit the photos and videos from S3. From your AWS console:

Creating a public key pair

You will need to upload a public key which you will use later. To make the below steps without extra friction, let's make the key first.

1. Go to the public key part of your CloudFront dashboard
2. Click Create public key
3. Type in any name that you want. This name will only be displayed internally in your AWS dashboard
4. Enter a description, if you want.
5. The Key field is the most important part of this form and the below steps are borrowed from this link:
   1. On your local machine, with openssl installed, navigate to a folder that you want to store your keys in. Don't store your keys in a public or easily accessed location.
   2. Invoke the following to create your private key

      openssl genrsa -out private_key.pem 2048

      Copy
   3. Next invoke the following to create yourpublic key

      openssl rsa -pubout -in private_key.pem -out public_key.pem

      Copy
   4. Take the contents of the public key in its entirety and paste it into the Key field back on your CloudFront dashboard
   5. Take the contents of the private key in its entirety and store it in the CLOUDFRONT_PRIVATE_KEY variable in your secrets.py file. Note that you will have to finangle the content a bit if you "paste" it in, as newline characters will have to be replaced by \n throughout the file. Alternatively, if you want to read in the contents from your private_key.pem file, you could do something like this:

      'CLOUDFRONT_PRIVATE_KEY': open('/absolute/path/to/your/private_key.pem').read()

6. Click Create public key to save your key
7. Now you will be taken back to your public key dashboard where you will see your new key and a value in the ID column
8. Copy the value in the ID column and store this value in the CLOUDFRONT_KEYPAIR_ID field in your secrets.py file.

Requesting a certificate

When you make your distribution, you will reach a point that you can set up an Alternative domain name (CNAME), which you will have to do for you to use the keypair ID you just created above. One prerequisite to this step is to request a certificate for this domain. The domain name you choose should be a subdomain of your main domain. For example if you purchased example.com above and plan to host Owlkins at this domain, you could now choose cdn.example.com as the subdomain to host your CloudFront distribution at. For this step, your DNS page open for your domain so that you can quickly copy some CNAME values into your DNS records.

1. Navigate to the certificate request dialog.
2. Choose Request a public certificate and click next
3. Enter your Fully qualified domain name, which following the above example, would be cdn.example.com
4. For Select validation method leave DNS validation - recommended selected
5. Click Request
6. Now you should be taken to your certificate dashboard where you see a message like Successfully requested certificate with ID XXXXX-XXXX-XX at the top, but no certificate in your dashboard!
   • Don't panic, typically it takes a few seconds for the certificate to be created. Try refreshing your dashboard.
7. Once your certificate populates the list on the dashboard, click it to go to the more detailed view
8. In the Domains section you should see a CNAME value that you will have to configure at your domain's DNS settings. Follow the steps as you did above for SMTP setup to add the CNAME to your domain.
9. Wait for the status to switch to Issued with a green checkmark before proceeding.

Creating the distribution

Now that you have a keypair ID to sign content and a certificate for your CDN subdomain, we can move on to creating the distribution.

1. Navigate now to the CloudFront dashboard
2. Click the Create distribution button
3. For the Origin domain select the S3 bucket you created above from the dropdown after clicking on the field.
4. You can skip the Origin path and Name fields.
5. S3 bucket access
   1. change to "Yes use OAI (bucket can restrict access to only CloudFront)"
   2. Click Create new OAI to create a new Origin access identity
   3. After creating the OAI, select it in the dropdown for Origin access identity
   4. Change the Bucket policy to Yes, update the bucket policy, so that AWS will handle the configuration needed for your S3 bucket.
6. Add custom header: this can be left alone, skip it
7. Enable Origin Shield, you can leave this as the default No
8. Additional settings you can skip
9. Now under Default cache behavior you can leave Path pattern as Default (*)
10. Compress objects automatically you can leave as Yes
11. Viewer protocol policy you should change to Redirect HTTP to HTTPS though you should never have anyone accessing a link via HTTP anyway from Owlkins
12. Allowed HTTP methods leave as GET, HEAD
13. Restrict Viewer Access: this is the portion that enforces access permissions to keep your content private.
    1. Change to Yes
    2. Trusted authorization type keep as Trusted key groups (recommended), and now you will have to make the key group by clicking the Create key group link and following it into a new tab.
       1. Enter anything you like in the Name, this should only be displayed on your dashboard
       2. Enter anything you like for the Description
       3. For the Public Keys dropdown, select the keypair ID you created earlier.
       4. Click Create key group
    3. Now with the key group created, you can go back to the tab for creating your distribution, and after clicking the refresh button next to the dropdown for Add key groups you should be able to select the key group you just created above.
14. Cache key and origin requests you can leave as Cache policy and origin request policy (recommended)
15. Function associations you can skip
16. Under Settings the default Price class should usually be changed to Use only North America and Europe if this describes the bulk of your users. If you have many users in Asia, consider leaving the default Use all edge locations (best performance) though note that this does come with an increased cost.
17. Alternate domain name (CNAME) though this is listed as (optional) it is not optional for Owlkins users who want to use the signed cookies and urls generated by the keypair ID we created above.
    1. Enter a domain name that is a subdomain of your main domain, for example if you purchased example.com above and plan to host Owlkins at this domain, and you followed the steps above for requesting a certificate your value here cdn.example.com.
    2. Select the certificate you made above.
    3. Leave Security Policy as the TLS version that is (recommended)
18. Supported HTTP versions leave HTTP/2 checked
19. Default root object leave blank and ignore
20. Standard logging leave off
21. IPv6 leave on
22. Fill in Description with anything you like
23. Click Create distribution to finalize these settings, where you will be taken back to your Cloudfront Distributions dashboard.
24. Now the distribution is spinning up and you can take the Alternative Domain Name you used in configuring your CloudFront distribution and save this as the AWS_CLOUDFRONT_DOMAIN variable in your config.py file. This is not in your secrets file because this will be a very public value.

Adding the CloudFront distribution to your DNS

From the CloudFront dashboard, find your "Domain Name" (which will look like d1XXXXXXX.cloudfront.net). Go to your DNS provider and add a new CNAME record that maps the domain you chose above (example was cdn.example.com) to this CloudFront Domain Name.

Most DNS UIs will have a form that lets you fill in the value.example.com, where value in our example would be cdn, such that if you were to put cdn.example.com in the form you would actually create a record for cdn.example.com.example.com. This mistake would be easy enough to fix but watch out for that!

Create a user to allow Owlkins to write to your S3 bucket

Now that you have a working S3 + CloudFront configuration, we need to add a "user" that will allow your Owlkins distriution to write photos and videos to it.

1. First, create a policy that will allow your user to access the bucket in the Policies dashboard
2. Once in the dashboard, click Create Policy
3. In the new view that appears, select the JSON tab.
4. Paste in a policy like the below, where you fill in the name of your bucket:

   
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "VisualEditor0",
               "Effect": "Allow",
               "Action": [
                   "s3:PutObject",
                   "s3:GetObjectAcl",
                   "s3:GetObject",
                   "s3:ListBucket",
                   "s3:DeleteObject",
                   "s3:PutObjectAcl"
               ],
               "Resource": [
                   "arn:aws:s3:::YOUR_BUCKET_HERE",
                   "arn:aws:s3:::YOUR_BUCKET_HERE/*"
               ]
           }
       ]
   }
       

   Copy
5. Click Next: Tags
6. You don't need to add any Tags. Click Next: Review
7. In the review dialog, enter a Name for the policy, perhaps s3-bucket-access-owlkins or something that will help you remember the purpose of the policy.
8. Click Create policy

Now that you have a policy created, you can create the user.

1. Navigate to your AWS IAM dashboard
2. Click Add users
3. In the wizard that your window now navigates to, enter a Uesr name that will help you easily identify the purpose of this user
4. For the Select AWS access type section, Check the box Access key - Programmatic access and leave Password - AWS Management Console access unchecked!
5. Click Next: permissions
6. On the permissions screen, select Attach existing policies directly
7. In the search text field next to Filter policies, type in the Name of the policy you just created.
8. Click Next: Tags
9. You don't have to enter any tags. Click Next: Review
10. Peruse the user you are about to create and then click Create user
11. You will now be taken to a screen that shows the user has been created. This screen contains the user's credentials which will not be available after you navigate away from this screen. If you navigate away from this screen without saving the credentials, either via clicking Download .csv or Show and copying the secret access key, then you will have to repeat the above steps.
    1. Copy the Access key ID and store this in the AWS_ACCESS_KEY_ID variable in your secrets.py file.
    2. Copy the Secret access key and store this in the AWS_SECRET_ACCESS_KEY variable in your secrets.py file.
12. Go back to the IAM Dashboard and you will see your new user. Click on your new user.
13. Now in the dashboard for your specific new user, find the User ARN value. Copy this and save it as the ARN variable in config.py

All finished with S3 + CloudFront

That was quite a few steps, but hopefully you've made it through without any issues. Soon we'll find out when we go to test your Owlkins installation!

If you already are using Vault by Hashicorp you can configure Owlkins to initialize all secret keys from that instead of using the secrets.py file.

In config.py, set VAULT_SEVER to point to your API endpoint and then configure the VAULT_KV_STORE and VAULT_ENVS variables.

For more information, see the comments in config_defaults.py.

Anytime there is a change to a file you will need to restart the gunicorn service to pick up the change. For example, if you change a parameter in config.py or secrets.py, or download an update from the git repo, you will need to invoke the following command:

sudo systemctl restart gunicorn

If you want to test your config.py and secrets.py before moving to the whole stack, you can execute the following from the root of your Owlkins installation (does not have to be within an activated virtual environment):

python3 test_config.py

your_user@your_server:~/owlkins python3 test_config.py
CRITICAL:test_config:    config.py passes checks
CRITICAL:test_config:    secrets store passes checks

One helpful way to check if everything is running fine is to go into the Django shell. For this, you will need to be in the virtual environment. From your Owlkins root, if you installed the virtual environment with the name we suggested above .venv, you can enter the virtual environment like the following:

source .venv/bin/activate

(.venv) your_user@your_server:~/owlkins 

python manage.py shell

(.venv) your_user@your_server:~/owlkins python manage.py shell
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

exit()

python manage.py migrate

After any update

Therefore given potential database changes combined with python code changes, anytime an update is downloaded it would be safe to execute the following two commands to restart the server and apply any database migrations:

python manage.py migrate
sudo systemctl restart gunicorn

This step could be considered optional if you don't want an email digest sent out to your users, but the truth is, a lot of your users will really appreciate the daily digest. I know some users that exclusively interact with the site through the daily emails and may not know how to get to the site otherwise!

If you are following the recommendation to run Ubuntu, and assuming you're on a modern version, then you're running systemd, and we will setup a systemd timer that will periodically send emails (we recommend once a day).

Slice file

First we will make the slice file which will determine how much of your resources are available to the process.

sudo nano /etc/systemd/system/daily_email_digest.slice

# /etc/systemd/system/daily_email_digest.slice
[Unit]
Description=Limited resources Slice
DefaultDependencies=no
Before=slices.target
[Slice]
CPUQuota=50%
MemoryLimit=2G

Service file

Next we can create the service file which will contain instructions for actually running the email:

sudo nano /etc/systemd/system/daily_email_digest.service

# /etc/systemd/system/daily_email_digest.service
[Unit]
Description=Run Daily Email Digest job for new photos
Wants=daily_email_digest.timer
[Service]
Type=oneshot
User=OWLKINS_INSTALL_USER
ExecStart=/home/OWLKINS_INSTALL_USER/owlkins/.venv/bin/python3 /home/OWLKINS_INSTALL_USER/owlkins/manage.py daily_email_digest
WorkingDirectory=/home/OWLKINS_INSTALL_USER/owlkins
Slice=daily_email_digest.slice

EnvironmentFile=/home/OWLKINS_INSTALL_USER/vault_envs

Timer file

Finally we will create the timer file which will instruct how often and when the service file runs. Create the timer like this:

sudo nano /etc/systemd/system/daily_email_digest.timer

# /etc/systemd/system/daily_email_digest.timer
[Unit]
Description=Run Daily Email Digest for photos
#Requires=daily_email_digest.service
[Timer]
Unit=daily_email_digest.service
OnCalendar=*-*-* 13:00:00
[Install]
WantedBy=timers.target

OnCalendar=*-*-* 08:00:00 America/Chicago

timedatectl list-timezones

Enable and start the timer

Now that your timer is set up you can enable it such that your system will always run the timer even after reboot with:

sudo systemctl enable daily_email_digest.timer

sudo systemctl list-timers

sudo systemctl start daily_email_digest

sudo systemctl status daily_email_digest

While Owlkins can successfully run the ffmpeg video conversion and upload logic within the web server process itself for a couple of videos, it will start to "drop" videos if many are uploaded at once (such as 8 or more). If you plan to upload videos regularly, I recommend you setup a daemon process to handle all video conversion tasks.

If you are following the recommendation to run Ubuntu, and assuming you're on a modern version, then you're running systemd, and we will setup a systemd timer that will periodically launch the Owlkins Django management command convert_videos.

Service file

First we can create the service file which will contain instructions for running the command:

sudo nano /etc/systemd/system/convert_videos.service

# /etc/systemd/system/convert_videos.service
[Unit]
Description=Run frequently to convert any pending video uploads from the native format to a more universal MP4 format
Wants=convert_videos.timer
[Service]
Type=oneshot
User=OWLKINS_INSTALL_USER
#EnvironmentFile=/home/OWLKINS_INSTALL_USER/vault_envs
ExecStart=/home/OWLKINS_INSTALL_USER/owlkins/.venv/bin/python3 /home/OWLKINS_INSTALL_USER/owlkins/manage.py convert_all_pending_videos
WorkingDirectory=/home/OWLKINS_INSTALL_USER/owlkins
[Install]
WantedBy=multi-user.target

EnvironmentFile=/home/OWLKINS_INSTALL_USER/vault_envs

Timer file

Finally we will create the timer file which will instruct how often and when the service file runs. Create the timer like this:

sudo nano /etc/systemd/system/convert_videos.timer

# /etc/systemd/system/convert_videos.timer
[Unit]
Description=Run frequently to convert any pending video uploads from the native format to a more universal MP4 format
[Timer]
Unit=convert_videos.service
OnCalendar=*:0/5
[Install]
WantedBy=timers.target

Enable and start the timer

Now that your timer is set up you can enable it such that your system will always run the timer even after reboot with:

sudo systemctl enable convert_videos.timer

sudo systemctl list-timers --all

sudo systemctl start convert_videos

sudo systemctl status convert_videos

Once you have verified that the daemon is properly running, change the line in your config.py to tell the web worker processes that this service is now available:

'VIDEO_DAEMON_SET_UP': True,