<<< Go back

May 10, 2022

Deploying Django to App Engine with Github Actions

Deploying to Google Cloud Platform (GCP) on Github actions has not been a straight forward process. The blog posts online are incomplete and hard to follow. So, hopefully I can help a little bit with what I found to make it easy to deploy this website to GCP App engine and perform migrations on a Cloud SQL database.

Deployment

from this blog post I was able to get the general structure of how to deploy from GitHub actions: https://www.ayobamiadewole.com/Blog/Github-Actions. My app.yaml varies a bit because I do not want to store my secrets in source control. This leads to some pretty interesting problems later on... But anyways here is the app.yaml file that I am using currently:

runtime: python39

handlers:
# This configures Google App Engine to serve the files in the app's static
# directory.
- url: /static
  static_dir: static/

# This handler routes all requests not caught above to your main app. It is
# required when static routes are defined, but can be omitted (along with
# the entire handlers section) when there are no static files defined.
- url: /.*
  script: auto

includes:
  - env_variables.yaml

The app.yaml is pretty basic and basically follows the app.yaml found in the GCP App Engine tutorial for Django from GCP's website. The only difference is the following part:

includes:
  - env_variables.yaml

Which allows me to not store my environment variables in source control. However, now I am also dependent on sourcing this env_variables.yaml file into the rest of the pipeline so I do not have to track too many duplicated secrets in GitHub actions and maintain their values.

Initially, I was not sure how to get this env_variables.yaml file accessible in the environment variables. So, I posted a question on the very much scary and every programmers favorite website to copy from; StackOverFlow!

https://stackoverflow.com/questions/72093178/how-do-i-read-a-yaml-file-and-outputting-the-yaml-file-values-as-environment-var

Basically the solution, someone suggested to me was to run this one-liner:

python -c 'from pathlib import Path;from ruamel.yaml import YAML; print( "".join( [f"{k}={v!r}\n" for k, v in YAML().load(Path("env_variables.yaml"))["env_variables"].items() if not k.__eq__("DATABASE_CONNECTION_ADDRESS")] ) )'

Which basically reads the environment variables keys and values and stores them in the following format KEY=VALUE and this gets sourced into the the environment using set -a; eval $(python -c ...); set +a.

So great! We can now turn a YAML file into environment variables! We now need to base64 encode our env_variables.yaml file and store that value in Github secrets. By running the following command in your terminal: base64 env_variables.yaml | pbcopy , it will base64 encode our yaml file and copy the contents to your clipboard (assuming you are using a mac). We can then follow the tutorial on Github's website on how to store this base64 value into a GitHub secret: https://docs.github.com/en/actions/security-guides/encrypted-secrets

Then we can start with this yaml file as base and put it at the following path in your project: .github/workflows/deploy.yaml

# .github/workflows/deploy

name: deploy-app-to-gcp
on:
 push:
   branches: [ main]
   paths:
     - '**'
jobs:
 deploy:
   name: Deploy
   runs-on: ubuntu-latest

   steps:
     - uses: actions/checkout@v1
     - uses: actions-hub/gcloud@master
       env:
         PROJECT_ID: ${{secrets.GCLOUD_PROJECT_PROD_ID}}
         APPLICATION_CREDENTIALS: ${{secrets.GCLOUD_GITHUB_CREDENTIALS}}
       with:
         args: app deploy app.yaml

We need to create two more secrets, now the we have a base for our GitHub actions workflow.

  • GCLOUD_PROJECT_PROD_ID
  • GCLOUD_GITHUB_CREDENTIALS

The GCLOUD_PROJECT_PROD_ID value should be the name of your project in the GCP console.

Screen Shot 2022-05-10 at 6.46.27 AM

The GCLOUD_GITHUB_CREDENTIALS is a json file that is base64 encoded version of a service account key from GCP with the correct permissions to run a deployment.

Read more about the GCP action here: https://github.com/actions-hub/gcloud

Cool, Cool! Now that we have those secrets we can add the following step so that the Django app can be deployed with the environment variables it needs to work

- name: get env file
        run: |
          echo "${{secrets.ENV_FILE}}" | base64 --decode > ./env_variables.yaml

And Boom! you have a working deployment!

# .github/workflows/deploy

name: deploy-app-to-gcp
on:
 push:
   branches: [ main]
   paths:
     - '**'
jobs:
 deploy:
   name: Deploy
   runs-on: ubuntu-latest

   steps:
     - uses: actions/checkout@v1
     - name: get env file
       run: |
         echo "${{secrets.ENV_FILE}}" | base64 --decode > ./env_variables.yaml
     - uses: actions-hub/gcloud@master
       env:
         PROJECT_ID: ${{secrets.GCLOUD_PROJECT_PROD_ID}}
         APPLICATION_CREDENTIALS: ${{secrets.GCLOUD_GITHUB_CREDENTIALS}}
       with:
         args: app deploy app.yaml

Screen Shot 2022-05-10 at 7.04.44 AM

Migration

Now that we can successfully deploy our Django app. We need a way to make sure our migrations get applied to our application. We can added another job to our yaml file and call it migrate .

migrate:
    name: Migrate Database
    runs-on: ubuntu-latest
    needs: deploy
    steps:
      - uses: actions/checkout@v1

We will load our env_variables.yaml from our secrets again and as well as our GCLOUD_GITHUB_CREDENTIALS. Then we will write both to disk so they are accessible in the rest of this job.

- name: get env file
        run: |
          echo "${{secrets.ENV_FILE}}" | base64 --decode > ./env_variables.yaml
          echo "${{secrets.GCLOUD_GITHUB_CREDENTIALS}}" | base64 --decode > ./secrets.json

To run our Django migration we will need to install our python dependancies.

- name: Set up Python 3.9
        uses: actions/setup-python@v2
        with:
          python-version: 3.9
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

After we install our dependancies, we will need a way to connect to our Cloud SQL database. We can connected to the database using Cloud SQL Proxy. We will just download it for now and use it in a later step.

- name: Get Cloud SQL Proxy
        run: |
          wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
          chmod +x cloud_sql_proxy

Now that we have everything setup, and downloaded. We can preform the actual migration. I am going to show the code and I will go line by line into what it means.

- name: migrate Database
        env:
          DATABASE_CONNECTION_ADDRESS: 127.0.0.1
        run: |
          pip install ruamel.yaml
          set -a; eval $(python -c 'from pathlib import Path;from ruamel.yaml import YAML; print( "".join( [f"{k}={v!r}\n" for k, v in YAML().load(Path("env_variables.yaml"))["env_variables"].items() if not k.__eq__("DATABASE_CONNECTION_ADDRESS")] ) )'); set +a
          ./cloud_sql_proxy -instances=${{secrets.GCLOUD_PROJECT_PROD_ID}}:us-central1:cms-db=tcp:5432 -credential_file secrets.json  &
          python manage.py migrate
          exit 0;

These two lines below read the env_variables.yaml file and source it into current environment to be accessible by Django when it runs its migration. We want to not include DATABASE_CONNECTION_ADDRESS from the env_variables.yaml file because we want to use the ip of the cloud sql proxy instead.Which we have specified in the env of this step.

pip install ruamel.yaml
set -a; eval $(python -c 'from pathlib import Path;from ruamel.yaml import YAML; print( "".join( [f"{k}={v!r}\n" for k, v in YAML().load(Path("env_variables.yaml"))["env_variables"].items() if not k.__eq__("DATABASE_CONNECTION_ADDRESS")] ) )'); set +a

This next step we are running the cloud sql proxy and giving it the credential file we wrote to disk earlier in this process. Additionally we are running the proxy in the background using the & symbol so that we can not block the main thread and continue on with the migration.

./cloud_sql_proxy -instances=${{secrets.GCLOUD_PROJECT_PROD_ID}}:us-central1:cms-db=tcp:5432 -credential_file secrets.json  &

Then finally we run the migration and exit!

python manage.py migrate
exit 0;

And wow our migration step is done!

Final Github Actions File

name: deploy-app-to-gcp
on:
  push:
    branches: [ main]
    paths:
      - '**'
jobs:
  deploy:
    name: Deploy
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v1
      - name: get env file
        run: |
          echo "${{secrets.ENV_FILE}}" | base64 --decode > ./env_variables.yaml
          echo "${{secrets.SECRET_JSON}}" | base64 --decode > ./secrets.json
      - uses: actions-hub/gcloud@master
        env:
          PROJECT_ID: ${{secrets.GCLOUD_PROJECT_PROD_ID}}
          APPLICATION_CREDENTIALS: ${{secrets.GCLOUD_GITHUB_CREDENTIALS}}
        with:
          args: app deploy app.yaml
  migrate:
    name: Migrate Database
    runs-on: ubuntu-latest
    needs: deploy
    steps:
      - uses: actions/checkout@v1
      - name: get env file
        run: |
          echo "${{secrets.ENV_FILE}}" | base64 --decode > ./env_variables.yaml
          echo "${{secrets.GCLOUD_GITHUB_CREDENTIALS}}" | base64 --decode > ./secrets.json
      - name: Set up Python 3.9
        uses: actions/setup-python@v2
        with:
          python-version: 3.9
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Get Cloud SQL Proxy
        run: |
          wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
          chmod +x cloud_sql_proxy
      - name: migrate Database
        env:
          DATABASE_CONNECTION_ADDRESS: 127.0.0.1
        run: |
          pip install ruamel.yaml
          set -a; eval $(python -c 'from pathlib import Path;from ruamel.yaml import YAML; print( "".join( [f"{k}={v!r}\n" for k, v in YAML().load(Path("env_variables.yaml"))["env_variables"].items() if not k.__eq__("DATABASE_CONNECTION_ADDRESS")] ) )'); set +a
          ./cloud_sql_proxy -instances=${{secrets.GCLOUD_PROJECT_PROD_ID}}:us-central1:cms-db=tcp:5432 -credential_file secrets.json  &
          python manage.py migrate
          exit 0;
1F48D778-78F1-4233-9663-E3A8378082E8

Josh Martin

Full Stack Engineer

Chicago, IL

Josh is a software engineer and consultant. He writes software in Go, python, javascript. Additionally, he has experience writing using Django, React.JS, React Native, Vue, Kubernetes, Docker.

Hire him as your next Consultant or New Hire!