General Linux Setup

While Amazon Web Services (AWS) provides the preferred hosting for SEED, running on a bare-bones linux server follows a similar setup, replacing the AWS services with their linux package counterparts, namely: PostgreSQL and Redis.

seed is a Django project and Django’s documentation is an excellent place to general understanding of this project’s layout.

Pre-requisites

Ubuntu server 14.04 or newer

We need to install the base packages needed to run the app:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install libpq-dev python-dev python-pip libatlas-base-dev \
gfortran build-essential g++ npm libxml2-dev libxslt1-dev git mercurial \
libssl-dev curl uwsgi-core uwsgi-plugin-python
$ sudo apt-get install redis-server
$ sudo apt-get install postgresql postgresql-contrib

Note

postgresql >=9.3 is required to support JSON Type

Configure PostgreSQL

$ sudo su - postgres
$ createdb "seed-deploy"
$ createuser -P DBUsername
$ psql
postgres=# GRANT ALL PRIVILEGES ON DATABASE "seed-deploy" TO DBUsername;
postgres=# \q;
$ exit

Note

Any database name and username can be used here in place of “seed-deploy” and DBUsername

Python Dependencies

clone the seed repository from github

$ git clone git@github.com:SEED-platform/seed.git

enter the repo and install the python dependencies from requirements.txt

$ cd seed
$ sudo pip install -r requirements.txt

JavaScript Dependencies

npm is required to install the JS dependencies. The bin/install_javascript_dependencies.sh script will download all JavaScript dependencies and build them. bower and grunt-cli will be installed globally by the install_javascript_dependencies script. The Ubuntu version 14.04 requires a cusomt install of nodejs/npm, and an install scrpt ( bin/node-and-npm-in-30s.sh) is provided to download a stable release and install npm assuming the prerequisites are met.

$ . bin/node-and-npm-in-30s.sh
$ bin/install_javascript_dependencies.sh

Django Database Configuration

Copy the local_untracked.py.dist file in the config/settings directory to config/settings/local_untracked.py, and add a DATABASES configuration with your database username, password, host, and port. Your database configuration can point to an AWS RDS instance or a postgresql 9.3 database instance you have manually installed within your infrastructure.

# Database
DATABASES = {
    'default': {
        'ENGINE':'django.db.backends.postgresql_psycopg2',
        'NAME': 'seed-deploy',
        'USER': 'DBUsername',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

Note

other databases could be used such as MySQL, but are not supported due to the postgres-specific JSON Type

In in the above database configuration, seed is the database name, this is arbitrary and any valid name can be used as long as the database exists. Enter the database name, user, password you set above.

The database settings can be tested using the Django management command, ./manage.py dbshell to conect to the configured database.

create the database tables and migrations:

$ python manage.py syncdb
$ python manage.py migrate

Note

running migrations can be shortened into a one-liner ./manage.py syncdb --migrate

Cache and Message Broker

The SEED project relies on redis for both cache and message brokering, and is available as an AWS ElastiCache service or with the redis-server linux package. (sudo apt-get install redis-server)

local_untracked.py should be updated with the CACHES and BROKER_URL settings.

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': "127.0.0.1:6379",
        'OPTIONS': {'DB': 1},
        'TIMEOUT': 300
    }
}
BROKER_URL = 'redis://127.0.0.1:6379/1'

Note

The popular memcached can also be used as a cache back-end, but is not supported and redis has a different cache key format, which could cause breakage and isn’t tested. Likewise, rabbitmq or AWS SQS are alternative message brokers, which could cause breakage and is not tested.

Creating the initial user

create a superuser to access the system

$ python manage.py create_default_user --username=demo@example.com --organization=example --password=demo123

Note

Every user must be tied to an organization, visit /app/#/profile/admin as the superuser to create parent organizations and add users to them.

Running celery the background task worker

Celery is used for background tasks (saving data, matching, creating projects, etc) and must be connected to the message broker queue. From the project directory, celery can be started:

$ python manage.py celery worker -B -c 2 --loglevel=INFO -E --maxtasksperchild=1000

Running the development web server

The Django dev server (not for production use) can be a quick and easy way to get an instance up and running. The dev server runs by default on port 8000 and can be run on any port. See Django’s runserver documentation for more options.

$ python manage.py runserver --settings=config.settings.dev

Running a production web server

Our recommended web server is uwsgi sitting behind nginx. The python package uwsgi is needed for this, and should install to /usr/local/bin/uwsgi Since AWS S3, is not being used here, we recommend using dj-static to load static files.

Note

The use of the dev settings file is production ready, and should be used for non-AWS installs with DEBUG set to False for production use.

$ sudo pip install uwsgi dj-static

Generate static files:

$ sudo ./manage.py collectstatic --settings=config.settings.dev

Update config/settings/local_untracked.py:

DEBUG = False
# static files
STATIC_ROOT = 'collected_static'
STATIC_URL = '/static/'

Start the web server:

$ sudo /usr/local/bin/uwsgi --http :80 --module standalone_uwsgi --max-requests 5000 --pidfile /tmp/uwsgi.pid --single-interpreter --enable-threads --cheaper-initial 1 -p 4

Warning

Note that uwsgi has port set to 80. In a production setting, a dedicated web server such as Nginx would be receiving requests on port 80 and passing requests to uwsgi running on a different port, e.g 8000.

environmental variables

The following environment variables can be set within the ~/.bashrc file to override default Django settings.

export SENTRY_DSN=https://xyz@app.getsentry.com/123
export DEBUG=False
export ONLY_HTTPS=True

SMTP service

In the AWS setup, we use SES to provide an email service Django can use as an email backend and configured it in our config/settings/main.py:

EMAIL_BACKEND = 'django_ses.SESBackend'

Many options for setting up your own SMTP service/server or using other SMTP third party services are available and compatible including gmail.

Django can likewsie send emails via python’s smtplib with sendmail or postfix installed. See their docs for more info.

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

local_untracked.py

# postgres DB config
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'seed',
        'USER': 'your-username',
        'PASSWORD': 'your-password',
        'HOST': 'your-host',
        'PORT': 'your-port',
    }
}

# config for local storage backend
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
STATICFILES_STORAGE = DEFAULT_FILE_STORAGE
DOMAIN_URLCONFS = {}
DOMAIN_URLCONFS['default'] = 'urls.main'

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': "127.0.0.1:6379",
        'OPTIONS': {'DB': 1},
        'TIMEOUT': 300
    }
}

# redis celery config
BROKER_URL = 'redis://127.0.0.1:6379/1'
CELERY_DEFAULT_QUEUE = 'seed-dev'
CELERY_QUEUES = (
    Queue(
        CELERY_DEFAULT_QUEUE,
        Exchange(CELERY_DEFAULT_QUEUE),
        routing_key=CELERY_DEFAULT_QUEUE
    ),
)

# SMTP config
EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

# static files
STATIC_ROOT = 'collected_static'
STATIC_URL = '/static/'