Development Setup¶

./manage.py create_default_user --username=demo@seed.lbl.gov --organization=lbl --password=demo123

sudo su - root
port install postgresql94-server postgresql94 postgresql94-doc
# init db
mkdir -p /opt/local/var/db/postgresql94/defaultdb
chown postgres:postgres /opt/local/var/db/postgresql94/defaultdb
su postgres -c '/opt/local/lib/postgresql94/bin/initdb -D /opt/local/var/db/postgresql94/defaultdb'

# At this point, you may want to add start/stop scripts or aliases to
# ~/.bashrc or your virtualenv ``postactivate`` script
# (in ``~/.virtualenvs/{env-name}/bin/postactivate``).

alias pg_start='sudo su postgres -c "/opt/local/lib/postgresql94/bin/pg_ctl \
    -D /opt/local/var/db/postgresql94/defaultdb \
    -l /opt/local/var/db/postgresql94/defaultdb/postgresql.log start"'
alias pg_stop='sudo su postgres -c "/opt/local/lib/postgresql94/bin/pg_ctl \
    -D /opt/local/var/db/postgresql94/defaultdb stop"'

pg_start

sudo su - postgres
PATH=$PATH:/opt/local/lib/postgresql94/bin/

brew install postgres
# follow the post install instructions to add to launchagents or call
# manually with `postgres -D /usr/local/var/postgres`
# Skip the remaining Postgres instructions!

createuser -P seeduser
createdb `whoami`
psql -c 'CREATE DATABASE "seeddb" WITH OWNER = "seeduser";'
psql -c 'GRANT ALL PRIVILEGES ON DATABASE "seeddb" TO seeduser;'
psql -c 'ALTER USER seeduser CREATEDB;'
psql -c 'ALTER USER seeduser CREATEROLE;'

workon seed

export PATH=$PATH:/opt/local/lib/postgresql94/bin

export CC=gcc

pip install -r requirements/local.txt

python setup.py install

sudo port install npm

brew install npm

cd config/settings
cp local_untracked.py.dist local_untracked.py

# postgres DB config
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'seeddb',
        'USER': 'seeduser',
        'PASSWORD': 'seedpass',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.cache.RedisCache',
        'LOCATION': "127.0.0.1:6379",
        'OPTIONS': {'DB': 1},
        'TIMEOUT': 300
    }
}
BROKER_URL = 'redis://127.0.0.1:6379/1'

export DJANGO_SETTINGS_MODULE=config.settings.dev
./manage.py migrate

./manage.py create_default_user --username=admin@my.org --organization=lbnl --password=badpass

psql seeddb seeduser
seeddb=> update landing_seeduser set api_key='DEADBEEF' where id=1;

sudo port install redis

brew install redis
# follow the post install instructions to add to launchagents or
# call manually with `redis-server`

./bin/install_javascript_dependencies.sh

export DJANGO_SETTINGS_MODULE=config.settings.dev

./bin/start-seed.sh

docker-machine create --virtualbox-disk-size 100000 -d virtualbox dev

docker-machine start dev  # if not already running
eval $(docker-machine env dev)

docker-compose build

docker-compose up

docker-compose run web ./manage.py create_default_user

username: demo@seed.lbl.gov
password: demo

docker run hello-world

docker ps -a

docker rm $(docker ps -a -q)

docker images

docker rmi -f $(docker images -q -a -f dangling=true)

sudo apt-get install python-pip
sudo pip install docker-compose

sudo apt-get install virtual-box
wget https://github.com/docker/machine/releases/download/v0.7.0/docker-machine-`uname -s`-`uname -m`
sudo mv docker-$(uname -s)-$(uname -m) /usr/local/bin/docker-machine
sudo chmod +x /usr/local/bin/docker-machine

pip install -e  'git+https://github.com/revsys/django-tos.git@aca823ccd12fdb897b2827832458b3c34e91dee6#egg=django_tos-master'

git update-index  --skip-worktree docker-compose.yml

cp config/settings/local_untracked.py.dist config/settings/local_untracked.py

# postgres DB config
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'seed',
        'USER': 'seed',
        'PASSWORD': 'seed',
        'HOST': '172.17.0.1',
        'PORT': '5432',
    }
}

docker-compose build

docker-compose up

docker exec -it "seed_web_1" bash

bin/postcompile

docker-compose run web ./manage.py create_default_user

username: demo@seed.lbl.gov
password: demo

Monitoring¶

flower --port=5555 --broker=redis://localhost:6379/1`

Authentication¶

Authentication is handled via an authorization token set in an http header. To request an API token, go to /app/#/profile/developer and click ‘Get a New API Key’.

Every request must include an ‘Authorization’ http header made up of your username (email) and your API key, separated with a ‘:’. For example, with curl:

curl -H Authorization:user@email_address.com:5edfd7f1f0696d4139118f8b95ab1f05d0dd418e https://seeddomain.com/app/api/get_api_schema/

Or using the Python Requests library:

headers = {'authorization': 'user@email_address.com:5edfd7f1f0696d4139118f8b95ab1f05d0dd418e'}
result = requests.get('https://seeddomain.com/app/api/get_api_schema/', headers=headers)
print result.json()

If authentication fails, the response’s status code will be 302, redirecting the user to /app/login.

Payloads¶

Many requests require a JSON-encoded payload and/or parameters in the query string of the url. A frequent requirement is including the organization_id of the org you belong to. E.g.:

curl -H Authorization:user@email_address.com:5edfd7f1f0696d4139118f8b95ab1f05d0dd418e \
  https://seeddomain.com/app/accounts/get_organization?organization_id={your org id here}

Or in a JSON payload:

curl -H Authorization:user@email_address.com:5edfd7f1f0696d4139118f8b95ab1f05d0dd418e \
  -d '{"organization_id":6, "user_id": 12, "role": "viewer"}' \
  https://seeddomain/app/accounts/update_role/

Using Python:

headers = {'authorization': 'user@email_address.com:5edfd7f1f0696d4139118f8b95ab1f05d0dd418e'}
params = json.dumps({'organization_id': 6, 'user_id': 12, 'role': 'viewer'})
result = requests.post('https://seeddomain.com/app/accounts/update_role/',
                       data=params,
                       headers=headers)
print result.json()

Responses¶

Responses from all requests will be JSON-encoded objects, as specified in each endpoint’s documentation. In the case of an error, most endpoints will return this instead of the expected payload (or an HTTP status code):

{
    'status': 'error',
    'message': 'explanation of the error here'
}

API-related Endpoints¶

seed.views.api.get_api_schema()¶

URI: /app/api/get_api_schema/

Returns a hash of all API endpoints and their descriptions.

Returns:

{
    '/example/url/here': {
        'name': endpoint name,
        'description': endpoint description
    }...
}

Todo

Should this require authentication? Should it limit the return to endpoints the user has authorization for?

Todo

Format docstrings better.

Account Management Endpoints¶

seed.views.accounts.add_org()¶

URI: /app/accounts/add_org/

Creates a new organization.

Payload:

{
    'organization_name': The name of the new org,
    'user_id': the user id of the owner of the new org,
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'message, if any',
    'organization_id': The ID of the new org, if created.
}

seed.views.accounts.add_user()¶

URI: /app/accounts/add_user/

Creates a new SEED user. One of ‘organization_id’ or ‘org_name’ is needed. Sends invitation email to the new user.

Payload:

{
    'organization_id': ID of an existing org to add the new user to,
    'org_name': Name of a new org to create with user as owner
    'first_name': First name of new user
    'last_name': Last name of new user
    'role': {
        'value': The permission level of new user within this org
            (one of member, viewer, owner)
    },
    'email': Email address of new user.
}

Returns:

{
    'status': 'success',
    'message': email address of new user,
    'org': name of the new org (or existing org),
    'org_created': True if new org created,
    'username': Username of new user
}

seed.views.accounts.add_user_to_organization()¶

URI: /app/accounts/add_user_to_organization/

Adds an existing user to an organization.

Payload:

{
    'organization_id': The ID of the organization,
    'user_id': the user id of the owner of the new org,
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'message, if any',
}

seed.views.accounts.create_sub_org()¶

URI: /app/accounts/create_sub_org/

Creates a child org of a parent org.

Payload:

{
    'parent_org_id': ID of the parent org,
    'sub_org': {
        'name': Name of new sub org,
        'email': Email address of owner of sub org, which
                must already exist
    }
}

Returns:

{
    'status': 'success' or 'error',
    'message': Error message, if any,
    'organization_id': ID of newly-created org
}

seed.views.accounts.get_cleansing_rules()¶

URI: /app/accounts/get_cleansing_rules/

Returns the cleansing rules for an org.

Parameters:	request –
GET:	Expects organization_id in the query string.

Returns:

{
 'status': 'success',
 'in_range_checking': An array of in-range error rules,
 'missing_matching_field': An array of fields to verify existence,
 'missing_values': An array of fields to ignore missing values
}

seed.views.accounts.get_organization()¶

URI: /app/accounts/get_organization/

Retrieves a single organization by id.

GET:	Expects ?organization_id=(:org_id)

Returns:

{'status': 'success or error', 'message': 'error message, if any',
 'organization':
    {'name': org name,
     'org_id': org's identifier (used with Authorization header),
     'id': org's identifier,
     'number_of_users': count of members of org,
     'user_is_owner': True if the user is owner of this org,
     'user_role': The role of user in this org (owner, viewer, member),
     'owners': [
         {
          'first_name': the owner's first name,
          'last_name': the owner's last name,
          'email': the owner's email address,
          'id': the owner's identifier (int)
          }
         ]
      'sub_orgs': [ a list of orgs having this org as parent, in
                    the same format...],
      'is_parent': True if this org contains suborgs,
      'num_buildings': Count of buildings belonging to this org
    }
}

seed.views.accounts.get_organizations()¶

URI: /app/accounts/get_organizations/

Retrieves all orgs the user has access to.

Returns:

{'organizations': [
    {'name': org name,
     'org_id': org's identifier (used with Authorization header),
     'id': org's identifier,
     'number_of_users': count of members of org,
     'user_is_owner': True if the user is owner of this org,
     'user_role': The role of user in this org (owner, viewer, member),
     'owners': [
                 {
                  'first_name': the owner's first name,
                  'last_name': the owner's last name,
                  'email': the owner's email address,
                  'id': the owner's identifier (int)
                 }
               ]
     'sub_orgs': [ a list of orgs having this org as parent, in
                the same format...],
     'is_parent': True if this org contains suborgs,
     'num_buildings': Count of buildings belonging to this org
    }...
   ]
}

seed.views.accounts.get_organizations_users()¶

URI: /app/accounts/get_organizations_users/

Retrieve all users belonging to an org.

Payload:

{'organization_id': org_id}

Returns:

{'status': 'success',
 'users': [
    {
     'first_name': the user's first name,
     'last_name': the user's last name,
     'email': the user's email address,
     'id': the user's identifier (int),
     'role': the user's role ('owner', 'member', 'viewer')
    }
  ]
}

Todo

check permissions that request.user is owner or admin and get more info about the users.

seed.views.accounts.get_query_threshold()¶

URI: /app/accounts/get_query_threshold/

Returns the “query_threshold” for an org. Searches from members of sibling orgs must return at least this many buildings from orgs they do not belong to, or else buildings from orgs they don’t belong to will be removed from the results.

GET:	Expects organization_id in the query string.

Returns:

{
 'status': 'success',
 'query_threshold': The minimum number of buildings that must be
     returned from a search to avoid squelching non-member-org results.
}

seed.views.accounts.get_shared_fields()¶

URI: /app/accounts/get_shared_fields/

Retrieves all fields marked as shared for this org tree.

GET:	Expects organization_id in the query string.

Returns:

{
 'status': 'success',
 'shared_fields': [
     {
      "title": Display name of field,
      "sort_column": database/search name of field,
      "class": css used for field,
      "title_class": css used for title,
      "type": data type of field,
          (One of: 'date', 'floor_area', 'link', 'string', 'number')
      "field_type": classification of field.  One of:
          'contact_information', 'building_information',
          'assessor', 'pm',
      "sortable": True if buildings can be sorted on this field,
     }
     ...
   ],
   'public_fields': [
       {
          "title": Display name of field,
          "sort_column": database/search name of field,
          "class": css used for field,
          "title_class": css used for title,
          "type": data type of field,
            (One of: 'date', 'floor_area', 'link', 'string', 'number')
          "field_type": classification of field.  One of:
              'contact_information', 'building_information',
              'assessor', 'pm',
          "sortable": True if buildings can be sorted on this field,
         }
         ...
   ]
}

seed.views.accounts.get_user_profile()¶

URI: /app/accounts/get_user_profile/

Retrieves the request’s user’s first_name, last_name, email and api key if exists.

Returns:

{
    'status': 'success',
    'user': {
        'first_name': user's first name,
        'last_name': user's last name,
        'email': user's email,
        'api_key': user's API key
    }
}

seed.views.accounts.remove_user_from_org()¶

URI: /app/accounts/remove_user_from_org/

Removes a user from an organization.

Payload:

{
    'organization_id': ID of the org,
    'user_id': ID of the user
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'error message, if any'
}

seed.views.accounts.reset_cleansing_rules()¶

URI: /app/accounts/reset_cleansing_rules/

Resets an organization’s data cleansing rules

Parameters:	request –
GET:	Expects organization_id in the query string.

Returns:

{
 'status': 'success',
 'in_range_checking': An array of in-range error rules,
 'missing_matching_field': An array of fields to verify existence,
 'missing_values': An array of fields to ignore missing values
}

seed.views.accounts.save_cleansing_rules()¶

URI: /app/accounts/save_cleansing_rules/

Saves an organization’s settings: name, query threshold, shared fields

Payload:

{
    'organization_id: 2,
    'cleansing_rules': {
        'missing_matching_field': [
            {
                'field': 'address_line_1',
                'severity': 'error'
            }
        ],
        'missing_values': [
            {
                'field': 'address_line_1',
                'severity': 'error'
            }
        ],
        'in_range_checking': [
            {
                'field': 'conditioned_floor_area',
                'enabled': true,
                'type': 'number',
                'min': null,
                'max': 7000000,
                'severity': 'error',
                'units': 'square feet'
            },
        ]
    }
}

Returns:

{
    'status': 'success',
}

seed.views.accounts.save_org_settings()¶

URI: /app/accounts/save_org_settings/

Saves an organization’s settings: name, query threshold, shared fields

Payload:

{
    'organization_id: 2,
    'organization': {
        'query_threshold': 2,
        'name': 'demo org',
        'fields': [  # All internal sibling org shared fields
            {
                'sort_column': database/search field name,
                    e.g. 'pm_property_id',
            }
        ],
        'public_fields': [  # All publicly shared fields
            {
                'sort_column': database/search field name,
                    e.g. 'pm_property_id',
            }
        ],
    }
}

Returns:

{
    'status': 'success or error',
    'message': 'error message, if any'
}

seed.views.accounts.update_role()¶

URI: /app/accounts/update_role/

Sets a user’s role within an organization.

Payload:

{
    'organization_id': organization's id,
    'user_id': user's id,
    'role': one of 'owner', 'member', 'viewer'
}

Returns:

{
    'status': 'success or error',
    'message': 'error message, if any'
}

seed.views.accounts.update_user()¶

URI: /app/accounts/update_user/

Updates the request’s user’s first name, last name, and email

Payload:

{
 'user': {
          'first_name': :first_name,
          'last_name': :last_name,
          'email': :email
        }
}

Returns:

{
    'status': 'success',
    'user': {
        'first_name': user's first name,
        'last_name': user's last name,
        'email': user's email,
        'api_key': user's API key
    }
}

File Upload Endpoints¶

These endpoints behave drastically differently depending on whether the system is configured to upload files to S3 or to the local file system.

seed.data_importer.views.handle_s3_upload_complete()¶

URI: /data/s3_upload_complete/

Notify the system that an upload to S3 has been completed. This is a necessary step after uploading to S3 or the SEED instance will not be aware the file exists.

Valid source_type values are found in seed.models.SEED_DATA_SOURCES

GET:	Expects the following in the query string: key: The full path to the file, within the S3 bucket. E.g. data_importer/buildings.csv source_type: The source of the file. E.g. ‘Assessed Raw’ or ‘Portfolio Raw’ source_program: Optional value from common.mapper.Programs source_version: e.g. “4.1” import_record: The ID of the ImportRecord this file belongs to.

Returns:

{
    'success': True,
    'import_file_id': The ID of the newly-created ImportFile object.
}

seed.data_importer.views.local_uploader()¶

URI: /data/upload/

Endpoint to upload data files to, if uploading to local file storage. Valid source_type values are found in seed.models.SEED_DATA_SOURCES

GET:	The following parameters are expected to be in the query string: import_record: the ID of the ImportRecord to associate this file with. qqfile: The name of the file source_type: A valid source type (e.g. ‘Portfolio Raw’ or ‘Assessed Raw’)

Payload:

The content of the file as a data stream.  Do not use multipart encoding.

Returns:

{
    'success': True,
    'import_file_id': The ID of the newly-uploaded ImportFile
}

seed.data_importer.views.get_upload_details()¶

URI: /data/get_upload_details/

Retrieves details about how to upload files to this instance.

Returns:

If S3 mode:

{
    'upload_mode': 'S3',
    'upload_complete': A url to notify that upload is complete,
    'signature': The url to post file details to for auth to upload to S3.
}

If local file system mode:

{
    'upload_mode': 'filesystem',
    'upload_path': The url to POST files to (see local_uploader)
}

seed.data_importer.views.sign_policy_document()¶

URI: /data/sign_policy_document/

Sign and return the policy document for a simple upload. http://aws.amazon.com/articles/1434/#signyours3postform

Payload:

{
 "expiration": ISO-encoded timestamp for when signature should expire,
               e.g. "2014-07-16T00:20:56.277Z",
 "conditions":
     [
         {"acl":"private"},
         {"bucket": The name of the bucket from get_upload_details},
         {"Content-Type":"text/csv"},
         {"success_action_status":"200"},
         {"key": filename of upload, prefixed with 'data_imports/',
                 suffixed with a unique timestamp.
                 e.g. 'data_imports/my_buildings.csv.1405469756'},
         {"x-amz-meta-category":"data_imports"},
         {"x-amz-meta-qqfilename": original filename}
     ]
}

Returns:

{
    "policy": A hash of the policy document. Using during upload to S3.
    "signature": A signature of the policy document.  Also used during upload to S3.
}

SEED (Building and Project) Endpoints¶

seed.views.main.create_dataset()¶

URI: /app/create_dataset/

Creates a new empty dataset (ImportRecord).

Payload:

{
    "name": Name of new dataset, e.g. "2013 city compliance dataset"
    "organization_id": ID of the org this dataset belongs to
}

Returns:

{
    'status': 'success',
    'id': The ID of the newly-created ImportRecord,
    'name': The name of the newly-created ImportRecord
}

seed.views.main.create_pm_mapping()¶

URI: /app/create_pm_mapping/

Create a mapping for PortfolioManager input columns.

Payload:

{
    columns: [ "name1", "name2", ... , "nameN"],
}

Returns:

{
    success: true,
    mapping: [
        [
            "name1",
            "mapped1", {
                bedes: true|false,
                numeric: true|false
            }
        ],
        [
            "name2",
            "mapped2", {
                bedes: true|false,
                numeric: true|false
            }
        ],
        ...
        [
            "nameN",
            "mappedN", {
                bedes: true|false,
                numeric: true|false
            }
        ]
    ]
}
-- OR --
{
    success: false,
    reason: "message goes here"
}

seed.views.main.delete_buildings()¶

URI: /app/delete_buildings/

Deletes all BuildingSnapshots the user has selected.

Does not delete selected_buildings where the user is not a member or owner of the organization the selected building belongs. Since search shows buildings across all the orgs a user belongs, it’s possible for a building to belong to an org outside of org_id.

DELETE:	Expects ‘org_id’ for the organization, and the search payload similar to add_buildings/create_project

Payload:

{
    'organization_id': 2,
    'search_payload': {
        'selected_buildings': [2, 3, 4],
        'select_all_checkbox': False,
        'filter_params': ... // see search_buildings
    }
}

Returns:

{
    'status': 'success' or 'error'
}

seed.views.main.delete_dataset()¶

URI: /app/delete_dataset/

Deletes all files from a dataset and the dataset itself.

DELETE:	Expects organization id and dataset id.

Payload:

{
    "dataset_id": 1,
    "organization_id": 1
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'error message, if any'
}

seed.views.main.delete_duplicates_from_import_file()¶

URI: /app/delete_duplicates_from_import_file/

Retrieves the number of matched and unmatched BuildingSnapshots for a given ImportFile record.

GET:	Expects import_file_id corresponding to the ImportFile in question.

Returns:

{
    "status": "success",
    "deleted": "Number of duplicates deleted"
}

seed.views.main.delete_file()¶

URI: /app/delete_file/

Deletes an ImportFile from a dataset.

Payload:

{
    "file_id": "ImportFile id",
    "organization_id": "current user organization id as integer"
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'error message, if any'
}

seed.views.main.delete_organization()¶

URI: /app/delete_organization/

Starts a background task to delete an organization and all related data.

GET:	Expects ‘org_id’ for the organization.

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.delete_organization_buildings()¶

URI: /app/delete_organization_buildings/

Starts a background task to delete all BuildingSnapshots in an org.

GET:	Expects ‘org_id’ for the organization.

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.export_buildings()¶

URI: /app/export_buildings/

Begins a building export process.

Payload:

{
    "export_name": "My Export",
    "export_type": "csv",
    "selected_buildings": [1234,], (optional list of building ids)
    "selected_fields": optional list of fields to export
    "select_all_checkbox": True // optional, defaults to False
}

Returns:

{
    "success": True,
    "status": "success",
    "export_id": export_id; see export_buildings_download,
    "total_buildings": count of buildings,
}

seed.views.main.export_buildings_download()¶

URI: /app/export_buildings/download/

Provides the url to a building export file.

Payload:

{
    "export_id": export_id from export_buildings
}

Returns:

{
    'success': True or False,
    'status': 'success or error',
    'message': 'error message, if any',
    'url': The url to the exported file.
}

seed.views.main.export_buildings_progress()¶

URI: /app/export_buildings/progress/

Returns current progress on building export process.

Payload:

{
    "export_id": export_id from export_buildings
}

Returns:

{
    'success': True,
    'status': 'success or error',
    'message': 'error message, if any',
    'buildings_processed': number of buildings exported
}

seed.views.main.get_PM_filter_by_counts()¶

URI: /app/get_PM_filter_by_counts/

Retrieves the number of matched and unmatched BuildingSnapshots for a given ImportFile record.

GET:	Expects import_file_id corresponding to the ImportFile in question.

Returns:

{
    'status': 'success',
    'matched': Number of BuildingSnapshot objects that have matches,
    'unmatched': Number of BuildingSnapshot objects with no matches.
}

seed.views.main.get_aggregated_building_report_data()¶

URI: /app/get_aggregated_building_report_data/

This method returns a set of aggregated building data for graphing. It expects as parameters

GET:

Parameters:

start_date – The starting date for the data series with the format YYYY-MM-DDThh:mm:ss+hhmm end_date – The starting date for the data series with the format YYYY-MM-DDThh:mm:ss+hhmm x_var – The variable name to be assigned to the “x” value in the returned data series y_var – The variable name to be assigned to the “y” value in the returned data series organization_id – The organization to be used when querying data.

The x_var values should be from the following set of variable names:

• site_eui
• source_eui
• site_eui_weather_normalized
• source_eui_weather_normalized
• energy_score

The y_var values should be from the following set of variable names:

• gross_floor_area
• use_description
• year_built

This method includes building record count information as part of the result JSON in a property called “building_counts.”

This property provides data on the total number of buildings available in each ‘year ending’ group, as well as the subset of those buildings that have actual data to graph. By sending these values in the result we allow the client to easily build a message like “200 of 250 buildings in this group have data.”

Returns:

{
    "status": "success",
    "chart_data": [
        {
            "yr_e": x - group by year ending
            "x": x, - median value in group
            "y": y - average value thing
        },
        {
            "yr_e": x
            "x": x,
            "y": y
        }
        ...
    ],
    "building_counts": [
        {
            "yr_e": string for year ending - group by
            "num_buildings": number of buildings in query results
            "num_buildings_w_data": number of buildings with valid data in this group, BOTH x and y?  # NOQA
        },
        ...
    ]
    "num_buildings": total number of buildings in query results,
    "num_buildings_w_data": total number of buildings with valid data in query results
}

—

parameters:

• name: x_var description: Name of column in building snapshot database to be used for “x” axis required: true type: string paramType: query
• name: y_var description: Name of column in building snapshot database to be used for “y” axis required: true type: string paramType: query
• start_date: description: The start date for the entire dataset. required: true type: string paramType: query
• end_date: description: The end date for the entire dataset. required: true type: string paramType: query
• name: organization_id description: User’s organization which should be used to filter building query results required: true type: string paramType: query

type:

status:

required: true type: string

chart_data:

required: true type: array

building_counts:

required: true type: array

num_buildings:

required: true type: string

num_buildings_w_data:

required: true type: string

status code:

• code: 400 message: Bad request, only GET method is available
• code: 401 message: Not authenticated
• code: 403 message: Insufficient rights to call this procedure

seed.views.main.get_building()¶

URI: /app/get_building/

Retrieves a building. If user doesn’t belong to the building’s org, fields will be masked to only those shared within the parent org’s structure.

GET:	Expects building_id and organization_id in query string. building_id should be the canonical_building ID for the building, not the BuildingSnapshot id.

Returns:

{
     'status': 'success or error',
     'message': 'error message, if any',
     'building': {'id': the building's id,
                  'canonical_building': the canonical building ID,
                  other fields this user has access to...
     },
     'imported_buildings': [ A list of buildings imported to create
                             this building's record, in the same
                             format as 'building'
                           ],
     'projects': [
        // A list of the building's projects
        {
            "building": {
                "approved_date":07/30/2014,
                "compliant": null,
                "approver": "demo@seed.lbl.gov"
                "approved_date": "07/30/2014"
                "compliant": null
                "label": {
                    "color": "red",
                    "name": "non compliant",
                    id: 1
                }
            }
            "description": null
            "id": 3
            "is_compliance": false
            "last_modified_by_id": 1
            "name": "project 1"
            "owner_id": 1
            "slug": "project-1"
            "status": 1
            "super_organization_id": 1
        },
        . . .
    ],
     'user_role': role of user in this org,
     'user_org_id': the org id this user belongs to
}

seed.views.main.get_building_report_data()¶

URI: /app/get_building_report_data/

This method returns a set of x,y building data for graphing. It expects as parameters

GET:

Parameters:

start_date – The starting date for the data series with the format YYYY-MM-DD end_date – The starting date for the data series with the format YYYY-MM-DD x_var – The variable name to be assigned to the “x” value in the returned data series # NOQA y_var – The variable name to be assigned to the “y” value in the returned data series # NOQA organization_id – The organization to be used when querying data.

The x_var values should be from the following set of variable names:

• site_eui
• source_eui
• site_eui_weather_normalized
• source_eui_weather_normalized
• energy_score

The y_var values should be from the following set of variable names:

• gross_floor_area
• use_description
• year_built

This method includes building record count information as part of the result JSON in a property called “building_counts.”

This property provides data on the total number of buildings available in each ‘year ending’ group, as well as the subset of those buildings that have actual data to graph. By sending these values in the result we allow the client to easily build a message like “200 of 250 buildings in this group have data.”

Returns:

{
    "status": "success",
    "chart_data": [
        {
            "id" the id of the building,
            "yr_e": the year ending value for this data point
            "x": value for x var,
            "y": value for y var,
        },
        ...
    ],
    "building_counts": [
        {
            "yr_e": string for year ending
            "num_buildings": number of buildings in query results
            "num_buildings_w_data": number of buildings with valid data in query results
        },
        ...
    ]
    "num_buildings": total number of buildings in query results,
    "num_buildings_w_data": total number of buildings with valid data in the query results  # NOQA
}

—

parameters:

• name: x_var description: Name of column in building snapshot database to be used for “x” axis required: true type: string paramType: query
• name: y_var description: Name of column in building snapshot database to be used for “y” axis required: true type: string paramType: query
• start_date: description: The start date for the entire dataset. required: true type: string paramType: query
• end_date: description: The end date for the entire dataset. required: true type: string paramType: query
• name: organization_id description: User’s organization which should be used to filter building query results required: true type: string paramType: query
• name: aggregate description: Aggregates data based on internal rules (given x and y var) required: true type: string paramType: query

type:

status:

required: true type: string

chart_data:

required: true type: array

num_buildings:

required: true type: string

num_buildings_w_data:

required: true type: string

status codes:

• code: 400 message: Bad request, only GET method is available
• code: 401 message: Not authenticated
• code: 403 message: Insufficient rights to call this procedure

seed.views.main.get_building_summary_report_data()¶

URI: /app/get_building_summary_report_data/

This method returns basic, high-level data about a set of buildings, filtered by organization ID.

It expects as parameters

GET:
Parameters:	start_date – The starting date for the data series with the format YYYY-MM-DD end_date – The starting date for the data series with the format YYYY-MM-DD

Returns:

{
    "status": "success",
    "summary_data":
    {
        "num_buildings": "number of buildings returned from query",
        "avg_eui": "average EUI for returned buildings",
        "avg_energy_score": "average energy score for returned buildings"
    }
}

Units for return values are as follows:

property	units
avg_eui	kBtu/ft2/yr

—

parameters:

• name: organization_id description: User’s organization which should be used to filter building query results required: true type: string paramType: query
• start_date: description: The start date for the entire dataset. required: true type: string paramType: query
• end_date: description: The end date for the entire dataset. required: true type: string paramType: query

type:

status:

required: true type: string

summary_data:

required: true type: object

status codes:

• code: 400 message: Bad request, only GET method is available
• code: 401 message: Not authenticated
• code: 403 message: Insufficient rights to call this procedure

seed.views.main.get_column_mapping_suggestions()¶

URI: /app/get_column_mapping_suggestions/

Returns suggested mappings from an uploaded file’s headers to known data fields.

Payload:

{
    'import_file_id': The ID of the ImportRecord to examine,
    'org_id': The ID of the user's organization
}

Returns:

{
    'status': 'success',
    'suggested_column_mappings': {
        column header from file: [ (destination_column, score) ...]
        ...
    },
    'building_columns': [ a list of all possible columns ],
    'building_column_types': [a list of column types corresponding to building_columns],
}

..todo: The response of this method may not be correct. verify.

seed.views.main.get_coparents()¶

URI: /app/get_coparents/

Returns the nodes in the BuildingSnapshot tree that can be unmatched.

GET:	Expects organization_id and building_id in the query string

Returns:

{
    'status': 'success',
    'coparents': [
        {
            "id": 333,
            "coparent": 223,
            "child": 443,
            "parents": [],
            "canonical_building_id": 1123
        },
        {
            "id": 223,
            "coparent": 333,
            "child": 443,
            "parents": [],
            "canonical_building_id": 1124
        },
        ...
    ]
}

seed.views.main.get_dataset()¶

URI: /app/get_dataset/

Retrieves ImportFile objects for one ImportRecord.

GET:	Expects dataset_id for an ImportRecord in the query string.

Returns:

{
    'status': 'success',
    'dataset': {
        'name': Name of ImportRecord,
        'number_of_buildings': Total number of buildings in all ImportFiles for this dataset,
        'id': ID of ImportRecord,
        'updated_at': Timestamp of when ImportRecord was last modified,
        'last_modified_by': Email address of user making last change,
        'importfiles': [
            {
               'name': Name of associated ImportFile, e.g. 'buildings.csv',
               'number_of_buildings': Count of buildings in this file,
               'number_of_mappings': Number of mapped headers to fields,
               'number_of_cleanings': Number of fields cleaned,
               'source_type': Type of file (see source_types),
               'id': ID of ImportFile (needed for most operations)
            }
         ],
         ...
    },
        ...
}

seed.views.main.get_datasets()¶

URI: /app/get_datasets/

Retrieves all datasets for the user’s organization.

GET:	Expects ‘organization_id’ of org to retrieve datasets from in query string.

Returns:

{
    'status': 'success',
    'datasets':  [
        {
            'name': Name of ImportRecord,
            'number_of_buildings': Total number of buildings in all ImportFiles,
            'id': ID of ImportRecord,
            'updated_at': Timestamp of when ImportRecord was last modified,
            'last_modified_by': Email address of user making last change,
            'importfiles': [
                {
                    'name': Name of associated ImportFile, e.g. 'buildings.csv',
                    'number_of_buildings': Count of buildings in this file,
                    'number_of_mappings': Number of mapped headers to fields,
                    'number_of_cleanings': Number of fields cleaned,
                    'source_type': Type of file (see source_types),
                    'id': ID of ImportFile (needed for most operations)
                }
            ],
            ...
        },
        ...
    ]
}

seed.views.main.get_datasets_count()¶

URI: /app/get_datasets_count/

Retrieves the number of datasets for an org.

GET:	Expects organization_id in the query string.

Returns:

{
    'status': 'success',
    'datasets_count': Number of datasets belonging to this org.
}

seed.views.main.get_first_five_rows()¶

URI: /app/get_first_five_rows/

Retrieves the first five rows of an ImportFile.

Payload:

{
    'import_file_id': The ID of the ImportFile
}

Returns:

{
    'status': 'success',
    'first_five_rows': [
        [list of strings of header row],
        [list of strings of first data row],
        ...
        [list of strings of fifth data row]
    ]
}

seed.views.main.get_import_file()¶

URI: /app/get_import_file/

Retrieves details about an ImportFile.

GET:	Expects import_file_id in the query string.

Returns:

{
    'status': 'success',
    'import_file': {
        "name": Name of the uploaded file,
        "number_of_buildings": number of buildings in the file,
        "number_of_mappings": number of mapped columns,
        "number_of_cleanings": number of cleaned fields,
        "source_type": type of data in file, e.g. 'Assessed Raw'
        "number_of_matchings": Number of matched buildings in file,
        "id": ImportFile ID,
        'dataset': {
            'name': Name of ImportRecord file belongs to,
            'id': ID of ImportRecord file belongs to,
            'importfiles': [  # All ImportFiles in this ImportRecord, with
                # requested ImportFile first:
                {'name': Name of file,
                 'id': ID of ImportFile
                }
                ...
            ]
        }
    }
}

seed.views.main.get_match_tree()¶

URI: /app/get_match_tree/

returns the BuildingSnapshot tree

GET:	Expects organization_id and building_id in the query string

Returns:

{
    'status': 'success',
    'match_tree': [ // array of all the members of the tree
        {
            "id": 333,
            "coparent": 223,
            "child": 443,
            "parents": [],
            "canonical_building_id": 1123
        },
        {
            "id": 223,
            "coparent": 333,
            "child": 443,
            "parents": [],
            "canonical_building_id": 1124
        },
        {
            "id": 443,
            "coparent": null,
            "child": 9933,
            "parents": [333, 223],
            "canonical_building_id": 1123
        },
        {
            "id": 9933,
            "coparent": null,
            "child": null,
            "parents": [443],
            "canonical_building_id": 1123
        },
        ...
    ]
}

seed.views.main.get_raw_column_names()¶

URI: /app/get_raw_column_names/

Retrieves a list of all column names from an ImportFile.

Payload:

{
    'import_file_id': The ID of the ImportFile
}

Returns:

{
    'status': 'success',
    'raw_columns': [
        list of strings of the header row of the ImportFile
    ]
}

seed.views.main.progress()¶

URI: /app/progress/

Get the progress (percent complete) for a task.

Payload:

{
    'progress_key': The progress key from starting a background task
}

Returns:

{
    'progress_key': The same progress key,
    'progress': Percent completion
}

seed.views.main.remap_buildings()¶

URI: /app/remap_buildings/

Re-run the background task to remap buildings as if it hadn’t happened at all. Deletes mapped buildings for a given ImportRecord, resets status.

NB: will not work if buildings have been merged into CanonicalBuilings.

Payload:

{
    'file_id': The ID of the ImportFile to be remapped
}

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.save_column_mappings()¶

URI: /app/save_column_mappings/

Saves the mappings between the raw headers of an ImportFile and the destination fields in the BuildingSnapshot model.

Valid source_type values are found in seed.models.SEED_DATA_SOURCES

Payload:

{
    "import_file_id": ID of the ImportFile record,
    "mappings": [
        ["destination_field": "raw_field"],  #direct mapping
        ["destination_field2":
            ["raw_field1", "raw_field2"],  #concatenated mapping
        ...
    ]
}

Returns:

{'status': 'success'}

seed.views.main.save_match()¶

URI: /app/save_match/

Adds or removes a match between two BuildingSnapshots. Creating a match creates a new BuildingSnapshot with merged data.

Payload:

{
    'organization_id': current user organization id,
    'source_building_id': ID of first BuildingSnapshot,
    'target_building_id': ID of second BuildingSnapshot,
    'create_match': True to create match, False to remove it,
    'organization_id': ID of user's organization
}

Returns:

{
    'status': 'success',
    'child_id': The ID of the newly-created BuildingSnapshot
                containing merged data from the two parents.
}

seed.views.main.save_raw_data()¶

URI: /app/save_raw_data/

Starts a background task to import raw data from an ImportFile into BuildingSnapshot objects.

Payload:

{
    'file_id': The ID of the ImportFile to be saved
}

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.search_building_snapshots()¶

URI: /app/search_building_snapshots/

Retrieves a paginated list of BuildingSnapshots matching search params.

Payload:

{
    'q': a string to search on (optional),
    'order_by': which field to order by (e.g. pm_property_id),
    'import_file_id': ID of an import to limit search to,
    'filter_params': {
        a hash of Django-like filter parameters to limit query.  See seed.search.filter_other_params.
    }
    'page': Which page of results to retrieve (default: 1),
    'number_per_page': Number of buildings to retrieve per page (default: 10),
}

Returns:

{
    'status': 'success',
    'buildings': [
        {
            'pm_property_id': ID of building (from Portfolio Manager),
            'address_line_1': First line of building's address,
            'property_name': Building's name, if any
        }...
    ]
    'number_matching_search': Total number of buildings matching search,
    'number_returned': Number of buildings returned for this page
}

seed.views.main.search_buildings()¶

URI: /app/search_buildings/

Retrieves a paginated list of CanonicalBuildings matching search params.

Payload:

{
    'q': a string to search on (optional),
    'show_shared_buildings': True to include buildings from other orgs in this user's org tree,
    'order_by': which field to order by (e.g. pm_property_id),
    'import_file_id': ID of an import to limit search to,
    'filter_params': {
        a hash of Django-like filter parameters to limit query.  See seed.search.filter_other_params.
        If 'project__slug' is included and set to a project's slug, buildings will include associated labels
        for that project.
    }
    'page': Which page of results to retrieve (default: 1),
    'number_per_page': Number of buildings to retrieve per page (default: 10),
}

Returns:

{
    'status': 'success',
    'buildings': [
        {
            all fields for buildings the request user has access to, e.g.:
                'canonical_building': the CanonicalBuilding ID of the building,
                'pm_property_id': ID of building (from Portfolio Manager),
                'address_line_1': First line of building's address,
                'property_name': Building's name, if any
            ...
        }...
    ]
    'number_matching_search': Total number of buildings matching search,
    'number_returned': Number of buildings returned for this page
}

seed.views.main.start_mapping()¶

URI: /app/start_mapping/

Starts a background task to convert imported raw data into BuildingSnapshots, using user’s column mappings.

Payload:

{
    'file_id': The ID of the ImportFile to be mapped
}

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.start_system_matching()¶

URI: /app/start_system_matching/

Starts a background task to attempt automatic matching between buildings in an ImportFile with other existing buildings within the same org.

Payload:

{
    'file_id': The ID of the ImportFile to be matched
}

Returns:

{
    'status': 'success' or 'error',
    'progress_key': ID of background job, for retrieving job progress
}

seed.views.main.update_building()¶

URI: /app/update_building/

Manually updates a building’s record. Creates a new BuildingSnapshot for the resulting changes.

PUT:

Payload:

{
    "organization_id": "organization id as integer",
    "building":
        {
            "canonical_building": "canonical building ID as integer"
            "fieldname": "value",
            "...": "Remaining fields in the BuildingSnapshot; see get_columns() endpoint for complete list."
        }
}

Returns:

{
    "status": "success",
    "child_id": "The ID of the newly-created BuildingSnapshot"
}

seed.views.main.update_dataset()¶

URI: /app/update_dataset/

Updates the name of a dataset.

Payload:

{
    'dataset': {
        'id': The ID of the Import Record,
        'name': The new name for the ImportRecord
    }
}

Returns:

{
    'status': 'success' or 'error',
    'message': 'error message, if any'
}

seed.views.main.version()¶

URI: /app/version/

Returns the SEED version and current git sha

parents and children¶

BuildingSnapshots also have linkage to other BuildingSnapshots in order to keep track of their parents and children. This is represented in the Django model as a many-to-many relation from BuildingSnapshot to BuildingSnapshot. It is represented in the PostgreSQL database as an additional seed_buildingsnapshot_children table.

In our case here, BS0 and BS1 would both have children BS2, and BS2 would have parents BS0 and BS1.

For illustration purposes let’s suppose BS2 and a new building BS3 match to form a child BS4.

And the corresponding tree would look like:

BS0  BS1
  \ /
  BS2  BS3
    \  /
     BS4 <-- CB0

BS0 --> CB0
BS1 --> CB0
BS2 --> CB0
BS3 --> CB0
BS4 --> CB0

manual-matching vs auto-matching¶

Since BuildingSnapshots can be manually matched, there is the possibility for two BuildingSnapshots each with an active CanonicalBuilding to match and the system has to choose to move only one CanonicalBuilding to the tip of the tree for the primary BuildingSnapshot and deactivate the secondary BuildingSnapshot’s CanonicalBuilding.

Take for example:

BS0  BS1
  \ /
  BS2  BS3
    \  /
     BS4 <-- CB0 (active: True)         BS5 <-- CB1 (active: True)

If a user decides to manually match BS4 and BS5, the system will take the primary BuildingSnapshot’s CanonicalBuilding and have it point to their child and deactivate CB1. The deactivation is handled by setting a field on the CanonicalBuilding instance, active, from True to False.

Here is what the tree would look like after the manual match of BS4 and BS5:

BS0  BS1
  \ /
  BS2  BS3
    \  /
     BS4  BS5 <-- CB1 (active: False)
       \  /
        BS6 <-- CB0 (active: True)

Even though BS5 is pointed to by a CanonicalBuilding, CB1, BS5 will not be returned by the normal search_buildings endpoint because the CanonicalBuilding pointing to it has its field active set to False.

what really happens to the BuildingSnapshot table on import (and when)¶

The above is conceptually what happens but sometimes the devil is in the details. Here is what happens to the BuildingSnapshot table in the database when records are imported.

Every time a record is added at least two BuildingSnapshot records are created.

Consider the following simple record:

The first thing the user is upload the file. When the user sees the “Successful Upload!” dialog one record has been added to the BuildingSnapshot table.

This new record has an id (73700 in this case) and a created and modified timestamp. Then there are a lot of empty fields and a source_type of 0. Then there is the extra_data column which contains the contents of the record in key-value form:

And a corresponding extra_data_sources that looks like

All of the fields that look like _source_id are also populated with 73700 E.G. owner_postal_code_source_id.

The other fields of interest are the super_organization field which is populated with the user’s default organization and the import_file_id field which is populated with a reference to a data_importer_importfile record.

At this point the record has been created before the user hits the “Continue to data mapping” button.

The second record (id = 73701) is created by the time the user gets to the screen with the “Save Mappings” button. This second record has the following fields populated:

• id
• created
• modified
• pm_property_id
• year_ending
• gross_floor_area
• address_line_1
• release_date
• source_type (this is 2 instead of 0 as with the other record)
• import_file_id
• super_organization_id.

That is all. All other fields are empty. In this case that is all that happens.

Now consider the same user uploading a new file from the next year that looks like

As before one new record is created on upload. This has id 73702 and follows the same pattern as 73700. And similarly 73703 is created like 73701 before the “Save Mappings” button appears.

However this time the system was able to make a match with an existing record. After the user clicks the “Confirm mappings & start matching” button a new record is created with ID 73704.

73704 is identical to 73703 (in terms of contents of the BuildingSnapshot table only) with the following exceptions:

• created and modified timestamps are different
• match type is populated and has a value of 1
• confidence is populated and has a value of .9
• source_type is 4 instead of 2
• canonical_building_id is populated with a value
• import_file_id is NULL
• last_modified_by_id is populated with value 2 (This is a key into the landing_seeduser table)
• address_line_1_source_id is 73701
• gross_floor_area_source_id is populated with value 73701
• pm_property_id_source_id is populated with 73701
• release_date_source_id is populated with 73701
• year_ending_source_id is populated with 73701

what really happens to the CanonicalBuilding table on import (and when)¶

In addition to the BuildingSnapshot table the CanonicalBuilding table is also updated during the import process. To summarize the above 5 records were created in the BuildingSnapshot table:

1. 73700 is created from the raw 2000 data
2. 73701 is the mapped 2000 data,
3. 73702 is created from the raw 2001 data
4. 73703 is the mapped 2001 data
5. 73704 is the result of merging the 2000 and 2001 data.

In this process CanonicalBuilding is updated twice. First when the 2000 record is imported the CanonicalBuilding gets populated with one new row at the end of the matching step. I.E. when the user sees the “Load More Data” screen. At this point there is a new row that looks like

At this point there is one new canonical building and that is the BuildingSnapshot with id 73701. Next the user uploads the 2001 data. When the “Matching Results” screen appears the CanonicalBuilding table has been updated. Now it looks like

There is still only one canonical building but now it is the BuildingSnapshot record that is the result of merging the 2000 and 2001 data: id = 73704.

organization¶

BuildingSnapshots belong to an Organization by virtue of a super_organization field that is a foreign key into the organization model (orgs_organization in Postgres). Many endpoints filter the buildings based on the organizations the requesting user belongs to. E.G. get_buildings changes which fields are returned based on the requesting user’s membership in the BuildingSnapshot’s organization.

*_source_id fields¶

Any field in the BuildingSnapshot table that is populated with data from a submitted record will have a corresponding _source_id field. E.G pm_property_id has pm_property_id_source_id, address_line_1 has address_line_1_source_id, etc...

These are foreign keys into the BuildingSnapshot that is the source of that value. To extend the above table

NOTE: The BuildingSnapshot records made from the raw input file have all the _source_id fields populated with that record’s ID. The non-canonical BuildingSnapshot records created from the mapped data have none set. The canonical BuildingSnapshot records that are the result of merging two records have only the _source_id fields set where the record itself has data. E.G. in the above address_line_1 is set to “1 fake st.” so there is a value in the canonical BuildingSnapshot’s address_line_1_source_id field. However there is no block number so block_number_source_id is empty. This is unlike the two raw BuildingSnapshot records who also have no block_number but nevertheless have a block_number_source_id populated.

extra_data¶

The BuildingSnapshot model has many “named” fields. Fields like “address_line_1”, “year_built”, and “pm_property_id”. However the users are allowed to submit files with arbitrary fields. Some of those arbitrary fields can be mapped to “named” fields. E.G. “Street Address” can usually be mapped to “Address Line 1”. For all the fields that cannot be mapped like that there is the extra_data field.

extra_data is Django json field that serves as key-value storage for other user-submitted fields. As with the other “named” fields there is a corresponding extra_data_sources field that serves the same role as the other _source_id fields. E.G. If a BuildingSnapshot has an extra_data field that looks like

It should have an extra_data_sources field that looks like

saving and possible data loss¶

When saving a BuildingSnapshot file some fields that are truncated if too long. The following are truncated to 128 characters

• tax_lot_id
• pm_property_id
• custom_id_1
• lot_number
• block_number
• district
• owner
• owner_email
• owner_telephone
• owner_address
• owner_city_state
• owner_postal_code

And the following are truncated to 255:

• property_name
• address_line_1
• address_line_2
• city
• postal_code
• state_province
• building_certification

No truncation happens to any of the fields stored in extra_data.

Import¶

From the web UI, the import process invokes seed.views.main.save_raw_data to save the data. When the data is done uploading, we need to know whether it is a Portfolio Manager file, so we can add metadata to the record in the database. The end of the upload happens in seed.data_importer.views.DataImportBackend.upload_complete or seed.data_importer.views.handle_s3_upload_complete, depending on whether it is using a local file system or Amazon S3-based backend. At this point, the request object has additional attributes for Portfolio Manager files. These are saved in the model seed.data_importer.models.ImportFile.

Mapping¶

After the data is saved, the UI invokes seed.views.main.get_column_mapping_suggestions to get the columns to display on the mapping screen. This loads back the model that was mentioned above as an ImportFile instance, and then the from_portfolio_manager property can be used to choose the branch of the code:

If it is a Portfolio Manager file the seed.common.mapper.get_pm_mapping method provides a high-level interface to the Portfolio Manager mapping (see comments in the containing file, mapper.py), and the result is used to populate the return value for this method, which goes back to the UI to display the mapping screen.

Otherwise the code does some auto-magical logic to try and infer the “correct” mapping.