Data Importer Package¶

Submodules¶

Managers¶

class seed.data_importer.managers.NotDeletedManager¶

Bases: django.db.models.manager.Manager

get_queryset(*args, **kwargs)¶

use_for_related_fields = True¶

Models¶

URLs¶

Utils¶

class seed.data_importer.utils.CoercionRobot¶

Bases: object

lookup_hash(uncoerced_value, destination_model, destination_field)¶

make_key(value, model, field)¶

seed.data_importer.utils.acquire_lock(name, expiration=None)¶

Tries to acquire a lock from the cache. Also sets the lock’s value to the current time, allowing us to see how long it has been held.

Returns False if lock already belongs by another process.

seed.data_importer.utils.chunk_iterable(iter, chunk_size)¶: Breaks an iterable (e.g. list) into smaller chunks, returning a generator of the chunk.

seed.data_importer.utils.get_core_pk_column(table_column_mappings, primary_field)¶

seed.data_importer.utils.get_lock_time(name)¶: Examines a lock to see when it was acquired.

seed.data_importer.utils.release_lock(name)¶: Frees a lock.

Views¶

class seed.data_importer.views.ImportFileViewSet(**kwargs)

Bases: rest_framework.viewsets.ViewSet

authentication_classes = (<class 'rest_framework.authentication.SessionAuthentication'>, <class 'seed.authentication.SEEDAuthentication'>)

cleansing_progress(request, *args, **kwargs)

Return the progress of the cleansing. — type:

status:

required: true type: string description: either success or error

progress:

type: integer description: status of background cleansing task

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

filtered_mapping_results(request, *args, **kwargs)

Retrieves a paginated list of Properties and Tax Lots for an import file after mapping. — parameter_strategy: replace parameters:

name: pk description: Import File ID (Primary key) type: integer required: true paramType: path

response_serializer: MappingResultsResponseSerializer

first_five_rows(request, *args, **kwargs)

Retrieves the first five rows of an ImportFile. — type:

status:

required: true type: string description: either success or error

first_five_rows:

type: array of strings description: list of strings for each of the first five rows for this import file

parameter_strategy: replace parameters:

name: pk description: “Primary Key” required: true paramType: path

get_cleansing_results(request, *args, **kwargs)

Retrieve the details of the cleansing script. — type:

status:

required: true type: string description: either success or error

message:

type: string description: additional information, if any

progress:

type: integer description: integer percent of completion

data:

type: JSON description: object describing the results of the cleansing

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

get_csv(request, *args, **kwargs)

Download a csv of the results. — type:

status:

required: true type: string description: either success or error

progress_key:

type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

mapping_done(request, *args, **kwargs)

Tell the backend that the mapping is complete. — type:

status:

required: true type: string description: either success or error

message:

required: false type: string description: error message, if any

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

matching_results(request, *args, **kwargs)

Retrieves the number of matched and unmatched BuildingSnapshots for a given ImportFile record.

GET:	Expects import_file_id corresponding to the ImportFile in question.

Returns:

{
    'status': 'success',
    'matched': Number of BuildingSnapshot objects that have matches,
    'unmatched': Number of BuildingSnapshot objects with no matches.
}

perform_mapping(request, *args, **kwargs)

Starts a background task to convert imported raw data into PropertyState and TaxLotState, using user’s column mappings. — type:

status:

required: true type: string description: either success or error

progress_key:

type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

raise_exception = True

raw_column_names(request, *args, **kwargs)

Retrieves a list of all column names from an ImportFile. — type:

status:

required: true type: string description: either success or error

raw_columns:

type: array of strings description: list of strings of the header row of the ImportFile

parameter_strategy: replace parameters:

name: pk description: “Primary Key” required: true paramType: path

retrieve(request, *args, **kwargs)

Retrieves details about an ImportFile. — type:

status:

required: true type: string description: either success or error

import_file:

type: ImportFile structure description: full detail of import file

parameter_strategy: replace parameters:

name: pk description: “Primary Key” required: true paramType: path

save_column_mappings(request, *args, **kwargs)

Saves the mappings between the raw headers of an ImportFile and the destination fields in the to_table_name model which should be either PropertyState or TaxLotState

Valid source_type values are found in seed.models.SEED_DATA_SOURCES

Payload:

{
    "import_file_id": ID of the ImportFile record,
    "mappings": [
        {
            'from_field': 'eui',  # raw field in import file
            'to_field': 'energy_use_intensity',
            'to_table_name': 'PropertyState',
        },
        {
            'from_field': 'gfa',
            'to_field': 'gross_floor_area',
            'to_table_name': 'PropertyState',
        }
    ]
}

Returns:

{'status': 'success'}

save_raw_data(request, *args, **kwargs)

Starts a background task to import raw data from an ImportFile into PropertyState objects as extra_data. If the cycle_id is set to year_ending then the cycle ID will be set to the year_ending column for each record in the uploaded file. Note that the year_ending flag is not yet enabled. — type:

status:

required: true type: string description: either success or error

message:

required: false type: string description: error message, if any

progress_key:

type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

name: cycle_id description: The ID of the cycle or the string “year_ending” paramType: string required: true

start_system_matching(request, *args, **kwargs)

Starts a background task to attempt automatic matching between buildings in an ImportFile with other existing buildings within the same org. — type:

status:

required: true type: string description: either success or error

progress_key:

type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

name: pk description: Import file ID required: true paramType: path

suffix = None

class seed.data_importer.views.LocalUploaderViewSet(**kwargs)

Bases: rest_framework.viewsets.GenericViewSet

Endpoint to upload data files to, if uploading to local file storage. Valid source_type values are found in seed.models.SEED_DATA_SOURCES

Returns:

{
    'success': True,
    'import_file_id': The ID of the newly-uploaded ImportFile
}

create(request, *args, **kwargs)

Upload a new file to an import_record. This is a multipart/form upload. — parameters:

name: import_record description: the ID of the ImportRecord to associate this file with. required: true paramType: body

name: source_type description: the type of file (e.g. ‘Portfolio Raw’ or ‘Assessed Raw’) required: false paramType: body

name: source_program_version description: the version of the file as related to the source_type required: false paramType: body

name: file or qqfile description: In-memory file object required: true paramType: Multipart

suffix = None

class seed.data_importer.views.MappingResultsPayloadSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

filter_params = <django.contrib.postgres.fields.jsonb.JSONField>

class seed.data_importer.views.MappingResultsPropertySerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs): Bases: rest_framework.serializers.Serializer

class seed.data_importer.views.MappingResultsResponseSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs): Bases: rest_framework.serializers.Serializer

class seed.data_importer.views.MappingResultsTaxLotSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs): Bases: rest_framework.serializers.Serializer

Data Importer Package¶

Submodules¶

Managers¶

Models¶

URLs¶

Utils¶

Views¶

Module contents¶