Data Importer Package

Submodules

Managers

class seed.data_importer.managers.NotDeletedManager

Bases: django.db.models.manager.Manager

get_queryset(*args, **kwargs)

Models

URLs

Utils

class seed.data_importer.utils.CoercionRobot

Bases: object

lookup_hash(uncoerced_value, destination_model, destination_field)
make_key(value, model, field)
seed.data_importer.utils.acquire_lock(name, expiration=None)

Tries to acquire a lock from the cache. Also sets the lock’s value to the current time, allowing us to see how long it has been held.

Returns False if lock already belongs by another process.

seed.data_importer.utils.chunk_iterable(iterlist, chunk_size)

Breaks an iterable (e.g. list) into smaller chunks, returning a generator of the chunk.

seed.data_importer.utils.get_core_pk_column(table_column_mappings, primary_field)
seed.data_importer.utils.get_lock_time(name)

Examines a lock to see when it was acquired.

seed.data_importer.utils.release_lock(name)

Frees a lock.

Views

class seed.data_importer.views.ImportFileViewSet(**kwargs)

Bases: rest_framework.viewsets.ViewSet

authentication_classes = (<class ‘rest_framework.authentication.SessionAuthentication’>, <class ‘seed.authentication.SEEDAuthentication’>)
available_matches(request, *args, **kwargs)
data_quality_progress(request, *args, **kwargs)

Return the progress of the data quality check. — type:

status:
required: true type: string description: either success or error
progress:
type: integer description: status of background data quality task

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
filtered_mapping_results(request, *args, **kwargs)

Retrieves a paginated list of Properties and Tax Lots for an import file after mapping. — parameter_strategy: replace parameters:

  • name: pk description: Import File ID (Primary key) type: integer required: true paramType: path

response_serializer: MappingResultsResponseSerializer

first_five_rows(request, *args, **kwargs)

Retrieves the first five rows of an ImportFile. — type:

status:
required: true type: string description: either success or error
first_five_rows:
type: array of strings description: list of strings for each of the first five rows for this import file

parameter_strategy: replace parameters:

  • name: pk description: “Primary Key” required: true paramType: path
get_csv(request, *args, **kwargs)

Download a csv of the results. — type:

status:
required: true type: string description: either success or error
progress_key:
type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
get_data_quality_results(request, *args, **kwargs)

Retrieve the details of the data quality check. — type:

status:
required: true type: string description: either success or error
message:
type: string description: additional information, if any
progress:
type: integer description: integer percent of completion
data:
type: JSON description: object describing the results of the data quality check

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
static has_coparent(state_id, inventory_type, fields=None)

Return the coparent of the current state id based on the inventory type. If fields are given (as a list), then it will only return the fields specified of the state model object as a dictionary.

Parameters:
  • state_id – int, ID of PropertyState or TaxLotState
  • inventory_type – string, either properties | taxlots
  • fields – list, either None or list of fields to return
Returns:

dict or state object, If fields is not None then will return state_object

mapping_done(request, *args, **kwargs)

Tell the backend that the mapping is complete. — type:

status:
required: true type: string description: either success or error
message:
required: false type: string description: error message, if any

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
mapping_suggestions(request, *args, **kwargs)

Returns suggested mappings from an uploaded file’s headers to known data fields. — type:

status:
required: true type: string description: Either success or error
suggested_column_mappings:

required: true type: dictionary description: Dictionary where (key, value) = (the column header from the file,

array of tuples (destination column, score))
building_columns:
required: true type: array description: A list of all possible columns
building_column_types:
required: true type: array description: A list of column types corresponding to the building_columns array

parameter_strategy: replace parameters:

  • name: pk description: import_file_id required: true paramType: path
  • name: organization_id description: The organization_id for this user’s organization required: true paramType: query
match(request, *args, **kwargs)
matching_results(request, *args, **kwargs)

Retrieves the number of matched and unmatched properties & tax lots for a given ImportFile record. Specifically for new imports

GET:Expects import_file_id corresponding to the ImportFile in question.

Returns:

{
    'status': 'success',
    'properties': {
        'matched': Number of PropertyStates that have been matched,
        'unmatched': Number of PropertyStates that are unmatched new imports
    },
    'tax_lots': {
        'matched': Number of TaxLotStates that have been matched,
        'unmatched': Number of TaxLotStates that are unmatched new imports
    }
}
matching_status(request, *args, **kwargs)

Retrieves the number and ids of matched and unmatched properties & tax lots for a given ImportFile record. Specifically for hand-matching

GET:Expects import_file_id corresponding to the ImportFile in question.

Returns:

{
    'status': 'success',
    'properties': {
        'matched': Number of PropertyStates that have been matched,
        'matched_ids': Array of matched PropertyState ids,
        'unmatched': Number of PropertyStates that are unmatched records,
        'unmatched_ids': Array of unmatched PropertyState ids
    },
    'tax_lots': {
        'matched': Number of TaxLotStates that have been matched,
        'matched_ids': Array of matched TaxLotState ids,
        'unmatched': Number of TaxLotStates that are unmatched records,
        'unmatched_ids': Array of unmatched TaxLotState ids
    }
}
perform_mapping(request, *args, **kwargs)

Starts a background task to convert imported raw data into PropertyState and TaxLotState, using user’s column mappings. — type:

status:
required: true type: string description: either success or error
progress_key:
type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
raise_exception = True
raw_column_names(request, *args, **kwargs)

Retrieves a list of all column names from an ImportFile. — type:

status:
required: true type: string description: either success or error
raw_columns:
type: array of strings description: list of strings of the header row of the ImportFile

parameter_strategy: replace parameters:

  • name: pk description: “Primary Key” required: true paramType: path
retrieve(request, *args, **kwargs)

Retrieves details about an ImportFile. — type:

status:
required: true type: string description: either success or error
import_file:
type: ImportFile structure description: full detail of import file

parameter_strategy: replace parameters:

  • name: pk description: “Primary Key” required: true paramType: path
save_column_mappings(request, *args, **kwargs)

Saves the mappings between the raw headers of an ImportFile and the destination fields in the to_table_name model which should be either PropertyState or TaxLotState

Valid source_type values are found in seed.models.SEED_DATA_SOURCES

Payload:

{
    "import_file_id": ID of the ImportFile record,
    "mappings": [
        {
            'from_field': 'eui',  # raw field in import file
            'to_field': 'energy_use_intensity',
            'to_table_name': 'PropertyState',
        },
        {
            'from_field': 'gfa',
            'to_field': 'gross_floor_area',
            'to_table_name': 'PropertyState',
        }
    ]
}

Returns:

{'status': 'success'}
save_raw_data(request, *args, **kwargs)

Starts a background task to import raw data from an ImportFile into PropertyState objects as extra_data. If the cycle_id is set to year_ending then the cycle ID will be set to the year_ending column for each record in the uploaded file. Note that the year_ending flag is not yet enabled. — type:

status:
required: true type: string description: either success or error
message:
required: false type: string description: error message, if any
progress_key:
type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
  • name: cycle_id description: The ID of the cycle or the string “year_ending” paramType: string required: true
start_system_matching(request, *args, **kwargs)

Starts a background task to attempt automatic matching between buildings in an ImportFile with other existing buildings within the same org. — type:

status:
required: true type: string description: either success or error
progress_key:
type: integer description: ID of background job, for retrieving job progress

parameter_strategy: replace parameters:

  • name: pk description: Import file ID required: true paramType: path
suffix = None
unmatch(request, *args, **kwargs)
class seed.data_importer.views.LocalUploaderViewSet(**kwargs)

Bases: rest_framework.viewsets.GenericViewSet

Endpoint to upload data files to, if uploading to local file storage. Valid source_type values are found in seed.models.SEED_DATA_SOURCES

Returns:

{
    'success': True,
    'import_file_id': The ID of the newly-uploaded ImportFile
}
create(request, *args, **kwargs)

Upload a new file to an import_record. This is a multipart/form upload. — parameters:

  • name: import_record description: the ID of the ImportRecord to associate this file with. required: true paramType: body
  • name: source_type description: the type of file (e.g. ‘Portfolio Raw’ or ‘Assessed Raw’) required: false paramType: body
  • name: source_program_version description: the version of the file as related to the source_type required: false paramType: body
  • name: file or qqfile description: In-memory file object required: true paramType: Multipart
suffix = None
class seed.data_importer.views.MappingResultsPayloadSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

filter_params = <django.contrib.postgres.fields.jsonb.JSONField>
class seed.data_importer.views.MappingResultsPropertySerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

class seed.data_importer.views.MappingResultsResponseSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

class seed.data_importer.views.MappingResultsTaxLotSerializer(instance=None, data=<class rest_framework.fields.empty>, **kwargs)

Bases: rest_framework.serializers.Serializer

Module contents