Data Importer Package¶
Submodules¶
Managers¶
Models¶
URLs¶
Utils¶
-
class
seed.data_importer.utils.
CoercionRobot
¶ Bases:
object
-
lookup_hash
(uncoerced_value, destination_model, destination_field)¶
-
make_key
(value, model, field)¶
-
-
seed.data_importer.utils.
acquire_lock
(name, expiration=None)¶ Tries to acquire a lock from the cache. Also sets the lock’s value to the current time, allowing us to see how long it has been held.
Returns False if lock already belongs by another process.
-
seed.data_importer.utils.
chunk_iterable
(iter, chunk_size)¶ Breaks an iterable (e.g. list) into smaller chunks, returning a generator of the chunk.
-
seed.data_importer.utils.
get_core_pk_column
(table_column_mappings, primary_field)¶
-
seed.data_importer.utils.
get_lock_time
(name)¶ Examines a lock to see when it was acquired.
-
seed.data_importer.utils.
release_lock
(name)¶ Frees a lock.
Views¶
-
class
seed.data_importer.views.
ImportFileViewSet
(**kwargs) Bases:
rest_framework.viewsets.ViewSet
-
authentication_classes
= (<class 'rest_framework.authentication.SessionAuthentication'>, <class 'seed.authentication.SEEDAuthentication'>)
-
cleansing_progress
(request, *args, **kwargs) Return the progress of the cleansing. — type:
- status:
- required: true type: string description: either success or error
- progress:
- type: integer description: status of background cleansing task
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
filtered_mapping_results
(request, *args, **kwargs) Retrieves a paginated list of Properties and Tax Lots for an import file after mapping. — parameter_strategy: replace parameters:
- name: pk description: Import File ID (Primary key) type: integer required: true paramType: path
response_serializer: MappingResultsResponseSerializer
-
first_five_rows
(request, *args, **kwargs) Retrieves the first five rows of an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- first_five_rows:
- type: array of strings description: list of strings for each of the first five rows for this import file
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
get_cleansing_results
(request, *args, **kwargs) Retrieve the details of the cleansing script. — type:
- status:
- required: true type: string description: either success or error
- message:
- type: string description: additional information, if any
- progress:
- type: integer description: integer percent of completion
- data:
- type: JSON description: object describing the results of the cleansing
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
get_csv
(request, *args, **kwargs) Download a csv of the results. — type:
- status:
- required: true type: string description: either success or error
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
mapping_done
(request, *args, **kwargs) Tell the backend that the mapping is complete. — type:
- status:
- required: true type: string description: either success or error
- message:
- required: false type: string description: error message, if any
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
matching_results
(request, *args, **kwargs) Retrieves the number of matched and unmatched BuildingSnapshots for a given ImportFile record.
GET: Expects import_file_id corresponding to the ImportFile in question. Returns:
{ 'status': 'success', 'matched': Number of BuildingSnapshot objects that have matches, 'unmatched': Number of BuildingSnapshot objects with no matches. }
-
perform_mapping
(request, *args, **kwargs) Starts a background task to convert imported raw data into PropertyState and TaxLotState, using user’s column mappings. — type:
- status:
- required: true type: string description: either success or error
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
raise_exception
= True
-
raw_column_names
(request, *args, **kwargs) Retrieves a list of all column names from an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- raw_columns:
- type: array of strings description: list of strings of the header row of the ImportFile
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
retrieve
(request, *args, **kwargs) Retrieves details about an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- import_file:
- type: ImportFile structure description: full detail of import file
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
save_column_mappings
(request, *args, **kwargs) Saves the mappings between the raw headers of an ImportFile and the destination fields in the to_table_name model which should be either PropertyState or TaxLotState
Valid source_type values are found in
seed.models.SEED_DATA_SOURCES
Payload:
{ "import_file_id": ID of the ImportFile record, "mappings": [ { 'from_field': 'eui', # raw field in import file 'to_field': 'energy_use_intensity', 'to_table_name': 'PropertyState', }, { 'from_field': 'gfa', 'to_field': 'gross_floor_area', 'to_table_name': 'PropertyState', } ] }
Returns:
{'status': 'success'}
-
save_raw_data
(request, *args, **kwargs) Starts a background task to import raw data from an ImportFile into PropertyState objects as extra_data. If the cycle_id is set to year_ending then the cycle ID will be set to the year_ending column for each record in the uploaded file. Note that the year_ending flag is not yet enabled. — type:
- status:
- required: true type: string description: either success or error
- message:
- required: false type: string description: error message, if any
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
- name: cycle_id description: The ID of the cycle or the string “year_ending” paramType: string required: true
-
start_system_matching
(request, *args, **kwargs) Starts a background task to attempt automatic matching between buildings in an ImportFile with other existing buildings within the same org. — type:
- status:
- required: true type: string description: either success or error
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
suffix
= None
-
-
class
seed.data_importer.views.
LocalUploaderViewSet
(**kwargs) Bases:
rest_framework.viewsets.GenericViewSet
Endpoint to upload data files to, if uploading to local file storage. Valid source_type values are found in
seed.models.SEED_DATA_SOURCES
Returns:
{ 'success': True, 'import_file_id': The ID of the newly-uploaded ImportFile }
-
create
(request, *args, **kwargs) Upload a new file to an import_record. This is a multipart/form upload. — parameters:
- name: import_record description: the ID of the ImportRecord to associate this file with. required: true paramType: body
- name: source_type description: the type of file (e.g. ‘Portfolio Raw’ or ‘Assessed Raw’) required: false paramType: body
- name: source_program_version description: the version of the file as related to the source_type required: false paramType: body
- name: file or qqfile description: In-memory file object required: true paramType: Multipart
-
suffix
= None
-
-
class
seed.data_importer.views.
MappingResultsPayloadSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
filter_params
= <django.contrib.postgres.fields.jsonb.JSONField>
-
-
class
seed.data_importer.views.
MappingResultsPropertySerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
class
seed.data_importer.views.
MappingResultsResponseSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
class
seed.data_importer.views.
MappingResultsTaxLotSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer