Data Importer Package¶
Submodules¶
Managers¶
Models¶
URLs¶
Utils¶
-
class
seed.data_importer.utils.
CoercionRobot
¶ Bases:
object
-
lookup_hash
(uncoerced_value, destination_model, destination_field)¶
-
make_key
(value, model, field)¶
-
-
seed.data_importer.utils.
acquire_lock
(name, expiration=None)¶ Tries to acquire a lock from the cache. Also sets the lock’s value to the current time, allowing us to see how long it has been held.
Returns False if lock already belongs by another process.
-
seed.data_importer.utils.
chunk_iterable
(iterlist, chunk_size)¶ Breaks an iterable (e.g. list) into smaller chunks, returning a generator of the chunk.
-
seed.data_importer.utils.
get_core_pk_column
(table_column_mappings, primary_field)¶
-
seed.data_importer.utils.
get_lock_time
(name)¶ Examines a lock to see when it was acquired.
-
seed.data_importer.utils.
release_lock
(name)¶ Frees a lock.
Views¶
-
class
seed.data_importer.views.
ImportFileViewSet
(**kwargs) Bases:
rest_framework.viewsets.ViewSet
-
authentication_classes
= (<class 'rest_framework.authentication.SessionAuthentication'>, <class 'seed.authentication.SEEDAuthentication'>)
-
available_matches
(request, *args, **kwargs)
-
data_quality_progress
(request, *args, **kwargs) Return the progress of the data quality check. — type:
- status:
- required: true type: string description: either success or error
- progress:
- type: integer description: status of background data quality task
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
destroy
(request, *args, **kwargs) Returns suggested mappings from an uploaded file’s headers to known data fields. — type:
- status:
- required: true type: string description: Either success or error
parameter_strategy: replace parameters:
- name: pk description: import_file_id required: true paramType: path
- name: organization_id description: The organization_id for this user’s organization required: true paramType: query
-
filtered_mapping_results
(request, *args, **kwargs) Retrieves a paginated list of Properties and Tax Lots for an import file after mapping. — parameter_strategy: replace parameters:
- name: pk description: Import File ID (Primary key) type: integer required: true paramType: path
response_serializer: MappingResultsResponseSerializer
-
first_five_rows
(request, *args, **kwargs) Retrieves the first five rows of an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- first_five_rows:
- type: array of strings description: list of strings for each of the first five rows for this import file
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
get_data_quality_results
(request, *args, **kwargs) Retrieve the details of the data quality check. — type:
- status:
- required: true type: string description: either success or error
- message:
- type: string description: additional information, if any
- progress:
- type: integer description: integer percent of completion
- data:
- type: JSON description: object describing the results of the data quality check
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
static
has_coparent
(state_id, inventory_type, fields=None) Return the coparent of the current state id based on the inventory type. If fields are given (as a list), then it will only return the fields specified of the state model object as a dictionary.
Parameters: - state_id – int, ID of PropertyState or TaxLotState
- inventory_type – string, either properties | taxlots
- fields – list, either None or list of fields to return
Returns: dict or state object, If fields is not None then will return state_object
-
mapping_done
(request, *args, **kwargs) Tell the backend that the mapping is complete. — type:
- status:
- required: true type: string description: either success or error
- message:
- required: false type: string description: error message, if any
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
mapping_suggestions
(request, *args, **kwargs) Returns suggested mappings from an uploaded file’s headers to known data fields. — type:
- status:
- required: true type: string description: Either success or error
- suggested_column_mappings:
required: true type: dictionary description: Dictionary where (key, value) = (the column header from the file,
array of tuples (destination column, score))- building_columns:
- required: true type: array description: A list of all possible columns
- building_column_types:
- required: true type: array description: A list of column types corresponding to the building_columns array
parameter_strategy: replace parameters:
- name: pk description: import_file_id required: true paramType: path
- name: organization_id description: The organization_id for this user’s organization required: true paramType: query
-
match
(request, *args, **kwargs)
-
matching_results
(request, *args, **kwargs) Retrieves the number of matched and unmatched properties & tax lots for a given ImportFile record. Specifically for new imports
GET: Expects import_file_id corresponding to the ImportFile in question. Returns:
{ 'status': 'success', 'properties': { 'matched': Number of PropertyStates that have been matched, 'unmatched': Number of PropertyStates that are unmatched new imports }, 'tax_lots': { 'matched': Number of TaxLotStates that have been matched, 'unmatched': Number of TaxLotStates that are unmatched new imports } }
-
matching_status
(request, *args, **kwargs) Retrieves the number and ids of matched and unmatched properties & tax lots for a given ImportFile record. Specifically for hand-matching
GET: Expects import_file_id corresponding to the ImportFile in question. Returns:
{ 'status': 'success', 'properties': { 'matched': Number of PropertyStates that have been matched, 'matched_ids': Array of matched PropertyState ids, 'unmatched': Number of PropertyStates that are unmatched records, 'unmatched_ids': Array of unmatched PropertyState ids }, 'tax_lots': { 'matched': Number of TaxLotStates that have been matched, 'matched_ids': Array of matched TaxLotState ids, 'unmatched': Number of TaxLotStates that are unmatched records, 'unmatched_ids': Array of unmatched TaxLotState ids } }
-
perform_mapping
(request, *args, **kwargs) Starts a background task to convert imported raw data into PropertyState and TaxLotState, using user’s column mappings. — type:
- status:
- required: true type: string description: either success or error
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
queryset
-
raise_exception
= True
-
raw_column_names
(request, *args, **kwargs) Retrieves a list of all column names from an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- raw_columns:
- type: array of strings description: list of strings of the header row of the ImportFile
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
retrieve
(request, *args, **kwargs) Retrieves details about an ImportFile. — type:
- status:
- required: true type: string description: either success or error
- import_file:
- type: ImportFile structure description: full detail of import file
parameter_strategy: replace parameters:
- name: pk description: “Primary Key” required: true paramType: path
-
save_column_mappings
(request, *args, **kwargs) Saves the mappings between the raw headers of an ImportFile and the destination fields in the to_table_name model which should be either PropertyState or TaxLotState
Valid source_type values are found in
seed.models.SEED_DATA_SOURCES
Payload:
{ "import_file_id": ID of the ImportFile record, "mappings": [ { 'from_field': 'eui', # raw field in import file 'from_units': 'kBtu/ft**2/year', # pint-parseable units, optional 'to_field': 'energy_use_intensity', 'to_table_name': 'PropertyState', }, { 'from_field': 'gfa', 'from_units': 'ft**2', # pint-parseable units, optional 'to_field': 'gross_floor_area', 'to_table_name': 'PropertyState', } ] }
Returns:
{'status': 'success'}
-
save_raw_data
(request, *args, **kwargs) Starts a background task to import raw data from an ImportFile into PropertyState objects as extra_data. If the cycle_id is set to year_ending then the cycle ID will be set to the year_ending column for each record in the uploaded file. Note that the year_ending flag is not yet enabled. — type:
- status:
- required: true type: string description: either success or error
- message:
- required: false type: string description: error message, if any
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
- name: cycle_id description: The ID of the cycle or the string “year_ending” paramType: string required: true
-
start_system_matching
(request, *args, **kwargs) Starts a background task to attempt automatic matching between buildings in an ImportFile with other existing buildings within the same org. — type:
- status:
- required: true type: string description: either success or error
- progress_key:
- type: integer description: ID of background job, for retrieving job progress
parameter_strategy: replace parameters:
- name: pk description: Import file ID required: true paramType: path
-
suffix
= None
-
unmatch
(request, *args, **kwargs)
-
-
class
seed.data_importer.views.
LocalUploaderViewSet
(**kwargs) Bases:
rest_framework.viewsets.ViewSet
Endpoint to upload data files to, if uploading to local file storage. Valid source_type values are found in
seed.models.SEED_DATA_SOURCES
Returns:
{ 'success': True, 'import_file_id': The ID of the newly-uploaded ImportFile }
-
create
(request, *args, **kwargs) Upload a new file to an import_record. This is a multipart/form upload. — parameters:
- name: import_record description: the ID of the ImportRecord to associate this file with. required: true paramType: body
- name: source_type description: the type of file (e.g. ‘Portfolio Raw’ or ‘Assessed Raw’) required: false paramType: body
- name: source_program_version description: the version of the file as related to the source_type required: false paramType: body
- name: file or qqfile description: In-memory file object required: true paramType: Multipart
-
suffix
= None
-
-
class
seed.data_importer.views.
MappingResultsPayloadSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
filter_params
= <django.contrib.postgres.fields.jsonb.JSONField>
-
-
class
seed.data_importer.views.
MappingResultsPropertySerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
class
seed.data_importer.views.
MappingResultsResponseSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
class
seed.data_importer.views.
MappingResultsTaxLotSerializer
(instance=None, data=<class rest_framework.fields.empty>, **kwargs) Bases:
rest_framework.serializers.Serializer
-
seed.data_importer.views.
get_upload_details
(request, *args, **kwargs) Retrieves details about how to upload files to this instance.
Returns:
If S3 mode: { 'upload_mode': 'S3', 'upload_complete': A url to notify that upload is complete, 'signature': The url to post file details to for auth to upload to S3. } If local file system mode: { 'upload_mode': 'filesystem', 'upload_path': The url to POST files to (see local_uploader) }
-
seed.data_importer.views.
handle_s3_upload_complete
(request, *args, **kwargs) Notify the system that an upload to S3 has been completed. This is a necessary step after uploading to S3 or the SEED instance will not be aware the file exists.
Valid source_type values are found in
seed.models.SEED_DATA_SOURCES
GET: Expects the following in the query string:
- key: The full path to the file, within the S3 bucket.
E.g. data_importer/buildings.csv
- source_type: The source of the file.
E.g. ‘Assessed Raw’ or ‘Portfolio Raw’
source_program: Optional value from common.mapper.Programs source_version: e.g. “4.1”
import_record: The ID of the ImportRecord this file belongs to.
Returns:
{ 'success': True, 'import_file_id': The ID of the newly-created ImportFile object. }
-
seed.data_importer.views.
sign_policy_document
(request, *args, **kwargs) Sign and return the policy document for a simple upload. http://aws.amazon.com/articles/1434/#signyours3postform
Payload:
{ "expiration": ISO-encoded timestamp for when signature should expire, e.g. "2014-07-16T00:20:56.277Z", "conditions": [ {"acl":"private"}, {"bucket": The name of the bucket from get_upload_details}, {"Content-Type":"text/csv"}, {"success_action_status":"200"}, {"key": filename of upload, prefixed with 'data_imports/', suffixed with a unique timestamp. e.g. 'data_imports/my_buildings.csv.1405469756'}, {"x-amz-meta-category":"data_imports"}, {"x-amz-meta-qqfilename": original filename} ] }
Returns:
{ "policy": A hash of the policy document. Using during upload to S3. "signature": A signature of the policy document. Also used during upload to S3. }