google.appengine.ext.bulkload.bulkloader_parser module

Summary

Bulkloader Config Parser and runner.

A library to read bulkloader yaml configs. Returns a BulkloaderEntry object which describes the bulkloader.yaml in object form, including some additional parsing of things like Python lambdas.

Contents

class google.appengine.ext.bulkload.bulkloader_parser.BulkloaderEntry(**attributes)source

Bases: google.appengine.api.validation.Validated

Root of the bulkloader configuration.

ATTRIBUTES = {'transformers': <google.appengine.api.validation.Repeated object>, 'python_preamble': <google.appengine.api.validation.Optional object>}
class google.appengine.ext.bulkload.bulkloader_parser.ConnectorOptions(**attributes)source

Bases: google.appengine.api.validation.Validated

Connector options.

ATTRIBUTES = {'xpath_to_nodes': <google.appengine.api.validation.Optional object>, 'template': <google.appengine.api.validation.Optional object>, 'encoding': <google.appengine.api.validation.Optional object>, 'skip_import_header_row': <google.appengine.api.validation.Optional object>, 'epilog': <google.appengine.api.validation.Optional object>, 'style': <google.appengine.api.validation.Optional object>, 'import_options': <google.appengine.api.validation.Optional object>, 'print_export_header_row': <google.appengine.api.validation.Optional object>, 'mode': <google.appengine.api.validation.Optional object>, 'column_list': <google.appengine.api.validation.Optional object>, 'export_options': <google.appengine.api.validation.Optional object>, 'columns': <google.appengine.api.validation.Optional object>, 'prolog': <google.appengine.api.validation.Optional object>}
CheckInitialized()source

Post-loading ‘validation’. Really used to fix up yaml hackyness.

class google.appengine.ext.bulkload.bulkloader_parser.ConnectorSubOptions(**attributes)source

Bases: google.appengine.api.validation.Validated

Connector options.

ATTRIBUTES = {'dialect': <google.appengine.api.validation.Optional object>, 'delimiter': <google.appengine.api.validation.Optional object>}
class google.appengine.ext.bulkload.bulkloader_parser.EvaluatedCallablesource

Bases: google.appengine.api.validation.Validator

Validates that a string evaluates to a Python callable.

Calls eval at validation time and stores the results as a ParsedMethod object. The ParsedMethod object can be used as a string (original value) or callable (parsed method). It also exposes supports_bulkload_state if the callable has a kwarg called ‘bulkload_state’, which is used to determine how to call the *_transform methods.

class ParsedMethod(value, key)source

Bases: object

Wrap the string, the eval’d method, and supports_bulkload_state.

ToValue(value)source

Returns the code string for this value.

Validate(value, key)source

Validates that the string compiles as a Python callable.

Parameters

  • value – String to compile as a regular expression.

  • key – The YAML field name.

Returns

Value wrapped in an object with properties ‘value’ and ‘fn’.

Raises

InvalidCodeInConfiguration when value does not compile.

class google.appengine.ext.bulkload.bulkloader_parser.ExportEntry(**attributes)source

Bases: google.appengine.api.validation.Validated

Describes the optional export transform for a single property.

ATTRIBUTES = {'external_name': <google.appengine.api.validation.Optional object>, 'export_transform': <google.appengine.api.validation.Optional object>}
class google.appengine.ext.bulkload.bulkloader_parser.PropertyEntry(**attributes)source

Bases: google.appengine.api.validation.Validated

Describes the transform for a single property.

ATTRIBUTES = {'default_value': <google.appengine.api.validation.Optional object>, 'export': <google.appengine.api.validation.Optional object>, 'import_template': <google.appengine.api.validation.Optional object>, 'external_name': <google.appengine.api.validation.Optional object>, 'import_transform': <google.appengine.api.validation.Optional object>, 'property': <google.appengine.api.validation.Type object>, 'export_transform': <google.appengine.api.validation.Optional object>}
CheckInitialized()source

Check that all required (combinations) of fields are set.

Also fills in computed properties.

Raises

InvalidConfiguration – If the config is invalid.

class google.appengine.ext.bulkload.bulkloader_parser.PythonPreambleEntry(**attributes)source

Bases: google.appengine.api.validation.Validated

Python modules to import at initialization time, typically models.

ATTRIBUTES = {'import': <google.appengine.api.validation.Type object>, 'as': <google.appengine.api.validation.Optional object>}
CheckInitialized()source

Check that all required fields are set, and update global state.

The imports specified in the preamble are imported at this time.

class google.appengine.ext.bulkload.bulkloader_parser.TransformerEntry(**attributes)source

Bases: google.appengine.api.validation.Validated

Describes the transform for an entity (or model) kind.

ATTRIBUTES = {'connector': <google.appengine.api.validation.Type object>, 'kind': <google.appengine.api.validation.Optional object>, 'sort_key_from_entity': <google.appengine.api.validation.Optional object>, 'name': <google.appengine.api.validation.Optional object>, 'connector_options': <google.appengine.api.validation.Optional object>, 'property_map': <google.appengine.api.validation.Repeated object>, 'model': <google.appengine.api.validation.Optional object>, 'post_export_function': <google.appengine.api.validation.Optional object>, 'post_import_function': <google.appengine.api.validation.Optional object>, 'use_model_on_export': <google.appengine.api.validation.Optional object>}
CheckInitialized()source

Check that all required (combinations) of fields are set.

Also fills in computed properties.

Raises

InvalidConfiguration – if the config is invalid.

google.appengine.ext.bulkload.bulkloader_parser.load_config(stream, config_globals)source

Load a configuration file and generate importer and exporter classes.

Parameters

  • stream – Stream containing config YAML.

  • config_globals – Dict to use to reference globals for code in the config.

Returns

BulkloaderEntry

Raises

InvalidConfiguration – If the config is invalid.