Module zillion.configs¶

AdHocFieldSchema ¶

Bases: zillion.configs.FormulaFieldConfigSchema

class zillion.configs.AdHocFieldSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

Base schema for an adhoc field

AdHocMetricSchema ¶

Bases: zillion.configs.AdHocFieldSchema

class zillion.configs.AdHocMetricSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of an adhoc metric

Attributes:

aggregation - (str, optional) A string representing the aggregation type to apply to this metric. See zillion.core.AggregationTypes.
technical - (str or dict, optional) A string or dict that will be parsed as a TechnicalField to define a technical computation to be applied to the metric.
rounding - (int, optional) If specified, the number of decimal places to round to
weighting_metric - (str, optional) A reference to a metric to use for weighting when aggregating averages
required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this metric.

BaseSchema ¶

Bases: marshmallow.schema.Schema

class zillion.configs.BaseSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

Base Schema with custom JSON module

Attributes:

meta - (*dict, optional) A dict of additional custom attributes for the config object

BollingerTechnical ¶

Bases: zillion.configs.RollingTechnical

class zillion.configs.BollingerTechnical(type, params, mode=None)

Compute a rolling average and bollinger bands for a column. This adds additional columns to the input dataframe.

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

ColumnConfigSchema ¶

Bases: zillion.configs.ColumnInfoSchema

class zillion.configs.ColumnConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a column configuration

ColumnFieldConfigField ¶

Bases: marshmallow.fields.Field

class zillion.configs.ColumnFieldConfigField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

A marshmallow field for the column's field attribute

ColumnFieldConfigSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.ColumnFieldConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a column's field attribute

Attributes:

name - (str) The name of the field
ds_formula - (str) A formula used to calculate the field value at the datasource query level. It must use syntax specific to the datasource.

ColumnInfo ¶

Bases: zillion.configs.ZillionInfo, tlbx.logging_utils.PrintMixin

class zillion.configs.ColumnInfo(**kwargs)

ZillionInfo for a column in a table. See ColumnInfoSchema for more details about fields.

add_field(self, field)

Add the field to the column's fields

create(zillion_info, unknown='raise')

Factory to create a ZillionInfo object from the class schema

field_ds_formula(self, name)

Get the datasource-level formula for a field if it exists

get_criteria_conversion(self, field_name, operation)

Get the datasource-level criteria conversion for a field/operation

get_field(self, name)

Get the reference to the field defined on this column. This may return a string or a dict depending on how the field was defined on the column.

Parameters:

name - (str) The name of the field

Returns:

(str or dict) - The name of the field or the dict defining the field

get_field_names(self)

Get the names of all fields mapped on this column

get_fields(self)

Get all fields mapped on this column

has_field(self, field)

Determine if the column supports the given field

has_field_ds_formula(self, name)

True if a datasource-level formula for a field exists

schema_load(zillion_info, unknown='raise')

Load an info dict with a marshmallow schema

Parameters:

zillion_info - (dict) A dict to load with the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

Returns:

(dict) - The loaded schema result

schema_validate(zillion_info, unknown='raise')

Validate an info dict against a schema.

Parameters:

zillion_info - (dict) A dict to validate against the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

ColumnInfoSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.ColumnInfoSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of column info that ends up in the zillion column metadata

Attributes:

fields - (list of ColumnFieldConfigField, optional) A list of field names or definitions
allow_type_conversions - (bool, optional) A flag denoting whether additional fields may be inferred from this column based on its column type (such as deriving year from a date).
type_conversion_prefix - (str, optional) A prefix to apply to all fields defined through automated type conversions.
active - (bool, optional) A flag denoting whether this column is active.
required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this column.

ConfigMixin ¶

class zillion.configs.ConfigMixin(*args, **kwargs)

Mixin to allow validation against a marshmallow schema

from_config(config)

Create a the object from a config

to_config(self)

Get the config for this object

DataSourceConfigField ¶

Bases: marshmallow.fields.Field

class zillion.configs.DataSourceConfigField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

The schema of a datasource configuration represented as a marshmallow Field

DataSourceConfigSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.DataSourceConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a datasource configuration

Attributes:

connect - (str or dict) A connection string or dict for establishing the datasource connection. This may have placeholders that get filled in from the DATASOURCE_CONTEXTS of the zillion config. See DataSourceConnectField for more details on passing a dict.
skip_conversion_fields - (bool, optional) Don't add any conversion fields when applying a config
metrics - (marshmallow field, optional) A list of MetricConfigSchema
dimensions - (marshmallow field, optional) A list of DimensionConfigSchema
tables - (marshmallow field, optional) A dict mapping of TableNameField -> TableConfigSchema

DataSourceConnectField ¶

Bases: marshmallow.fields.Field

class zillion.configs.DataSourceConnectField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

The schema of a datasource connect field

DataSourceConnectSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.DataSourceConnectSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a technical configuration

DataSourceCriteriaConversionsField ¶

Bases: marshmallow.fields.Field

class zillion.configs.DataSourceCriteriaConversionsField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

A field for defining column-level criteria conversions. This allows for optimizing queries by converting values instead of applying a function on the column to evaluate criteria, which can otherwise prevent index usage.

DiffTechnical ¶

Bases: zillion.configs.PandasTechnical

class zillion.configs.DiffTechnical(type, params, mode=None)

A Technical that computes a periodic diff on a DataFrame

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

DimensionConfigSchema ¶

Bases: zillion.configs.FieldConfigSchema

class zillion.configs.DimensionConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a dimension configuration

Attributes:

values - (str or list, optional) A list of allowed dimension values or a name of a callable to provide a list of values. If a string representing a callable is passed, it must be importable and the callable must accept two arguments: (warehouse ID, dimension object). An example callable would be zillion.field.values_from_db which reads allowed dimension values from the dimension_values table in the Zillion database.
sorter - (str, optional) A reference to an importable callable that accepts three arguments: (warehouse ID, dimension object, values). Currently values is a pandas Series and the callable is expected to return a Series. See zillion.field.sort_by_value_order for an example.

DimensionValuesField ¶

Bases: marshmallow.fields.Field

class zillion.configs.DimensionValuesField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

A field for defining dimension values

FieldConfigSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.FieldConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The based schema of a field configuration

Attributes:

name - (str) The name of the field
type - (str) A string representing the data type of the field. This will be converted to a SQLAlchemy type via ast.literal_eval.
display_name - (str, optional) The display name of the field
description - (str, optional) The description of the field

FormulaFieldConfigSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.FormulaFieldConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The based schema of a formula field configuration

Attributes:

name - (str) The name of the field
formula - (str, optional) A formula used to compute the field value. Formula fields are applied at the combined query layer, rather than in datasources queries, so the syntax must match that of the combined query layer database.
display_name - (str, optional) The display name of the field
description - (str, optional) The description of the field

FormulaMetricConfigSchema ¶

Bases: zillion.configs.FormulaFieldConfigSchema, zillion.configs.MetricConfigSchemaMixin

class zillion.configs.FormulaMetricConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a formula metric configuration

MetricConfigSchema ¶

Bases: zillion.configs.FieldConfigSchema, zillion.configs.MetricConfigSchemaMixin

class zillion.configs.MetricConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a metric configuration

MetricConfigSchemaMixin ¶

class zillion.configs.MetricConfigSchemaMixin()

Common attributes and logic for metric configs

Attributes:

aggregation - (str, optional) A string representing the aggregation type to apply to this metric. See zillion.core.AggregationTypes.
rounding - (int, optional) If specified, the number of decimal places to round to
weighting_metric - (str, optional) A reference to a metric to use for weighting when aggregating averages
technical - (str or dict, optional) A string or dict that will be parsed as a TechnicalField to define a technical computation to be applied to the metric.
required_grain - (list of str, optional) If specified, a list of dimensions that must be present in the dimension grain of any report that aims to include this metric.

PandasTechnical ¶

Bases: zillion.configs.Technical

class zillion.configs.PandasTechnical(type, params, mode=None)

A generic Technical runs a pandas method

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

PolyNested ¶

Bases: marshmallow.fields.Nested

class zillion.configs.PolyNested(nested, *, default=, only=None, exclude=(), many=False, unknown=None, **kwargs)

A polytype nested field that iterates through a list of possible types

RankTechnical ¶

Bases: zillion.configs.PandasTechnical

class zillion.configs.RankTechnical(type, params, mode=None)

A Technical specific to the pandas rank function

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

RollingTechnical ¶

Bases: zillion.configs.Technical

class zillion.configs.RollingTechnical(type, params, mode=None)

A Technical that uses the pandas rolling feature

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

TableConfigSchema ¶

Bases: zillion.configs.TableInfoSchema

class zillion.configs.TableConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a table configuration

Attributes:

columns - (dict, optional) A dict mapping of column name to ColumnConfigSchema
data_url - (str, optional) A url used to download table data if this is an adhoc table
if_exists - (str, optional) Control whether to replace, fail, or ignore when the table data already exists.
drop_dupes - (bool, optional) Drop duplicate primary key rows when loading a table from a data_url
convert_types - (dict, optional) A mapping of column names to types to convert to when loading a table from a data url. The types must be strings representing valid sqlalchemy types. Ex: {"col1": "date", "col2": "integer"}
primary_key - (list of str, optional) A list of fields representing the primary key of the table
adhoc_table_options - (dict, optional) A dict of additional params to pass to the adhoc table class as kwargs

TableInfo ¶

Bases: zillion.configs.ZillionInfo, tlbx.logging_utils.PrintMixin

class zillion.configs.TableInfo(**kwargs)

ZillionInfo for a table. See TableInfoSchema for more details about fields.

create(zillion_info, unknown='raise')

Factory to create a ZillionInfo object from the class schema

schema_load(zillion_info, unknown='raise')

Load an info dict with a marshmallow schema

Parameters:

zillion_info - (dict) A dict to load with the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

Returns:

(dict) - The loaded schema result

schema_validate(zillion_info, unknown='raise')

Validate an info dict against a schema.

Parameters:

zillion_info - (dict) A dict to validate against the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

TableInfoSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.TableInfoSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of table info that ends up in the zillion table metadata

Attributes:

type - (str) Specifies the TableType
active - (bool, optional) A flag denoting whether this table is active or not.
parent - (str, optional) A reference to the full name of a parent table. This impacts the possible join relationships of this table. It is assumed to be safe to join back to any parent or ancestor table via shared keys (the child table must have the primary key of the parent table).
siblings - (list, optional) A list of references to the full names of sibling tables. This impacts the possible join relationships of this table. It is assumed to be safe to join back to any sibling table via shared keys (the child table must have the primary key of the sibling table).
create_fields - (bool, optional) If true, try to create Field objects from all columns in the table. Specifying the fields in a column config will override this behavior. Metric vs Dimension fields are inferred from the type. It is generally better to be explicit about your fields and field types, but this option provides convenience for special cases, particularly adhoc use cases.
use_full_column_names - (bool, optional) If True and create_fields is True, fully qualify the created field names using the full table and column names. If false, assume it is safe to simply use the column name as the field name.
primary_key - (list of str) A list of fields representing the primary key of the table
incomplete_dimensions - (list of str, optional) If specified, a list of dimensions that are not safe to use for joins.
priority - (int, optional) Set the priority of this table relative to other tables. All tables default to priority=1. When choosing the best table, lower numbers are considered higher priority. Tables at the same priority level use the length of their TableSet for the given query as the tie-breaker. See Warehouse._choose_best_table_set.

TableNameField ¶

Bases: marshmallow.fields.String

class zillion.configs.TableNameField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

The schema of a table configuration represented as a marshmallow Field

TableTypeField ¶

Bases: marshmallow.fields.Field

class zillion.configs.TableTypeField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

A field for the type of a table

Technical ¶

Bases: tlbx.object_utils.MappingMixin, tlbx.logging_utils.PrintMixin

class zillion.configs.Technical(type, params, mode=None)

A technical computation on a DataFrame column

Parameters:

type - (str) The TechnicalType
params - (dict) Params for the technical computation
mode - (str) The mode that controls how to apply the technical computation across the data's dimensions. See TechnicalModes for options. If None, the default mode will be set based on the technical type.

Attributes:

allowed_params - (set) Define the allowed technical parameters

apply(self, df, column, rounding=None)

Apply a technical computation to a dataframe. If the dataframe has a multilevel index and the technical is being applied in group mode, then the data will be sliced along the second to last level and the technical applied to each subgroup. Otherwise the technical is applied across the entire dataframe. The technical is applied to the dataframe in place.

Parameters:

df - (DataFrame) A DataFrame to apply a technical computation to
column - (str) The name of the target column for the technical computation
rounding - (dict, optional) The rounding settings for the report's columns

get_default_mode()

Get the default mode for applying the technical calculation

parse_technical_string_params(val)

Return named params from a technical string

TechnicalField ¶

Bases: marshmallow.fields.Field

class zillion.configs.TechnicalField(*, default=, missing=, data_key=None, attribute=None, validate=None, required=False, allow_none=None, load_only=False, dump_only=False, error_messages=None, **metadata)

A field for defining technical calculations

TechnicalInfoSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.TechnicalInfoSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a technical configuration

WarehouseConfigSchema ¶

Bases: zillion.configs.BaseSchema

class zillion.configs.WarehouseConfigSchema(*, only=None, exclude=(), many=False, context=None, load_only=(), dump_only=(), partial=False, unknown=None)

The schema of a warehouse configuration.

Attributes:

includes - (marshmallow field, optional) A list of warehouse files to import. Later items in the list will override earlier items for overlapping keys. Any settings in the warehouse config will take precedence.
metrics - (marshmallow field, optional) A list of MetricConfigSchema
dimensions - (marshmallow field, optional) A list of DimensionConfigSchema
datasources - (marshmallow field) A dict mapping of datasource name -> DataSourceConfigField

ZillionInfo ¶

Bases: tlbx.object_utils.MappingMixin

class zillion.configs.ZillionInfo(**kwargs)

Information that defines a part of the zillion configuration. The information may come from a JSON config file or directly from the SQLALchemy object's info.zillion attribute. The JSON schema is parsed with a marshmallow schema object. See the particular schema used with each subclass for details on fields.

Parameters:

kwargs - Parameters that will be parsed with the given marshmallow schema.

Attributes:

schema - (marshmallow schema) A class attribute that specifies the marshmallow schema used to parse the input args on init.

create(zillion_info, unknown='raise')

Factory to create a ZillionInfo object from the class schema

schema_load(zillion_info, unknown='raise')

Load an info dict with a marshmallow schema

Parameters:

zillion_info - (dict) A dict to load with the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

Returns:

(dict) - The loaded schema result

schema_validate(zillion_info, unknown='raise')

Validate an info dict against a schema.

Parameters:

zillion_info - (dict) A dict to validate against the schema
unknown - (optional) A flag passed through to marshmallow's schema processing

create_technical ¶

zillion.configs.create_technical(info)

Create a technical instance from the input object

Parameters:

info - (str or dict) If str, parse as atechnical string. If a dict, parse as TechnicalInfoSchema.

Returns:

(Technical) - A Technical object based on the input.

default_field_display_name ¶

zillion.configs.default_field_display_name(name)

Determine a default display name from the field name

Parameters:

name - (str) The field name to process

Returns:

(str) - The field display name

default_field_name ¶

zillion.configs.default_field_name(column)

Get the default field name from a SQLAlchemy column

Parameters:

column - (SQLAlchemy column) A column to get the default field name for

Returns:

(str) - The default field name for the column

field_safe_name ¶

zillion.configs.field_safe_name(name)

Replace characters with underscores if they are not in FIELD_NAME_ALLOWED_CHARS

Parameters:

name - (str) The field name to process

Returns:

(str) - The "safe" field name

Parameters:

cfg - (dict, str, or buffer) A datasource config dict or a file path/buffer to read the config contents from.

Returns:

(dict) - The parsed datasource config

load_datasource_config_from_env ¶

zillion.configs.load_datasource_config_from_env(var)

Parse a datasource JSON config from a location stored in an environment variable

load_warehouse_config ¶

zillion.configs.load_warehouse_config(cfg)

Parse a warehouse JSON config

Parameters:

cfg - (dict, str, or buffer) A warehouse config dict or a file path/buffer to read the config contents from.

Returns:

(dict) - The parsed warehouse config

load_warehouse_config_from_env ¶

zillion.configs.load_warehouse_config_from_env(var)

Parse a warehouse JSON config from a location stored in an environment variable

parse_schema_file ¶

zillion.configs.parse_schema_file(f, schema)

Parse a marshmallow schema file

Parameters:

f - (str or buffer) A file path or buffer to read the raw schema contents from. Both JSON and YAML are supported.
schema - (marshmallow schema) The marshmallow schema to use to parse the data

Returns:

(dict) - A dict structure loaded from the schema file

parse_technical_string ¶

zillion.configs.parse_technical_string(val)

Parse Technical args from a shorthand string

Parameters:

val - (str) The technical string to parse. The general format is: type(*args):mode. The type must be a valid value in TechnicalTypes. The argument requirements vary by type, and are optional in some cases. The mode controls whether the computation is done across the last group or the full data. The mode is optional, and will default to a value specific to that technical type (usually "group" mode). Examples:
- "mean(5)" for moving average, window=5
- "mean(5,2)" for moving average, window=5, min_period=2
- "cumsum" for cumulative sum (no args)
- "cumsum:all" for cumulative sum across all data, regardless of dimension

Returns:

(dict) - A dict of Technical args