Skip to content

Basic functions

open_mastr.Mastr

Mastr is used to download the MaStR database.

An SQL database is used to mirror the MaStR database. It is filled by downloading and parsing the MaStR via bulk download.

Example

from open_mastr import Mastr

db = Mastr()
db.download()
PARAMETER DESCRIPTION
engine

Defines the engine of the database where the MaStR is mirrored to. Default is 'sqlite'.

TYPE: (sqlite, Engine) DEFAULT: 'sqlite'

output_dir

Default is the content of env var OUTPUT_PATH, else ~/.open-MaStR

TYPE: Top-level directory of produced output (downloaded files, created database) DEFAULT: None

browse_available_downloads()

Browse available MaStR downloads from the website without starting the download.

This method fetches and displays all available download dates from the MaStR website, allowing users to see what historical data is available before deciding to download.

RETURNS DESCRIPTION
list of dict

List of available downloads with date, version, and type information.

Examples:

>>> from open_mastr import Mastr
>>> db = Mastr()
>>> available_downloads = db.browse_available_downloads()

download(method='bulk', data=None, date=None, select_date_interactively=False, bulk_cleansing=True, keep_old_downloads=False, mastr_table_to_db_table=None, alter_database_tables=True, english=False, add_views_for_old_table_names=True, **kwargs)

Download the MaStR registry and write it to a local database.

PARAMETER DESCRIPTION
method

Only "bulk" is a valid value. The download via the MaStR SOAP API has been removed. Defaults to 'bulk'.

TYPE: bulk DEFAULT: 'bulk'

data

Specifies which tables to download.

Possible values:

  • "wind"
  • "solar"
  • "biomass"
  • "hydro"
  • "gsgk"
  • "combustion"
  • "nuclear"
  • "gas"
  • "storage"
  • "storage_units"
  • "electricity_consumer"
  • "location"
  • "market"
  • "grid"
  • "balancing_area"
  • "permit"
  • "deleted_units"
  • "deleted_market_actors"
  • "retrofit_units"

Usage:

  • If None, all data is downloaded.
  • If a string, only the specified table is downloaded (e.g., "wind").
  • If a list, multiple tables are downloaded (e.g., ["wind", "solar"]).

TYPE: str or list or None DEFAULT: None

date
date description
"20230101" If file from this date exists locally, it is used. Otherwise, it tries to get it from markstammdatenregister.de
"today" Shorthand for specify today's date in YYYYMMDD format
None set date="today"
"existing" Deprecated since 0.16, see #616

Defaults to None.

TYPE: None or `datetime.datetime` or str DEFAULT: None

select_date_interactively

If set to True, the user will be presented with a list of available download dates from the MaStR website and can interactively select which date to download. This allows downloading historical data instead of just the latest available data. When True, the date parameter is ignored. Defaults to False.

TYPE: bool DEFAULT: False

bulk_cleansing

If set to True, data cleansing is applied after the download (which is recommended). In its original format, many entries in the MaStR are encoded with IDs. Columns like state or fueltype do not contain entries such as "Hessen" or "Braunkohle", but instead only contain IDs. Cleansing replaces these IDs with their corresponding original entries. Defaults to True.

TYPE: bool DEFAULT: True

keep_old_downloads

If set to True, prior downloaded MaStR zip files will be kept. Defaults to False.

TYPE: bool DEFAULT: False

mastr_table_to_db_table

If given, downloaded data from a MaStR file will be stored in the SQLAlchemy table associated with that file. The tables must exist already; they are not created. Example: {"EinheitenWind": Table(...), "EinheitenSolar": Table(...), ...}

If None / not given, the mapping will be generated by calling Mastr.generate_data_model (and thus downloading the MaStR documentation). In this case, the tables will be created.

You can pass this parameter to use your own mapping.

Defaults to None.

TYPE: Mapping from MaStR table name (str) to SQLALchemy Table, or None DEFAULT: None

alter_database_tables

If set to True, ALTER statememts to add database columns will be issued if there are fields in the downloaded XML that are not yet present in the database tables that have been generated or given. If set to False and such unexpected fields are found, those fields are not imported.

Defaults to True.

TYPE: bool DEFAULT: True

english

If set to True and no mastr_table_to_db_table mapping is given, the generated tables will have English names and English columns. (Some untranslated German names may remain if they haven't been added to open-mastr's translation info yet.)

Defaults to False.

TYPE: bool = False, optional, DEFAULT: False

add_views_for_old_table_names

If set to True, database views will be generated for tables renamed in version 1.0 so that the previous table names still work. Only has an effect if mastr_table_to_db_table is not given.

Defaults to True.

TYPE: bool = True, optional, DEFAULT: True

generate_data_model(data=None, date=None, url=None, metadata=None, catalog_value_as_str=True, english=False)

Generate data model from MaStR documentation.

Download the MaStR documentation, extract the XSD files that describe the format of the MaStR XML files, and generate SQLAlchemy tables from those XSD files. The tables are not created in the database.

You can use this method to create a mapping from the MaStR data model to your own data model. This mapping should then be passed to the Mastr.download method.

Example

from open_mastr import Mastr, format_mastr_table_to_db_table
db = Mastr()
mapping = db.generate_data_model()
# Print the tables so that you can see what was generated.
print(format_mastr_table_to_db_table(mapping))
# edit your mapping here
db.download(mastr_table_to_db_table=mapping)
PARAMETER DESCRIPTION
data

Specifies which tables to generate. See the download method for details.

TYPE: str or list or None DEFAULT: None

date

Specifies date for the docs download. See the download method for details.

TYPE: None or `datetime.datetime` or str DEFAULT: None

url

The URL to download the MaStR documentation from. If this is given, the date parameter is not used.

TYPE: str or None DEFAULT: None

metadata

SQLAlchemy MetaData object to use for the tables. If not given, a new MetaData object is created.

TYPE: SQLAlchemy MetaData or None DEFAULT: None

catalog_value_as_str

If set to True, columns which contain values from the MaStR catalog (Katalogwerte) are generated as string/VARCHAR columns. If set to False, they are generated as int columns. This should usually be set to True if you want to use the bulk_cleansing option of the download method to convert the catalog IDs to their values. Defaults to True.

TYPE: bool DEFAULT: True

english

If set to True, table and column names are translated from their MaStR name to an English name if open-mastr already has a translation stored for that table/column.

The English name and original MaStR name are always stored in the table's/column's info attribute.

Defaults to False.

TYPE: bool DEFAULT: False

RETURNS DESCRIPTION
dict from str to SQLAlchemy Table

Dict mapping original MaStR table name to SQLAlchemy table Example: {"EinheitenWind": Table(...), "EinheitenSolar": Table(...), ...}

to_csv(db_table_names=None, chunksize=500000)

Export tables from existing database to CSV.

PARAMETER DESCRIPTION
db_table_names

The names of the database tables to export. If None, all tables in the database will be exported. Defaults to None.

TYPE: Iterable of str or None DEFAULT: None

chunksize

Number of rows to retrieve from the database before dumping them to the CSV file. Defaults to 500000.

TYPE: int DEFAULT: 500000

translate()

The translate method has been removed. You can use the english option in the Mastr.download method to get English table and column names.