Advanced

For most users, the functionalites described in Getting Started are sufficient. If you want to examine how you can configure the package's behavior for your own needs, check out Configuration. Or you can explore the two main functionalities of the package, namely the Bulk Download or the SOAP API download.

Configuration

Database settings

Configure your database with the engine parameter of Mastr. It defines the engine of the database where the MaStR is mirrored to. Default is 'sqlite'.

The possible databases are:

• sqlite: By default the database will be stored in $HOME/.open-MaStR/data/sqlite/open-mastr.db.
• own database: The Mastr class accepts a sqlalchemy.engine.Engine object as engine which enables the user to use any other desired database such as PostgreSQL. The tables are created in the default DB schema, in PostgreSQL this is public. If you use an own database so, you need to insert the connection parameter into the engine variable. In the example below, the following parameters are used: user open-mastr, password open-mastr-pw, database open-mastr-db. Make sure it exists and the user has sufficient permissions.

  from sqlalchemy import create_engine

  # SQLite DB
  engine_sqlite = create_engine("sqlite:///path/to/sqlite/database.db")
  # postgreSQL DB
  engine_postgres = create_engine("postgresql+psycopg2://open-mastr:open-mastr-pw@localhost:55443/open-mastr-db")
  db = Mastr(engine=engine_sqlite)

Project directory

The directory $HOME/.open-MaStR is automatically created. It is used to store configuration files and save data. You can change this default path, see environment variables. Default config files are copied to this directory which can be modified - but with caution. The project home directory is structured as follows (files and folders below data/ just an example).

    .open-MaStR/
    ├── config
    │   ├── credentials.cfg
    │   ├── filenames.yml
    │   ├── logging.yml
    ├── data
    │   ├── dataversion-<date>
    │   ├── sqlite
    │       └── open-mastr.db
        └── xml_download
            └── Gesamtdatenexport_<date>.zip
    └── logs
        └── open_mastr.log

• config
  • credentials.cfg
    Credentials used to access Marktstammdatenregister (MaStR) API
  • filenames.yml
    File names are defined here.
  • logging.yml
    Logging configuration. For changing the log level to increase or decrease details of log messages, edit the level of the handlers. See below for details on logging.
• data
  • dataversion-<date>
    Contains exported data as csv files from method to_csv
  • sqlite
    Contains the sqlite database in open-mastr.db
  • xml_download
    Contains the bulk download in Gesamtdatenexport_<date>.zip
    New bulk download versions overwrite older versions.
• logs
  • open_mastr.log
    The files stores the logging information from executing open-mastr.

Logs

For the download via the API, logs are stored in a single file in /$HOME/<user>/.open-MaStR/logs/open_mastr.log. New logging messages are appended. It is recommended to delete the log file from time to time because of its required disk space.

By default, the log level is set to INFO. You can increase or decrease the verbosity by either changing logging.yml (see above) or adjusting it manually in your code. E.g. to enable DEBUG messages in open_mastr.log you can use the following snippet:

  import logging
  from open_mastr import Mastr

  # Increase to DEBUG to show more details in open_mastr.log
  # Must be called after importing open_mastr to have the open-MaStR logger imported
  logging.getLogger("open-MaStR").setLevel(logging.DEBUG)

Data

If the zipped dump of the MaStR is downloaded, it is saved in the folder $HOME/.open-MaStR/data/xml_download.

The data can then be written to any sql database supported by sqlalchemy. The type of the sql database is determined by the parameter engine in the Mastr class.

For more information regarding the database see Database settings.

Environment variables

There are some environment variables to customize open-MaStR:

Variable	Description	Example
SQLITE_DATABASE_PATH	Path to the SQLite file. This allows to use to use multiple instances of the MaStR database. The database instances exist in parallel and are independent of each other.	/home/mastr-rabbit/.open-MaStR/data/sqlite/your_custom_instance_name.db
OUTPUT_PATH	Path to user-defined output directory for CSV data, XML file and database. If not specified, output directory defaults to $HOME/.open-MaStR/	Linux: /home/mastr-rabbit/open-mastr-user-defined-output-path, Windows: C:\\Users\\open-mastr-user-defined-output-path
USE_RECOMMENDED_NUMBER_OF_PROCESSES	If set to True, the number of processes used for the bulk download is set to the recommended number of processes. The recommended number is min(the number of available CPUs - 1, 4). If set to False, the number of processes is 1 if not otherwise configured via NUMBER_OF_PROCESSES.	True or False
NUMBER_OF_PROCESSES	Number of processed to be used for the bulk download processing.	4

Note: When using USE_RECOMMENDED_NUMBER_OF_PROCESSES or NUMBER_OF_PROCESSES on Windows or MacOS, please make sure that you are wrapping the method call under an if __name__ == "__main__" as shown in the example.

Bulk download

On the homepage MaStR/Datendownload a zipped folder containing the whole MaStR is offered. The data is delivered as xml-files. The official documentation can be found on the same page (in german). This data is updated on a daily base.

flowchart LR
  id1("📥 Download raw data
  from marktstammdatenregister.de") --> id2(Write xml files to database)
  id2 --> id3[("📗 open-mastr database")]
  id3 --> id4("🔧 Decode and cleanse data")
  id4 --> id3
  id3 --> id5("Merge corresponding tables
  and save as csv")
  id5 --> id6>"📜 open-mastr csv files"]
  click id1 "https://www.marktstammdatenregister.de/MaStR/Datendownload" _blank
  click id2 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/xml_download/utils_write_to_database.py#L17C6-L17C6" _blank
  click id4 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/xml_download/utils_cleansing_bulk.py#L10" _blank
  click id5 "https://github.com/OpenEnergyPlatform/open-MaStR/blob/7b155a9ebdd5204de8ae6ba7a96036775a1f4aec/open_mastr/mastr.py#L288" _blank
  click id6 "https://doi.org/10.5281/zenodo.6807425" _blank

In the following, the process is described that is started when calling the Mastr.download function with the parameter method="bulk". First, the zipped files are downloaded and saved in $HOME/.open-MaStR/data/xml_download. The zipped folder contains many xml files, which represent the different tables from the MaStR. Those tables are then parsed to a sqlite database. If only some specific tables are of interest, they can be specified with the parameter data. Every table that is selected in data will be deleted from the local database, if existent, and then filled with data from the xml files.

In the next step, a basic data cleansing is performed. Many entries in the MaStR from the bulk download are replaced by numbers. As an example, instead of writing the german states where the unit is registered (Saxony, Brandenburg, Bavaria, ...) the MaStR states corresponding digits (7, 2, 9, ...). One major step of cleansing is therefore to replace those digits with their original meaning. Moreover, the datatypes of different entries are set in the data cleansing process and corrupted files are repaired.

If needed, the tables in the database can be obtained as csv files. Those files are created by first merging corresponding tables (e.g all tables that contain information about solar) and then dumping those tables to .csv files with the to_csv method.

Note: By default, existing zip files in $HOME/.open-MaStR/data/xml_download are deleted when a new file is downloaded. You can change this behavior by setting keep_old_downloads=True in Mastr.download().

SOAP API download

MaStR account and credentials

For downloading data from the Marktstammdatenregister (MaStR) database via its API a registration is mandatory.

To download data from the MaStR API using the open-MaStR, the credentials (MaStR user and token) need to be provided in a certain way. Three options exist:

1. Credentials file: Both, user and token, are stored in plain text in the credentials file. The file is located at '~/.open-MaStR/config/credentials.cfg'. Fill in your user and token like this:

       [MaStR]
       user = SOM123456789012
       token = msöiöo8u2o29933n31733m§=§1n33§304n... # optional, 540 characters
   

   The token should be written in one line, without line breaks.

2. Credentials file + keyring: The user is stored in the credentials file, while the token is stored encrypted in the keyring.

   Read in the documentation of the keyring library how to store your token in the keyring.

3. Don't store: Just use the password for one query and forget it

   The latter option is only available when using MaStRAPI. Instantiate with

   MaStRAPI(user='USERNAME', key='TOKEN')
   

   to provide user and token in a script and use these credentials in subsequent queries.

MaStRAPI

You can access the MaStR data via API by using the class MaStRAPI directly if you have the API credentials configured correctly. Use the code snippet below for queries.

from open_mastr.soap_api.download import MaStRAPI

if __name__ == "__main__":

    mastr_api = MaStRAPI()
    print(mastr_api.GetLokaleUhrzeit())

The MaStR API has different models to query from, the default are power units ("Anlage"). To change this, you can pass the desired model to MaStRAPI. E.g. to query market actors instantiate it using MaStRAPI(service_port="Akteur").

For API calls, models and optional parameters refer to the API documentation.

Example queries and their responses (for model 'Anlage')

mastr_api.GetLokaleUhrzeit()mastr_api.GetListeAlleEinheiten(limit=1)mastr_api.GetEinheitSolar(einheitMastrNummer="SEE984033548619")

Response:

{
'Ergebniscode': 'OK',
'AufrufVeraltet': False,
'AufrufLebenszeitEnde': None,
'AufrufVersion': 1,
'LokaleUhrzeit': datetime.datetime(2023, 9, 25, 8, 1, 46, 24766, tzinfo=<FixedOffset '+02:00'>)
}

API function name: GetLokaleUhrzeit
Example query: mastr_api.GetLokaleUhrzeit()
Parameter: None

Response:

{
"Ergebniscode": "OkWeitereDatenVorhanden",
"AufrufVeraltet": False,
"AufrufLebenszeitEnde": None,
"AufrufVersion": 1,
"Einheiten": [
    {
        "EinheitMastrNummer": "SEE984033548619",
        "DatumLetzeAktualisierung": datetime.datetime(
            2020, 2, 20, 16, 28, 35, 250812
        ),
        "Name": "Photovoltaikanlage ERWin4",
        "Einheitart": "Stromerzeugungseinheit",
        "Einheittyp": "Solareinheit",
        "Standort": "48147 Münster",
        "Bruttoleistung": Decimal("3.960"),
        "Erzeugungsleistung": None,
        "EinheitSystemstatus": "Aktiv",
        "EinheitBetriebsstatus": "InBetrieb",
        "Anlagenbetreiber": "ABR949444220202",
        "EegMastrNummer": "EEG920083771065",
        "KwkMastrNummer": None,
        "SpeMastrNummer": None,
        "GenMastrNummer": None,
        "BestandsanlageMastrNummer": None,
        "NichtVorhandenInMigriertenEinheiten": None,
    }
],
}

API function name: GetEinheitSolar
Example query: mastr_api.GetListeAlleEinheiten(limit=1)

Parameter	Description
marktakteurMastrNummer	The MaStR number of the requested unit
startAb	Sets the beginning of the records to be delivered [default value: 1]
datumAb	Restrict the amount of data to be retrieved to changed data from the specified date [Default value: NULL]
limit	Limit of the maximum data records to be delivered [default/maximum value: maximum of own limit]
einheitMastrNummern[]

Response:

{
"Ergebniscode": "OK",
"AufrufVeraltet": False,
"AufrufLebenszeitEnde": None,
"AufrufVersion": 1,
"EinheitMastrNummer": "SEE984033548619",
"DatumLetzteAktualisierung": datetime.datetime(2020, 2, 20, 16, 28, 35, 250812),
"LokationMastrNummer": "SEL948991715391",
"NetzbetreiberpruefungStatus": "Geprueft",
"Netzbetreiberzuordnungen": [
    {
        "NetzbetreiberMastrNummer": "SNB980883363112",
        "NetzbetreiberpruefungsDatum": datetime.date(2020, 2, 25),
        "NetzbetreiberpruefungsStatus": "Geprueft",
    }
],
"NetzbetreiberpruefungDatum": datetime.date(2020, 2, 25),
"AnlagenbetreiberMastrNummer": "ABR949444220202",
"NetzbetreiberMastrNummer": ["SNB980883363112"],
"Land": "Deutschland",
"Bundesland": "NordrheinWestfalen",
"Landkreis": "Münster",
"Gemeinde": "Münster",
"Gemeindeschluessel": "05515000",
"Postleitzahl": "48147",
"Gemarkung": None,
"FlurFlurstuecknummern": None,
"Strasse": None,
"StrasseNichtGefunden": False,
"Hausnummer": {"Wert": None, "NichtVorhanden": False},
"HausnummerNichtGefunden": False,
"Adresszusatz": None,
"Ort": "Münster",
"Laengengrad": None,
"Breitengrad": None,
"UtmZonenwert": None,
"UtmEast": None,
"UtmNorth": None,
"GaussKruegerHoch": None,
"GaussKruegerRechts": None,
"Registrierungsdatum": datetime.date(2019, 2, 1),
"GeplantesInbetriebnahmedatum": None,
"Inbetriebnahmedatum": datetime.date(2007, 7, 20),
"DatumEndgueltigeStilllegung": None,
"DatumBeginnVoruebergehendeStilllegung": None,
"DatumWiederaufnahmeBetrieb": None,
"EinheitSystemstatus": "Aktiv",
"EinheitBetriebsstatus": "InBetrieb",
"BestandsanlageMastrNummer": None,
"NichtVorhandenInMigriertenEinheiten": None,
"AltAnlagenbetreiberMastrNummer": None,
"DatumDesBetreiberwechsels": None,
"DatumRegistrierungDesBetreiberwechsels": None,
"NameStromerzeugungseinheit": "Photovoltaikanlage ERWin4",
"Weic": {"Wert": None, "NichtVorhanden": False},
"WeicDisplayName": None,
"Kraftwerksnummer": {"Wert": None, "NichtVorhanden": False},
"Energietraeger": "SolareStrahlungsenergie",
"Bruttoleistung": Decimal("3.960"),
"Nettonennleistung": Decimal("3.960"),
"Schwarzstartfaehigkeit": None,
"Inselbetriebsfaehigkeit": None,
"Einsatzverantwortlicher": None,
"FernsteuerbarkeitNb": False,
"FernsteuerbarkeitDv": None,
"FernsteuerbarkeitDr": None,
"Einspeisungsart": "Volleinspeisung",
"PraequalifiziertFuerRegelenergie": None,
"GenMastrNummer": None,
"zugeordneteWirkleistungWechselrichter": Decimal("4.000"),
"GemeinsamerWechselrichterMitSpeicher": "KeinStromspeicherVorhanden",
"AnzahlModule": 22,
"Lage": "BaulicheAnlagen",
"Leistungsbegrenzung": "Nein",
"EinheitlicheAusrichtungUndNeigungswinkel": True,
"Hauptausrichtung": "Sued",
"HauptausrichtungNeigungswinkel": "Grad20Bis40",
"Nebenausrichtung": "None",
"NebenausrichtungNeigungswinkel": "None",
"InAnspruchGenommeneFlaeche": None,
"ArtDerFlaeche": [],
"InAnspruchGenommeneAckerflaeche": None,
"Nutzungsbereich": "Haushalt",
"Buergerenergie": None,
"EegMastrNummer": "EEG920083771065",
}

API function name: GetEinheitSolar
Example query: mastr_api.GetEinheitSolar(einheitMastrNummer="SEE984033548619")

Parameter	Description
apiKey	The web service key for validation
marktakteurMastrNummer	The MaStR number of the market actor used by the web service user
einheitMastrNummer	The MaStR number of the requested unit

Why can't I just query all information of all units of a specific power plant type?

As the example queries above demonstrate, the API is structured so that units of power plants types (e.g. wind turbine, solar PV systems, gas power plant) have to be queried directly by their unique identifier ( EinheitMastrNummer") and a distinct API query. To download all unit information of a specific power plant you need to know the "EinheitMastrNummer".

Firstly, by querying for all units with mastr_api.GetListeAlleEinheiten() you'll get all units, their unique identifier (EinheitMastrNummer) and their power plant type (Einheitentyp). You can then sort them by power plant type and use the power plant type specific API query to retrieve information about it.

MaStRDownload

MaStRDownload has been removed

In versions > v0.16.0 the MaStRDownload class cannot be used anymore.

MaStRMirror

MaStRMirror has been removed

In versions > v0.16.0 the MaStRMirror class cannot be used anymore.