Data Connectors, Interfaces, & Outerfaces

OpenGameData uses classes called connectors to aid in loading input data and writing output data. Specific connector subclasses, called ``interface``s, perform **in**put, and subclasses called ``outerface``s perform **out**put.

Introduction to Data Storage and Interfaces

`StorageConnector`s, `Interface`s, and `Outerface`s

OpenGameData uses classes called StorageConnectors to establish connections to data stores.

Data stores may include a local file, an external file host, or a database:

---
title: Basic Data Stores
---
flowchart TD

StorageConnector --> local[(Local File)]
StorageConnector --> remote[(Remote Fileserver)]
StorageConnector --> Database[(Database)]

However, each data store might in turn contain multiple resources, such as files or database tables:

---
title: Detailed Data Stores
---
flowchart TD

StorageConnector --> local[(Local File)]
StorageConnector --> remote[(Remote Fileserver)]
remote[(Remote Fileserver)] -.-> file1.tsv
remote[(Remote Fileserver)] -.-> file2.tsv
remote[(Remote Fileserver)] -.-> file3.tsv
StorageConnector --> Database[(Database)]
Database[(Database)] -.-> database1
database1 -.-> table1
database1 -.-> table2
Database[(Database)] -.-> database2
database2 -.-> table3

We use Interface and Outerface classes are used for data input and output, where an Interface class has specific functions for retrieving data from a store, and an Outerface has functions for writing data to a store. For any given storage medium, then, there may be a Connector, Interface, and Outerface class. In the example below, we show how the MySQLConnector, MySQLInterface, and MySQLOuterface are related:

---
title: Storage Connector Hierarchy
---
classDiagram

StorageConnector <|-- MySQLConnector
Interface <|-- MySQLInterface
MySQLConnector <|-- MySQLInterface
MySQLConnector <|-- MySQLOuterface
Outerface <|-- MySQLOuterface

In this hierarchy, we separate connection logic from data I/O, with a common base class called StorageConnector to handle connection logic. Interface and Outerface base classes exist independently as mixin classes that define a set of functions for reading or writing data. Then for any data storage medium we want to support, we write a subclass of StorageConnector, e.g. MySQLConnector, to take in the corresponding config and call appropriate functions from whatever MySQL library is in use. To create an actual Interface or Outerface for MySQL, we would create MySQLInterface or MySQLOuterface that inherits from MySQLConnector and Interface/Outerface.

Summary of Interface Class Relationships

Putting together all the various classes described in the sections below, and hiding direct connections where certain configs/schemas are passed to an upper-level class through lower-level classes, the proposed refactor/redesign can be summarized as follows:

---
title: Full Hierarchy of Storage Classes
---
classDiagram

StorageConnector <|-- MySQLConnector
MySQLConnector <|-- MySQLInterface
Interface <|-- MySQLInterface
StorageConnector <|-- FileConnector
FileConnector <|-- FileOuterface
Outerface <|-- FileOuterface

DataStoreConfig *-- "1" CredentialConfig
DataStoreConfig *-- "1" LocationConfig
CredentialConfig <|-- PasswordCredentialConfig
CredentialConfig <|-- KeyCredentialConfig
LocationConfig <|-- HostLocationConfig
LocationConfig <|-- FileLocationConfig

TableSchema *-- "1" TableStructureSchema
EventTableStructureSchema <|-- TableStructureSchema
FeatureTableStructureSchema <|-- TableStructureSchema
TableSchema *-- "1" TableLocationSchema
TableLocationSchema <|-- DatabaseTableLocationSchema

GameDataConfig *-- "1" DataStoreConfig
GameDataConfig *-- "1..2" TableSchema
MySQLInterface *-- "1" GameDataConfig
FileOuterface *-- "1" GameDataConfig

(for additional details, see Proposal #61 on GitHub)

Storage Connectors

STUB: This section is on the to-do list.

Interfaces

STUB: This section is on the to-do list.

Writing an `Interface`

All interfaces inherit from an Interface base class, which in turn inherits from StorageConnector. Thus, each interface must implement the functions required by its base classes.

`StorageContainer` Functions

First, StorageConnector subclasses must implement an _open and _close function for connecting to and disconnecting from a corresponding storage resource (such as a file or database). Typically, this is as simple as importing and using an appropriate package for the resource you want to connect to. For example, the MySQLEventInterface imports the following:

from mysql.connector import connection, cursor

It then uses the imported connection module to establish and close connections to a MySQL database.

`Interface` Functions

Second, every Interface subclass must implement functions to allow a user to first collect some information on the data contained in the connected resource.

Work In Progress below

Mostly this is for Events, but we also will allow reading Features. Their functions allow querying of the ID, version, and timing information available in the source.

In particular, they have the following functions:

AvailableIDs - gets list of session or player IDs available, subject to date and version filters
AvailableDates - gets range of dates, subject to ID and version filters
AvailableVersions - gets list of log or app versions available, subject to ID or date filters

These functions allow higher-level logic to pre-check what’s available before loading it.

Event & Feature Interfaces

These define actual data retrieval.

They have the following function:

GetEventCollection - takes an EventTableSchema, gets a collection of events, subject to ID, date, and version filters; only ID filter is required. Generally a good idea to use other filtering at AvailableIDs step to get session/player IDs, rather than trying to re-filter here, but option is available.
GetFeatureCollection - takes FeatureTableSchema, gets a collection of feature data objects, subject to ID, date, and version filters.

Event & Feature Outerfaces

These define how to send data out of the system.

Writing an `Outerface`

In the opengamedata-core code, the DataOuterface base class is itself a subclass of the Interface base class. Thus, like any Interface, it must implement the _open and _close functions. In addition, DataOuterface adds its own set of abstract functions for implementing the writing of data. We’ll discuss this in greater detail below.

`init` Data

The DataOuterface base class is initialized from two objects, a GameSourceSchema and a set of ExportModes:

  def __init__(self, game_id, config:GameSourceSchema, export_modes:Set[ExportMode]):

The GameSourceSchema and ExportMode set can be accessed by self._config and self.ExportModes, respectively (to modify the modes, you should use the private data member self._modes) You can optionally add parameters for your DataOuterface’s constructor, but this is generally not recommended as there is no built-in way to ensure the values of those parameters can be set in a config file.

The `GameSourceSchema`

The GameSourceSchema class contains four pieces:

A DataHostConfig object, (self._config.DataHost) which includes the location and credential information needed to access a data host, such as a database or file system.
A database name (self._config.DatabaseName) and table name (self._config.TableName), which can be used for cases where the data host is a database system, to locate the data once the config information has been used to establish a connection. These can be ignored for cases where they are not applicable, such as when the data host is just a local file.
A TableSchema, which indicates what columns are available in the format supported on the data host (such as the columns of the specific database table indicated by self._config.TableName), and how those columns map to the elements of the specific data type handled by the DataOuterface.

When writing a DataOuterface subclass, you do not need to directly interact with the TableSchema, but the DataHostConfig and (optionally) database/table names will be used in your code for connecting to the game source.

The set of ExportMode enum objects indicates which types of data output have been requested. You typically will not need to interact directly with this set, but in some cases it may be useful. Specifically, if you are writing a DataOuterface for a file system, you can elect to open only the files corresponding to the requested output modes.

Implementing the `Interface` Functions

The Interface base class defines abstract functions _open and _close. These must be implemented by each new interface or outerface.

    def _open(self) -> bool:

    def _close(self) -> bool:

These functions are simple in concept but nontrivial in implementation. The _open function should create and store (as an element of self) an open connection to the DataHost, and _close should close that connection.

The `_open` function

Practically, _open should use whatever Python package is available for communicating with the given database or file type, and return True if the connection was opened without errors. In case of errors, it is acceptable to simply raise an error and let the system cancel the data export, though if there are any means to retry a connection, we recommend implementing a short retry loop of up to 5 retry attempts. It is also acceptable to print a reasonable error message and return False.

As discussed above, the DataHostConfig should contain the location and any necessary credentials to open a connection. The specific instance received should be from a subclass of DataHostConfig specific to the system for which you are writing a DataOuterface. Refer to that class’ interface to see what data members are available. If no such class exists, it may need to be written before the DataOuterface can be fully implemented.

If you are writing a DataOuterface that needs separate connections for each type of output (e.g. outputting to static files), you can use self.ExportModes to determine which connections to open.

The `_close` function

Like _open, the _close function should return True if the connection to the data host was terminated without errors. In case of errors, we prefer to handle within the _close function rather than raising an error to a higher level, although raising an error is acceptable.

Implementing the `DataOuterface` Functions

The DataOuterface base class defines a significant number of abstract functions to be implemented. These cover five different kinds of output:

“Raw” events
“Processed” events
“Session” features
“Player” features
“Population” features

From an implementation perspective, there is little to no difference in how “raw” and “processed” events are output, nor is there typically any difference between outputting “session”, “player”, and “population” features. The primary difference is that, in several cases (such as outputting data to files), it is useful to separate these kinds of data into different output locations. Thus, while it is not necessarily the most efficient organization of functions, we implement one function for each kind of output, where the functions within the event and feature categories are largely identical to one another.

    def _destination(self, mode:ExportMode) -> str:

    def _removeExportMode(self, mode:ExportMode) -> str:

    def _writeRawEventsHeader(self, header:List[str]) -> None:

    def _writeProcessedEventsHeader(self, header:List[str]) -> None:

    def _writeSessionHeader(self, header:List[str]) -> None:

    def _writePlayerHeader(self, header:List[str]) -> None:

    def _writePopulationHeader(self, header:List[str]) -> None:

    def _writeRawEventLines(self, events:List[Event]) -> None:

    def _writeProcessedEventLines(self, events:List[Event]) -> None:

    def _writeSessionLines(self, sessions:List[List[FeatureData]]) -> None:

    def _writePlayerLines(self, players:List[List[FeatureData]]) -> None:

    def _writePopulationLines(self, populations:List[List[FeatureData]]) -> None:

Currently-Supported Storage Connectors

At present, OpenGameData has built-in support for the following types of storage, via corresponding classes in opengamedata-common:

Local TSV Files
MariaDB MySQL
BigQuery

Data Connectors, Interfaces, & Outerfaces

Introduction to Data Storage and Interfaces

StorageConnectors, Interfaces, and Outerfaces

Summary of Interface Class Relationships

Storage Connectors

Interfaces

Writing an Interface

StorageContainer Functions

Interface Functions

Work In Progress below

Event & Feature Interfaces

Event & Feature Outerfaces

Writing an Outerface

__init__ Data

The GameSourceSchema

Implementing the Interface Functions

The _open function

The _close function

Implementing the DataOuterface Functions