Architecture Overview

Overview

This chapter covers a high-level architectural view of the OpenGameData infrastructure, including the various tools provided by the OpenGameData comunity. Additionally, it covers the

Data Storage

Data is currently spread across multiple systems:

• OpenGameData: The primary, “state of the art” storage system logs event data to a MySQL database. Within this database, there is one table for each game, though all tables share a common schema. Once nightly, the data from this MySQL database are moved to BigQuery for long-term warehousing and storage. Games using this system: Wake (Aqualab), Mashopolis, IceCube VR, Waddle (Penguins)

• Firebase + BigQuery: This system logs event data to Google Firebase, using the Firebase “universal” schema. Any data that does not fit into an existing column within this schema is relegated to the event_params field. Once nightly, the Firebase events are moved to BigQuery for long-term warehousing and storage. Firebase events can not be accessed directly. Games using this system: Legend of the Lost Emerald (Shipwrecks)

• Logger: This system uses a pair of MySQL database servers. Events are initially logged to fieldday-logger, and are copied via MySQL’s replication feature for long-term storage to fieldday-store. Both database servers have a MySQL database named logger, which in turn have a single table for all games, named log. Games using this system: Jo Wilder, Lakeland, all Yard Games

Data Processing

Requests

The Request class contains data defining a set of data to export, the set of generators to use, and the outputs. In particular, it has the following properties:

Property	Description
GameID	The game whose data will be exported
Interface	The interface for accessing the data
Range	The set of dates, session IDs, or player IDs to be retrieved from the interface for export
Outerfaces	A set of outerfaces, where exported data will be sent
Export[Type]	[Type] is any of RawEvents, ProcessedEvents, Sessions, Players, or Population. Each is a boolean indicating whether that type of data should be output to the outerfaces

Export Manager

The processing of data begins at the ExportManager class. This class is responsible for overseeing the execution of given Request instances; most of the actual work is delegated down to lower-level managers. The work done directly by ExportManager consists of pre-processing, processing, and post-processing phases.

Pre-processing

This phase mostly consists of opening the outerfaces and setting up the other managers for delegation. The ExportManager also attempts to locate the [Game]Loader class for the game whose data is being exported, if the loader class exists.

Processing

Post-processing

Event Manager

Feature Manager

Keeps a PopulationProcessor, plus PlayerProcessors for each player ID encountered in the dataset, and SessionProcessors for each session ID.

Processors

At their most basic, all processors simply have a game schema, process events, and return lines.

Generator Processors

Processors for generators add a loader class, an instantiation of that loader, and a registry.

Registries

Generators

Data processing is outlined by the graph below.

Automation

APIs

Dashboards & Web Tools

Game Schemas

This documents the format and rules for writing a game schema (GAME_ID.json) file.