SE6MEN7AT0R Auto-Coding Tool

Introduction

The auto-coder is a cool tool for auto-coding.

Built for Built for OpenGameData{:width 40}

Or, built for OGD Logo

Data Tab

In the data tab, you load data to use in the system.

Table of contents

Loading Datasets

Load TSV is for uploading data in TSV format to work with.

You can also delete the datasets, including all the associated files like models, preprocessors…

Datasets can be downloaded from any game Raw Data at the OGD website. It should have the following columns:

  • session_id

  • app_id

  • timestamp

  • event_name

  • event_data

  • event_source

  • app_version

  • app_branch

  • log_version

  • offset

  • user_id

  • user_data

  • game_state

  • index

Dataset Info

Presents the filename, the date range of the dataset and the number of models trained. And the table for number of rows, segments, users, sessions in the filtered (explained next) and original datasets.

Event description

To provide a more informative event description, you can use the Load Event Descriptions button that allows you to upload a JSON file as follows:

[
    "switch_job": "switched from job '{event_data[prev_job_name]}' to job '{game_state[job_name]}'",
        "receive_fact": "received new fact: '{event_data[fact_id]}'",
    ...
]

That maps each value from event_name to a wrapping description, with replacement variables for event_data and game_state columns and writes the results to the event_description column

Event Filtering

To apply the changes Floppy Disk icon needs to be pressed.

A new auxiliary column is added, filtered_in. Therefore, we can exclude rows with certain values for event_name column setting filtered_in = False.

Pie charts

Labeling progress pie: how many segments are not labels, and how many segments are excluded. (if not segmented)

Labels count pie: how many segments are inside each class (also available at the top as text).

Downloading Dataset

If you ever want to “export” the dataset you have been working on, click the Download button.

Segment Tab

In the segment tab, you segment data, so multiple rows belong to the same segment.

Table of contents

IMPORTANT: the following operations overwrite the affected segments’ labels. You can lose your progress, download dataset before experimenting.

Manual segmenting

We select the user from the dropdown (events from different users will never be inside one segment) that also displays the number of segments.

Then you have to select the rows that you want to apply a segment_id value. Apart from just clicking, you can drag and scroll with the mouse, use Click and Shift for faster multiple events selection.

After selecting, you can set the Segment Id and hit Apply. The segment is saved to the segment_id column.

Automatic segmenting

To make segmenting much faster (or at least the prior segmenting in bigger segments), you can use the Cutoff Event Types.

Basically, the segments are formed by “cutting” the dataframe in segments using those rows that have an event_name value among the selected as the dividers. This cutoff event itself goes into the previous segment after dividing.

In addition, if the new session is started, it also goes to a new segment. So there are no different sessions events in the same segment, even if the user is the same.

Label Tab

In the label tab, you perform qualitative analysis and assign labels for specific behaviours identified in the segment

Table of contents

Labeling

You can either create a new label when typing and hitting enter or either select from existing labels.

The user can provide a description for each label either using the pencil button to edit existing or just importing a codebook that can look like this:

[
    {
        "code": "Struggle",
        "definition": "Indicates moments when a player encounters significant difficulty in completing a job."
    },
    {
        "code": "No Struggle",
        "definition": "Applied when a player progresses through a task smoothly without evident setbacks."
    }
]

Also, you can use the Justification control to make notes on why you chose that label.

After hitting Apply, the information is saved to the segment_labels column.

After applying, the top section Labels count updates after recounting.

Train Tab

In the train tab, you build machine learning pipelines (preprocessing and modeling) and perform model comparison.

All the values of the form are saved when sent to server after you hit Train button, and are restored when clicked on the model button in the left side panel in the Models Bar View.

Model (hyper)parameters

Target labels: Use if you want to exclude segments with certain labels when training the model.

Ratio of training data: instead of using 80/20 train-test split, you can modify the proportion in case your full dataset is bigger or smaller.

Class weights: in cases when number of objects within one category is much more than in other categories, to avoid model bias to improve scores on one label, it’s suggested to use class weights, so that errors in any class are penalized equally.

Another hyperparameter is the model type, which are explained in the next subsections

Logistic Regression

The simplest and fastest model, can be useful as a baseline

Regularization: used to reduce overfitting (good train performance, poor test performance), increasing this parameter’s value.

Penalty: type of the penalization, L1 or L2.

Random Forest

Slightly more complex model, doesn’t need scaling and null handling.

Number of Estimators: number of trees that will be used to generate the prediction, used to reduce overfitting.

Max Depth: increase to reduce bias.

Neural Network

Most complex models, needs more time to train, but can discover complex patterns in the data.

Epochs: number of training epochs, increase to improve accuracy.

Learning rate: very low can lead to very slow learning, very high can lead to not converging to the optima.

Hidden layers: setting the shape of the neural network with ReLU activation.

Dimensionality reduction

More complex model can lead to overfitting or multicolinearity. We need some techniques to reduce the size.

Scaling

To keep the data ranges similar and avoid giving more importance to one variable because the range is bigger, you can use scaling. Usually, Standard Scaling will work out well.

PCA

A common technique to reduce dimensions in high-dimensional data.

You can visualize the plot and select the number of components when the variance explained changed the most abrupt way.

Standar Scaling is useful to make the plot look more smooth.

If set to 0, PCA is disabled.

Features Selection

Selection of features can be performed manually (by default all the ogd features are excluded).

The columns are colored using the maximum Pearson correlation coefficients with the included features. The greener is the selection, the less collinearity, means less duplicate information which is bad for the model.

The Autoselect button build a logistic regression with all features included and uses L1 penalization to exclude features with coefficient = 0, therefore performing and automatic features selection.

Model comparison

We have a more visual Bar View and the classic Table View for model comparison.

The left panel is used for model selection to restore the parameters on the page. The percentage is the Test Accuracy.

The center panel has the plot of all the models’ metrics, that can be changed using the top dropdown to use label-specific metrics (Precision, Recall) or label-aggregated metrics(Test and Train Accuracy, F1, AUC).

The right panel has the model summary, that has a confusion matrix, row count, columns count and the time performance.

Applying models

Apply Selected Model is used to apply the last trained model or the currently selected in the models sidebar.

Apply Best Model is used to apply the best model by test accuracy.

Apply Tab

In the apply tab, you perform the revision of the labels that were output by the model.

Code Organization

The tool uses a bunch of libraries, and the code is split into several files.

Table of contents:

Backend

The flask webserver is defined inside main.py with all the routes.

Dependencies:

  • flask

  • joblib

  • numpy

  • pandas

  • polars-lts-cpu

  • scikit-learn

  • torch

uploads: main data folder

  1. Contains the uploaded datasets TSV files.

  2. In addition, for each dataset_file a <dataset_file>_models.json file is created to store all the training experiments information.

  3. And under uploads/models subdirectory, the models and preprocessors are stored (using joblib and torch).

Detailed file structure

1. DATA TAB

dataset.py

  • Read the dataframe

  • Get dataset info

  • Get users list

events.py

  • Create extended description for event_name column

  • Filter by event_name column

2. SEGMENT TAB

segment.py

  • Manual and automatic segmenting, writing to segment_id column

  • Get segments list for user_id

3. LABEL TAB

label.py

  • Labels rows, writing to segment_labels column

4. TRAIN TAB

ogd_features.py

  • Calculates OGD numeric features (float, int, bool)

dim_reduction.py

  • Calculates PCA metrics with scaling as argument

  • Calculates correlation matrix

  • Autoselects features building a logistic regression with regularization to exclude useless features with zero coefficients.

preprocess.py

  • One hot encodes the event_name column and calculates sum and average grouping by user_id and segment_id, so that we have 1 row for each segment.

  • Adds more columns like segment_duration in seconds and events_counts.

  • Merges the new dataframe with the OGD features.

train.py

  • Train models (specific implementations are in train_logistic.py, train_random_forest.py and train_neural_net.py) with certain hyperparameters, saving the model and the results.

metrics.py

  • Calculates evaluation metrics saved on training

5. APPLY TAB

inference.py

  • Loads existing models using the *_models.json from the uploads folder

  • Applies selected model to get the label with max confidence (saved to predicted_labels and prediction_confidence).

Frontend

  • static folder for the assets

  • templates folder for html templates

Dependencies:

  • Bootstrap

    • Bootstrap icons

  • jQuery

  • Select2

  • Chart.js

  • DataTables with plugins