OGD ReadTheDocs Self-Documentation

Overview

Inevitably, for any sufficiently large project, a few words of that project’s documentation must be spent describing how the project is documented. This section of the OpenGameData (OGD) documentation does just that.

The tech stack for the opengamedata-doc project can be roughly broken down as follows:

Storage : GitHub
Implementation : reStructuredText (ReST) and Markdown
Build : Sphinx
Deployment : readthedocs (rtd)

Storage

The documentation source is stored in a project GitHub. There’s not a lot else to say about this. If you don’t have experience with Git, you might start with the git-scm docs. From there, you can learn about the services offered by GitHub with its getting started guide.

Implementation

Languages

To write the OGD documentation, we use a combination of reStructuredText (ReST), Markdown, and a small handful of other special-use formats. ReST is the default format for writing documentation used by the Sphinx build system. Other formats, including Markdown, are supported via Sphinx extensions (e.g. the myst_parser extension).

ReST has a steep learning curve but offers significant features for controlling the structure of a document, as its name suggests. Markdown is a markup language that is simple, easy to understand, and fast to write. Thus, use Markdown to write most major sections of documents, and use ReST to assemble the individual Markdown docs into well-structured pages.

Markdown

We use Markdown as the primary way to write documentation of the OpenGameData system. As mentioned above, Sphinx prefers reStructuredText, so we utilize the myst-parser extension to render Markdown documents. This extension actually uses an implementation of Markedly Structured Text, or MyST. MyST is built on the CommonMark specification, which is a particular dialect of Markdown, but adds some functionality of its own.

In general, we strive to only use the features of Markdown that are common across most or all Markdown dialects. Any cases requiring advanced functionality should just be written in ReST. The main exception to this rule is the use of \(\LaTeX\)-style math parsing, which is covered in the “Special Cases” section below. That said, it may be helpful to be aware of non-standard things done by the parser we use. For nitty-gritty details, see the latest version of the CommonMark specification, as well as MyST’s further extensions of CommonMark.

Markdown Primer

The Markdown Guide website is a good reference on the (relatively simple) syntax of Markdown, including documentation of several popular extensions to the core Markdown language. In addition, the Markdown Guide has a nice quick reference on the most common syntax across Markdown and Markdown-like dialects.

Below, we include our own quick reference, adapted from the quick reference linked above, for the elements we most commonly use. The “Our Use” column indicates our conventions for when/how to use the various bits of syntax to create a visual language within the documentation. Most uses are obvious, but some are scoped to specific uses:

Markdown	Syntax	Our Use
Heading	# H1	Doc Headers
	## H2	Section Headers
	### H3	Subsection Headers
	etc…	etc…
Bold	Bold	System/Tool Names
Italic	Italic	Keywords
Quote	> Quote	Keyword Definitions
`code`	`code`	Variable/Class/Library Names
`code block`	```code block```	Example Code
Link	[Link](Target URL)	Links to Other Pages
Unordered List	- Item 1	Lists of Items
	- Item 2
	- …
Table	\| Column 1 \| Column 2 \| … \|	Tables of information
	\| — \| — \| … \|
	\| Value 1 \| Value 2 \| … \|

reStructuredText

ReST Primer

Special Cases

As mentioned, we have a few special cases that use special formats outside of ReST and Markdown:

\(\LaTeX\) formatting: The myst-parser extension supports further extensions of its own, described on myst-parser’s own readthedocs site. We use dollarmath and amsmath to support inline use of \(\LaTeX\)-style implementation of mathematics content (myst-parser page).
Code-level documentation: We use the autodoc extension of Sphinx, which turns module and function comment blocks into readable API reference (sphinx-doc page).
Graphs and Visualizations: We use Graphviz, a graph definition language enabled by the graphviz extension (sphinx-doc page). Note that documents using Graphviz heavily must be implemented in ReST, as Markdown does not have a means for rendering in-line Graphviz graphics.

Documentation Organization

Now that we’ve covered the languages used to write the documentation, we can discuss the organization of the documentation implementation. From a high-level point of view, the documentation is made up of several Units (listed in the overview). Each unit is, in turn, made up of several Chapters.

Units

The documentation source contains one folder for each unit, containing the section chapters, as well as a unit index. The unit index is always named index.rst and lists each chapter, acting as a table of contents for the unit. There is a unit index template available.

The unit folder should also include a markdown file named _<unit_name>.md, which contains a brief overview of the unit contents. In particular, it should contain a single second-level header called “Overview” with one or more paragraphs summarizing the contents of the unit. This will be the first item included in the index.rst and serves a dual purpose:

Provides a uniform style of introduction to each unit.
Acts as an easily-identifiable “name” for the unit within the source directory, allowing us to leave unit names numeric, which should reduce the number of truly broken links whenever we add, rename, merge, or otherwise reorganize units.

There is a unit overview template available.

Chapters

Chapters are implemented with a similar structure to units. However, since they contain true content, they are made up of an index.rst document merging multiple Markdown shards.

The pieces making up the chapter will be placed in a subfolder, with a numeric name like chapter_01/. The “assembly” reStructuredText file is then named index.rst, and uses the .. mdinclude directive to assemble the individual Markdown shards, shown in the reStructuredText chapter index template. The Markdown shards, in turn, will each begin with level-2 headers, and include only their section of the content. This is demonstrated in the Markdown shard template. The first shard should be named beginning with an underscore, followed by the chapter name (e.g. _chapter_name.md), similar to the convention for units. A chapter may have a single shard; it is not necessary to break the content down further for short chapters.

Folder Structure

Taking all the information given above, the structure of the documentation source has a structure similar to that demonstrated below:

unit_01/
- chapter_1/
  - _ch1_name.md
  - index.rst
  - ch1_shard_1.md
  - ch1_shard_2.md
- chapter_2/
  - _ch2_name.md
  - index.rst
  - ch2_shard_1.md
  - ch2_shard_2.md
  - ch2_shard_3.md
- …
- _unit_a.md
- index.rst
unit_02/
- chapter_1/
  - _ch1_name.md
  - index.rst
  - ch1_shard_1.md
- chapter_2/
  - _ch2_name.md
  - index.rst
  - ch2_shard_1.md
  - ch2_shard_2.md
- …
- _unit_b.md
- index.rst
…

Build

Sphinx

We use the Sphinx system to build the opengamedata-doc documentation from source into a usable HTML website. The Sphinx build process is configured in source/conf.py There’s nothing particularly special about our configuration, but you can read about Sphinx configuration generally on the sphinx-doc website.

We use several Sphinx extensions to support building various implementation languages into our documentation. We will link to individual extensions’ pages as they are discussed below; for general information on Sphinx extensions, see the sphinx-doc extensions page

sphinx-apidoc

We use the sphinx-apidoc command to prepare/update the reference section of the documentation from submodules containing the code from the various OpenGameData Python repositories.

We do the following, for example, to document the ogd.core module:

  > sphinx-apidoc -o src/reference/core --tocfile index-core --remove-old --implicit-namespaces opengamedata-core/src/ogd/

readthedocs

Our build and deploy process uses the readthedocs provider to build and host our documentation project. The opengamedata-doc repository communicates with readthedocs via a GitHub webhook. Any changes pushed to the opengamedata-doc repository will trigger an automatic build and update on readthedocs.

Submodules

At times, it is useful to include submodules in the opengamedata-doc repository, which contain code or documentation that can be parsed or otherwise included by the main Sphinx project. Existing examples include the opengamedata-core and platform submodules included in the base opengamedata-doc directory.

While these are useful, it is important to note that when adding the submodule from a repository on GitHub:

> git submodule add <repository GH URL> <folder name>

You must use the https URL, not the more-common ssh URL. If you use the ssh URL, the automatic build on readthedocs will fail on its git submodule update step, with a permissions error.

Deployment

The documentation is deployed to our readthedocs site after building. Further, the project is configured via webhook to rebuild and deploy every time changes are pushed to the opengamedata-doc GitHub project. Configuration of the readthedocs build process lives in the .readthedocs.yaml file (see here for rtd’s config documentation)