.soy

pysoy's data format

<!> This document is a draft and under active development

Concept & Purpose

PySoy's design as a cloud game engine requires a file format to distribute game objects to clients in the cloud. The previous strategy of PySoy object creation and management soley by Python will not work in the cloud;

• Embedding Python would greatly increase the size of cloud clients
• Its cumbersome to embed binary values such as mesh and texture data in a Python source file or decode from existing universal formats such as COLLADA and PNG
• As Python has no security layer, we would have to take responsibility for ensuring the client was "sandboxed" against rouge game code being distributed in the cloud

It was determined that a format is needed which can create, link, and populate PySoy objects without embedding Python in cloud gaming clients. This format should also be usable within Python in much the same way as a Python module, to the extent that data in .soy files can be brought into the Python namespace with a simple...

import soy
sys.path.append("data")
import MyGame

... and the objects contained in data/MyGame.soy then available at MyGame as standard Python objects. If MyGame included a Window, widgets, a Scene, bodies, controllers, etc they would immediately appear on the screen with full interactivity. As .soy format contains no actual Python code, the same files could be made available on the cloud for game clients to use directly.

While its conceivable that .soy could contain a full interactive game by itself, in most cases the .soy files will contain PySoy network objects which connect to a server running Python.

Another issue we attempt to address with .soy is starting a game without having to wait for the full file to be downloaded. Muxing different types of data together in a progressive manner would allow objects to be used before they're fully transferred. For example, textures can transfer lower resolution versions (mipmap layers) early in the stream and leave the full resolution to the end of the stream after everything else has been created. Players will be far less annoyed by low resolution textures the first time they start a game than having to wait for the full game to download before playing.

.soy Importer

The .soy format is intended to be used within Python with the standard import command. To achieve this PySoy adds its importer to sys.path_hooks on initialization. This hook applies to any path added after import soy.

We aim to support network protocols such as HTTP through this same importer, thus the following code could work:

import sys
import soy
sys.path.append('http://media.pysoy.org/tests/data')
import blocks

To best support cloud gaming a single .small soy file which handles game initialization with larger game content stored in separate .soy archives is preferable for faster startup. Most of these clients will need to download the first .soy file in its entirety before launching the game through a browser plugin or standalone plugin, though later .soy archives will typically be loaded progressively by the client.

.soy Exporter

While Python offers a standard import mechanism it does not offer a mechanism to export a namespace from memory to a file (its easy enough to write .py files in a text editor). Therefore, we will offer a soy.Exporter class.

Objects of this class can be populated by PySoy objects and exports them to a file when called. Example usage:

export = soy.Exporter()
export.mainwindow = soy.widgets.Window('My Game')
(...)
export('data/MyGame')

An Exporter object could be created based on an existing namespace as well for editing existing .soy files, ie

import MyGame
MyGame.window.size = (640,480)
soy.Exporter(MyGame)('data/MyGame')

Its intended that Exporter objects can be added to other Exporter objects or other namespaces containing PySoy objects and other math functions. This can be used for development, to save game state, or even as part of PySoy-based game developer tools.

Implementation

Each PySoy object which can be stored provides two methods in libsoy, "import" and "export" which are exposed in PySoy as "import" and "export".

The libsoy backend does not contain any facilities for reading or writing .soy files, this is the responsibility of PySoy or cloud game client.

On import, chapter 0 is used to construct objects on the frontend (which also initializes them on the backend) and store references to them. Each chapter thereafter is sent to that object's import method for processing. When the stream is fully processed the import command returns.

On export, chapter 0 provides the object construction parameters and each chapter thereafter is a page for the .soy file. The Exporter is responsible for combining these into a valid .soy file and writing it to disk.

Exportable objects are duck typed; any object with an export method can be stored in a .soy file, though its class must be available before the object is later imported since .soy files may not contain new classes or other instruction code.

Soy Format Overview

Each object stored in a .soy archive consists of one or more "pages", each page placed in a sequential "chapter". Each .soy archive begins with chapter 0 (the "Table of Contents") which contains the first page of each object, followed by chapter 1 which contains the second page of every object which has a second page and so on.

Chapter 0 consists of the .soy signature, license, and pages for every object in the file. These pages each consist the object type name and the arguments used to construct the object. To simplify ordering, no object should require another object as a construction argument.

Each further chapter consists of pages whos formats are specific to the object they're intended for and as little overhead as is reasonable. Other objects in the same archive may be linked to here as they were already defined in chapter 0, for example a Window may list widgets within it on page 1.

Table of Contents

Also known as "chapter 0", the table of contents consists of the .soy signature, license, copyright, authors, date, the name of each object, its type, and the arguments needed to construct it.

The first 3 bytes of the stream should read "soy" signature followed by a byte identifying the major PySoy version that generated the file. This can be used as mime-magic to identify the file. The major version of this format version is 0. Decoding should not be attempted for an unsupported major version.

Following the signature is the metadata; authors, date, version, documentation, etc. These are made available in the resulting Python module as __author__, __date___, __version__ and __doc__ respectively.

The license field is required and must contain the string "GNU AGPLv3", any other license field will be rejected as incompatible. This field is included because future PySoy versions may be licensed under the AGPLv4 or other AGPLv3-compatible licenses which have not been written yet, and because the AGPLv3 requires that the license be stated. See [Licensing] for more details.

The __credits__ string will be generated from the copyright and the boilerplate for the stated license. The first line of __credits__ is stored as copyright on export, the license field is automatically the license of the version of PySoy used to export the archive (currently "GNU AGPLv3").

Metadata is stored in key,value pairs of null terminated strings. A pair of zero-length strings signals the end of metadata.

The chapter 0 header is sequenced as follows:

char[3] "s", "o", "y"
uint8   Major Version (0)
uint16  Authors Length
char*   Authors String
uint16  Copyright Length
char*   Copyright String
uint16  Date Length
char*   Date String
uint16  Docs Length
char*   Docs String
uint16  License Length
char*   License String
uint16  Version Length
char*   Version String
uint32  Number of Pages
loop    Pages

Pages:
uint16  Object Name Length
char*   Object Name String
uint16  Type Name Length
char*   Type Name String
uint8   Number of Arguments
loop    Arguments 

Arguments:
uint8   Data Type (0 = int, 1 = float, 2 = string)
type    Data

Int Data:
int32   Stored

Float Data:
double  Stored as "binary64" as per IEEE 754

String Data:
uint16  Length of Data
char*   Data

Authors

• ArcRiley
• BlaisePascal
• PalleRaabjerg