Integrating Verse to PySoy

a Summer of Code proposal for the Python Software Foundation by Toni Alatalo

Basic personal information

I am a student of information processing sciences in the University of Oulu, Finland. The program covers software engineering and information systems design, and I have been in a special curriculum for digital media where we have done e.g. interaction and graphic design, and hypermedia theory and computer game studies. I have also studied film and tv studies in the humanities faculty on the side. I have now practically finished my master thesis, where I implemented a system for controlling character animations in Python in Blender, but have some remaining courses still that will do next term. Last summer I was privileged to work on the IPython notebook system in a summer of code project. During last winter I learned more of 3d graphics work practice in the open movie project Orange, for which did mostly Python programming. Now I wish to continue on that track and contribute more to the pythonic open source 3d creative commons with this project, and later after graduation also benefit from the advances in the business we are planning to launch. More information about my interests is at the end of this document, whereas the following parts go the actual matter, that is the present the idea for the work and the plan to make it real.

Project Abstract / Summary

Repeated saving of files in authoring tools and re-loading them in the real-time engine is a basic bottleneck in the creation process of interactive content, such as games. When modeling 3d geometry, the details of how the result looks are typically different in the modeling application compared to the final result in the engine used. In texture painting, the 2d image applications do not show how the bitmap is wrapped on the 3d surface, not to mention how it affects it (e.g. specularity or bump mapping), so the artist has to paint blind.

So in typical cases, the file conversion based preview of the results disrupts the work-flow and makes getting the details right a tedious job. A solution is provided by a network protocol called Verse, that specifies and implements synchronizing geometry and image data live between applications. The prominent open source authoring applications Blender and Gimp support it already, but our Pythonic game engine of choice --- PySoy --- does not yet (nor does any other open source engine at the moment). This proposal lays out a plan how to integrate the necessary parts of Verse to PySoy as a Summer of Code project. The Blender Verse integration, by the way, was largely done as a SoC2005 project.

When this is done, a PySoy view can be used to preview the final look of the models being worked on in Blender, with their textures being painted in Gimp, in real-time without any user interaction needed for getting the updates. Verse supports collaboration so that in a larger team any number of artists can be simultaneously working, also from anywhere over the Internet, the overall result getting build on the server. For example, Blender can be used as a level editor for a game that is simultaously running on other computers, where game testing is done, so that changes in the level design can be "felt" immediately. This leaves unprecedented space for innovative changes of workflow that we probably can not even fully realize now. On the other hand, answering the demands of Verse integration lead to also otherwise useful improvements to dynamic mesh editing and animated textures in the PySoy core.

copied from old Soya section during site cleanup

Introduction

PySoy is a relatively mature Python 3d game engine, that has been used in a fully finalized GPLd game Slune, and several other projects. It is written in Pyrex, and uses OpenGL for all graphics on that level (from compiled code), thus achieving good performance while simultaneously providing a nice pythonic API.

Verse is a network protocol for sharing different kinds of data needed in 3d production, such as geometry and textures, in real-time between different applications (or different instances of the same application). Besides native Verse tools developed by the core team, support for the protocol has been added in the open source 3d modelling, animation and rendering suite Blender, and in the Gimp image editor. PySoy users often use exactly these tools for making the models, animations, levels / worlds and textures to be run in the engine, but as PySoy has no Verse support yet, they can not benefit from the real-time preview of the final result in the engine. Apart from adding Verse support to open source tools, the Uni-Verse consortium has also prototyped geometry edit synchronization in two prominent proprietary modelling tools, namely Maya and 3ds Max.

In game production it is not enough to have the textures visible in the modelling application, as the shading algorithms etc. used in the game engine are different and hence result in a different final image. Hence it is necessary to add Verse support to game engines too, to have the final result --- textures from the painting application such as Gimp, and geometry and the UV-coordinates for the texture mapping from the modelling application --- in the game engine so that it can visualize them in the actual way. Having all this data update live, when e.g. a texture artist is painting and a modeller is tweaking the geometry, is a great improvement in the workflow, where today typically the workers have to run their data thru a (series of) export(s) .. it can even take minutes for them to see how the work done actually looks like, and often they then need to return back to the authoring application to correct mistakes etc.

The Orange open movie project tested the gimp-blender connection, and after the improvements that have been done in the verse bitmap performance etc., that cross-application connection is already useful for movie production. The Verse developers are also in the talks with the Gimp developers to further improve this support. For games, where smaller textures are used, the performance of the verse-gimp plugin is already certainly good enough.

Blender is the primary application used for creating the models and animations to be used in the PySoy engine, via the typical BlenderPython -scripting using export route. Verse-Blender already supports very efficient live update of geometry changes over Verse. It does not support animation yet, but that will be hopefully added sooner than later by the Verse-Blender developers. Blender is also often used as a world / level editor for games run in PySoy. Verse does not directly support game level special data, and it would not make sense for it to do so because it is highly gameworld dependent, for example AI waypoint markers etc. They can be synchronized, however, from Blender via Verse to PySoy and other tools with the Verse tag mechanism (basically custom metadata fields).

Requirements

For live preview of modelling and texturing in PySoy via the Verse protocol, the things to do are: (this list is to be prioritized together with the mentor!)

  • getting object and geometry nodes from Verse to PySoy
    • Blender already sends both of these fully (object nodes are the location and rotation of the thing in space, and geometry the shape of it)
  • getting textures (bitmap nodes) from Verse to PySoy
    • Gimp support for this is adequate already
  • getting UV coordinates from Verse to PySoy
    • Verse-Blender does not support UV coords yet, but it is probably a relative easy addition that we (PySoy developers) can request from the Blender developers or even do ourselves. Verse itself supports UVs and some of the verse tools from the uni-verse project have already working implementations of them.
  • getting material settings from Verse to PySoy
    • note: Verse-Blender does not support materials yet, and the whole Blender material system has been restructured for the upcoming 2.42 release (also to match the Verse material model better)

Object, geometry and bitmap synchronization from Verse to PySoy are the first targets, because they are both supported already by Blender and Gimp respectively, and as they are useful features in the creation workflow. Adequate support for these is also the absolute minimum requirement for the completion of this project. The priorization of the other features is to be done with the mentor, also depending how Verse-Blender will get them (e.g. material settings vs. UV coordinates).

Design and Implementation plan

My plan to add Verse support to PySoy is the following:

As pushing geometry and texture changes (efficiently) from PySoy is not needed at this stage (the first need is to pull and show them), hooking Verse calls to the geometry changing parts of the Pyrex-written PySoy core is not required. The Verse developers already provide a well-working Python module, made with SWIG but targeted directly for Python only, which exposes the c functions transparently and uses the same official code for the networking as the native Verse tools do. Using those in a Python-written PySoy add-on module is sufficient at least for prototyping the first needed functionality, and possibly as a permanent solution too. It is still some non-trivial work, because Verse is a very low-level protocol.

PySoy already supports changes in object geometry at run-time, with objects of class World, so implementing geometry updates live from Verse for those objects is a straightforward first step. However, as PySoy applications generally always use the optimized Shape class for geometry, it would be desireable to enhance the core so that also these so-called shapified objects could be modified to reflect changes from Verse. This needs changes in the internal Pyrex code and involves design decisions which I must coordinate with the project leaders. Object node support, i.e. changing the position and rotation (and scale) of a thing in space, is relatively simple and straightforward and can be fully done with the existing PySoy API.

With texture changes I can first quickly make an early prototype by just having a component get Verse bitmap updates, and show them in PySoy by simply assigning new image using the new version of the data for to the texture of the test shape. However, decent support requires again enchancing the core to support updateable / animated textures. It willl help to learn from how this has been recently done in both Verse-Blender, where texture support was added (in c) early this year, and also from the native Verse tools (the original Verse creator Eskil Steenberg's C-written renderer just got live texture updating, featuring progressive drawing of the updated textures, during the last week of April 2006). Also the gimp-plugin not only sends but also receives bitmap node changes from Verse, but is probably not the best example of it because it has to work with the limitations of the plugin API there. Also as both Blender and the native Verse tools use OpenGL for graphics, like PySoy does too, the implementations there are closer to what is needed here. Obviously, adding animated textures to PySoy will be useful and fun for many other purposes too.

For both geometry and texture synchronization, it probably makes sense to only do the (minimal) necessary additions to the Pyrex written core, and provide them as clean, non Verse specific API additions in PySoy. The actual Verse support can the be an optional add-on module, written in pure Python. This is possible because PySoy already features an extensible mainloop mechanism, the Idler, that has already been used for networked games. The Verse networking support added in this proposed project does not overlap with the existing game networking code, because both the purpose of the creation-tools-integrating Verse support and the nature of the protocols (e.g. bandwitdh use) are substantially different.

Schedule

  • May 23rd - May 30th: first prototypes of both object + geometry and texture data updates from Verse to PySoy
  • May 31st - June 22nd: actual base overall PySoy Verse support, and beginnings of all areas of functionality targeted on top of it. Besides the main Python-written module, includes first necessary changes in the Pyrex-written core. And hopefully can be done well in a test-driven fashion.

(a summer holiday break)

  • June 26th - June 30th: fixing and polishing the base, also documenting it more for review. making plans for the remaining part.
  • July 1st - August 4th: attacking the hard problems in the Pyrex-written core: implement decent support for animated / updateable textures if that was not solved earlier, implement modifyable Shape objects, or a sensible robust mechanism around that problem. keep improving the Python-written verse module as necessary. study the feasibility of getting material settings and UV coordinate support to Blender unless someone else has done it already.
  • August 7th - August 12th: tie up loose ends, document, get people to test as much as possible and fix everything.
  • August 14th - August 21st: sit back, relax, have free time and let the great reviews and feedback come. that is: a week is kept as a buffer to cope with delays in the project, fighting with problems and fixing nasty issues someone finds lurking in the code.. (despite the nice unit tests i made!)

Process

I would like to use test driven development in the process, after the first quick throwaway prototypes, but at this point it is somewhat unclear how to write tests for Verse operations --- also for the core Verse developers themselves, so that is something we can hopefully come up with together. Also otherwise my style of development derives largely from the extreme programming / agile development approaches, altho / because I know the traditional 'big design up front' approaches too. That is the usual mantras many of us happily share now: do the simplest thing that can possibly work, have always working code at the end of the day, release early - release often, refactor mercilessly. And yes, I plan to have a summer holiday, and certainly hope don't need to work overtime much (also an XP principle! - and I think all-nighters easily just end up in messy code, i.e. are not efficient).

Finally, if you let me go on the wild side for a sentence, it is possible that this work develops also into a general purpose Python Verse package, that can be useful in integrating other tools that have Python-bindings (like Ogre3d) or are written in Python (like PythonCAD) to Verse. But this is not set as a primary goal in this Summer of Code plan, so no unnecessary framework building will be done before the actual things needed here work adequately --- refactoring this to something more general can be done later if it makes sense.

References

New PySoy site (trac),  http://www.pysoy.org

  • http://www.pysoy.org/wiki/SummerOfCode/2006/Verse is the on-line version of this document there, and it will develop during the project to reflect the status (being integrated with the overall PySoy roadmap and timeline in that trac, and the coded in Subversion of course).

The Verse protocol,  http://www.blender.org/modules/verse/

Brecht van Lommel's Verse Gimp plugin, and PyVerse examples,  http://users.pandora.be/blendix/verse/

The Uni-Verse consortium,  http://www.uni-verse.org/

The First and the Last, an open world building project being drafted,  http://www.firstandlast.org/ (just a link to the wiki at  http://studio.kyperjokki.fi/FirstAndLast/ so far)

Personal experience and motivation

I am an experienced Python programmer, having earlier done the technical part of my master thesis in Python (real-time control of character animations, using the Blender game engine), and know PySoy, from both using it (e.g. the plant toy features real-time mesh changes in PySoy, which is also required for this Verse integration - see  http://studio.kyperjokki.fi/engine/SoyaPlant ) and under the hood from doing the macosx port and helping with some bugfixes. Am also familiar with 3d tool programming in Blender, mostly from working via and on the Python API there (am a Blender contributor with CVS commit rights), but also doing some fixes in the c written core. Recently I have also studied the Verse model, tested using it with Gimp and Blender, and writing own Verse clients using the Python bindings. Also I succesfully completed my Summer of Code project last year for IPython Notebooks --- a major lesson learned from there being the niceness of test-driven development (how it enabled refactoring), our university courses do not teach that approach (yet). Besides recent experience from using C in Blender, I wrote some games in C as a kid (when was 14) so do feel pretty confident about the Pyrex work necessary for this PySoy integration, even though am weaker there than on the Python side (so am looking forward to doing this also to ramp up my pointer handling and memory management skills!)

After finishing the remaining courses at the university, I plan to fully focus on our own enterprise where we create pictures, videos and 3d worlds and develop tools for making and using / showing them. I have already used PySoy together with Blender in early prototyping of our ideas (like opengl image showing apps), and believe that having them work together more closely in the future will be both great fun for experimenting and working with the tools. One project in which Verse may be a cool way of world building is our open project called First and Last, at  http://www.firstandlast.org/ (only points to the wiki with planning / drafting now - no worlds are on-line yet).

There are many other individuals, groups and at least one company now too using PySoy for game development, and I really hope I get this chance to implement this functionality to make using these great tools even better for them as well. This will also show the Uni-Verse consortium, that does not use other languages than C or C++ much yet, how Python rocks! ;)

Attachments

  • verse.png Download (122.4 KB) - added by ArcRiley 10 years ago. copied from old Soya section during site cleanup