From ArcRiley at pysoy.org Thu May 1 07:55:56 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Thu, 1 May 2008 07:55:56 -0400 Subject: [PySoy-Dev] PyMill update In-Reply-To: <200804220921.41603.evan@theunixman.com> References: <200804220921.41603.evan@theunixman.com> Message-ID: Erps, this bounced for some reason. Resending.. > I was noticing that this morning as I was playing with it. My questions are > mainly about what the process is supposed to look like. Ok "big picture" time. .pxd's are redundant. All the information contained therein can be described in the .pym files directly. When a third party app compiles to extend PySoy they'd use an interface file, not a .pxd. We use .pxd's still only because it seems a faster way to get PySoy building in a stable manner again. Remember, beta-3 work is left to be done. What I would like to see in the end is the interface files for each extension built first in a specific directory under build, from distutils, before compile() is ever run. Then as compile runs it knows that cimport always describes an interface file either local or installed system-wide (ie, in site-packages next to the module itself). I've already resolved with robert, of the cython crew, that the convention for loading a C header will be cinclude, not cimport, so all those C library .pxd's are also obsolete once this code is written. Our entire include/ directory will, someday, be able to be deleted. > I noticed in setup.py (from pysoy) that the include directory is being set, and that > from the errors the mill compiler is finding the .pyd files. But then instead of > treating the .pyd files as an interface definition, it tries to compile them > as self-contained modules. Yea. Cython basically makes the sources=[] list redundant, since each Cython extension is built from exactly one file. This is the behaviour we need to change. > So, is the intent to somehow have the compiler treat declared but not defined > objects as forward declarations implicitly? Exactly. You'll see how the Cython.Compiler, via the mill.distutils, is milling all the .c sources to build/temp*/ right next to where the object files for the same are built by GCC, in the same way we'll need the C declarations written to a .h in the same directory and locally #include ""'d in each generated C file. > Also, would it be prudent to create a new distutils Command for mill so that > we aren't stepping on the existing build_ext? It might make it easier to > isolate the changes we want in the build process since we'll end up without a > lot of the implicit behavior in build_ext already (if there is any > interfering), and we can always go back to overriding build_ext if it turns > out that it's not a problem and still useful to have. There's no interfering. :-) Plus, as SWIG .i files are currently understood by distutils, I'd eventually like to see PyMill done the same way such that we don't need to implement our own distutils at all. When a .pym file is found in sources it'd just import mill and call the appropriate functions. From ArcRiley at pysoy.org Thu May 1 08:29:53 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Thu, 1 May 2008 08:29:53 -0400 Subject: [PySoy-Dev] Trac 0.11rc1 Message-ID: As of yesterday 0.11rc1 has been released, marking a feature freeze. I've started the upgrade process by installing PyMill at http://pysoy.org:8000/pymill/ and a quick and dirty plugin to change the projectname WikiWord to a link to /wiki/ (attached) At least two more plugins are needed, a download plugin doing such useful things as OS detection and a download-redirect+infopage (working on it now) and a docs plugin that parses the svn source for pydocs and formats it with a shiny navigation interface (which I haven't started yet). These are not essential for upgrading pysoy.org to 0.11 but if you're looking for a way to contribute, here's yet another one. These can, of course, be used by other projects using Trac and can be published on trac-hacks.org The only thing holding back pysoy.org going 0.11 is my own familiarity with the genshi template system. From ArcRiley at pysoy.org Thu May 1 08:31:22 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Thu, 1 May 2008 08:31:22 -0400 Subject: [PySoy-Dev] Trac 0.11rc1 In-Reply-To: References: Message-ID: Erps, forgot the attachement -------------- next part -------------- A non-text attachment was scrubbed... Name: ProjectNameLink.py Type: text/x-python Size: 842 bytes Desc: not available Url : http://www.pysoy.org/pipermail/pysoy-dev/attachments/20080501/5874ed30/attachment-0001.py From evan at theunixman.com Sat May 10 13:53:33 2008 From: evan at theunixman.com (Evan Cofsky) Date: Sat, 10 May 2008 10:53:33 -0700 Subject: [PySoy-Dev] PyMill update In-Reply-To: References: <200804220921.41603.evan@theunixman.com> Message-ID: <6b279dc60805101053p7c3a295ci94b4deecf39d5bda@mail.gmail.com> On Thu, May 1, 2008 at 4:55 AM, Arc Riley wrote: > Ok "big picture" time. > > .pxd's are redundant. All the information contained therein can be > described in the .pym files directly. When a third party app compiles > to extend PySoy they'd use an interface file, not a .pxd. We use > .pxd's still only because it seems a faster way to get PySoy building > in a stable manner again. Remember, beta-3 work is left to be done. I wonder if bypassing all of Cython except the parts we need for beta-3/4 is a possibility. Then, when we have it working well enough, work on bringing it back into Cython. > What I would like to see in the end is the interface files for each > extension built first in a specific directory under build, from > distutils, before compile() is ever run. Then as compile runs it > knows that cimport always describes an interface file either local or > installed system-wide (ie, in site-packages next to the module > itself). > So basically it would generate the headers in a packagable and well-defined place, and then create the C code implementation files that include their own generated headers, and also any that were explicitly "cinclude"-ed? I've already resolved with robert, of the cython crew, that the > convention for loading a C header will be cinclude, not cimport, so > all those C library .pxd's are also obsolete once this code is > written. Our entire include/ directory will, someday, be able to be > deleted. I think I see a quick and dirty way or two to at least get the code compiling. It won't necessarily be Cythonic, but maybe at least Milleresque anyway. And it might give more ideas to talk and/or argue about. Exactly. You'll see how the Cython.Compiler, via the mill.distutils, > is milling all the .c sources to build/temp*/ right next to where the > object files for the same are built by GCC, in the same way we'll need > the C declarations written to a .h in the same directory and locally > #include ""'d in each generated C file. > Ah, got it (in theory). > There's no interfering. :-) > > Plus, as SWIG .i files are currently understood by distutils, I'd > eventually like to see PyMill done the same way such that we don't > need to implement our own distutils at all. When a .pym file is found > in sources it'd just import mill and call the appropriate functions. > Maybe breaking this down into smaller tasks will help, at least conceptually: 1. Mill single files with the right header generation. 2. Compile all the way to a loadable Python module. 3. Repeat for a single package. 4. Repeat for multiple packages. 5. Mill some soy. 6. ??? 7. Profit. -- Evan Cofsky, Horn Player e at lahorn.com Check out my horn playing: http://tinyurl.com/47bvxr -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.pysoy.org/pipermail/pysoy-dev/attachments/20080510/18c45d92/attachment.htm From ArcRiley at pysoy.org Sat May 10 18:20:44 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Sat, 10 May 2008 18:20:44 -0400 Subject: [PySoy-Dev] PyMill update In-Reply-To: <6b279dc60805101053p7c3a295ci94b4deecf39d5bda@mail.gmail.com> References: <200804220921.41603.evan@theunixman.com> <6b279dc60805101053p7c3a295ci94b4deecf39d5bda@mail.gmail.com> Message-ID: > I wonder if bypassing all of Cython except the parts we need for beta-3/4 is > a possibility. Then, when we have it working well enough, work on bringing > it back into Cython. Things are changing rapidly on the Pyrex/Cython front. Let me summarize: Cython was founded with the primary goal of remaining compatible with Pyrex. We, likewise, wanted to remain compatible with Cython. Greg (the author of Pyrex) recently changed the syntax for "for" in Pyrex 0.9.7 and has said that he'll remove support for the old syntax in the next micro release (0.9.8). This means that code that compiled with 0.9.6.4 will not compile with 0.9.8 and produces numerous obnoxious depreciation warnings with 0.9.7 - this is an ongoing pattern and part of why we need to migrate away from Pyrex. Cython guys are obviously upset, because Greg didn't consult -anyone- for even opinions before doing so, a behavior many of us may recognize the author of Soya demonstrating frequently, and they are realizing that their goal of remaining Pyrex-compatible puts them at Greg's mercy as far as language changes. Several recent discussions in the Cython crowd about doing a fairly major rewrite and changing the Cython language in an incompatible way from Pyrex (to more closely mimic Python syntax). I agree with their language goals, but the core problem is that any work we do based on Cython.Compiler may need to be redone several times over the Summer, the PyMill repository is already severely out of sync with Cython's repository due to code being moved around on both ends. So in summary, I'm not worried about Pyrex or Cython compatibility at this point. I think we should do what we need to do for PySoy, what makes sense for *us*, and figure out what to do as the dust settles more around this Fall. > So basically it would generate the headers in a packagable and well-defined > place, and then create the C code implementation files that include their > own generated headers, and also any that were explicitly "cinclude"-ed? Mostly right There's three phases, the first two are part of distutils "build_ext" command: 1) Parse .pym sources and generate one .h file for each extension and potentially many .c files within the build/ directory directly (not to the system) 2) Compile .c files which include the headers generated in step 1 as needed (along with any headers installed on the system) and link them into one extension (for each instance of Extension) 3) Install the extension(s) and headers during the distutils install command Note that the headers cannot be generated directly to some system-wide path, this would break many conventions and make many people upset with us. > Maybe breaking this down into smaller tasks will help, at least > conceptually: I agree, but given the above, the steps need to be a bit more first - we need to define the language. this was not done with Pyrex, and we see what kind of mess it is internally second - build the main mill package from scratch, using the wisdom of the Cython guys (reverse engineering in some cases), even the people working on Cython agree it needs to be rewritten since the barrier to entry for new developers is obscenely high third - test small extensions, as you suggested, or possibly unit tests fourth - migrate PySoy's codebase, in whatever ways needed, to the language spec'ed above (changes should be MINOR!) fifth - debug PyMill using PySoy as a guide, and then optimisations on PyMill can continue while PySoy dev continues unhindered I'll note that we have a trac site for PyMill now at http://pysoy.org:8000/ From ArcRiley at pysoy.org Sat May 10 21:20:33 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Sat, 10 May 2008 21:20:33 -0400 Subject: [PySoy-Dev] .pym language draft Message-ID: I think Evan's suggestion earlier on IRC about using the tokenize module is a good one. It's MUCH more robust than Pyrex.Plex and we can do just about anything with it. See http://docs.python.org/dev/3.0/library/tokenize.html This is my attempt at a very early draft for the .pym language, and nothing in here should be read as dictation - just ideas. * 100% compatibility with Python 3.0 syntax for parsers - will allow pydoc/etc parsers to work without a hitch - lowers the learning curve for existing Python coders significantly - as tokenize returns comments (it can be used for highlighters), we can implement cinclude as #include, cdef as #def, IF as #if, etc - as those elements wouldn't be visible to Python in a compiled module, they will not effect parsing at all * C variables, types, struct elements, functions, etc retain the same names - provides predictability for external access - requires using unique names in some places for #def's * C vs Python optimizations handled transparently - ie, "for i in range(4,10,2) :" given previous "#def int i" will result in "for (i=4; i<10; i+=2) {" - special class methods (ie, __sub__ __rsub__) handled via standard test/dispatch code in C - all classes and functions implemented in C, Python-side calls these unless overridden (ie, after inheritance) I would like to see, for example, PySoy's mutexes handled using the "with" statement introduced with PEP 0343, and same with GIL. thus, "with GIL:" could continue to work as it does in Pyrex. http://www.python.org/dev/peps/pep-0343/ Questions unanswered (that I can think of now): - how to handle class "properties"? - do special context managers (ie, gil/nogil) need to be defined in the pym code at all, and what to do when they are - does there need to be a separate syntax for #def (constants) and C variable declarations? - what syntax should be use to control whether a Python object is automatically refcounted, or when it should/should not be refcounted? This is by no means complete, just a starting place for further discussion From ArcRiley at pysoy.org Sun May 11 00:38:34 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Sun, 11 May 2008 00:38:34 -0400 Subject: [PySoy-Dev] .pym language draft In-Reply-To: References: Message-ID: A bit of a follow-up, I wrote a short experiment with tokenizer - in summary, class, def, cdef, property, all the things I was worried about are all token type "NAME". Comments, on the other hand, are one token per comment line My earlier leaning on making all our language additions a comment is wavering; on the pydoc front, having things like properties behind comments will not expose them to pydoc, and in some sample code it made it really hard to notice the C stuff. it would also be /some/ work to change all our cdefs to something else, and we would have trouble with indentation with cdef functions. From ArcRiley at pysoy.org Thu May 15 12:12:23 2008 From: ArcRiley at pysoy.org (Arc Riley) Date: Thu, 15 May 2008 12:12:23 -0400 Subject: [PySoy-Dev] Beta-3 sprint this weekend (May 16-18) Message-ID: Hey listen up Greg just released Pyrex 0.9.8 which he says fixes the INCREF/DECREF pairs added on Python arguments to cdef class methods when nogil is declared. In short, this means that our multicore race conditions can now be fixed. If not, then he's apparently willing to handle any remaining fixes in quickly. He also included a workaround for our circular import problems - we can now forward declare classes in .pxds and thus replace the manual typecasting everywhere. An example of forward declaring was added to soy.widgets.pxd last night This means, barring any other blocking bugs, we can release Beta-3 next week! I'll be working through the weekend on this, and anyone who wants to help get 1.0_beta3 released is encouraged to help. There's a number of easy tasks (docs, code cleanup, etc) that anyone with a text editor should be able to do, and the more people helping means the more advanced tasks (ie, Blender mesh loading) will get handled.