Cythonize setup.py for several .pyx files

Concept

Distutils is the standard way to build Cython extension modules for large projects. Typically one invokes setup(…) which then indirectly invokes Cython when a .pyx source file is encountered. This limits the control we have over the pyx to c compilation process. This proposal is to make a standard Cython method that “preparses” the list of Extension modules and converts all .pyxsource files to .c(pp) ones before passing them off to distutils. Doing this step ourselves, all at once, would have several advantages including:

• Cython-aware dependency tracking (Done)
• in-file directives for language, libraries, compiler flags, etc. (which could be transitive) (Done)
• Efficient multi-file compilation (especially with shared deep cimports) (Done)
• sharing common utility code in a single .h file
• shared cimported type declaration code
• whole-program type inference or other analysis

This will resolve other nuisances such as the difficulty in supporting parallel Pyrex and Cython installs or relying on distutils to know about Cython at all. Disentangling the Cythonization phase from the compile process and using in-file directives will allow easier support for other build systems as well.

Example

Rather than writing a setup.py that looks like

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules=[
Extension("primes", ["primes.pyx"]),
Extension("spam", ["spam.pyx"]),
...

one would write

from distutils.core import setup
from Cython.Build import cythonize

setup(
name = 'MyProject',
ext_modules = cythonize(["*.pyx"]),
)

The cythonize command takes a list of filenames/filename patterns and returns a list of Extension classes ready to be compiled by distutils, using timestamps and its understanding of Cython’s include and cimport statements to compile all and only the necessary extensions. It is also permissible to do:

ext_modules=[
Extension("primes", ["primes.pyx"]),
Extension("spam", ["spam.pyx"], libraries = ["spam"]),
...
]

setup(
name = 'MyProject',
ext_modules = cythonize(ext_modules),
)

however, the preferred way to specify distutils options is directly in the .pyx or .pxd file itself, via a comment similar to how module-level cython directives are given. For example, one can write:

# distutils: language = c++
# distutils: libraries = spam eggs
# distutils: include_dirs = /opt/food/include

When such declarations appear in .pxd files, they are transitive (if it makes sense), so modules cimporting from them will automatically propagate the right compile flags to use them as well.

The cythonize command takes several options, including an nthreads parameter to use multiprocessing to compile .pyx files in parallel.

See http://hg.cython.org/cython-devel/file/5a033978dd11/Cython/Build/Dependencies.py#l442.

Issues

• One still may need to read and understand sys.argv to know whether or not to invoke Cython.