This page describes how to compile a Cython extension (works for CPython extensions also) for Windows.
If you encounter one of the following errors:
C:\your\project\path python setup.py build_ext --inplace --compiler=msvc running build_ext building 'helloworld' extension error: Unable to find vcvarsall.bat
Then you are at the right place! These errors simply mean that you lack a C compiler to compile the .c files generated by Cython from your .pyx files. On Linux, the solution is easy, and most users do not even have to install anything special because the “platform compiler”, gcc, is already installed. On Windows, it is a little bit more complicated, since the platform compiler is not installed by default, so you have to do it by yourself.
Below we describe the two currently available ways to install the platform compiler on Windows and set it up so that you can compile the C extension for Python on Windows.
NOTE: the following works only for Python 2.7 and 3.2, for Python 3.3 and 3.4 you should try the older method with the SDK.
The easiest way to compile correctly working C and Cython extensions for Python on Windows is to use Visual Studio SDK, because incompatibilities may arise when using other compilers. However, the process (outlined below) can be quite daunting and will consume several GB of installation, only to get a working compiler.
Microsoft fairly recently released a freely distributed version of their compiler specifically for building Python extensions for the python.org 2.7 series, called Microsoft Visual C++ Compiler for Python, which is only ~85MB and should work for most Python compiling purposes.
This version does not have a couple of bugs that made it hard to get set up properly for 64-bit extensions. Before proceeding to the instructions below, you should update
setuptools to at least 6.0.
Here is how to compile C extensions using Microsoft Visual C++ Compiler for Python:
from distutils.core import setup from distutils.extension import Extension
But the following will:
try: from setuptools import setup from setuptools import Extension except ImportError: from distutils.core import setup from distutils.extension import Extension
For the rest of setup.py, read the tutorial at <http://www.sefidian.com/2017/06/17/basic-tutorial-of-cython/>.
When using an IPython Notebook, you can apply the same trick:
#First cell: import setuptools %load_ext Cython #Second cell: %%cython -a import cython cimport cython cdef int alpha = 255 print alpha
Now, you should be able to compile your C extensions, and even Cython extensions if you have Cython installed. If you still get an error about not finding vcvarsall.bat, you may try the following workarounds:
Steps 2 and 3 must be reproduced every time before building your C extensions. This is because of an issue with the VCForPython27.msi which installs the header files and vcvarsall.bat in folders with a different path layout than the VS2008 SDK and thus confuses the compiler detection of distutils. This will maybe get fixed in distutils for Python 2.7.11.
However, if you want to use IPython/Jupyter and the
%%cython magic, you cannot specify the
--compiler nor enter the required variables to workaround the detection issue (thus
%%cython will always try to compile using mingw).
Another way to work around this issue, without needing to set variables every time, is to patch the file
C:\yourpythoninstall\Lib\distutils\msvc9compiler.py by adding the following code at the top of the
def find_vcvarsall(version): """Find the vcvarsall.bat file At first it tries to find the productdir of VS 2008 in the registry. If that fails it falls back to the VS90COMNTOOLS env var. """ vsbase = VS_BASE % version vcpath = os.environ['ProgramFiles'] vcpath = os.path.join(vcpath, 'Common Files', 'Microsoft', 'Visual C++ for Python', '9.0', 'vcvarsall.bat') if os.path.isfile(vcpath): return vcpath vcpath = os.path.join(os.environ['LOCALAPPDATA'], 'Programs', 'Common', 'Microsoft', 'Visual C++ for Python', '9.0', 'vcvarsall.bat') if os.path.isfile(vcpath): return vcpath ...
Then, create a file
distutils.cfg in the same folder, and put this inside:
Now you can compile C extensions without any prior manipulations. Thus you can now call the
%%cython magic inside IPython/Jupyter, by placing this inside a cell:
%load_ext Cython %%cython
For more information about this detection issue, see the bug report and workaround by Gregory Szorc: http://bugs.python.org/issue23246
Note: Enthought Canopy users, see here
Note2: don’t try the GccWithBinaries project, even though it will install with a 32-bit Python (whatever the bits of the installed Windows), The Cython extension won’t compile. Same with standard MinGW, and same with GCC. https://github.com/develersrl/gccwinbinaries
Note3: to compile your Cython scripts, you need to either use pyximport or make a setup.py as described at http://docs.cython.org/src/tutorial/cython_tutorial.html> and <http://docs.cython.org/src/reference/compilation.html.
You can also try distcontrib <http://distcontrib.readthedocs.org/en/latest/> which seems to generate setup.py compatible with Cython.
The standard way to compile C extensions for CPython (including Cython extensions) is to install the compiler that was used to compile the CPython interpreter. CPython is always compiled using Visual Studio.
In order to do this, you either need the Visual Studio that compiled CPython or a Windows SDK C/C++ compiler. Now the problem is that depending on which Python version you want to compile your extension for, you need a different version of the SDK because they are not cross-compatible.
Luckily, people have compiled the list of CPython versions with their corresponding SDK, and in fact, you can use the free Visual Studio Express v9.0 package to compile Python between versions 2.6.9 – 3.3.4, VSE v10.0 (2010) to compile Python 3.3.5 – 3.4.x, and Visual Studio 2015 Community Edition for Python 3.5. Make sure that you check the “Common tools for Visual C++” option during the Visual Studio install (in order to avoid the dreaded vcvarsall.bat error). Note that this only works for compiling 32-bits extensions, for 64-bits extensions you will need to install in addition the corresponding Windows SDK below.
Here are two good listings where you will find all the necessary info to install the correct Visual Studio for your Python:
If you don’t want to install Visual Studio, you can choose to install the Windows SDK C/C++ compiler, as indicated in the table below:
|Python||Windows SDK / .NET||Download the GRMSDKX_EN_DVD.iso|
|MS Windows SDK for Windows 7 and .NET Framework 3.5 SP1||Windows SDK (.NET 3.5 SP1)|
|3.3||MS Windows SDK for Windows 7 and .NET Framework 4||Windows SDK (.NET 4)|
At this point, it does not matter if you want to compile for 64 or 32 bit. The version you downloaded can do both.
Before you run something along the lines of
pip install package_name
python setup.py command_name
to install or build a package, you do have to start an SDK Command Prompt or CMD Shell and set some environment variables.
By default, the shell starts at the installation path of the Windows SDK (C:\Program Files\Microsoft SDKs\Windows\v7.0. There, we have to do two things:
Thus, we have to enter two commands
set DISTUTILS_USE_SDK=1 setenv /x64 /release
Assuming you haven’t changed the Windows SDK installation path, this looks something like this
C:\Program Files\Microsoft SDKs\Windows\v7.0>set DISTUTILS_USE_SDK=1 C:\Program Files\Microsoft SDKs\Windows\v7.0>setenv /x64 /release
As you might have guessed, you can swap out
/debug and vice versa.
Then, switch to your project directory and try to
python setup.py command_name or
pip install package_name. This should work now.
Finally, you should distribute your binary package as a .whl file to PyPi.
The C/C++ compiler and linker that comes with the Windows SDK should be used because Python itself is compiled with it. That way your extensions will always be binary compatible with Python and other installed system components and libraries.
** Do not use MinGW-w64. As you will notice, the MinGW import library for Python (e.g. libpython27.a) is omitted from the AMD64 version of Python. This is deliberate. Do not try to make one using dlltool. There is no official MinGW-w64 release yet, it is still in “beta” and considered unstable, although you can get a 64-bit build from e.g. TDM-GCC. There have also been issues with the mingw runtime conflicting with the MSVC runtime; this can happen from places you don’t expect, such as inside runtime libraries for g++ or gfortran. To stay on the safe side, avoid MinGW-w64 for now.
** Those with experience in Linux or Apple MacOSX might pick GCC (MinGW) as their C/C++ compiler of choice on Windows. Note that GCC is not the platform C compiler on Windows. To be binary compatible with most libraries and applications on Windows, use Microsoft’s compiler. This is no different from selecting GCC as the compiler of choice on Linux. Microsoft’s platform C/C++ compiler for Windows is free of charge.
** For C++, observe that GCC (Mingw) and MSVC use different C++ runtimes. If you use C++ in your extension, the C++ runtime must also be compatible with Python’s C runtime. For this reason, always use Microsoft’s compiler for C++, even on 32-bit x86. This is very important on a platform like Windows where C++ is pervasive (cf. COM and ActiveX). If you use GCC as a C++ compiler your extension will not be binary compatible with other C++ libraries or COM, not be compatible with Python’s C runtime, and differences in exception handling between GCC and MSVC can crash your application.
** If you need Fortran (e.g. for f2py or fwrap), Intel, Absoft and Portland have compilers compatible with Microsoft’s C compiler and linker. Be very careful if you use an unofficial MinGW-w64 build of gfortran, as libgfortran is linked against an incompatible C runtime. Unlike C and C++, there is no freely available Fortran compiler except MinGW-w64 gfortran on Windows 64.
** If you prefer to use Microsoft Visual C++ for editing and debugging, the SDK can be registered with the commercial version of Microsoft Visual Studio. For Python 2.6, 2.7, and 2.8, you will need Visual Studio 2008.