Your Linux Data Center Experts

It's been years since I played with Pyrex. I thought it was great at the time, but didn't really have much of a need for it until recently. Pyrex is a system by Greg Ewing which is a kind of hybrid of Python and C.

Pyrex is a language very much a combination of C and Python. It looks much like Python, and can easily interact with both Python and C data-types. It can be used to interface easily between Python and C code, including existing C libraries, for example:

cdef extern from "time.h":
   ctypedef int time_t
   time_t time(time_t *p)

def ctime():
   return(time(NULL))

This is a Pyrex module which calls the C library “time()” function and returns the result. This is compiled into C code, which is then compiled into a shared Python module which you can call directly from Python. In this case, we're wrapping a C library, much like you could do with SWIG, but this is (to me) much easier than SWIG.

It can also be used for building performance critical code. This is the reason I was using it. Usually, Python is pretty fast, but I was building a program which was doing traffic accounting kind of like Cisco NetFlows, in real-time, and had some concerns for the performance of the system, while at the same time not wanting to drop fully down into C for building it, at least not until I had a prototype up and working.

In the end, the prototype with some of the code built using Pyrex (around 113 lines in Pyrex) is performing quite nicely. Just this morning it was using less than 5% CPU doing accounting on 300mbps of bulk data traffic. These were large packets, 1500 bytes each probably, but my target is to be able to handle 30mbps peak, and never more than 100mbps of, worst case, 64 byte packets. Oh, and that's on a system with around 6x the CPU clocks of the test system I was running on.

I realize now that I may have been doing premature optimization. Raw Python may have handled this just fine. I probably should have tried it in pure Python, but it would have required about the same amount of code, and probably only half the time to code (since I wasn't familiar with Pyrex). I could have dropped fully down into C, but that would probably have taken 2x as long at least. The biggest issue is the decomposition of packets to get the TCP/IP header information, which there's a Python module for.

Pyrex is insanely cool. I'll happily use it in the future for place where I need higher performance, or access to a C library from Python. It's real easy. SWIG never was.

comments powered by Disqus

Join our other satisfied clients. Contact us today.