October 18, 2012

Python3 is still slow

Filed under: Uncategorized — heavytailed @ 9:02 pm

Short post today. Anecdotal. Nontechnical.

So I tend to do development in python, because it seems to be optimized for swift development. I’m discovering, though, that for anything that requires serious work, there’s a second “phase” of development that involves slowly converting all of the functions into a c program via extensions.

Case in point: I’ve written a basic program to calculate genetic distance given certain constraints. I did it in python for being able to better manage the IO (because it has to read formats for which none of the tools have generic readers: GATK can’t read bed, PLINK can’t read VCF, and nobody likes to convert.)

It was slow. Painfully slow. So I optimized the python: replace loops with list comprehensions where possible, define complex functions rather than using lambda with maps, even little tricks like (x+x+x) rather than 3*x (the former is actually faster in python). Still, unusable.

So I profiled it, and converted the heavy hitting methods to a C extension (both of them, by the way, numpy methods: applyOverDimension, and diag()). I did not alter the algorithmic complexity at all. Nevertheless:

Without Extensions:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.034 0.034 483.779 483.779 <string>:1(<module>)

With Extensions:

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.041 0.041 31.006 31.006 <string>:1(<module>)

So a 94% reduction in runtime just by shifting two methods from python to c. The question is, was my development of the algorithm framework in python sufficiently quick as to make up for the extra day of writing c-extensions, or should I have written it all in c from the start?

This was all done in python3, which I’ve switched over to. Unlike all the people who seem to hate it…I don’t think it’s all that different. I guess I never do the kind of programming that would make me notice differences. But, like python 2, python3 is brutally slow and needs a helping hand from C.


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: