This is the second post on how to accelerate Python with Cython. The previous post used Cython to declare static C data types in Python, to speed up the runtime of a prime number generator. In this post we shall be modifying a more complex program, one that performs an image transform on a map. This will allow me to demonstrate some more advanced Cython techniques, such as importing C functions from the C math library, using memory views of Numpy arrays and turning off the Python global interpreter lock (GIL).
As with the previous post, we shall be making a series of Cython modifications to Python code, noting the speed improvement with each step. As we go, we’ll be using the Cython compiler’s annotation feature to see which lines are converted into C at each stage, and which lines are still using Python objects and functions. And as we tune the code to run at increasingly higher speeds, we shall be profiling it to see what’s still holding us up, and where to refocus our attention.
Although I will be using Python 3 on a Mac, the instructions I give will mostly be platform agnostic: I will assume you have installed Cython on your system (on Windows, Linux or OS/X) and have followed and understood the installation and testing steps in my previous post. This will be essential if you are to follow the steps I outline below. As stated in the previous post, Cython is not for Python beginners.
Continue reading “Faster Image Transforms With Cython”
This longer post will show you some of the coding skills you’ll need for turning your existing Python code into the Python-C hybrid we call Cython. In doing so, we’ll be digging into some C static data types, to see how much faster Python code will run, and restructuring some Python code along the way for maximum speed.
With Cython, all the benefits of Python are still yours – easily readable code, fast development cycles, powerful high level commands, maintainability, a suite of web development frameworks, a huge standard library for data science, machine learning, imaging, databases and security, plus easy manipulation of files, documents and strings. You should still use Python for all these things – these are what Python does best. But you should also consider combining them with Cython to speed up the computationally intensive Python functions that needs to be fast. Continue reading “From Python To Cython”
This post continues my investigation into running standard Python on multiple cores using the concurrent.futures module. This time, I will be taking some image processing code from a previous post on the transverse Mercator transform and attempting to scale it to multiple CPUs.
Continue reading “Parallel Python – 2: Image Transforms”
After using Spyder for a couple of years, I recently changed my Python IDE from Spyder to PyCharm Community Edition (CE). And since I’ve now used both, I thought I’d share my impressions of each with you.
Continue reading “IDE Comparison: Spyder vs. PyCharm CE”
This post describes the process I used to design an algorithm that allows you to implement a modified Sieve of Eratosthenes to bypass the memory limitations of your computer and, in the process, to find big primes well beyond your 64-bit computer’s supposed numerical limit of 2.0e63 (9.223e18). Beyond that, with this algorithm, the only limitation is the speed of your CPU.
Continue reading “Eratosthenes 2: Swifter, Further, Cooler”
The Sieve of Eratosthenes is a beautifully elegant way of finding all the prime numbers up to any limit. The goal of this post is to implement the algorithm as efficiently as possible in standard Python 3.5, without resorting to importing any modules, or to running it on faster hardware.
Eratosthenes was a Greek scholar who lived in Alexandria (276BC to 194BC) in the so-called Hellenistic period. He was working about a century after Alexander, and about a century before the Romans arrived to impose their cultural desert and call it peace. And then do nothing with the body of knowledge they discovered. Literally. For over 1,600 years, if you count Constantinople. Not a damn thing.
So much for overly religious, centralised, bureaucratic superstates, obsessed with conquest. But I digress.
Continue reading “A Faster Sieve of Eratosthenes”
OK, one for the Mac users. Continuing the theme of user interfaces, here’s a simple but powerful way of using AppleScript to create a user interface for your Python programs and shell scripts and sending the results to just about any application installed on your Mac.
This solution has the advantage over Python’s native Tkinter in that the development time is much faster, and uses the speech synthesis features of OS/X to make your code much easier to use for the non-technical, elderly or visually impaired.
Continue reading “Using AppleScript To Launch Python”
With complex numbers, I always feel as if I’m getting a glimpse of something truly awesome that lies hidden within mathematics. The first time I understood how they worked, I thought it was some form of magic.
I get the same feeling with prime numbers. Like many, I’ve looked at them from all angles – prime gaps, large primes, prime densities, prime sieves – and they continue to fascinate. A few months ago I was thumbing through Henry Warren’s programmers cookbook Hacker’s Delight (A) and discovered a whole chapter on the various formulas for (some of) them. Mind-bending stuff.
Continue reading “Finding Primes Using Complex Numbers”
This is the second of two posts on how to quickly create a Tkinter dashboard for your command line Python programs. The Tkinter widgets and programming techniques it introduces are a sequel to the previous post.
So far, you have an interactive graphical way of opening a file to analyse it in some way with your own logic, entering text to use as triggers or search strings, setting your own program flags on/off using check boxes, switching between two or more mutually exclusive program flags using radio boxes, controlling access to widgets and the variables they control, calling your own logic, and saving your results in a new file.
This post will build on these skills by showing how to create a dashboard to accept numerical input, perform different kinds of type- and value-checking, and select multiple input files simultaneously using a Tkinter GUI file selector. The solution will be multi-platform, and is shown running above on (from left) Windows 10, Mac OS/X and Linux Mint above. This post will explain how to create the same thing for your own program. Before proceeding, make sure you’ve read and understood the previous post.
Continue reading “GUI Template For Python: Part 2”
The next problem I needed to solve was to come up with a simple graphical user interface (GUI) template as a front-end for configuring and launching any Python code or module I may wish to write or run. Initial impetus: I didn’t want to have to write a user interface from scratch every time I wrote some Python code to manipulate text, data or files. Bonus reason: if I made the GUI template generic enough, others might be able to use it to create their own user interfaces.
This would solve a problem that occurs in many technical fields. A university professor may have a post-doc researcher on her team, one who has written a complex command line program performing, e.g. image processing, AI or genetic analysis. At some stage, there may be some highly repetitive tests that can be performed by someone less technical, freeing up the researcher. She wouldn’t want him running these repetitive command-line tests with code only he knows how to run or, worse, sitting around designing complex user interfaces for others to use it. It would be better to get an intern or research assistant (or even a temp) to run the tests using a GUI that the researcher can knock up in a day or two. This would free him up to concentrate on his research. And finish it faster. Continue reading “GUI Template For Python: Part 1”
The first problem I wanted to solve was to write a short program that would allow me to perform basic textual analysis of any work of literature.
I wanted to be able to study the richness of different authors’ language by looking at how they used neologisms (their own made up words), pseudo-archaisms, invented their own contractions for authentic speech, or used hyphenated compound words, etc. I also wanted to be able to list all the characters and place names (proper nouns) mentioned in a text.
Continue reading “Analysis Tool For Literary Texts”
OK, first things first. What tools will I be using?
After talking to a good friend who is an experienced coder, I decided on the following:
Spyder, running Python 3. It seems to have everything I need, including a good debugger, a variable explorer, hot-linking to function definitions, auto-completion typing, Matplotlib, QT, plus a choice of either a Python and iPython console (each with their different strengths). The bundle I went with is Spyder for WinPython-64bit (WinPython-64bit-126.96.36.199Qt5). The QT will be useful later.
Continue reading “The Toolkit”