Using iPython for Sys Admin

Python on mainframe terminal

Manipulating files and programs using Unix shell file programs can often be a bit of a pain, even for seasoned programmers. This can be due to how infrequently you use them, or because you are often moving between OS/X, Windows and Linux, and their subtle differences can often trip you up. 

I used to be fairly proficient at them, but nowadays find I use them so rarely that I often have to revise what I used to know, even to achieve the most basic tasks. For many coders, the Unix shell programming language has become like an obscure language you only brush up on when you need to speak to a distant relative at Christmas time.

Fortunately, if you know and love Python, most of what you need to do with Unix shell for-loops and file permissions can easily be done with iPython, without having to spend an entire day every twelve months re-learning what you first learnt to do in Unix shell files ten years ago. 

Many Python users don’t realise iPython gives them access to three wonderful things in one place: all the shell commands you know and love, plus the full suite of Python functions,  and access to the weird world of hybrid commands that allow you to mix both on the same line. The result is a powerful set of sys admin tools which you can use to perform some quite complex file, directory and process admin tasks. 

This post is essentially a quick tour of ten common system administration jobs you can perform with iPython to manage the files, folders and processes on your system. There is so much more you can do, but this is just a taster for what you can achieve with iPython when you need to put aside your normal Python coding and roll up your sleeves to sort out your system. Without revising too much of the shell programming language syntax before you begin.

Installation & Simple Commands

If you use a sophisticated IDE such as Spyder or PyCharm, you will already have iPython as part of your IDE’s interface. If you’re running Mac or Linux and you already have Python installed, just use pip, or what ever package installation software you normally use: 

$ pip install ipython 

As you might expect, on Windows it’s a bit more painful, but a good way to install it is also to use pip. If you don’t have it, you’ll first need to launch a terminal window (cmd.exe) as the administrator, then install pip using easy_install, then install iPython using pip:

$ easy_install --upgrade pip
$ pip install ipython 

Who said Windows was difficult? Your persistence will be rewarded though, as iPython on Windows actually looks pretty cool. 

However you install it, once you’ve got it working you always run it the same way. Here’s how it looks when you launch it (the version numbers will vary):

$ ipython 
Python 3.7.1 (default, Dec 14 2018, 13:28:58) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:   

As with its simpler sibling the standard Python REPL (the one with the >>> prompt), iPython is great for testing Python ideas and functions, but has many more features. The most obvious difference with the standard Python REPL is that your commands are numbered to allow you to reference and reuse useful command sequences later. 

To view your command history with its numerical references, you just type: 

In [1]: history -n  

Or to see the previous session’s entire history, use:

In [2]: history ~1/  

Another difference is the set of so-called magic commands, which are quite powerful and worth taking the time to learn. The magic commands are invoked using the % prefix. The command numbers work the same way as for the history command, and can also access the commands from previous sessions. For example, to recall a previous sequence of useful commands, you just use the recall command %recall (or %rep for repeat- they do the same thing):

In   [5]: %recall 12-24  

which presents the requested commands, allowing you to edit and rerun them if you wish.

If you are quite pleased with some commands you put together in your third last session, you can recall them too:

In   [5]: %recall ~3/10-15  

You also may want to edit a block of commands without running them, and then save them to a file. It’s all included in the edit command, which drops you and the commands into an active Vim editor session (or Notepad, if you’re using Windows) where you can edit the commands and save them to a .py file:

In   [5]: %edit ~1/  

But if you’re quite happy with a sequence of commands, you can save them to a Python file for later incorporation into a program:

In   [87]:  %save 69-79 

OK, intro over. If you think you need a longer introduction to iPython, it’s well worth checking out the online documentation.

1. Manipulating Folder Content Lists 

A. Turning a folder listing into a Python list:   I’ll start with some simple but powerful commands that give you access to the current folder contents as a Python list. First, to turn the contents of a folder into a list, use this:

In [1]: file_list = !ls                                    
In [2]: file_list   

The hybrid !ls command is superior to os.listdir() because it ignores the hidden .files, giving the list of only the normally visible filenames as a Python list. This format opens up an enormous number of possibilities for you – you can then use the members of the list any way you like, filtering them through any Python function, even using your own lambdas:

In [7]: file_list.grep(lambda x : '.JPG' in x)             

The single Python grep line is a succinct way of saving on a Python for-loop.

B. Turning a Folder Listing into a CSV text string: This is useful if you are moving back and forward between folder lists and strings. Again, a couple of ways to do this. Here’s a good one:

In [2]: dirlist = !ls                                      
In [3]: dirlist_text = ', '.join(dirlist)       
In [4]: with open('Actors.txt', "wt") as f:
   ...:     f.write(dirlist_text)                          
Out[4]: 1882

In [5]: dirlist_text                                       
Out[5]: "Titus Welliver, Tom Cruise, Tom Hanks, 
Tommy Lee Jones, Uma Thurman, Walton Goggins, 
Woody Harrelson"


2. Extracting a Filename From a Path

This is fairly easy with standard Python, and doesn’t need any of iPython’s tricks:

In [1]: import os
In [2]: myfilepath = ‘/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest/ShiftedWorldMap.png’
In [3]: pathbits = os.path.split(myfilepath)
In [4]: pathbits
('/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest', 'ShiftedWorldMap.png')

In [5]: myfile = os.path.split(myfilepath)[-1]
In [6]: print (myfile)

3. Renaming Files in a Batch

This can be very useful indeed. It is slightly different on different shells, but in iPython, it’s always the same:

In [1]: import os
In [2]: files = !ls
In [3]: for file in files:   # can't use !ls here
    ...:    if '.png' in file:
    ...:        os.rename(file, file.replace('IMG_', 'NZ_Foto))

Note how the !ls command could not be used in the for loop, and we had to set it up first.

4. Deleting Files That Match a Pattern

This is almost identical, but must be used with caution:

In [1]: import os 
In [2]: files = !ls
In [3]: for myfile in files: 
   ...:     if ".jpg" in myfile: 
   ...:         os.remove(myfile) 

Note how you can delete the file while using its name as the loop variable.


5. Finding and Executing Your Programs

There are a few ways to do this. Here is a simple iPython string version, which in a folder of Python programs will run every Python script it finds:

In [1]: import os
In [2]: progs = !ls
In [3]: for prog_name in progs: 
    ...:     if ".py" in prog_name: 
    ...:         os.system("./" + prog_name)

This can be adapted to work on any kind of program.

6. Gathering Process Information

The key to understanding the power of using iPython for system administration is that you can read any system variable and do anything with it, using any of the Python string, file, print, IO, web or database libraries and their methods.

Once you realise that you can feed the output of any shell command into iPython, anything is possible. Using a shell command to find all processes relating to a particular user, we would normally feed ps into a pipe (noting that that syntax may vary between platforms):

$ ps aux | awk '{if ($1 == "Elon Tusk") print $2}'


But with iPython, you can grab the output and parse it to your heart’s content:

In [1]: ps = !ps aux 
In [2]: len(ps)                                            
Out[2]: 311


To find all processes relating to, for example, Bluetooth, we would simply say:

In [2]: ps.grep('bluetooth')
['root  127  0.3  0.1  4334916   13008 ??  Ss  7:24am   0:35.53 /usr/sbin/bluetoothd']]

7. Monitoring Processes Over Time

There might also be a situation were you need to monitor system information over time. This short iPython loop achieves this by using Python index variables to create incremental system snapshots. Again, this can be adapted to do almost anything. As with ‘ls’ above, note how a hybrid command takes the output of a shell command and modifies it with a Python variable to create incremental filenames. With shell command, this is one of those tasks that often needs slightly different syntax on different platforms. 

In iPython, the command is the same for Mac, Linux or Windows:

In [1]: import time                                        
In [2]: for i in range(1, 10): 
   ...:     !ps aux > ps_snapshot{i}.txt 
   ...:     time.sleep(60) 

In [3]: ls
ps_snapshot1.txt  ps_snapshot4.txt  ps_snapshot7.txt   
ps_snapshot2.txt  ps_snapshot5.txt  ps_snapshot8.txt
ps_snapshot3.txt  ps_snapshot6.txt  ps_snapshot9.txt 

Think of the mixed Python/Shell ‘ps aux’ line as essentially a shell line, allowing you to insert Python variables into the {} field, just as you would with the Python string.format() function.

Note that no exclamation point was used before the ls command [3] here. This is because iPython recognises the simple form of most common shell commands, such as ls, pwd, cd & cat – but only when typed on their own line. In a multi-line code block, iPython needs the !prefix before the shell command.

To create a whole host of 0-byte files with specific filenames for testing, you can combine iPython with the Unix touch command:

In [1]: for i in range (10): 
   ...:     !touch Foto{i}.png 

In [2]: ls Fo*                                             
Foto0.png  Foto2.png  Foto4.png  Foto6.png  Foto8.png
Foto1.png  Foto3.png  Foto5.png  Foto7.png  Foto9.png


8. Duplicating a Massive File Structure For Testing

Suppose the above list of files is real, but that there are a million of them. And suppose that we need to test a file manipulation program on this number of files, but not on the original files. Rather than copying gigabytes of data to another location just to text our program on, we can quickly duplicate any number of file names  as empty files to another folder.

There are a couple of ways to do this. Here’s one:

In [1]: photo_list = !ls
In [2]: cd /TestFolder                                     

In [3]: for file in photo_list: 
   ...:     !touch {file} 
In [4]: ls
Foto0.png  Foto2.png  Foto4.png  Foto6.png  Foto8.png
Foto1.png  Foto3.png  Foto5.png  Foto7.png  Foto9.png

All of which are 0 byte files. We have therefore duplicated a large file structure using (almost) no disk space.

Note: be careful with spaces in your filenames if you’re handing Python strings to the shell. If your filenames have some whitespace, use this on the touch line:

!touch {file.replace(' ', '\ ')}

9. Executing an External Command with Arguments 

The goal this time is to execute a shell-level command with command line arguments from within iPython.  In general the format is as follows:

In [1]: import subprocess 
In [2]: cmd_arg_list = [‘cmd',  ‘arg1',  'arg2']
In [2]: 

One application of this could be to change file permissions of the Python programs in the current folder:

In [1]: import subprocess                                 In [2]: files = !ls                                       In [3]: for file in files: 
   ...:     if '.py' in file: 
   ...:['chmod', '744', file]) 


This command can also be useful if you are administering a SQLite database and need to execute a command from Python that is only available as a shell command, or as a ‘dot-command’ from the sqlite interactive command line tool. For example, if you want to dump a SQL table to a CSV text file, you can use it in this way:

In [1]: import subprocess 
In [2]:["sqlite3", "hr_database.db",  
            ".headers off",
            ".mode csv", 
            '.separator " :: "', 
            '.nullvalue ""', 
            "select * from {0} order by Name;".format('staff')

which (as far as I know) is something you cannot yet do with the SQLite3 API in Python.

A variation on this is if you want to collect the output of a shell-level command. To do this, you use subprocess.check_output(). In general the format is almost the same as for

In [1]: import subprocess 
In [2]: output = subprocess.check_output([‘cmd',  ‘arg1',  'arg2']) 

Note, however, that in Python 3 the output will be a byte string, accessible using:

In [3]: print(output.decode('utf8'))


10. Launching a Web Browser Session from Python 

This can be useful if you are writing an app that needs to launch a browser with a page already loaded, perhaps to display some information for users. From inside a terminal running iPython, you call the Python function webbrowser():

In [1]: import webbrowser                                  
In [2]: c = webbrowser.get("firefox")                      
In [3]: c.open_new_tab('')        
Out[3]: True

This will launch Firefox, or any browser you have installed on your system, open at the site you choose, with the URL route you need already set. If you’re a developer, this can save you a lot of time and typing if you’re testing, say,  a particular URL route for a Flask web framework with a different port each time:

In [1]: import webbrowser                                  
In [2]: port = 5007
In [3]: c = webbrowser.get("firefox")                      
In [4]: c.open_new_tab('localhost/index:{}'.format(port))  
Out[4]: True

In Summary

This has only been a taster for what is possible using iPython for system administration, but hopefully even seasoned Python users will find something useful here. Some of the commands may seem more complex than the shell commands, but iPython has a few advantages over shell commands:

  • You have access to the entire library of Python string and file manipulation functions, plus
  • You don’t have to re-familiarise yourself with any shell command manual pages, and
  • You can still access all your favourite shell commands, combined with the power of iPython.

Feel free to bookmark this page for reference if you think this list is useful. If your appetite has been whetted for more, please check out the references below.   

Useful References

For a more comprehensive introduction to this area, see Jason Brittain and Ian F. Darwin’s Python for Unix and Linux System Administration (O’Reilly, 2008).  

For a list of Python system administration tricks arranged thematically by problem, see Chapter 13 of  David Beazley and Brian K. Jones’ book The Python Cookbook, entitled Utility Scripting and System Administration (O’Reilly, 2013)

Faster Image Transforms With Cython

Python to Cython

This is the second post on how to accelerate Python with Cython. The previous post used Cython to declare static C data types in Python, to speed up the runtime of a prime number generator. In this post we shall be modifying a more complex program, one that performs an image transform on a map. This will allow me to demonstrate some more advanced Cython techniques, such as importing C functions from the C math library, using memory views of Numpy arrays and turning off the Python global interpreter lock (GIL).

As with the previous post, we shall be making a series of Cython modifications to Python code, noting the speed improvement with each step. As we go, we’ll be using the Cython compiler’s annotation feature to see which lines are converted into C at each stage, and which lines are still using Python objects and functions. And as we tune the code to run at increasingly higher speeds, we shall be profiling it to see what’s still holding us up, and where to refocus our attention.

Although I will be using Python 3 on a Mac, the instructions I give will mostly be platform agnostic:  I will assume you have installed Cython on your system (on Windows, Linux or OS/X) and have followed and understood the installation and testing steps in my previous post. This will be essential if you are to follow the steps I outline below. As stated in the previous post, Cython is not for Python beginners.

Continue reading “Faster Image Transforms With Cython”

From Python To Cython

This longer post will show you some of the coding skills you’ll need for turning your existing Python code into the Python-C hybrid we call Cython. In doing so, we’ll be digging into some C static data types, to see how much faster Python code will run, and restructuring some Python code along the way for maximum speed.

With Cython, all the benefits of Python are still yours – easily readable code, fast development cycles, powerful high level commands, maintainability, a suite of web development frameworks, a huge standard library for data science, machine learning, imaging, databases and security, plus easy manipulation of files, documents and strings. You should still use Python for all these things – these are what Python does best. But you should also consider combining them with Cython to speed up the computationally intensive Python functions that needs to be fast. Continue reading “From Python To Cython”

Parallel Python – 1: Prime Numbers

With the impending demise of Moore’s Law, multiple cores are a common manufacturers’ workaround for improving hardware performance, whether or not your installed apps can use the parallel architecture.

And with each new release of Python, parallel programming gets even easier. But the degree to which your code can use your multiple cores will depend on the kind of problem you are trying to solve, on the implementation of Python you are running and, as it turns out, how truly parallel the underlying architecture of your system actually is.

The goal of this series of posts is to see how adaptable some of my existing code is to take advantage of multi-core hardware, to see what changes need be made to scale it, and to measure the performance improvements from the exercise. Continue reading “Parallel Python – 1: Prime Numbers”

Transverse Mercator With Python

Mercator mapWith global warming melting the icecaps and opening up the poles for oil exploration and tourism, I think it’s time for a new standard wall map, one that shifts those distorted map regions away from major land masses, and places the polar regions where we can see them. That way, our cruise ship and oil tanker captains can navigate more easily through the clear, blue Arctic Ocean, unimpeded by any tiresome ice-pack.

I particularly love that oil companies want to use the new Arctic Ocean sea lanes to transport their oil to market faster. Is it irony, or some form of rare, extinction-level stupidity that only comes around one every few thousand years? Hard to tell. But I digress.

Continue reading “Transverse Mercator With Python”

The Toolkit – Updated

SpannersAlas, dear Windows, it was not to be. I’m afraid I’ve been seeing other platforms. Specifically, I’ve been spending time with OS/X behind your back.  It was just too painful to be with you.  All those arguments, the shouting, the hair-pulling, the throwing things across the room.

Sure, you’re a lot less volatile than you used to be. And you don’t do the tearful breakdown thing any more. Yes, I know I can do almost anything with you that I can with OS/X, but everything just takes longer. OK, you want me to be honest? Fine. I find you excruciatingly frustrating to be with.  Why is it always ME navigating around YOUR moods? I mean, why is it that after 25 years, everything with you is STILL a workaround?

Continue reading “The Toolkit – Updated”

Eratosthenes 2: Swifter, Further, Cooler

Human & alien hands


This post describes the process I used to design an algorithm that allows you to implement a modified Sieve of Eratosthenes to bypass the memory limitations of your computer and, in the process, to find big primes well beyond your 64-bit computer’s supposed numerical limit of 2.0e63 (9.223e18). Beyond that, with this algorithm, the only limitation is the speed of your CPU.

Continue reading “Eratosthenes 2: Swifter, Further, Cooler”

A Faster Sieve of Eratosthenes


The Sieve of Eratosthenes is a beautifully elegant way of finding all the prime numbers up to any limit. The goal of this post is to implement the algorithm as efficiently as possible in standard Python 3.5, without resorting to importing any modules, or to running it on faster hardware.

Eratosthenes was a Greek scholar who lived in Alexandria (276BC to 194BC) in the so-called Hellenistic period. He was working about a century after Alexander, and about a century before the Romans arrived to impose their cultural desert and call it peace. And then do nothing with the body of knowledge they discovered. Literally. For over 1,600 years, if you count Constantinople. Not a damn thing.

So much for overly religious, centralised, bureaucratic superstates, obsessed with conquest. But I digress.

Continue reading “A Faster Sieve of Eratosthenes”

Using AppleScript To Launch Python

OK, one for the Mac users. Continuing the  theme of user interfaces, here’s a simple but powerful way of using AppleScript to create a user interface for your Python programs and shell scripts and sending the results to just about any application installed on your Mac.

This solution has the advantage over Python’s native Tkinter in that the development time is much faster, and uses the speech synthesis features of OS/X to make your code much easier to use for the non-technical, elderly or visually impaired.

Continue reading “Using AppleScript To Launch Python”

Finding Primes Using Complex Numbers

Pythagoras In LegoWith complex numbers, I always feel as if I’m getting a glimpse of something truly awesome that lies hidden within mathematics. The first time I understood how they worked, I thought it was some form of magic.

I get the same feeling with prime numbers. Like many, I’ve looked at them from all angles – prime gaps, large primes, prime densities, prime sieves – and they continue to fascinate. A few months ago I was thumbing through Henry Warren’s programmers cookbook Hacker’s Delight (A) and discovered a whole chapter on the various formulas for (some of) them. Mind-bending stuff.

Continue reading “Finding Primes Using Complex Numbers”

GUI Template For Python: Part 2

This is the second of two posts on how to quickly create a Tkinter dashboard for your command line Python programs. The Tkinter widgets and programming techniques it introduces are a sequel to the previous post.

So far, you have an interactive graphical way of opening a file to analyse it in some way with your own logic, entering text to use as triggers or search strings, setting your own program flags on/off using check boxes, switching between two or more mutually exclusive program flags using radio boxes, controlling access to widgets and the variables they control, calling your own logic, and saving your results in a new file.

This post will build on these skills by showing how to create a dashboard to accept numerical input, perform different kinds of type- and value-checking, and select multiple input files simultaneously using a Tkinter GUI file selector. The solution will be multi-platform, and is shown running above on (from left) Windows 10, Mac OS/X and Linux Mint above. This post will explain how to create the same thing for your own program. Before proceeding, make sure you’ve read and understood the previous post.

Continue reading “GUI Template For Python: Part 2”

GUI Template For Python: Part 1

The next problem I needed to solve was to come up with a simple graphical user interface (GUI) template as a front-end for configuring and launching any Python code or module I may wish to write or run. Initial impetus: I didn’t want to have to write a user interface from scratch every time I wrote some Python code to manipulate text, data or files. Bonus reason: if I made the GUI template generic enough, others might be able to use it to create their own user interfaces.

This would solve a problem that occurs in many technical fields. A university professor may have a post-doc researcher on her team, one who has written a complex command line program performing, e.g. image processing, AI or genetic analysis. At some stage, there may be some highly repetitive tests that can be performed by someone less technical, freeing up the researcher. She wouldn’t want him running these repetitive command-line tests with code only he knows how to run or, worse, sitting around designing complex user interfaces for others to use it. It would be better to get an intern or research assistant (or even a temp) to run the tests using a GUI that the researcher can knock up in a day or two. This would free him up to concentrate on his research. And finish it faster. Continue reading “GUI Template For Python: Part 1”

Analysis Tool For Literary Texts

The first problem I wanted to solve was to write a short program that would allow me to perform basic textual analysis of any work of literature.

I wanted to be able to study the richness of different authors’ language by looking at how they used neologisms (their own made up words), pseudo-archaisms, invented their own contractions for authentic speech, or used hyphenated compound words, etc. I also wanted to be able to list all the characters and place names (proper nouns) mentioned in a text.

Continue reading “Analysis Tool For Literary Texts”

The Toolkit

SpannersOK, first things first. What tools will I be using?

After talking to a good friend who is an experienced coder, I decided on the following:


Spyder, running Python 3. It seems to have everything I need, including a good debugger, a variable explorer, hot-linking to function definitions, auto-completion typing, Matplotlib, QT, plus a choice of either a Python and iPython console (each with their different strengths). The bundle I went with is Spyder for WinPython-64bit (WinPython-64bit- The QT will be useful later.

Continue reading “The Toolkit”