Using iPython for Sys Admin

Python on mainframe terminal

Manipulating files and programs using Unix shell file programs can often be a bit of a pain, even for seasoned programmers. This can be due to how infrequently you use them, or because you are often moving between OS/X, Windows and Linux, and their subtle differences can often trip you up. 

I used to be fairly proficient at them, but nowadays find I use them so rarely that I often have to revise what I used to know, even to achieve the most basic tasks. For many coders, the Unix shell programming language has become like an obscure language you only brush up on when you need to speak to a distant relative at Christmas time.

Fortunately, if you know and love Python, most of what you need to do with Unix shell for-loops and file permissions can easily be done with iPython, without having to spend an entire day every twelve months re-learning what you first learnt to do in Unix shell files ten years ago. 

Many Python users don’t realise iPython gives them access to three wonderful things in one place: all the shell commands you know and love, plus the full suite of Python functions,  and access to the weird world of hybrid commands that allow you to mix both on the same line. The result is a powerful set of sys admin tools which you can use to perform some quite complex file, directory and process admin tasks. 

This post is essentially a quick tour of ten common system administration jobs you can perform with iPython to manage the files, folders and processes on your system. There is so much more you can do, but this is just a taster for what you can achieve with iPython when you need to put aside your normal Python coding and roll up your sleeves to sort out your system. Without revising too much of the shell programming language syntax before you begin.

Installation & Simple Commands

If you use a sophisticated IDE such as Spyder or PyCharm, you will already have iPython as part of your IDE’s interface. If you’re running Mac or Linux and you already have Python installed, just use pip, or what ever package installation software you normally use: 

$ pip install ipython 

As you might expect, on Windows it’s a bit more painful, but a good way to install it is also to use pip. If you don’t have it, you’ll first need to launch a terminal window (cmd.exe) as the administrator, then install pip using easy_install, then install iPython using pip:

$ easy_install --upgrade pip
$ pip install ipython 

Who said Windows was difficult? Your persistence will be rewarded though, as iPython on Windows actually looks pretty cool. 

However you install it, once you’ve got it working you always run it the same way. Here’s how it looks when you launch it (the version numbers will vary):

$ ipython 
Python 3.7.1 (default, Dec 14 2018, 13:28:58) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:   

As with its simpler sibling the standard Python REPL (the one with the >>> prompt), iPython is great for testing Python ideas and functions, but has many more features. The most obvious difference with the standard Python REPL is that your commands are numbered to allow you to reference and reuse useful command sequences later. 

To view your command history with its numerical references, you just type: 

In [1]: history -n  

Or to see the previous session’s entire history, use:

In [2]: history ~1/  

Another difference is the set of so-called magic commands, which are quite powerful and worth taking the time to learn. The magic commands are invoked using the % prefix. The command numbers work the same way as for the history command, and can also access the commands from previous sessions. For example, to recall a previous sequence of useful commands, you just use the recall command %recall (or %rep for repeat- they do the same thing):

In   [5]: %recall 12-24  

which presents the requested commands, allowing you to edit and rerun them if you wish.

If you are quite pleased with some commands you put together in your third last session, you can recall them too:

In   [5]: %recall ~3/10-15  

You also may want to edit a block of commands without running them, and then save them to a file. It’s all included in the edit command, which drops you and the commands into an active Vim editor session (or Notepad, if you’re using Windows) where you can edit the commands and save them to a .py file:

In   [5]: %edit ~1/  

But if you’re quite happy with a sequence of commands, you can save them to a Python file for later incorporation into a program:

In   [87]:  %save scrape_to_csv.py 69-79 

OK, intro over. If you think you need a longer introduction to iPython, it’s well worth checking out the online documentation.

1. Manipulating Folder Content Lists 

A. Turning a folder listing into a Python list:   I’ll start with some simple but powerful commands that give you access to the current folder contents as a Python list. First, to turn the contents of a folder into a list, use this:

In [1]: file_list = !ls                                    
In [2]: file_list   
Out[2]: 
['IMG_3941.png',
 'IMG_4446.jpg',
 'IMG_4447.png',
 'IMG_4448.jpg',
 'IMG_4449.jpg',
 'IMG_4770.JPG',
 'IMG_5138.JPG',
 'IMG_5158.JPG',
 'IMG_5311.JPG',
 'IMG_5441.JPG'
 'IMG_6150.png',
 'IMG_6151.jpg',
 'IMG_6153.png',
 'IMG_6179.jpg']

The hybrid !ls command is superior to os.listdir() because it ignores the hidden .files, giving the list of only the normally visible filenames as a Python list. This format opens up an enormous number of possibilities for you – you can then use the members of the list any way you like, filtering them through any Python function, even using your own lambdas:

In [7]: file_list.grep(lambda x : '.JPG' in x)             
Out[7]: 
['IMG_4770.JPG',
 'IMG_5138.JPG',
 'IMG_5158.JPG',
 'IMG_5311.JPG',
 'IMG_5441.JPG']

The single Python grep line is a succinct way of saving on a Python for-loop.

B. Turning a Folder Listing into a CSV text string: This is useful if you are moving back and forward between folder lists and strings. Again, a couple of ways to do this. Here’s a good one:

In [2]: dirlist = !ls                                      
In [3]: dirlist_text = ', '.join(dirlist)       
In [4]: with open('Actors.txt', "wt") as f:
   ...:     f.write(dirlist_text)                          
   ...: 
Out[4]: 1882

In [5]: dirlist_text                                       
Out[5]: "Titus Welliver, Tom Cruise, Tom Hanks, 
Tommy Lee Jones, Uma Thurman, Walton Goggins, 
Woody Harrelson"

 

2. Extracting a Filename From a Path

This is fairly easy with standard Python, and doesn’t need any of iPython’s tricks:

In [1]: import os
In [2]: myfilepath = ‘/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest/ShiftedWorldMap.png’
In [3]: pathbits = os.path.split(myfilepath)
In [4]: pathbits
Out[4]: 
('/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest', 'ShiftedWorldMap.png')

In [5]: myfile = os.path.split(myfilepath)[-1]
In [6]: print (myfile)
Out[6]: 
ShiftedWorldMap.png

3. Renaming Files in a Batch

This can be very useful indeed. It is slightly different on different shells, but in iPython, it’s always the same:

In [1]: import os
In [2]: files = !ls
In [3]: for file in files:   # can't use !ls here
    ...:    if '.png' in file:
    ...:        os.rename(file, file.replace('IMG_', 'NZ_Foto))
    ...:

Note how the !ls command could not be used in the for loop, and we had to set it up first.

4. Deleting Files That Match a Pattern

This is almost identical, but must be used with caution:

In [1]: import os 
In [2]: files = !ls
In [3]: for myfile in files: 
   ...:     if ".jpg" in myfile: 
   ...:         os.remove(myfile) 
   ...:

Note how you can delete the file while using its name as the loop variable.

 

5. Finding and Executing Your Programs

There are a few ways to do this. Here is a simple iPython string version, which in a folder of Python programs will run every Python script it finds:

In [1]: import os
In [2]: progs = !ls
In [3]: for prog_name in progs: 
    ...:     if ".py" in prog_name: 
    ...:         os.system("./" + prog_name)

This can be adapted to work on any kind of program.

6. Gathering Process Information

The key to understanding the power of using iPython for system administration is that you can read any system variable and do anything with it, using any of the Python string, file, print, IO, web or database libraries and their methods.

Once you realise that you can feed the output of any shell command into iPython, anything is possible. Using a shell command to find all processes relating to a particular user, we would normally feed ps into a pipe (noting that that syntax may vary between platforms):

$ ps aux | awk '{if ($1 == "Elon Tusk") print $2}'

 

But with iPython, you can grab the output and parse it to your heart’s content:

In [1]: ps = !ps aux 
In [2]: len(ps)                                            
Out[2]: 311

 

To find all processes relating to, for example, Bluetooth, we would simply say:

In [2]: ps.grep('bluetooth')
Out[2]: 
['root  127  0.3  0.1  4334916   13008 ??  Ss  7:24am   0:35.53 /usr/sbin/bluetoothd']]

7. Monitoring Processes Over Time

There might also be a situation were you need to monitor system information over time. This short iPython loop achieves this by using Python index variables to create incremental system snapshots. Again, this can be adapted to do almost anything. As with ‘ls’ above, note how a hybrid command takes the output of a shell command and modifies it with a Python variable to create incremental filenames. With shell command, this is one of those tasks that often needs slightly different syntax on different platforms. 

In iPython, the command is the same for Mac, Linux or Windows:

In [1]: import time                                        
In [2]: for i in range(1, 10): 
   ...:     !ps aux > ps_snapshot{i}.txt 
   ...:     time.sleep(60) 
   ...:      

In [3]: ls
ps_snapshot1.txt  ps_snapshot4.txt  ps_snapshot7.txt   
ps_snapshot2.txt  ps_snapshot5.txt  ps_snapshot8.txt
ps_snapshot3.txt  ps_snapshot6.txt  ps_snapshot9.txt 

Think of the mixed Python/Shell ‘ps aux’ line as essentially a shell line, allowing you to insert Python variables into the {} field, just as you would with the Python string.format() function.

Note that no exclamation point was used before the ls command [3] here. This is because iPython recognises the simple form of most common shell commands, such as ls, pwd, cd & cat – but only when typed on their own line. In a multi-line code block, iPython needs the !prefix before the shell command.

To create a whole host of 0-byte files with specific filenames for testing, you can combine iPython with the Unix touch command:

In [1]: for i in range (10): 
   ...:     !touch Foto{i}.png 
   ...:

In [2]: ls Fo*                                             
Foto0.png  Foto2.png  Foto4.png  Foto6.png  Foto8.png
Foto1.png  Foto3.png  Foto5.png  Foto7.png  Foto9.png

 

8. Duplicating a Massive File Structure For Testing

Suppose the above list of files is real, but that there are a million of them. And suppose that we need to test a file manipulation program on this number of files, but not on the original files. Rather than copying gigabytes of data to another location just to text our program on, we can quickly duplicate any number of file names  as empty files to another folder.

There are a couple of ways to do this. Here’s one:

In [1]: photo_list = !ls
In [2]: cd /TestFolder                                     
/Users/Me/TestFolder   

In [3]: for file in photo_list: 
   ...:     !touch {file} 
   ...:
In [4]: ls
Foto0.png  Foto2.png  Foto4.png  Foto6.png  Foto8.png
Foto1.png  Foto3.png  Foto5.png  Foto7.png  Foto9.png

All of which are 0 byte files. We have therefore duplicated a large file structure using (almost) no disk space.

Note: be careful with spaces in your filenames if you’re handing Python strings to the shell. If your filenames have some whitespace, use this on the touch line:

!touch {file.replace(' ', '\ ')}

9. Executing an External Command with Arguments 

The goal this time is to execute a shell-level command with command line arguments from within iPython.  In general the format is as follows:

In [1]: import subprocess 
In [2]: cmd_arg_list = [‘cmd',  ‘arg1',  'arg2']
In [2]: subprocess.call(cmd_arg_list) 

One application of this could be to change file permissions of the Python programs in the current folder:

In [1]: import subprocess                                 In [2]: files = !ls                                       In [3]: for file in files: 
   ...:     if '.py' in file: 
   ...:         subprocess.call(['chmod', '744', file]) 
   ...:        

 

This command can also be useful if you are administering a SQLite database and need to execute a command from Python that is only available as a shell command, or as a ‘dot-command’ from the sqlite interactive command line tool. For example, if you want to dump a SQL table to a CSV text file, you can use it in this way:

In [1]: import subprocess 
In [2]: subprocess.call(["sqlite3", "hr_database.db",  
            ".headers off",
            ".mode csv", 
            '.separator " :: "', 
            '.nullvalue ""', 
            '.once', 
            'Data/DB_table_data/{0}_table_data.txt',
            "select * from {0} order by Name;".format('staff')
            ])  

which (as far as I know) is something you cannot yet do with the SQLite3 API in Python.

A variation on this is if you want to collect the output of a shell-level command. To do this, you use subprocess.check_output(). In general the format is almost the same as for subprocess.call():

In [1]: import subprocess 
In [2]: output = subprocess.check_output([‘cmd',  ‘arg1',  'arg2']) 

Note, however, that in Python 3 the output will be a byte string, accessible using:

In [3]: print(output.decode('utf8'))

 

10. Launching a Web Browser Session from Python 

This can be useful if you are writing an app that needs to launch a browser with a page already loaded, perhaps to display some information for users. From inside a terminal running iPython, you call the Python function webbrowser():

In [1]: import webbrowser                                  
In [2]: c = webbrowser.get("firefox")                      
In [3]: c.open_new_tab('https://idlecoding.com/from-python-to-cython/')        
Out[3]: True

This will launch Firefox, or any browser you have installed on your system, open at the site you choose, with the URL route you need already set. If you’re a developer, this can save you a lot of time and typing if you’re testing, say,  a particular URL route for a Flask web framework with a different port each time:

In [1]: import webbrowser                                  
In [2]: port = 5007
In [3]: c = webbrowser.get("firefox")                      
In [4]: c.open_new_tab('localhost/index:{}'.format(port))  
Out[4]: True

In Summary

This has only been a taster for what is possible using iPython for system administration, but hopefully even seasoned Python users will find something useful here. Some of the commands may seem more complex than the shell commands, but iPython has a few advantages over shell commands:

  • You have access to the entire library of Python string and file manipulation functions, plus
  • You don’t have to re-familiarise yourself with any shell command manual pages, and
  • You can still access all your favourite shell commands, combined with the power of iPython.

Feel free to bookmark this page for reference if you think this list is useful. If your appetite has been whetted for more, please check out the references below.   

Useful References

For a more comprehensive introduction to this area, see Jason Brittain and Ian F. Darwin’s Python for Unix and Linux System Administration (O’Reilly, 2008).  

For a list of Python system administration tricks arranged thematically by problem, see Chapter 13 of  David Beazley and Brian K. Jones’ book The Python Cookbook, entitled Utility Scripting and System Administration (O’Reilly, 2013)

From Python To Cython

This longer post will show you some of the coding skills you’ll need for turning your existing Python code into the Python-C hybrid we call Cython. In doing so, we’ll be digging into some C static data types, to see how much faster Python code will run, and restructuring some Python code along the way for maximum speed.

With Cython, all the benefits of Python are still yours – easily readable code, fast development cycles, powerful high level commands, maintainability, a suite of web development frameworks, a huge standard library for data science, machine learning, imaging, databases and security, plus easy manipulation of files, documents and strings. You should still use Python for all these things – these are what Python does best. But you should also consider combining them with Cython to speed up the computationally intensive Python functions that needs to be fast. Continue reading “From Python To Cython”

The Toolkit – Updated

SpannersAlas, dear Windows, it was not to be. I’m afraid I’ve been seeing other platforms. Specifically, I’ve been spending time with OS/X behind your back.  It was just too painful to be with you.  All those arguments, the shouting, the hair-pulling, the throwing things across the room.

Sure, you’re a lot less volatile than you used to be. And you don’t do the tearful breakdown thing any more. Yes, I know I can do almost anything with you that I can with OS/X, but everything just takes longer. OK, you want me to be honest? Fine. I find you excruciatingly frustrating to be with.  Why is it always ME navigating around YOUR moods? I mean, why is it that after 25 years, everything with you is STILL a workaround?

Continue reading “The Toolkit – Updated”