Manipulating files and programs using Unix shell file programs can often be a bit of a pain, even for seasoned programmers. This can be due to how infrequently you use them, or because you are often moving between OS/X, Windows and Linux, and their subtle differences can often trip you up.
I used to be fairly proficient at them, but nowadays find I use them so rarely that I often have to revise what I used to know, even to achieve the most basic tasks. For many coders, the Unix shell programming language has become like an obscure language you only brush up on when you need to speak to a distant relative at Christmas time.
Fortunately, if you know and love Python, most of what you need to do with the Unix shell for filename searching, for-loops and file permissions can easily be done with iPython, without having to spend hours revising what you first learnt to do in Unix shell years ago.
Many Python users don’t realise iPython gives them access to three wonderful things in one place: all the shell commands you know and love, plus the full suite of Python functions, and access to the weird world of hybrid commands that allow you to mix both on the same line. The result is a powerful set of sys admin tools which you can use to perform some quite complex file, directory and process admin tasks.
This post is essentially a quick tour of ten common system administration jobs you can perform with iPython to manage the files, folders and processes on your system. There is so much more you can do, but this is just a taster for what you can achieve with iPython when you need to put aside your normal Python coding and roll up your sleeves to sort out your system. Without revising too much of the shell programming language syntax before you begin.
Installation & Simple Commands
If you use a sophisticated IDE such as Spyder or PyCharm, you will already have iPython as part of your IDE’s interface. If you’re running Mac or Linux and you already have Python installed, just use pip, or what ever package installation software you normally use:
$ pip install ipython
As you might expect, on Windows it’s a bit more painful, but a good way to install it is also to use pip. If you don’t have it, you’ll first need to launch a terminal window (cmd.exe) as the administrator, then install pip using easy_install, then install iPython using pip:
$ easy_install --upgrade pip
$ pip install ipython
Who said Windows was difficult? Your persistence will be rewarded though, as iPython on Windows actually looks pretty cool.
However you install it, once you’ve got it working you always run it the same way. Here’s how it looks when you launch it (the version numbers will vary):
$ ipython Python 3.7.1 (default, Dec 14 2018, 13:28:58) Type 'copyright', 'credits' or 'license' for more information IPython 7.6.1 -- An enhanced Interactive Python. Type '?' for help. In [1]:
As with its simpler sibling the standard Python REPL (the one with the >>> prompt), iPython is great for testing Python ideas and functions, but has many more features. The most obvious difference with the standard Python REPL is that your commands are numbered to allow you to reference and reuse useful command sequences later.
To view your command history with its numerical references, you just type:
In [1]: history -n
Or to see the previous session’s entire history, use:
In [2]: history ~1/
Another difference is the set of so-called magic commands, which are quite powerful and worth taking the time to learn. The magic commands are invoked using the % prefix. The command numbers work the same way as for the history command, and can also access the commands from previous sessions. For example, to recall a previous sequence of useful commands, you just use the recall command %recall (or %rep for repeat- they do the same thing):
In [5]: %recall 12-24
which presents the requested commands, allowing you to edit and rerun them if you wish.
If you are quite pleased with some commands you put together in your third last session, you can recall them too:
In [5]: %recall ~3/10-15
You also may want to edit a block of commands without running them, and then save them to a file. It’s all included in the edit command, which drops you and the commands into an active Vim editor session (or Notepad, if you’re using Windows) where you can edit the commands and save them to a .py file:
In [5]: %edit ~1/
But if you’re quite happy with a sequence of commands, you can save them to a Python file for later incorporation into a program:
In [87]: %save scrape_to_csv.py 69-79
OK, intro over. If you think you need a longer introduction to iPython, it’s well worth checking out the online documentation.
1. Manipulating Folder Content Lists
A. Turning a folder listing into a Python list: I’ll start with some simple but powerful commands that give you access to the current folder contents as a Python list. First, to turn the contents of a folder into a list, use this:
In [1]: file_list = !ls In [2]: file_list Out[2]: ['IMG_3941.png', 'IMG_4446.jpg', 'IMG_4447.png', 'IMG_4448.jpg', 'IMG_4449.jpg', 'IMG_4770.JPG', 'IMG_5138.JPG', 'IMG_5158.JPG', 'IMG_5311.JPG', 'IMG_5441.JPG' 'IMG_6150.png', 'IMG_6151.jpg', 'IMG_6153.png', 'IMG_6179.jpg']
The hybrid !ls command is superior to os.listdir() because it ignores the hidden .files, giving the list of only the normally visible filenames as a Python list. This format opens up an enormous number of possibilities for you – you can then use the members of the list any way you like, filtering them through any Python function, even using your own lambdas:
In [7]: file_list.grep(lambda x : '.JPG' in x) Out[7]: ['IMG_4770.JPG', 'IMG_5138.JPG', 'IMG_5158.JPG', 'IMG_5311.JPG', 'IMG_5441.JPG']
The single Python grep line is a succinct way of saving on a Python for-loop.
B. Turning a Folder Listing into a CSV text string: This is useful if you are moving back and forward between folder lists and strings. Again, a couple of ways to do this. Here’s a good one:
In [2]: dirlist = !ls In [3]: dirlist_text = ', '.join(dirlist) In [4]: with open('Actors.txt', "wt") as f: ...: f.write(dirlist_text) ...: Out[4]: 1882 In [5]: dirlist_text Out[5]: "Titus Welliver, Tom Cruise, Tom Hanks, Tommy Lee Jones, Uma Thurman, Walton Goggins, Woody Harrelson"
2. Extracting a Filename From a Path
This is fairly easy with standard Python, and doesn’t need any of iPython’s tricks:
In [1]: import os In [2]: myfilepath = ‘/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest/ShiftedWorldMap.png’ In [3]: pathbits = os.path.split(myfilepath) In [4]: pathbits Out[4]: ('/Users/Me/Desktop/Coding/CodingStuffMASTER/PythonStuff/iPythonTest', 'ShiftedWorldMap.png') In [5]: myfile = os.path.split(myfilepath)[-1] In [6]: print (myfile) Out[6]: ShiftedWorldMap.png
3. Renaming Files in a Batch
This can be very useful indeed. It is slightly different on different shells, but in iPython, it’s always the same:
In [1]: import os In [2]: files = !ls In [3]: for file in files: # can't use !ls here ...: if '.png' in file: ...: os.rename(file, file.replace('IMG_', 'NZ_Foto)) ...:
Note how the !ls command could not be used in the for loop, and we had to set it up first.
4. Deleting Files That Match a Pattern
This is almost identical, but must be used with caution:
In [1]: import os In [2]: files = !ls In [3]: for myfile in files: ...: if ".jpg" in myfile: ...: os.remove(myfile) ...:
Note how you can delete the file while using its name as the loop variable.
5. Finding and Executing Your Programs
There are a few ways to do this. Here is a simple iPython string version, which in a folder of Python programs will run every Python script it finds:
In [1]: import os In [2]: progs = !ls In [3]: for prog_name in progs: ...: if ".py" in prog_name: ...: os.system("./" + prog_name)
This can be adapted to work on any kind of program.
6. Gathering Process Information
The key to understanding the power of using iPython for system administration is that you can read any system variable and do anything with it, using any of the Python string, file, print, IO, web or database libraries and their methods.
Once you realise that you can feed the output of any shell command into iPython, anything is possible. Using a shell command to find all processes relating to a particular user, we would normally feed ps into a pipe (noting that that syntax may vary between platforms):
$ ps aux | awk '{if ($1 == "Elon Tusk") print $2}'
But with iPython, you can grab the output and parse it to your heart’s content:
In [1]: ps = !ps aux
In [2]: len(ps)
Out[2]: 311
To find all processes relating to, for example, Bluetooth, we would simply say:
In [2]: ps.grep('bluetooth')
Out[2]:
['root 127 0.3 0.1 4334916 13008 ?? Ss 7:24am 0:35.53 /usr/sbin/bluetoothd']]
7. Monitoring Processes Over Time
There might also be a situation were you need to monitor system information over time. This short iPython loop achieves this by using Python index variables to create incremental system snapshots. Again, this can be adapted to do almost anything. As with ‘ls’ above, note how a hybrid command takes the output of a shell command and modifies it with a Python variable to create incremental filenames. With shell command, this is one of those tasks that often needs slightly different syntax on different platforms.
In iPython, the command is the same for Mac, Linux or Windows:
In [1]: import time In [2]: for i in range(1, 10): ...: !ps aux > ps_snapshot{i}.txt ...: time.sleep(60) ...: In [3]: ls ps_snapshot1.txt ps_snapshot4.txt ps_snapshot7.txt ps_snapshot2.txt ps_snapshot5.txt ps_snapshot8.txt ps_snapshot3.txt ps_snapshot6.txt ps_snapshot9.txt
Think of the mixed Python/Shell ‘ps aux’ line as essentially a shell line, allowing you to insert Python variables into the {} field, just as you would with the Python string.format() function.
Note that no exclamation point was used before the ls command [3] here. This is because iPython recognises the simple form of most common shell commands, such as ls, pwd, cd & cat – but only when typed on their own line. In a multi-line code block, iPython needs the !prefix before the shell command.
To create a whole host of 0-byte files with specific filenames for testing, you can combine iPython with the Unix touch command:
In [1]: for i in range (10): ...: !touch Foto{i}.png ...: In [2]: ls Fo* Foto0.png Foto2.png Foto4.png Foto6.png Foto8.png Foto1.png Foto3.png Foto5.png Foto7.png Foto9.png
8. Duplicating a Massive File Structure For Testing
Suppose the above list of files is real, but that there are a million of them. And suppose that we need to test a file manipulation program on this number of files, but not on the original files. Rather than copying gigabytes of data to another location just to text our program on, we can quickly duplicate any number of file names as empty files to another folder.
There are a couple of ways to do this. Here’s one:
In [1]: photo_list = !ls In [2]: cd /TestFolder /Users/Me/TestFolder In [3]: for file in photo_list: ...: !touch {file} ...: In [4]: ls Foto0.png Foto2.png Foto4.png Foto6.png Foto8.png Foto1.png Foto3.png Foto5.png Foto7.png Foto9.png
All of which are 0 byte files. We have therefore duplicated a large file structure using (almost) no disk space.
Note: be careful with spaces in your filenames if you’re handing Python strings to the shell. If your filenames have some whitespace, use this on the touch line:
!touch {file.replace(' ', '\ ')}
9. Executing an External Command with Arguments
The goal this time is to execute a shell-level command with command line arguments from within iPython. In general the format is as follows:
In [1]: import subprocess In [2]: cmd_arg_list = [‘cmd', ‘arg1', 'arg2'] In [2]: subprocess.call(cmd_arg_list)
One application of this could be to change file permissions of the Python programs in the current folder:
In [1]: import subprocess In [2]: files = !ls In [3]: for file in files: ...: if '.py' in file: ...: subprocess.call(['chmod', '744', file]) ...:
This command can also be useful if you are administering a SQLite database and need to execute a command from Python that is only available as a shell command, or as a ‘dot-command’ from the sqlite interactive command line tool. For example, if you want to dump a SQL table to a CSV text file, you can use it in this way:
In [1]: import subprocess In [2]: subprocess.call(["sqlite3", "hr_database.db", ".headers off", ".mode csv", '.separator " :: "', '.nullvalue ""', '.once', 'Data/DB_table_data/{0}_table_data.txt', "select * from {0} order by Name;".format('staff') ])
which (as far as I know) is something you cannot yet do with the SQLite3 API in Python.
A variation on this is if you want to collect the output of a shell-level command. To do this, you use subprocess.check_output(). In general the format is almost the same as for subprocess.call():
In [1]: import subprocess In [2]: output = subprocess.check_output([‘cmd', ‘arg1', 'arg2'])
Note, however, that in Python 3 the output will be a byte string, accessible using:
In [3]: print(output.decode('utf8'))
10. Launching a Web Browser Session from Python
This can be useful if you are writing an app that needs to launch a browser with a page already loaded, perhaps to display some information for users. From inside a terminal running iPython, you call the Python function webbrowser():
In [1]: import webbrowser In [2]: c = webbrowser.get("firefox") In [3]: c.open_new_tab('https://idlecoding.com/from-python-to-cython/') Out[3]: True
This will launch Firefox, or any browser you have installed on your system, open at the site you choose, with the URL route you need already set. If you’re a developer, this can save you a lot of time and typing if you’re testing, say, a particular URL route for a Flask web framework with a different port each time:
In [1]: import webbrowser In [2]: port = 5007 In [3]: c = webbrowser.get("firefox") In [4]: c.open_new_tab('localhost/index:{}'.format(port)) Out[4]: True
In Summary
This has only been a taster for what is possible using iPython for system administration, but hopefully even seasoned Python users will find something useful here. Some of the commands may seem more complex than the shell commands, but iPython has a few advantages over shell commands:
- You have access to the entire library of Python string and file manipulation functions, plus
- You don’t have to re-familiarise yourself with any shell command manual pages, and
- You can still access all your favourite shell commands, combined with the power of iPython.
Feel free to bookmark this page for reference if you think this list is useful. If your appetite has been whetted for more, please check out the references below.
Useful References
For a more comprehensive introduction to this area, see Jason Brittain and Ian F. Darwin’s Python for Unix and Linux System Administration (O’Reilly, 2008).
For a list of Python system administration tricks arranged thematically by problem, see Chapter 13 of David Beazley and Brian K. Jones’ book The Python Cookbook, entitled Utility Scripting and System Administration (O’Reilly, 2013)
