Using the pathlib module

Using the `pathlib` module#

The pathlib module is one of two ways of manipulating and using file paths in Python. See the path manipulation page for more discussion, and os.path page for a tutorial on an alternative approach.

The primary documentation for pathlib is https://docs.python.org/3/library/pathlib.html.

The standard way to use the Pathlib module is to import the Path class from the module:

from pathlib import Path

In Jupyter or IPython, you can tab complete on Path to list the methods (functions) and attributes attached to it.

An object (value) of type Path represents a pathname. As you remember , a pathname is a string that identifies a particular file or directory on a computer filesystem.

Let us start by making a default object from the Path class, like this:

p = Path()
p

PosixPath('.')

By default, the path object, here p, refers to our current working directory, or . for short. . is a relative path, meaning that we specify where we are relative to our current directory. . means we are exactly in our current directory.

Because the . is a relative path, it does not tell us where we are in the filesystem, only where we are relative to the current directory.

Path objects have an absolute function attached to them. Another way of saying this is that Path objects have an absolute method. Calling this method gives us the absolute location of the path, meaning, the filesystem position relative to the base location of the disk the file is on.

abs_p = p.absolute()
abs_p

PosixPath('/home/runner/work/textbook/textbook')

Notice the / in front of the absolute filename (on Unix), meaning the base location for all files. You will see a drive location like C: or similar, at the front of the absolute path, if you are on Windows.

We can always convert the Path object to a simple string, using the str function. str() converts anything to a string, if it can:

# The path, as a string
str(abs_p)

'/home/runner/work/textbook/textbook'

Sometimes we want to get a path referring the directory containing a path. The Path object has a parent attribute attached to it (an attribute is data attached to an object). The parent attribute is a Path object for the containing directory:

abs_p.parent

PosixPath('/home/runner/work/textbook')

The parent attribute of the Path object gives the directory name from a full file path. It works correctly for Unix paths on Unix machines, and Windows paths on Windows machines.

# On Unix
a_path = Path('/a/full/path/then_filename.txt')
# Show the directory containing the file.
a_path.parent

PosixPath('/a/full/path')

parent also works for relative paths.

# On Unix
rel_path = Path('relative/path/then_filename.txt')
rel_path.parent

PosixPath('relative/path')

Use the name attribute of the Path object to get the filename rather than the directory name:

# On Unix
rel_path.name

'then_filename.txt'

Sometimes you want to join one or more directory names with a filename to get a path. Path objects have a clever way of doing this, by overriding the / (division) operator.

To remind you about operator overloading, remember that addition means different things for numbers and strings. For numbers, addition means arithmetic addition:

# Addition for numbers
2 + 2

For strings, addition means concatenation — sticking the strings together:

# Addition for strings.
"first" + "second"

'firstsecond'

Path objects use the division operator / to mean “stick the path fragments together to make a new path, where the / separates directories”.

# On Unix
Path('relative') / 'path' / 'then_filename.txt'

PosixPath('relative/path/then_filename.txt')

This also works on Windows and Unix in the same way.

Sometimes you want to get the filename extension. Use the suffix attribute for this:

rel_path

PosixPath('relative/path/then_filename.txt')

rel_path.suffix

'.txt'

You will often find yourself wanting to replace the file extension. You can do this with the with_suffix method:

rel_path.with_suffix('.md')

PosixPath('relative/path/then_filename.md')

Path objects also have methods that allow you to read and write text characters and raw bytes.

Let us make a new path to point to a file we will write in the current directory.

new_path = Path() / 'a_test_file.txt'
new_path

PosixPath('a_test_file.txt')

We can write text characters (strings) to this file, with the write_text method:

a_multiline_string = """Some text.
More text.
Last text."""
new_path.write_text(a_multiline_string)

We can read the text out of a file using read_text:

new_path.read_text()

'Some text.\nMore text.\nLast text.'

Similarly, we can write and read raw byte data, using write_bytes and read_bytes.

It is often useful to read in a text file, and split the result into lines. We do this with read_text, and then we use the splitlines method of string object to split the read text into lines.

text = new_path.read_text()
text.splitlines()

['Some text.', 'More text.', 'Last text.']

Listing files in a directory#

We can use the glob method of the Path object to give a list of all, or some files in a directory.

For example, to see all files in the current directory, we could do this:

cwd = Path()
list(cwd.glob('*'))

[PosixPath('arrays_and_images.Rmd'),
 PosixPath('my_image_again.nii'),
 PosixPath('packages_namespaces.Rmd'),
 PosixPath('coding_style.md'),
 PosixPath('saving_images.Rmd'),
 PosixPath('mean_test_example.Rmd'),
 PosixPath('dot_outer.Rmd'),
 PosixPath('slice_timing.Rmd'),
 PosixPath('the_software.md'),
 PosixPath('git_walk_through.Rmd'),
 PosixPath('numpy_transpose.Rmd'),
 PosixPath('a_test_file.txt'),
 PosixPath('regress_one_voxel.Rmd'),
 PosixPath('whole_image_statistics.Rmd'),
 PosixPath('git_workflow_exercises.md'),
 PosixPath('git_videos.md'),
 PosixPath('dipy_registration.py'),
 PosixPath('truthiness.Rmd'),
 PosixPath('validating_data.Rmd'),
 PosixPath('list_comprehensions.Rmd'),
 PosixPath('my_script.py'),
 PosixPath('printing_floating.Rmd'),
 PosixPath('rotation_2d_3d.md'),
 PosixPath('classes.md'),
 PosixPath('numpy_logical.Rmd'),
 PosixPath('my_pair_image.img'),
 PosixPath('test_one_voxel.Rmd'),
 PosixPath('nibabel_apply_affine.Rmd'),
 PosixPath('github_pca_homework.md'),
 PosixPath('logistics.md'),
 PosixPath('plot_lines.Rmd'),
 PosixPath('participate.md'),
 PosixPath('mymodule.py'),
 PosixPath('reading_text.Rmd'),
 PosixPath('test_rdmodule.py'),
 PosixPath('convolution_background.Rmd'),
 PosixPath('participate_grading.md'),
 PosixPath('on_loops.Rmd'),
 PosixPath('voxels_by_time.Rmd'),
 PosixPath('data'),
 PosixPath('arange.Rmd'),
 PosixPath('my_image.nii'),
 PosixPath('numpy_meshgrid.Rmd'),
 PosixPath('boolean_indexing_nd.Rmd'),
 PosixPath('subtract_mean_math.md'),
 PosixPath('keyword_arguments.Rmd'),
 PosixPath('multi_multiply.Rmd'),
 PosixPath('CNAME'),
 PosixPath('os_path.Rmd'),
 PosixPath('model_one_voxel.Rmd'),
 PosixPath('_todo.md'),
 PosixPath('path_manipulation.Rmd'),
 PosixPath('reshape_and_4d.Rmd'),
 PosixPath('_toc.yml'),
 PosixPath('numpy_squeeze.Rmd'),
 PosixPath('installation_on_mac.md'),
 PosixPath('Makefile'),
 PosixPath('convolution_matrices.Rmd'),
 PosixPath('slicing_with_booleans.Rmd'),
 PosixPath('rdmodule.py'),
 PosixPath('using_jupyter.Rmd'),
 PosixPath('why_mse_correlation.Rmd'),
 PosixPath('more_on_jupyter.Rmd'),
 PosixPath('reshape_and_3d.Rmd'),
 PosixPath('nibabel_affines.Rmd'),
 PosixPath('subtract_means.Rmd'),
 PosixPath('_build'),
 PosixPath('syllabus.md'),
 PosixPath('hypothesis_tests.Rmd'),
 PosixPath('nans.Rmd'),
 PosixPath('string_literals.Rmd'),
 PosixPath('numpy_random.Rmd'),
 PosixPath('otsu_threshold.Rmd'),
 PosixPath('subplots.Rmd'),
 PosixPath('rotations.py'),
 PosixPath('allclose.Rmd'),
 PosixPath('to_code.md'),
 PosixPath('images_3d.Rmd'),
 PosixPath('glm_intro.Rmd'),
 PosixPath('_config.yml'),
 PosixPath('.gitignore'),
 PosixPath('dot_and_outer.md'),
 PosixPath('images'),
 PosixPath('brisk_python.Rmd'),
 PosixPath('another_example.py'),
 PosixPath('pathlib.Rmd'),
 PosixPath('anterior_cingulate.md'),
 PosixPath('non_tr_onsets.Rmd'),
 PosixPath('intro_to_4d.Rmd'),
 PosixPath('sequence_unpacking.Rmd'),
 PosixPath('map_coordinates.Rmd'),
 PosixPath('comparing_arrays.Rmd'),
 PosixPath('installation_on_linux.md'),
 PosixPath('newaxis.Rmd'),
 PosixPath('floating_in_text.Rmd'),
 PosixPath('project_grading.md'),
 PosixPath('bonferroni_correction.Rmd'),
 PosixPath('example_module.py'),
 PosixPath('methods_vs_functions.Rmd'),
 PosixPath('ds114_sub009_t2r1_conv.txt'),
 PosixPath('example_data.md'),
 PosixPath('bib'),
 PosixPath('on_modules.Rmd'),
 PosixPath('images_and_memory.Rmd'),
 PosixPath('.github'),
 PosixPath('reshape_and_2d.Rmd'),
 PosixPath('.git'),
 PosixPath('regression_notation.Rmd'),
 PosixPath('assert.Rmd'),
 PosixPath('numpy_intro.Rmd'),
 PosixPath('dunders.Rmd'),
 PosixPath('matrix_rank.Rmd'),
 PosixPath('choosing_editor.md'),
 PosixPath('nibabel_images.Rmd'),
 PosixPath('nipype_settings.py'),
 PosixPath('string_formatting.Rmd'),
 PosixPath('diag_inverse.md'),
 PosixPath('projects.md'),
 PosixPath('on_testing.Rmd'),
 PosixPath('introducing_nipype.Rmd'),
 PosixPath('validate_against_scipy.Rmd'),
 PosixPath('dipy_registration.Rmd'),
 PosixPath('_scripts'),
 PosixPath('diagonal_zooms.Rmd'),
 PosixPath('github_dummies_homework.md'),
 PosixPath('what_is_an_image.Rmd'),
 PosixPath('topics.md'),
 PosixPath('tr_and_headers.Rmd'),
 PosixPath('docstrings.Rmd'),
 PosixPath('module_directories.Rmd'),
 PosixPath('image_header_and_affine.Rmd'),
 PosixPath('simplemod.py'),
 PosixPath('classes_and_labs.md'),
 PosixPath('voxel_time_courses.Rmd'),
 PosixPath('nipype_ds114_sub009_t2r1.py'),
 PosixPath('scripts'),
 PosixPath('comparing_floats.md'),
 PosixPath('images_and_affines.Rmd'),
 PosixPath('installation.md'),
 PosixPath('full_scripting.md'),
 PosixPath('exercises.md'),
 PosixPath('mydirmod'),
 PosixPath('glossary.md'),
 PosixPath('optimizing_space.Rmd'),
 PosixPath('check_install.py'),
 PosixPath('using_module.Rmd'),
 PosixPath('boolean_arrays.Rmd'),
 PosixPath('numpy_diag.Rmd'),
 PosixPath('.codespellrc'),
 PosixPath('bibliography.md'),
 PosixPath('resampling_with_ndimage.Rmd'),
 PosixPath('code'),
 PosixPath('installation_on_windows.md'),
 PosixPath('slope_with_deviations.Rmd'),
 PosixPath('.pytest_cache'),
 PosixPath('index_reshape.Rmd'),
 PosixPath('arrays_3d.Rmd'),
 PosixPath('requirements.txt'),
 PosixPath('boolean_indexing.Rmd'),
 PosixPath('on_regression.Rmd'),
 PosixPath('intro.md'),
 PosixPath('on_correlation.Rmd'),
 PosixPath('length_one_tuples.Rmd'),
 PosixPath('surviving_computers.md'),
 PosixPath('mentors.md'),
 PosixPath('volmeans.py'),
 PosixPath('pearson_functions.md'),
 PosixPath('__pycache__'),
 PosixPath('build_requirements.txt'),
 PosixPath('introducing_python.Rmd'),
 PosixPath('github_glm_homework.md'),
 PosixPath('multi_model_homework.md'),
 PosixPath('my_pair_image.hdr'),
 PosixPath('on_convolution.Rmd'),
 PosixPath('sys_path.Rmd'),
 PosixPath('array_reductions.Rmd'),
 PosixPath('the_problem.md')]

Notice two things here.

Selecting files with `glob`#

The argument to the glob method above is '*'. The '*' tells glob to get all files and directories, using what is called a Glob match. This is a powerful feature that allows you to be selective in asking for the files that glob returns. For example, if you wanted to see only the files ending with .txt you could do:

list(cwd.glob('*.txt'))

[PosixPath('a_test_file.txt'),
 PosixPath('ds114_sub009_t2r1_conv.txt'),
 PosixPath('requirements.txt'),
 PosixPath('build_requirements.txt')]

There are more detail in the page linked above.

`list` around the output of `glob`#

Notice that we used list around the output of glob, as in, for example:

list(cwd.glob('*.txt'))

[PosixPath('a_test_file.txt'),
 PosixPath('ds114_sub009_t2r1_conv.txt'),
 PosixPath('requirements.txt'),
 PosixPath('build_requirements.txt')]

This is because glob returns something called a generator which can return all the Path objects, but will not do that until we ask it to.

cwd.glob('*.txt')

<generator object Path.glob at 0x7fe21b1544a0>

The list call converts the result into a list, and in doing so, asks the generator to return all the Path objects:

list(cwd.glob('*.txt'))

[PosixPath('a_test_file.txt'),
 PosixPath('ds114_sub009_t2r1_conv.txt'),
 PosixPath('requirements.txt'),
 PosixPath('build_requirements.txt')]

Deleting files#

And finally, to be tidy, we use the unlink method to delete the temporary file we were using. unlink is strangely named, where the name refers to the way the computer disk system stores files, but does always have the effect of deleting the file.

new_path.unlink()

Using the pathlib module

Contents

Using the pathlib module#

Listing files in a directory#

Selecting files with glob#

list around the output of glob#

Deleting files#

Using the `pathlib` module#

Selecting files with `glob`#

`list` around the output of `glob`#