A Collection of Utilities

Math

pytools.levi_civita(tup: tuple[int, ...]) → int

Compute an entry of the Levi-Civita symbol for the indices tuple.

Assertive accessors

pytools.one(iterable: Iterable[T]) → T

Return the first entry of iterable. Assert that iterable has only that one entry.

pytools.is_single_valued(iterable: ~collections.abc.Iterable[~pytools.T], equality_pred: ~collections.abc.Callable[[~pytools.T, ~pytools.T], bool] = <built-in function eq>) → bool

pytools.all_roughly_equal(iterable, threshold)

pytools.single_valued(iterable: ~collections.abc.Iterable[~pytools.T], equality_pred: ~collections.abc.Callable[[~pytools.T, ~pytools.T], bool] = <built-in function eq>) → T

Return the first entry of iterable; Assert that other entries are the same with the first entry of iterable.

Memoization

pytools.memoize(*args: F, **kwargs: Any) → F

Stores previously computed function values in a cache.

Two keyword-only arguments are supported:

Parameters:

• use_kwargs – Allows the caller to use keyword arguments. Defaults to False. Setting this to True has a non-negligible performance impact.

• key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

pytools.memoize_on_first_arg(function: Callable[[Concatenate[T, P]], R], *, cache_dict_name: str | None = None) → Callable[[Concatenate[T, P]], R]

Like memoize_method(), but for functions that take the object in which do memoization information is stored as first argument.

Supports cache deletion via function_name.clear_cache(self).

pytools.memoize_method(method: Callable[[Concatenate[T, P]], R]) → Callable[[Concatenate[T, P]], R]

Supports cache deletion via method_name.clear_cache(self).

Changed in version 2021.2: Can memoize methods on classes that do not allow setting attributes (e.g. by overwriting __setattr__), e.g. frozen dataclasses.

pytools.memoize_in(container: Any, identifier: Hashable) → None

Adds a cache to the function it decorates. The cache is attached to container and must be uniquely specified by identifier (i.e. all functions using the same container and identifier will be using the same cache). The decorated function may only receive positional arguments.

Note

This function works well on nested functions, which do not have stable global identifiers.

Changed in version 2020.3: identifier no longer needs to be a str, but it needs to be hashable.

Changed in version 2021.2.1: Can now use instances of classes as container that do not allow setting attributes (e.g. by overwriting __setattr__), e.g. frozen dataclasses.

pytools.keyed_memoize_on_first_arg(key: Callable[P, Hashable], *, cache_dict_name: str | None = None) → None

Like memoize_method(), but for functions that take the object in which memoization information is stored as first argument.

Supports cache deletion via function_name.clear_cache(self).

Parameters:

• key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

• cache_dict_name – The name of the dict attribute in the instance used to hold the cache.

Added in version 2020.3.

pytools.keyed_memoize_method(key: Callable[P, Hashable], *, cache_dict_name: str | None = None) → None

Like memoize_method, but additionally uses a function key to compute the key under which the function result is stored.

Supports cache deletion via method_name.clear_cache(self).

Parameters:

key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

Added in version 2020.3.

Changed in version 2021.2: Can memoize methods on classes that do not allow setting attributes (e.g. by overwriting __setattr__), e.g. frozen dataclasses.

pytools.keyed_memoize_in(container: Any, identifier: Hashable, key: Callable[P, Hashable]) → None

Like memoize_in, but additionally uses a function key to compute the key under which the function result is memoized.

Parameters:

key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

Added in version 2021.2.1.

Argmin/max

pytools.argmin2(iterable, return_value=False)

pytools.argmax2(iterable, return_value=False)

pytools.argmin(iterable)

pytools.argmax(iterable)

Cartesian products

pytools.cartesian_product(*args)

pytools.distinct_pairs(list1, list2)

Permutations, Tuples, Integer sequences

pytools.wandering_element(length, wanderer=1, landscape=0)

pytools.generate_nonnegative_integer_tuples_below(n, length=None, least=0)

n may be a sequence, in which case length must be None.

pytools.generate_nonnegative_integer_tuples_summing_to_at_most(n, length)

Enumerate all non-negative integer tuples summing to at most n, exhausting the search space by varying the first entry fastest, and the last entry the slowest.

pytools.generate_all_integer_tuples_below(n, length, least_abs=0)

pytools.generate_permutations(original)

Generate all permutations of the list original.

Nicked from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252178

pytools.generate_unique_permutations(original)

Generate all unique permutations of the list original.

Formatting

class pytools.Table(alignments: tuple[str, ...] | None = None)

An ASCII table generator.

__init__(alignments: tuple[str, ...] | None = None) → None

Create a new Table.

Parameters:

alignments – A tuple of alignments of each column: "l", "c", or "r", for left, center, and right alignment, respectively). Columns which have no alignment specifier will use the last specified alignment. For example, with alignments=("l", "r"), the third and all following columns will use right alignment.

add_row(row: tuple[Any, ...]) → None

Add row to the table. Note that all rows must have the same number of columns.

property nrows: int

The number of rows currently in the table.

property ncolumns: int

The number of columns currently in the table.

__str__() → str

Returns a string representation of the table.

>>> tbl = Table(alignments=['l', 'r', 'l'])
>>> tbl.add_row([1, '|'])
>>> tbl.add_row([10, '20||'])
>>> print(tbl)
1  |    |
---+------
10 | 20||

github_markdown() → str

Returns a string representation of the table formatted as GitHub-Flavored Markdown.

>>> tbl = Table(alignments=['l', 'r', 'l'])
>>> tbl.add_row([1, '|'])
>>> tbl.add_row([10, '20||'])
>>> print(tbl.github_markdown())
1  |     \|
:--|-------:
10 | 20\|\|

csv(dialect: str = 'excel', csv_kwargs: dict[str, Any] | None = None) → str

Returns a string containing a CSV representation of the table.

Parameters:

• dialect – String passed to csv.writer().

• csv_kwargs – Dict of arguments passed to csv.writer().

>>> tbl = Table()
>>> tbl.add_row([1, ","])
>>> tbl.add_row([10, 20])
>>> print(tbl.csv())
1,","
10,20

latex(skip_lines: int = 0, hline_after: tuple[int, ...] | None = None) → str

Returns a string containing the rows of a LaTeX representation of the table.

Parameters:

• skip_lines – number of lines to skip at the start of the table.

• hline_after – list of row indices after which to add an hline (the indices must subtract skip_lines, if non-zero).

>>> tbl = Table()
>>> tbl.add_row([0, "skipped"])
>>> tbl.add_row([1, "apple"])
>>> tbl.add_row([2, "pear"])
>>> print(tbl.latex(skip_lines=1))
1 & apple \\
2 & pear \\

text_without_markup() → str

Returns a string representation of the table without markup.

>>> tbl = Table()
>>> tbl.add_row([0, "orange"])
>>> tbl.add_row([1111, "apple"])
>>> tbl.add_row([2, "pear"])
>>> print(tbl.text_without_markup())
0    orange
1111 apple
2    pear

pytools.merge_tables(*tables: Table, skip_columns: tuple[int, ...] | None = None) → Table

Parameters:

skip_columns – a tuple of column indices to skip in all the tables except the first one.

pytools.string_histogram(iterable, min_value=None, max_value=None, bin_count=20, width=70, bin_starts=None, use_unicode=True)

pytools.word_wrap(text, width, wrap_using='\n')

A word-wrap function that preserves existing line breaks and most spaces in the text. Expects that existing line breaks are posix newlines (\n).

Debugging

pytools.typedump(val: Any, max_seq: int = 5, special_handlers: Mapping[type, Callable] | None = None, fully_qualified_name: bool = True) → str

Return a string representation of the type of val, recursing into iterable objects.

Parameters:

• val – The object for which the type should be returned.

• max_seq – For iterable objects, the maximum number of elements to include in the return string. Lower this value if you get a RecursionError.

• special_handlers – An optional mapping of specific types to special handlers.

• fully_qualified_name – Return fully qualified names, that is, include module names and use __qualname__ instead of __name__.

Returns:

A string representation of the type of val.

pytools.invoke_editor(s, filename='edit.txt', descr='the file')

Progress bars

class pytools.ProgressBar(descr, total, initial=0, length=40)

draw()

progress(steps=1)

set_progress(done)

finished()

__enter__()

__exit__(exc_type, exc_val, exc_tb)

Name generation

pytools.generate_unique_names(prefix)

pytools.generate_numbered_unique_names(prefix: str, num: int | None = None) → Iterable[tuple[int, str]]

class pytools.UniqueNameGenerator(existing_names: Collection[str] | None = None, forced_prefix: str = '')

Class that creates a new str on each __call__() that is unique to the generator.

__init__(existing_names: Collection[str] | None = None, forced_prefix: str = '')

Create a new UniqueNameGenerator.

Parameters:

• existing_names – a set of existing names that will be skipped when generating new names.

• forced_prefix – all generated str have this prefix.

is_name_conflicting(name: str) → bool

Returns True if name conflicts with an existing str.

add_name(name: str, *, conflicting_ok: bool = False) → None

Parameters:

conflicting_ok – A flag to dictate the behavior when name is conflicting with the set of existing names. If True, a conflict is silently passed. If False, a ValueError is raised on encountering a conflict.

add_names(names: Iterable[str], *, conflicting_ok: bool = False) → None

Parameters:

conflicting_ok – Plainly passed to UniqueNameGenerator.add_name().

__call__(based_on: str = 'id') → str

Returns a new unique name.

Deprecation Warnings

pytools.deprecate_keyword(oldkey: str, newkey: str | None = None, *, deadline: str | None = None)

Decorator used to deprecate function keyword arguments.

Parameters:

• oldkey – deprecated argument name.

• newkey – new argument name that serves the same purpose, if any.

• deadline – expected time frame for the removal of the deprecated argument.

pytools.module_getattr_for_deprecations(module_name: str, depr_name_to_replacement_and_obj: Mapping[str, tuple[str, object, str | int]], name: str) → object

A helper to construct module-level object.__getattr__() functions so that deprecated names can still be found but raise a warning.

The typical usage pattern is as follows:

__getattr__ = partial(module_getattr_for_deprecations, __name__, {
    "OldName": ("NewName", NewName, 2026),
    })

Functions for dealing with (large) auxiliary files

pytools.download_from_web_if_not_present(url, local_name=None)

Added in version 2017.5.

Helpers for numpy

pytools.reshaped_view(a, newshape)

Create a new view object with shape newshape without copying the data of a. This function is different from numpy.reshape by raising an exception when data copy is necessary.

Parameters:

• a – a numpy.ndarray object.

• newshape – an int object or a tuple of int objects.

Added in version 2018.4.

Timing data

class pytools.ProcessTimer

Measures elapsed wall time and process time.

__enter__()

__exit__(exc_type, exc_val, exc_tb)

done()

Timing data attributes:

wall_elapsed

process_elapsed

Added in version 2018.5.

Log utilities

class pytools.ProcessLogger(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)

Logs the completion time of a (presumably) lengthy process to logging. Only uses a high log level if the process took perceptible time.

__init__(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)

done(extra_msg=None, *extra_fmt_args)

__enter__()

__exit__(exc_type, exc_val, exc_tb)

class pytools.DebugProcessLogger(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)

class pytools.log_process(logger, description=None, long_threshold_seconds=None)

A decorator that uses ProcessLogger to log data about calls to the wrapped function.

__init__(logger, description=None, long_threshold_seconds=None)

__call__(wrapped)

Call self as a function.

Sorting in natural order

pytools.natorder(item)

Return a key for natural order string comparison.

See natsorted().

Added in version 2020.1.

pytools.natsorted(iterable, key=None, reverse=False)

Sort using natural order [1], as opposed to lexicographic order.

Example:

>>> sorted(["_10", "_1", "_9"]) == ["_1", "_10", "_9"]
True
>>> natsorted(["_10", "_1", "_9"]) == ["_1", "_9", "_10"]
True

Parameters:

• iterable – an iterable to be sorted. It must only have strings, unless key is specified.

• key – if provided, a key function that returns strings for ordering using natural order.

• reverse – if True, sorts in descending order.

Returns:

a sorted list

[1]

https://en.wikipedia.org/wiki/Natural_sort_order

Added in version 2020.1.

Backports of newer Python functionality

pytools.resolve_name(name)

A backport of pkgutil.resolve_name() (added in Python 3.9).

Added in version 2021.1.2.

Hashing

pytools.unordered_hash(hash_instance: Any, iterable: Iterable[Any], hash_constructor: Callable[[], Any] | None = None) → Any

Using a hash algorithm given by the parameter-less constructor hash_constructor, return a hash object whose internal state depends on the entries of iterable, but not their order. If hash is the instance returned by evaluating hash_constructor(), then the each entry i of the iterable must permit hash.update(i) to succeed. An example of hash_constructor is hashlib.sha256 from hashlib. hash.digest_size must also be defined. If hash_constructor is not provided, hash_instance.name is used to deduce it.

Returns:

the updated hash_instance.

Warning

The construction used in this function is likely not cryptographically secure. Do not use this function in a security-relevant context.

Added in version 2021.2.

Sampling

pytools.sphere_sample_equidistant(npoints_approx: int, r: float = 1.0)

Generate points regularly distributed on a sphere based on https://www.cmu.edu/biolphys/deserno/pdf/sphere_equi.pdf.

Returns:

an ndarray of shape (3, npoints), where npoints does not generally equal npoints_approx.

pytools.sphere_sample_fibonacci(npoints: int, r: float = 1.0, *, optimize: str | None = None)

Generate points on a sphere based on an offset Fibonacci lattice from [2].

[2]

http://extremelearning.com.au/how-to-evenly-distribute-points-on-a-sphere-more-effectively-than-the-canonical-fibonacci-lattice/

Parameters:

optimize – takes the values: None to use the standard Fibonacci lattice, "minimum" to minimize the nearest neighbor distances in the lattice and "average" to minimize the average distances in the lattice.

Returns:

an ndarray of shape (3, npoints).

String utilities

pytools.strtobool(val: str | None, default: bool | None = None) → bool

Convert a string representation of truth to True or False. True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Uppercase versions are also accepted. If default is None, raises ValueError if val is anything else. If val is None and default is not None, returns default. Based on distutils.util.strtobool().

Parameters:

• val – Value to convert.

• default – Value to return if val is None.

Returns:

Truth value of val.

pytools.to_identifier(s: str) → str

Convert a string to a valid Python identifier, by removing non-alphanumeric, non-underscore characters, and prepending an underscore if the string starts with a numeric character.

Parameters:

s – The string to convert to an identifier.

Returns:

The converted string.

Set-like functions for iterables

These functions provide set-like operations on iterables. In contrast to Python’s built-in set type, they maintain the internal order of elements.

pytools.unique(seq: Iterable[T]) → Collection[T]

Return unique elements in seq, removing all duplicates. The internal order of the elements is preserved. See also itertools.groupby() (which removes consecutive duplicates).

pytools.unique_difference(*args: Iterable[T]) → Collection[T]

Return unique elements that are in the first iterable in *args but not in any of the others. The internal order of the elements is preserved.

pytools.unique_intersection(*args: Iterable[T]) → Collection[T]

Return unique elements that are common to all iterables in *args. The internal order of the elements is preserved.

pytools.unique_union(*args: Iterable[T]) → Collection[T]

Return unique elements that are in any iterable in *args. The internal order of the elements is preserved.

Functionality for dataclasses

pytools.opt_frozen_dataclass(*, init: bool = True, repr: bool = True, eq: bool = True, order: bool = False, unsafe_hash: bool | None = None, match_args: bool = True, kw_only: bool = False, slots: bool = False, weakref_slot: bool = False) → Callable[[type[T]], type[T]]

Like dataclasses.dataclass(), but marks the dataclass frozen only if __debug__ is active. Frozen dataclasses have a ~20% cost penalty (on creation, from having to call object.__setattr__()) that this decorator avoids when the interpreter runs with “optimization” enabled.

The resulting dataclass supports hashing, even when it is not actually frozen, if unsafe_hash is left at the default or set to True.

Note

Python prevents non-frozen dataclasses from inheriting from frozen ones, and vice versa. To ensure frozen-ness is applied predictably in all scenarios (mainly __debug__ on and off), it is strongly recommended that all dataclasses inheriting from ones with this decorator also use this decorator. There are no run-time checks to make sure of this.

Added in version 2024.1.18.

Type Variables Used

class pytools.T

class pytools.R

Generic unbound invariant typing.TypeVar.

class pytools.F

Generic invariant typing.TypeVar bound to a typing.Callable.

class pytools.P

Generic unbound invariant typing.ParamSpec.

An in-memory relational database table

class pytools.datatable.DataTable(column_names: Sequence[str], column_data: list[Any] | None = None)

An in-memory relational database table.

__init__(column_names: Sequence[str], column_data: list[Any] | None = None) → None

Construct a new table, with the given C{column_names}.

Parameters:

• column_names – An indexable of column name strings.

• column_data – None or a list of tuples of the same length as column_names indicating an initial set of data.

copy() → DataTable

Make a copy of the instance, but leave individual rows untouched.

If the rows are modified later, they will also be modified in the copy.

deep_copy() → DataTable

Make a copy of the instance down to the row level.

The copy’s rows may be modified independently from the original.

join(column: str, other_column: str, other_table: DataTable, outer: bool = False) → DataTable

Return a table joining this and the C{other_table} on C{column}.

The new table has the following columns: - C{column}, titled the same as in this table. - the columns of this table, minus C{column}. - the columns of C{other_table}, minus C{other_column}.

Assumes both tables are sorted ascendingly by the column by which they are joined.

Dot helper functions

pytools.graphviz.dot_escape(s: str) → str

Escape the string s for compatibility with the dot language, particularly backslashes and HTML tags.

Parameters:

s – The input string to escape.

Returns:

s with special characters escaped.

pytools.graphviz.show_dot(dot_code: str, output_to: str | None = None) → str | None

Visualize the graph represented by dot_code.

Parameters:

• dot_code – An instance of str in the dot language to visualize.

• output_to –

  An instance of str that can be one of:

  • "xwindow" to visualize the graph as an X window.

  • "browser" to visualize the graph as an SVG file in the system’s default web-browser.

  • "svg" to store the dot code as an SVG file on the file system. Returns the path to the generated SVG file.

  Defaults to "xwindow" if X11 support is present, otherwise defaults to "browser".

Returns:

Depends on output_to. If "svg", returns the path to the generated SVG file, otherwise returns None.