API Reference

class redun_psij.CommonJobSpec(*, tool, args, stdout_path, stderr_path, cwd=None, custom_attributes=<factory>)

Bases: object

The spec in common for redun_psij.Job1Spec and redun_psij.JobNSpec.

Parameters:

  • tool (str)

  • args (list[str])

  • stdout_path (str)

  • stderr_path (str)

  • cwd (str | None)

  • custom_attributes (dict[str, str])

tool: str

The tool name, to be found in the job configuration file.

args: list[str]

Command and its arguments.

stdout_path: str

Path to file which captures stdout of the job.

stderr_path: str

Path to file which captures stderr of the job.

cwd: str | None = None

Optional directory to change to before running the job.

If this is used, ensure that all paths in args are passed via os.path.abspath(), since relative paths in args would otherwise be broken.

It is not necessary to make stdout_path nor stderr_path absolute, nor any in expected output, since the executor handles these.

custom_attributes: dict[str, str]

Custom attributes to pass to PSI/J, e.g. from redun_psij.JobContext.

exception redun_psij.ConfigError(message, path=None, tool=None, cause=None)

Bases: Exception

Executor configuration error.

Parameters:

  • message (str)

  • path (str | None)

  • tool (str | None)

  • cause (Exception | None)

class redun_psij.ExpectedPaths(*, required: dict[str, str] = <factory>, optional: dict[str, str] = <factory>)

Bases: object

Parameters:

  • required (dict[str, str])

  • optional (dict[str, str])

required: dict[str, str]

Paths of any expected output files which are required to be created by the job. Any missing file constitutes an error. Keys are arbitrary strings and may be used to match up output files with what was expected.

optional: dict[str, str]

As per required where failure to create a file is not an error.

class redun_psij.FilteredGlob(glob, reject_re=None)

Bases: object

A file glob optionally without any paths matched by reject_re.

Parameters:

  • glob (str)

  • reject_re (str | None)

glob: str

File glob to match job output files.

reject_re: str | None = None

Regular expression in Python regex syntax to exclude unwanted glob matches.

class redun_psij.Job1Spec(*, tool, args, stdout_path, stderr_path, cwd=None, custom_attributes=<factory>, expected_path)

Bases: CommonJobSpec

For jobs which produce a single file whose path is known in advance.

Parameters:

  • tool (str)

  • args (list[str])

  • stdout_path (str)

  • stderr_path (str)

  • cwd (str | None)

  • custom_attributes (dict[str, str])

  • expected_path (str)

expected_path: str

Path of the single expected output file, it being an error if this is not created by the job.

class redun_psij.JobContext(*context)

Bases: object

Provides a hierarchical job context suitable for inclusion as PSI/J custom attributes when running e.g. Slurm jobs.

Parameters:

context (str)

with_sub(sub)

Return a new sub-context by extending the previous with a new element.

Parameters:

sub (str) – name of sub-context

Return type:

Self

property custom_attributes: dict[str, str]

Format the context as a comment custom attribute, suitable for a JobSpec, to annotate a PSI/J (Slurm) job with the context in the comment field.

exception redun_psij.JobError(message)

Bases: Exception

Executor job error.

Parameters:

message (str)

class redun_psij.JobFailure(exit_code, stderr)

Bases: object

Returned by redun_psij.run_job_1_returning_failure() and redun_psij.run_job_n_returning_failure() on error, rather than throwing an exception.

Contains stderr as an actual file so that deleting that file will trigger redun to rerun the failed task.

Parameters:

  • exit_code (int)

  • stderr (File)

class redun_psij.JobNSpec(*, tool, args, stdout_path, stderr_path, cwd=None, custom_attributes=<factory>, expected_paths=<factory>, expected_globs=<factory>)

Bases: CommonJobSpec

For jobs which produce multiple files with paths matching the glob, exluding any matched by the regex.

As many conbinations of globs and paths may be expected, in a dict whose keys are arbitrary strings. Result files will be returned in a dict with the same keys.

Parameters:

  • tool (str)

  • args (list[str])

  • stdout_path (str)

  • stderr_path (str)

  • cwd (str | None)

  • custom_attributes (dict[str, str])

  • expected_paths (ExpectedPaths)

  • expected_globs (dict[str, FilteredGlob])

expected_paths: ExpectedPaths

Expected output files whose paths are known in advance.

expected_globs: dict[str, FilteredGlob]

Expected output files whose paths are globbed after the event.

class redun_psij.ResultFiles(expected_files, globbed_files)

Bases: object

The output files produced by redun_psij.run_job_n().

Parameters:

  • expected_files (dict[str, File])

  • globbed_files (dict[str, list[File]])

expected_files: dict[str, File]

Files as defined by redun_psij.ExpectedPaths.

globbed_files: dict[str, list[File]]

Files as defined by redun_psij.FilteredGlob.

redun_psij.get_tool_config(tool)

Return tool config for the named tool from the Jsonnet configuration.

Parameters:

tool (str) – name of tool

Return type:

dict[str, Any]

redun_psij.run_job_1(spec)

Run a job which is expected to produce the single file expected_path Errors are thrown as redun_psij.JobError.

Parameters:

spec (Job1Spec) – defines the job parameters

Return type:

File

redun_psij.run_job_1_returning_failure(spec)

Run a job which is expected to produce the single file expected_path Errors are returned as redun_psij.JobFailure.

Parameters:

spec (Job1Spec) – defines the job parameters

Return type:

File | JobFailure

redun_psij.run_job_n(spec)

Run a job which is expected to produce files matching redun_psij.ResultFiles

Errors are thrown as redun_psij.JobError.

Parameters:

spec (JobNSpec)

Return type:

ResultFiles

redun_psij.run_job_n_returning_failure(spec)

Run a job which is expected to produce files matching redun_psij.ResultFiles

Errors are returned as redun_psij.JobFailure.

Parameters:

spec (JobNSpec) – defines the job parameters

Return type:

ResultFiles | JobFailure