Process substitution

In computing, process substitution is a form of inter-process communication that allows the input or output of a command to appear as a file. The command is substituted in-line, where a file name would normally occur, by the command shell. This allows programs that normally only accept files to directly read from or write to another program.

History

Process substitution was available as a compile-time option for ksh88, the 1988 version of the KornShell from Bell Labs.[1] The rc shell provides the feature as "pipeline branching" in Version 10 Unix, released in 1990.[2] The Bash shell provided process substitution no later than version 1.14, released in 1994.[3]

Example

The following examples use KornShell syntax.

The Unix diff command normally accepts the names of two files to compare, or one file name and standard input. Process substitution allows one to compare the output of two programs directly: 

$ diff <(sort file1) <(sort file2)

The <(command) expression tells the command interpreter to run command and make its output appear as a file. The command can be any arbitrarily complex shell command.

Without process substitution, the alternatives are:

  1. Save the output of the command(s) to a temporary file, then read the temporary file(s). 
    $ sort file2 > /tmp/file2.sorted
$ sort file1 | diff - /tmp/file2.sorted
$ rm /tmp/file2.sorted
  2. Create a named pipe (also known as a FIFO), start one command writing to the named pipe in the background, then run the other command with the named pipe as input. 
    $ mkfifo /tmp/sort2.fifo
$ sort file2 > /tmp/sort2.fifo &
$ sort file1 | diff - /tmp/sort2.fifo
$ rm /tmp/sort2.fifo

Both alternatives are more cumbersome.

Process substitution can also be used to capture output that would normally go to a file, and redirect it to the input of a process. The Bash syntax for writing to a process is >(command). Here is an example using the tee, wc and gzip commands that counts the lines in a file with wc -l and compresses it with gzip in one pass: 

$ tee >(wc -l >&2) < bigfile | gzip > bigfile.gz

Advantages

The main advantages of process substitution over its alternatives are:

  • Simplicity: The commands can be given in-line; there is no need to save temporary files or create named pipes first.
  • Performance: Reading directly from another process is often faster than having to write a temporary file to disk, then read it back in. This also saves disk space.
  • Parallelism: The substituted process can be running concurrently with the command reading its output or writing its input, taking advantage of multiprocessing to reduce the total time for the computation.

Mechanism

Under the hood, process substitution has two implementations. On systems which support /dev/fd (most Unix-like systems) it works by calling the pipe() system call, which returns a file descriptor $fd for a new anonymous pipe, then creating the string /dev/fd/$fd, and substitutes that on the command line. On systems without /dev/fd support, it calls mkfifo with a new temporary filename to create a named pipe, and substitutes this filename on the command line. To illustrate the steps involved, consider the following simple command substitution on a system with /dev/fd support: 

$ diff file1 <(sort file2)

The steps the shell performs are:

  1. Create a new anonymous pipe. This pipe will be accessible with something like /dev/fd/63; you can see it with a command like echo <(true).
  2. Execute the substituted command in the background (sort file2 in this case), piping its output to the anonymous pipe.
  3. Execute the primary command, replacing the substituted command with the path of the anonymous pipe. In this case, the full command might expand to something like diff file1 /dev/fd/63.
  4. When execution is finished, close the anonymous pipe.

For named pipes, the execution differs solely in the creation and deletion of the pipe; they are created with mkfifo (which is given a new temporary file name) and removed with unlink. All other aspects remain the same.

Limitations

The "files" created are not seekable, which means the process reading or writing to the file cannot perform random access; it must read or write once from start to finish. Programs that explicitly check the type of a file before opening it may refuse to work with process substitution, because the "file" resulting from process substitution is not a regular file. Additionally, up to Bash 4.4 (released September 2016), it was not possible to obtain the exit code of a process substitution command from the shell that created the process substitution. [4]

See also

References

  1. ^ Rosenblatt, Bill; Robbins, Arnold (April 2002). "Appendix A.2". Learning the Korn Shell (2nd ed.). O'Reilly & Associates. ISBN 0-596-00195-9.
  2. ^ Duff, Tom (1990). Rc — A Shell for Plan 9 and UNIX Systems. CiteSeerX 10.1.1.41.3287.
  3. ^ Ramey, Chet (August 18, 1994). Bash 1.14 release notes. Free Software Foundation. Available in the  Gnu source archive of version 1.14.7 as of 12 February 2016.
  4. ^ "ProcessSubstitution". Greg's Wiki. 22 Sep 2016. Retrieved 2021-02-06.

Further reading