# libpipeline

## NAME



           **libpipeline** — pipeline manipulation library
    


## SYNOPSIS



           <_pipeline.h_ >
    


## DESCRIPTION



           **libpipeline** is a C library for setting up and running pipelines of
           processes, without needing to involve shell command-line parsing
           which is often error-prone and insecure.  This relieves
           programmers of the need to laboriously construct pipelines using
           lower-level primitives such as **fork** and **execve**.
    
           The general way to use **libpipeline** involves constructing a
           _pipeline_ structure and adding one or more _pipecmd_ structures to
           it.  A _pipecmd_ represents a subprocess (or “command”), while a
           _pipeline_ represents a sequence of subprocesses each of whose
           outputs is connected to the next one's input, as in the example **ls**
           | **grep pattern** | **less**.  The calling program may adjust certain
           properties of each command independently, such as its environment
           and _nice_(3) priority, as well as properties of the entire pipeline
           such as its input and output and the way signals are handled while
           executing it.  The calling program may then start the pipeline,
           read output from it, wait for it to complete, and gather its exit
           status.
    
           Strings passed as _const char *_ function arguments will be copied
           by the library.
    
       **Functions to build individual commands**
           _pipecmd *_ **pipecmd_new**(_const char *name_)
    
                 Construct a new command representing execution of a program
                 called _name_.
    
           _pipecmd *_ **pipecmd_new_argv**(_const char *name_ , _va_list argv_)
           _pipecmd *_ **pipecmd_new_args**(_const char *name_ , _..._)
    
                 Convenience constructors wrapping **pipecmd_new**() and
                 **pipecmd_arg**().  Construct a new command representing
                 execution of a program called _name_ with arguments.
                 Terminate arguments with **nullptr**(before C23: **(void *) 0**).
    
           _pipecmd *_ **pipecmd_new_argstr**(_const char *argstr_)
    
                 Split _argstr_ on whitespace to construct a command and
                 arguments, honouring shell-style single-quoting, double-
                 quoting, and backslashes, but not other shell evilness like
                 wildcards, semicolons, or backquotes.  This is included only
                 to support situations where command arguments are encoded
                 into configuration files and the like.  While it is safer
                 than _system_(3), it still involves significant string parsing
                 which is inherently riskier than avoiding it altogether.
                 Please try to avoid using it in new code.
    
           _typedef void pipecmd_function_type (void *)_ ;
           _typedef void pipecmd_function_free_type (void *)_ ;
           _pipecmd *_ **pipecmd_new_function**(_const char *name_ ,
                 _pipecmd_function_type *func_ ,
                 _pipecmd_function_free_type *free_func_ , _void *data_)
    
                 Construct a new command that calls a given function rather
                 than executing a process.
    
                 The data argument is passed as the function's only argument,
                 and will be freed before returning using free_func (if
                 non-_NULL_).
    
                 **pipecmd_*** functions that deal with arguments cannot be used
                 with the command returned by this function.
    
           _pipecmd *_ **pipecmd_new_sequencev**(_const char *name_ , _va_list cmdv_)
           _pipecmd *_ **pipecmd_new_sequence**(_const char *name_ , _..._)
    
                 Construct a new command that itself runs a sequence of
                 commands, supplied as _command *_ arguments following _name_ and
                 terminated by **nullptr**(before C23: **(void *) 0**).  The
                 commands will be executed in forked children; if any exits
                 non-zero then it will terminate the sequence, as with "&&"
                 in shell.
    
                 **pipecmd_*** functions that deal with arguments cannot be used
                 with the command returned by this function.
    
           _pipecmd *_ **pipecmd_new_passthrough**(_void_)
    
                 Return a new command that just passes data from its input to
                 its output.
    
           _pipecmd *_ **pipecmd_dup**(_pipecmd *cmd_)
    
                 Return a duplicate of a command.
    
           _void_ **pipecmd_arg**(_pipecmd *cmd_ , _const char *arg_)
    
                 Add an argument to a command.
    
           _void_ **pipecmd_argf**(_pipecmd *cmd_ , _const char *format_ , _..._)
    
                 Convenience function to add an argument with printf
                 substitutions.
    
           _void_ **pipecmd_argv**(_pipecmd *cmd_ , _va_list argv_)
           _void_ **pipecmd_args**(_pipecmd *cmd_ , _..._)
    
                 Convenience functions wrapping **pipecmd_arg**() to add multiple
                 arguments at once.  Terminate arguments with **nullptr**(before
                 C23: **(void *) 0**).
    
           _void_ **pipecmd_argstr**(_pipecmd *cmd_ , _const char *argstr_)
    
                 Split _argstr_ on whitespace to add a list of arguments,
                 honouring shell-style single-quoting, double-quoting, and
                 backslashes, but not other shell evilness like wildcards,
                 semicolons, or backquotes.  This is included only to support
                 situations where command arguments are encoded into
                 configuration files and the like.  While it is safer than
                 _system_(3), it still involves significant string parsing
                 which is inherently riskier than avoiding it altogether.
                 Please try to avoid using it in new code.
    
           _void_ **pipecmd_get_nargs**(_pipecmd *cmd_)
    
                 Return the number of arguments to this command.  Note that
                 this includes the command name as the first argument, so the
                 command ‘echo foo bar’ is counted as having three arguments.
    
                 Added in **libpipeline** 1.1.0.
    
           _void_ **pipecmd_nice**(_pipecmd *cmd_ , _int value_)
    
                 Set the _nice_(3) value for this command.  Defaults to 0.
                 Errors while attempting to set the nice value are ignored,
                 aside from emitting a debug message.
    
           _void_ **pipecmd_discard_err**(_pipecmd *cmd_ , _int discard_err_)
    
                 If _discard_err_ is non-zero, redirect this command's standard
                 error to _/dev/null_.  Otherwise, and by default, pass it
                 through.  This is usually a bad idea.
    
           _void_ **pipecmd_chdir**(_pipecmd *cmd_ , _const char *directory_)
    
                 Change the working directory to _directory_ while running this
                 command.
    
                 Added in **libpipeline** 1.3.0.
    
           _void_ **pipecmd_fchdir**(_pipecmd *cmd_ , _int directory_fd_)
    
                 Change the working directory to the directory given by the
                 open file descriptor _directory_fd_ while running this
                 command.
    
                 Added in **libpipeline** 1.4.0.
    
           _void_ **pipecmd_setenv**(_pipecmd *cmd_ , _const char *name_ , _const char_
                 _*value_)
    
                 Set environment variable _name_ to _value_ while running this
                 command.
    
           _void_ **pipecmd_unsetenv**(_pipecmd *cmd_ , _const char *name_)
    
                 Unset environment variable _name_ while running this command.
    
           _void_ **pipecmd_clearenv**(_pipecmd *cmd_)
    
                 Clear the environment while running this command.  (Note
                 that environment operations work in sequence;
                 pipecmd_clearenv followed by pipecmd_setenv causes the
                 command to have just a single environment variable set.)
                 Beware that this may cause unexpected failures, for example
                 if some of the contents of the environment are necessary to
                 execute programs at all (say, **PATH**).
    
                 Added in **libpipeline** 1.1.0.
    
           _void_ **pipecmd_pre_exec**(_pipecmd *cmd_ , _pipecmd_function_type *func_ ,
                 _pipecmd_function_free_type *free_func_ , _void *data_)
    
                 Install a pre-exec handler.  This will be run immediately
                 before executing the command's payload (process or
                 function).  Pass NULL to clear any existing pre-exec
                 handler.  The data argument is passed as the function's only
                 argument, and will be freed before returning using free_func
                 (if non-NULL).
    
                 This is similar to pipeline_install_post_fork, except that
                 is specific to a single command rather than installing a
                 global handler, and it runs slightly later (immediately
                 before exec rather than immediately after fork).
    
                 Added in **libpipeline** 1.5.0.
    
           _void_ **pipecmd_sequence_command**(_pipecmd *cmd_ , _pipecmd *child_)
    
                 Add a command to a sequence created using
                 **pipecmd_new_sequence**().
    
           _void_ **pipecmd_dump**(_pipecmd *cmd_ , _FILE *stream_)
    
                 Dump a string representation of a command to stream.
    
           _char *_ **pipecmd_tostring**(_pipecmd *cmd_)
    
                 Return a string representation of a command.  The caller
                 should free the result.
    
           _void_ **pipecmd_exec**(_pipecmd *cmd_)
    
                 Execute a single command, replacing the current process.
                 Never returns, instead exiting non-zero on failure.
    
                 Added in **libpipeline** 1.1.0.
    
           _void_ **pipecmd_free**(_pipecmd *cmd_)
    
                 Destroy a command.  Safely does nothing if _cmd_ is **NULL**.
    
       **Functions to build pipelines**
           _pipeline *_ **pipeline_new**(_void_)
    
                 Construct a new pipeline.
    
           _pipeline *_ **pipeline_new_commandv**(_pipecmd *cmd1_ , _va_list cmdv_)
           _pipeline *_ **pipeline_new_commands**(_pipecmd *cmd1_ , _..._)
    
                 Convenience constructors wrapping **pipeline_new**() and
                 **pipeline_command**().  Construct a new pipeline consisting of
                 the given list of commands.  Terminate commands with **nullptr**
                 (before C23: **(void *) 0**).
    
           _pipeline *_ **pipeline_new_command_argv**(_const char *name_ , _va_list_
                 _argv_)
           _pipeline *_ **pipeline_new_command_args**(_const char *name_ , _..._)
    
                 Construct a new pipeline and add a single command to it.
    
           _pipeline *_ **pipeline_join**(_pipeline *p1_ , _pipeline *p2_)
    
                 Joins two pipelines, neither of which are allowed to be
                 started.  Discards _want_out_ , _want_outfile_ , and _outfd_ from
                 _p1_ , and _want_in_ , _want_infile_ , and _infd_ from _p2_.
    
           _void_ **pipeline_connect**(_pipeline *source_ , _pipeline *sink_ , _..._)
    
                 Connect the input of one or more sink pipelines to the
                 output of a source pipeline.  The source pipeline may be
                 started, but in that case **pipeline_want_out**() must have been
                 called with a negative _fd_ ; otherwise, calls
                 **pipeline_want_out**(_source_ , _-1_).  In any event, calls
                 **pipeline_want_in**(_sink_ , _-1_) on all sinks, none of which are
                 allowed to be started.  Terminate arguments with **nullptr**
                 (before C23: **(void *) 0**).
    
                 This is an application-level connection; data may be
                 intercepted between the pipelines by the program before
                 calling **pipeline_pump**(), which sets data flowing from the
                 source to the sinks.  It is primarily useful when more than
                 one sink pipeline is involved, in which case the pipelines
                 cannot simply be concatenated into one.
    
                 The result is similar to _tee_(1), except that output can be
                 sent to more than two places and can easily be sent to
                 multiple processes.
    
           _void_ **pipeline_command**(_pipeline *p_ , _pipecmd *cmd_)
    
                 Add a command to a pipeline.
    
           _void_ **pipeline_command_argv**(_pipeline *p_ , _const char *name_ , _va_list_
                 _argv_)
           _void_ **pipeline_command_args**(_pipeline *p_ , _const char *name_ , _..._)
    
                 Construct a new command and add it to a pipeline in one go.
    
           _void_ **pipeline_command_argstr**(_pipeline *p_ , _const char *argstr_)
    
                 Construct a new command from a shell-quoted string and add
                 it to a pipeline in one go.  See the comment against
                 **pipecmd_new_argstr**() above if you're tempted to use this
                 function.
    
           _void_ **pipeline_commandv**(_pipeline *p_ , _va_list cmdv_)
           _void_ **pipeline_commands**(_pipeline *p_ , _..._)
    
                 Convenience functions wrapping **pipeline_command**() to add
                 multiple commands at once.  Terminate arguments with **nullptr**
                 (before C23: **(void *) 0**).
    
           _void_ **pipeline_want_in**(_pipeline *p_ , _int fd_)
           _void_ **pipeline_want_out**(_pipeline *p_ , _int fd_)
    
                 Set file descriptors to use as the input and output of the
                 whole pipeline.  If non-negative, _fd_ is used directly as a
                 file descriptor.  If negative, **pipeline_start**() will create
                 pipes and store the input writing half and the output
                 reading half in the pipeline's _infd_ or _outfd_ field as
                 appropriate.  The default is to leave input and output as
                 stdin and stdout unless **pipeline_want_infile**() or
                 **pipeline_want_outfile**() respectively has been called.
    
                 Calling these functions supersedes any previous call to
                 **pipeline_want_infile**() or **pipeline_want_outfile**()
                 respectively.
    
           _void_ **pipeline_want_infile**(_pipeline *p_ , _const char *file_)
           _void_ **pipeline_want_outfile**(_pipeline *p_ , _const char *file_)
    
                 Set file names to open and use as the input and output of
                 the whole pipeline.  This may be more convenient than
                 supplying file descriptors, and guarantees that the files
                 are opened with the same privileges under which the pipeline
                 is run.
    
                 Calling these functions (even with **NULL** , which returns to
                 the default of leaving input and output as stdin and stdout)
                 supersedes any previous call to **pipeline_want_in**() or
                 **pipeline_want_outfile**() respectively.
    
                 The given files will be opened when the pipeline is started.
                 If an output file does not already exist, it is created
                 (with mode 0666 modified in the usual way by umask); if it
                 does exist, then it is truncated.
    
           _void_ **pipeline_ignore_signals**(_pipeline *p_ , _int ignore_signals_)
    
                 If _ignore_signals_ is non-zero, ignore **SIGINT** and **SIGQUIT** in
                 the calling process while the pipeline is running, like
                 _system_(3).  Otherwise, and by default, leave their
                 dispositions unchanged.
    
           _int_ **pipeline_get_ncommands**(_pipeline *p_)
    
                 Return the number of commands in this pipeline.
    
           _pipecmd *_ **pipeline_get_command**(_pipeline *p_ , _int n_)
    
                 Return command number _n_ from this pipeline, counting from
                 zero, or **NULL** if _n_ is out of range.
    
           _pipecmd *_ **pipeline_set_command**(_pipeline *p_ , _int n_ , _pipecmd *cmd_)
    
                 Set command number _n_ in this pipeline, counting from zero,
                 to _cmd_ , and return the previous command in that position.
                 Do nothing and return **NULL** if _n_ is out of range.
    
           _pid_t_ **pipeline_get_pid**(_pipeline *p_ , _int n_)
    
                 Return the process ID of command number _n_ from this
                 pipeline, counting from zero.  The pipeline must be started.
                 Return **-1** if _n_ is out of range or if the command has already
                 exited and been reaped.
    
                 Added in **libpipeline** 1.2.0.
    
           _FILE *_ **pipeline_get_infile**(_pipeline *p_)
           _FILE *_ **pipeline_get_outfile**(_pipeline *p_)
    
                 Get streams corresponding to _infd_ and _outfd_ respectively.
                 The pipeline must be started.
    
           _void_ **pipeline_dump**(_pipeline *p_ , _FILE *stream_)
    
                 Dump a string representation of _p_ to stream.
    
           _char *_ **pipeline_tostring**(_pipeline *p_)
    
                 Return a string representation of _p_.  The caller should free
                 the result.
    
           _void_ **pipeline_free**(_pipeline *p_)
    
                 Destroy a pipeline and all its commands.  Safely does
                 nothing if _p_ is **NULL**.  May wait for the pipeline to complete
                 if it has not already done so.
    
       **Functions to run pipelines and handle signals**
           _typedef void pipeline_post_fork_fn (void)_ ;
           _void_ **pipeline_install_post_fork**(_pipeline_post_fork_fn *fn_)
    
                 Install a post-fork handler.  This will be run in any child
                 process immediately after it is forked.  For instance, this
                 may be used for cleaning up application-specific signal
                 handlers.  Pass **NULL** to clear any existing post-fork
                 handler.
    
                 See pipecmd_pre_exec for a similar facility limited to a
                 single command rather than global to the calling process.
    
           _void_ **pipeline_start**(_pipeline *p_)
    
                 Start the processes in a pipeline.  Installs this library's
                 **SIGCHLD** handler if not already installed.  Calls **error**
                 **(FATAL)** on error.
    
                 The standard file descriptors (0, 1, and 2) must be open
                 before calling this function.
    
           _int_ **pipeline_wait_all**(_pipeline *p_ , _int **statuses_ , _int_
                 _*n_statuses_)
    
                 Wait for a pipeline to complete.  Set *_statuses_ to a newly-
                 allocated array of wait statuses, as returned by _waitpid_(2),
                 and *_n_statuses_ to the length of that array.  The return
                 value is similar to the exit status that a shell would
                 return, with some modifications.  If the last command exits
                 with a signal (other than **SIGPIPE** , which is considered
                 equivalent to exiting zero), then the return value is 128
                 plus the signal number; if the last command exits normally
                 but non-zero, then the return value is its exit status; if
                 any other command exits non-zero, then the return value is
                 127; otherwise, the return value is 0.  This means that the
                 return value is only 0 if all commands in the pipeline exit
                 successfully.
    
           _int_ **pipeline_wait**(_pipeline *p_)
    
                 Wait for a pipeline to complete and return its combined exit
                 status, calculated as for **pipeline_wait_all**().
    
           _int_ **pipeline_run**(_pipeline *p_)
    
                 Start a pipeline, wait for it to complete, and free it, all
                 in one go.
    
           _void_ **pipeline_pump**(_pipeline *p_ , _..._)
    
                 Pump data among one or more pipelines connected using
                 **pipeline_connect**() until all source pipelines have reached
                 end-of-file and all data has been written to all sinks (or
                 failed).  All relevant pipelines must be supplied: that is,
                 no pipeline that has been connected to a source pipeline may
                 be supplied unless that source pipeline is also supplied.
                 Automatically starts all pipelines if they are not already
                 started, but does not wait for them.  Terminate arguments
                 with **nullptr**(before C23: **(void *) 0**).
    
       **Functions to read output from pipelines**
           In general, output is returned as a pointer into a buffer owned by
           the pipeline, which is automatically freed when **pipeline_free**() is
           called.  This saves the caller from having to explicitly free
           individual blocks of output data.
    
           _const char *_ **pipeline_read**(_pipeline *p_ , _size_t *len_)
    
                 Read _len_ bytes of data from the pipeline, returning the data
                 block.  _len_ is updated with the number of bytes read.
    
           _const char *_ **pipeline_peek**(_pipeline *p_ , _size_t *len_)
    
                 Look ahead in the pipeline's output for _len_ bytes of data,
                 returning the data block.  _len_ is updated with the number of
                 bytes read.  The starting position of the next read or peek
                 is not affected by this call.
    
           _size_t_ **pipeline_peek_size**(_pipeline *p_)
    
                 Return the number of bytes of data that can be read using
                 **pipeline_read**() or **pipeline_peek**() solely from the peek
                 cache, without having to read from the pipeline itself (and
                 thus potentially block).
    
           _void_ **pipeline_peek_skip**(_pipeline *p_ , _size_t len_)
    
                 Skip over and discard _len_ bytes of data from the peek cache.
                 Asserts that enough data is available to skip, so you may
                 want to check using **pipeline_peek_size**() first.
    
           _const char *_ **pipeline_readline**(_pipeline *p_)
    
                 Read a line of data from the pipeline, returning it.
    
           _const char *_ **pipeline_peekline**(_pipeline *p_)
    
                 Look ahead in the pipeline's output for a line of data,
                 returning it.  The starting position of the next read or
                 peek is not affected by this call.
    
       **Signal handling**
           **libpipeline** installs a signal handler for **SIGCHLD** , and collects
           the exit status of child processes in **pipeline_wait**().
           Applications using this library must either refrain from changing
           the disposition of **SIGCHLD**(in other words, must rely on
           **libpipeline** for all child process handling) or else must make sure
           to restore 's **SIGCHLD** handler before calling any of its functions.
    
           If the _ignore_signals_ flag is set in a pipeline (which is the
           default), then the **SIGINT** and **SIGQUIT** signals will be ignored in
           the parent process while child processes are running.  This
           mirrors the behaviour of _system_(3).
    
           **libpipeline** leaves child processes with the default disposition of
           **SIGPIPE** , namely to terminate the process.  It ignores **SIGPIPE** in
           the parent process while running **pipeline_pump**().
    
       **Reaping of child processes**
           **libpipeline** installs a **SIGCHLD** handler that will attempt to reap
           child processes which have exited.  This calls _waitpid_(2) with **-1** ,
           so it will reap any child process, not merely those created by way
           of this library.  At present, this means that if the calling
           program forks other child processes which may exit while a
           pipeline is running, the program is not guaranteed to be able to
           collect exit statuses of those processes.
    
           You should not rely on this behaviour, and in future it may be
           modified either to reap only child processes created by this
           library or to provide a way to return foreign statuses to the
           application.  Please contact the author if you have an example
           application and would like to help design such an interface.
    


## ENVIRONMENT



           If the PIPELINE_DEBUG environment variable is set to “1”, then
           **libpipeline** will emit debugging messages on standard error.
    
           If the PIPELINE_QUIET environment variable is set to any value,
           then **libpipeline** will refrain from printing an error message when
           a subprocess is terminated by a signal.  Added in **libpipeline**
           1.4.0.
    


## EXAMPLES



           In the following examples, function names starting with **pipecmd_**
           or **pipeline_** are real **libpipeline** functions, while any other
           function names are pseudocode.  These examples use the C23-style
           _nullptr_ : https://en.cppreference.com/w/c/language/nullptr keyword
           to terminate variadic argument lists; on earlier versions of C,
           use **(void *) 0** instead.
    
           The simplest case is simple.  To run a single command, such as **mv**
           _source dest_ :
    
                 pipeline *p = pipeline_new_command_args ("mv", source, dest, nullptr);
                 int status = pipeline_run (p);
    
           **libpipeline** is often used to mimic shell pipelines, such as the
           following example:
    
                 **zsoelim < input-file **| **tbl** | **nroff -mandoc -Tutf8**
    
           The code to construct this would be:
    
                 pipeline *p;
                 int status;
    
                 p = pipeline_new ();
                 pipeline_want_infile (p, "input-file");
                 pipeline_command_args (p, "zsoelim", nullptr);
                 pipeline_command_args (p, "tbl", nullptr);
                 pipeline_command_args (p, "nroff", "-mandoc", "-Tutf8", nullptr);
                 status = pipeline_run (p);
    
           You might want to construct a command more dynamically:
    
                 pipecmd *manconv = pipecmd_new_args ("manconv", "-f", from_code,
                                                      "-t", "UTF-8", nullptr);
                 if (quiet)
                         pipecmd_arg (manconv, "-q");
                 pipeline_command (p, manconv);
    
           Perhaps you want an environment variable set only while running a
           certain command:
    
                 pipecmd *less = pipecmd_new ("less");
                 pipecmd_setenv (less, "LESSCHARSET", lesscharset);
    
           You might find yourself needing to pass the output of one pipeline
           to several other pipelines, in a “tee” arrangement:
    
                 pipeline *source, *sink1, *sink2;
    
                 source = make_source ();
                 sink1 = make_sink1 ();
                 sink2 = make_sink2 ();
                 pipeline_connect (source, sink1, sink2, nullptr);
                 /* Pump data among these pipelines until there's nothing left. */
                 pipeline_pump (source, sink1, sink2, nullptr);
                 pipeline_free (sink2);
                 pipeline_free (sink1);
                 pipeline_free (source);
    
           Maybe one of your commands is actually an in-process function,
           rather than an external program:
    
                 pipecmd *inproc = pipecmd_new_function ("in-process", &func,
                                                         NULL, NULL);
                 pipeline_command (p, inproc);
    
           Sometimes your program needs to consume the output of a pipeline,
           rather than sending it all to some other subprocess:
    
                 pipeline *p = make_pipeline ();
                 const char *line;
    
                 pipeline_want_out (p, -1);
                 pipeline_start (p);
                 line = pipeline_peekline (p);
                 if (!strstr (line, "coding: UTF-8"))
                         printf ("Unicode text follows:0);
                 while (line = pipeline_readline (p))
                         printf ("  %s", line);
                 pipeline_free (p);
    


## BUGS



           Using this library in a program which runs any other child
           processes and/or installs its own **SIGCHLD** handler is unlikely to
           work.
    


* * *