semvec/pypi-packages huggingface dataset:
steed 0.2.0
A drop-in replacement for tee with file buffering and output queue management.
Requires Python: <4.0,>=3.7

Pipelines may wish to use `tee` to cache intermediate steps as a failsafe in case of a downstream failure. This strategy is limited by the fact that the speed at which `tee` writes to disk is bottlenecked by the speed at which the downstream pipeline reads from its stdout. Therefore `tee` cannot finish writing to disk before its input has been completely output by the downstream pipeline. An option is to store all input from stdin in an in-memory buffer, but this can lead to out-of-memory failures. The `steed` command overcomes this limitation by writing to the on-disk file continuously, while simultaneously reading from it to obtain a size-limited buffer queue that is output to stdout. This allows `steed` to complete writing to disk before it has finished outputting to downstream commands, improving its utility as a cacheing mechanism.