*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to stream responses from an LLM

On this page

How to stream responses from an LLM
===================================

All `LLM`s implement the [Runnable interface](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.Runnable.html#langchain_core.runnables.base.Runnable), which comes with **default** implementations of standard runnable methods (i.e. `ainvoke`, `batch`, `abatch`, `stream`, `astream`, `astream_events`).

The **default** streaming implementations provide an`Iterator` (or `AsyncIterator` for asynchronous streaming) that yields a single value: the final output from the underlying chat model provider.

The ability to stream the output token-by-token depends on whether the provider has implemented proper streaming support.

See which [integrations support token-by-token streaming here](/v0.2/docs/integrations/llms/).

note

The **default** implementation does **not** provide support for token-by-token streaming, but it ensures that the model can be swapped in for any other model as it supports the same standard interface.

Sync stream[​](#sync-stream "Direct link to Sync stream")
---------------------------------------------------------

Below we use a `|` to help visualize the delimiter between tokens.

    from langchain_openai import OpenAIllm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0, max_tokens=512)for chunk in llm.stream("Write me a 1 verse song about sparkling water."):    print(chunk, end="|", flush=True)

**API Reference:**[OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

    |Spark|ling| water|,| oh| so clear||Bubbles dancing|,| without| fear||Refreshing| taste|,| a| pure| delight||Spark|ling| water|,| my| thirst|'s| delight||

Async streaming[​](#async-streaming "Direct link to Async streaming")
---------------------------------------------------------------------

Let's see how to stream in an async setting using `astream`.

    from langchain_openai import OpenAIllm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0, max_tokens=512)async for chunk in llm.astream("Write me a 1 verse song about sparkling water."):    print(chunk, end="|", flush=True)

**API Reference:**[OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

    |Spark|ling| water|,| oh| so clear||Bubbles dancing|,| without| fear||Refreshing| taste|,| a| pure| delight||Spark|ling| water|,| my| thirst|'s| delight||

Async event streaming[​](#async-event-streaming "Direct link to Async event streaming")
---------------------------------------------------------------------------------------

LLMs also support the standard [astream events](https://api.python.langchain.com/en/latest/runnables/langchain_core.runnables.base.Runnable.html#langchain_core.runnables.base.Runnable.astream_events) method.

tip

`astream_events` is most useful when implementing streaming in a larger LLM application that contains multiple steps (e.g., an application that involves an `agent`).

    from langchain_openai import OpenAIllm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0, max_tokens=512)idx = 0async for event in llm.astream_events(    "Write me a 1 verse song about goldfish on the moon", version="v1"):    idx += 1    if idx >= 5:  # Truncate the output        print("...Truncated")        break    print(event)

**API Reference:**[OpenAI](https://api.python.langchain.com/en/latest/llms/langchain_openai.llms.base.OpenAI.html)

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/how_to/streaming_llm.ipynb)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to stream runnables

](/v0.2/docs/how_to/streaming/)[

Next

How to use a time-weighted vector store retriever

](/v0.2/docs/how_to/time_weighted_vectorstore/)

*   [Sync stream](#sync-stream)
*   [Async streaming](#async-streaming)
*   [Async event streaming](#async-event-streaming)