viruPVS/langchain-docs huggingface dataset:
Useful for checking if an input will fit in a model’s context window.
Parameters
text – The string input to tokenize.
Returns
The integer number of tokens in the text.
get_num_tokens_from_messages(messages: List[BaseMessage]) → int¶
Get the number of tokens in the messages.
Useful for checking if an input will fit in a model’s context window.
Parameters
messages – The message inputs to tokenize.
Returns
The sum of the number of tokens across the messages.
get_output_schema(config: Optional[RunnableConfig] = None) → Type[BaseModel]¶
Get a pydantic model that can be used to validate output to the runnable.
Runnables that leverage the configurable_fields and configurable_alternatives
methods will have a dynamic output schema that depends on which
configuration the runnable is invoked with.
This method allows to get an output schema for a specific configuration.
Parameters
config – A config to use when generating the schema.
Returns
A pydantic model that can be used to validate output.
get_token_ids(text: str) → List[int]¶
Return the ordered ids of the tokens in a text.
Parameters
text – The string input to tokenize.
Returns
A list of ids corresponding to the tokens in the text, in order they occurin the text.
invoke(input: Union[PromptValue, str, List[BaseMessage]], config: Optional[RunnableConfig] = None, *, stop: Optional[List[str]] = None, **kwargs: Any) → str¶
Transform a single input into an output. Override to implement.
Parameters
input – The input to the runnable.
config – A config to use when invoking the runnable.
The config supports standard keys like ‘tags’, ‘metadata’ for tracing
purposes, ‘max_concurrency’ for controlling how much work to do
predict(text: str, *, stop: Optional[Sequence[str]] = None, **kwargs: Any) → str¶
Pass a single string input to the model and return a string prediction.
Use this method when passing in raw text. If you want to pass in specifictypes of chat messages, use predict_messages.
Parameters
text – String input to pass to the model.
stop – Stop words to use when generating. Model output is cut off at the
first occurrence of any of these substrings.
**kwargs – Arbitrary additional keyword arguments. These are usually passed
to the model provider API call.
Returns
Top model prediction as a string.
predict_messages(messages: List[BaseMessage], *, stop: Optional[Sequence[str]] = None, **kwargs: Any) → BaseMessage¶
Pass a message sequence to the model and return a message prediction.
Use this method when passing in chat messages. If you want to pass in raw text,use predict.
Parameters
messages – A sequence of chat messages corresponding to a single model input.
stop – Stop words to use when generating. Model output is cut off at the
first occurrence of any of these substrings.
**kwargs – Arbitrary additional keyword arguments. These are usually passed
to the model provider API call.
Returns
Top model prediction as a message.
save(file_path: Union[Path, str]) → None¶
Save the LLM.
Parameters
file_path – Path to file to save the LLM to.
Example:
.. code-block:: python
llm.save(file_path=”path/llm.yaml”)
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
property config_specs: List[langchain_core.runnables.utils.ConfigurableFieldSpec]¶
List configurable fields for this runnable.
property input_schema: Type[pydantic.main.BaseModel]¶
The type of input this runnable accepts specified as a pydantic model.
property lc_attributes: Dict¶
List of attribute names that should be included in the serialized kwargs.
These attributes must be accepted by the constructor.
property lc_secrets: Dict[str, str]¶
A map of constructor argument names to secret ids.
For example,{“openai_api_key”: “OPENAI_API_KEY”}
property output_schema: Type[pydantic.main.BaseModel]¶
The type of output this runnable produces specified as a pydantic model.
Examples using SelfHostedHuggingFaceEmbeddings¶
Self Hosted
langchain_community.embeddings.gradient_ai.TinyAsyncGradientEmbeddingClient¶
class langchain_community.embeddings.gradient_ai.TinyAsyncGradientEmbeddingClient(access_token: Optional[str] = None, workspace_id: Optional[str] = None, host: str = 'https://api.gradient.ai/api', aiosession: Optional[ClientSession] = None)[source]¶
A helper tool to embed Gradient. Not part of Langchain’s or Gradients stable API,
direct use discouraged.
To use, set the environment variable GRADIENT_ACCESS_TOKEN with your
API token and GRADIENT_WORKSPACE_ID for your gradient workspace,
or alternatively provide them as keywords to the constructor of this class.
Example
mini_client = TinyAsyncGradientEmbeddingClient(
    workspace_id="12345614fc0_workspace",
    access_token="gradientai-access_token",
)
embeds = mini_client.embed(
    model="bge-large",
    text=["doc1", "doc2"]
)
# or
embeds = await mini_client.aembed(
    model="bge-large",
    text=["doc1", "doc2"]
)
Methods
__init__([access_token, workspace_id, host, ...])
aembed(model, texts)
call the embedding of model, async method
embed(model, texts)
call the embedding of model
__init__(access_token: Optional[str] = None, workspace_id: Optional[str] = None, host: str = 'https://api.gradient.ai/api', aiosession: Optional[ClientSession] = None) → None[source]¶
async aembed(model: str, texts: List[str]) → List[List[float]][source]¶
call the embedding of model, async method
Parameters
model (str) – to embedding model
texts (List[str]) – List of sentences to embed.
texts (List[str]) – List of sentences to embed.
Returns
List of vectors for each sentence
Return type
List[List[float]]
embed(model: str, texts: List[str]) → List[List[float]][source]¶
call the embedding of model
Parameters
model (str) – to embedding model
texts (List[str]) – List of sentences to embed.
Returns
List of vectors for each sentence
Return type
List[List[float]]
langchain_community.embeddings.huggingface.HuggingFaceBgeEmbeddings¶
class langchain_community.embeddings.huggingface.HuggingFaceBgeEmbeddings[source]¶
Bases: BaseModel, Embeddings
HuggingFace BGE sentence_transformers embedding models.
To use, you should have the sentence_transformers python package installed.
Example
from langchain_community.embeddings import HuggingFaceBgeEmbeddings
model_name = "BAAI/bge-large-en"
model_kwargs = {'device': 'cpu'}
encode_kwargs = {'normalize_embeddings': True}
hf = HuggingFaceBgeEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)
Initialize the sentence_transformer.
param cache_folder: Optional[str] = None¶
Path to store models.
Can be also set by SENTENCE_TRANSFORMERS_HOME environment variable.
param encode_kwargs: Dict[str, Any] [Optional]¶
Keyword arguments to pass when calling the encode method of the model.
param model_kwargs: Dict[str, Any] [Optional]¶
Keyword arguments to pass to the model.
param model_name: str = 'BAAI/bge-large-en'¶
Model name to use.
param query_instruction: str = 'Represent this question for searching relevant passages: '¶
Instruction to use for embedding query.
async aembed_documents(texts: List[str]) → List[List[float]]¶
Asynchronous Embed search docs.
async aembed_query(text: str) → List[float]¶
Asynchronous Embed query text.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Parameters
text – The text to embed.
Returns
Embeddings for the text.
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using HuggingFaceBgeEmbeddings¶
BGE on Hugging Face
langchain_community.embeddings.openai.OpenAIEmbeddings¶
class langchain_community.embeddings.openai.OpenAIEmbeddings[source]¶
Bases: BaseModel, Embeddings
OpenAI embedding models.
To use, you should have the openai python package installed, and the
environment variable OPENAI_API_KEY set with your API key or pass it
as a named parameter to the constructor.
Example
from langchain_community.embeddings import OpenAIEmbeddings
openai = OpenAIEmbeddings(openai_api_key="my-api-key")
In order to use the library with Microsoft Azure endpoints, you need to set
the OPENAI_API_TYPE, OPENAI_API_BASE, OPENAI_API_KEY and OPENAI_API_VERSION.
The OPENAI_API_TYPE must be set to ‘azure’ and the others correspond to
the properties of your endpoint.
In addition, the deployment name must be passed as the model parameter.
Example
import os
os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_BASE"] = "https://<your-endpoint.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "your AzureOpenAI key"
os.environ["OPENAI_API_VERSION"] = "2023-05-15"
os.environ["OPENAI_PROXY"] = "http://your-corporate-proxy:8080"
from langchain_community.embeddings.openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(
    deployment="your-embeddings-deployment-name",
    model="your-embeddings-model-name",
    openai_api_base="https://your-endpoint.openai.azure.com/",
    openai_api_type="azure",
)
text = "This is a test query."
query_result = embeddings.embed_query(text)
Create a new model by parsing and validating input data from keyword arguments.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param allowed_special: Union[Literal['all'], Set[str]] = {}¶
param chunk_size: int = 1000¶
Maximum number of texts to embed in each batch
param default_headers: Optional[Mapping[str, str]] = None¶
param default_query: Optional[Mapping[str, object]] = None¶
param deployment: Optional[str] = 'text-embedding-ada-002'¶
param disallowed_special: Union[Literal['all'], Set[str], Sequence[str]] = 'all'¶
param embedding_ctx_length: int = 8191¶
The maximum number of tokens to embed at once.
param headers: Any = None¶
param http_client: Optional[Any] = None¶
Optional httpx.Client.
param max_retries: int = 2¶
Maximum number of retries to make when generating.
param model: str = 'text-embedding-ada-002'¶
param model_kwargs: Dict[str, Any] [Optional]¶
Holds any model parameters valid for create call not explicitly specified.
param openai_api_base: Optional[str] = None (alias 'base_url')¶
Base URL path for API requests, leave blank if not using a proxy or service
emulator.
param openai_api_key: Optional[str] = None (alias 'api_key')¶
Automatically inferred from env var OPENAI_API_KEY if not provided.
param openai_api_type: Optional[str] = None¶
param openai_api_version: Optional[str] = None (alias 'api_version')¶
Automatically inferred from env var OPENAI_API_VERSION if not provided.
param openai_organization: Optional[str] = None (alias 'organization')¶
param openai_organization: Optional[str] = None (alias 'organization')¶
Automatically inferred from env var OPENAI_ORG_ID if not provided.
param openai_proxy: Optional[str] = None¶
param request_timeout: Optional[Union[float, Tuple[float, float], Any]] = None (alias 'timeout')¶
Timeout for requests to OpenAI completion API. Can be float, httpx.Timeout or
None.
param retry_max_seconds: int = 20¶
Max number of seconds to wait between retries
param retry_min_seconds: int = 4¶
Min number of seconds to wait between retries
param show_progress_bar: bool = False¶
Whether to show a progress bar when embedding.
param skip_empty: bool = False¶
Whether to skip empty strings when embedding or raise an error.
Defaults to not skipping.
param tiktoken_enabled: bool = True¶
Set this to False for non-OpenAI implementations of the embeddings API, e.g.
the –extensions openai extension for text-generation-webui
param tiktoken_model_name: Optional[str] = None¶
The model name to pass to tiktoken when using this class.
Tiktoken is used to count the number of tokens in documents to constrain
them to be under a certain limit. By default, when set to None, this will
be the same as the embedding model name. However, there are some cases
where you may want to use this Embedding class with a model name not
supported by tiktoken. This can include when using Azure embeddings or
when using one of the many model providers that expose an OpenAI-like
API but with different models. In those cases, in order to avoid erroring
when tiktoken is called, you can specify a model name to use here.
when tiktoken is called, you can specify a model name to use here.
async aembed_documents(texts: List[str], chunk_size: Optional[int] = 0) → List[List[float]][source]¶
Call out to OpenAI’s embedding endpoint async for embedding search docs.
Parameters
texts – The list of texts to embed.
chunk_size – The chunk size of embeddings. If None, will use the chunk size
specified by the class.
Returns
List of embeddings, one for each text.
async aembed_query(text: str) → List[float][source]¶
Call out to OpenAI’s embedding endpoint async for embedding query text.
Parameters
text – The text to embed.
Returns
Embedding for the text.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
embed_documents(texts: List[str], chunk_size: Optional[int] = 0) → List[List[float]][source]¶
Call out to OpenAI’s embedding endpoint for embedding search docs.
Parameters
texts – The list of texts to embed.
chunk_size – The chunk size of embeddings. If None, will use the chunk size
specified by the class.
Returns
List of embeddings, one for each text.
embed_query(text: str) → List[float][source]¶
Call out to OpenAI’s embedding endpoint for embedding query text.
Parameters
text – The text to embed.
Returns
Embedding for the text.
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using OpenAIEmbeddings¶
OpenAI
AzureOpenAI
RePhraseQueryRetriever
Cohere Reranker
kNN
DocArray Retriever
SVM
Pinecone Hybrid Search
LOTR (Merger Retriever)
Xata chat memory
Confident
Azure OpenAI
Document Comparison
Vectorstore
LanceDB
Weaviate
Xata
Redis
PGVector
Rockset
DingoDB
Zilliz
SingleStoreDB
Typesense
Atlas
Activeloop Deep Lake
Neo4j Vector Index
Chroma
Alibaba Cloud OpenSearch
Baidu Cloud VectorSearch
StarRocks
scikit-learn
DocArray HnswSearch
MyScale
ClickHouse
scikit-learn
DocArray HnswSearch
MyScale
ClickHouse
Qdrant
Tigris
Supabase (Postgres)
OpenSearch
Pinecone
Azure Cognitive Search
Cassandra
USearch
Milvus
Elasticsearch
DocArray InMemorySearch
Postgres Embedding
Faiss
Epsilla
AnalyticDB
Hologres
MongoDB Atlas
Meilisearch
Loading documents from a YouTube url
Psychic
Docugami
LLM Caching integrations
Set env var OPENAI_API_KEY or load from a .env file:
Set env var OPENAI_API_KEY or load from a .env file
Question Answering
Perform context-aware text splitting
Conversational Retrieval Agent
Retrieve from vector stores directly
Retrieve as you generate with FLARE
Improve document indexing with HyDE
Analysis of Twitter the-algorithm source code with LangChain, GPT4 and Activeloop’s Deep Lake
Use LangChain, GPT and Activeloop’s Deep Lake to work with code base
Structure answers with OpenAI functions
QA using Activeloop’s DeepLake
Agents
AutoGPT
BabyAGI User Guide
BabyAGI with Tools
!pip install bs4
Plug-and-Plai
SalesGPT - Your Context-Aware AI Sales Assistant With Knowledge Base
Custom Agent with PlugIn Retrieval
Generative Agents in LangChain
SQL
Indexing
Caching
MultiVector Retriever
MultiQueryRetriever
Parent Document Retriever
WebResearchRetriever
Supabase
Deep Lake
Memory in the Multi-Input Chain
Combine agents and vector stores
Custom agent with tool retrieval
Select by maximal marginal relevance (MMR)
Few-shot examples for chat models
Loading from LangChainHub
First we add a step to load memory
langchain_community.embeddings.tensorflow_hub.TensorflowHubEmbeddings¶
class langchain_community.embeddings.tensorflow_hub.TensorflowHubEmbeddings[source]¶
Bases: BaseModel, Embeddings
TensorflowHub embedding models.
To use, you should have the tensorflow_text python package installed.
Example
from langchain_community.embeddings import TensorflowHubEmbeddings
url = "https://tfhub.dev/google/universal-sentence-encoder-multilingual/3"
tf = TensorflowHubEmbeddings(model_url=url)
Initialize the tensorflow_hub and tensorflow_text.
param model_url: str = 'https://tfhub.dev/google/universal-sentence-encoder-multilingual/3'¶
Model name to use.
async aembed_documents(texts: List[str]) → List[List[float]]¶
Asynchronous Embed search docs.
async aembed_query(text: str) → List[float]¶
Asynchronous Embed query text.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
embed_documents(texts: List[str]) → List[List[float]][source]¶
Compute doc embeddings using a TensorflowHub embedding model.
Parameters
texts – The list of texts to embed.
Returns
List of embeddings, one for each text.
embed_query(text: str) → List[float][source]¶
Compute query embeddings using a TensorflowHub embedding model.
Parameters
text – The text to embed.
Returns
Embeddings for the text.
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using TensorflowHubEmbeddings¶
TensorflowHub
ScaNN