viruPVS/langchain-docs huggingface dataset:
Create graph index from text.
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using GraphIndexCreator¶
Graph QA
langchain.indexes.vectorstore.VectorstoreIndexCreator¶
class langchain.indexes.vectorstore.VectorstoreIndexCreator[source]¶
Bases: BaseModel
Logic for creating indexes.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embedding: langchain_core.embeddings.Embeddings [Optional]¶
param text_splitter: langchain.text_splitter.TextSplitter [Optional]¶
param vectorstore_cls: Type[langchain_core.vectorstores.VectorStore] = <class 'langchain_community.vectorstores.chroma.Chroma'>¶
param vectorstore_kwargs: dict [Optional]¶
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
from_documents(documents: List[Document]) → VectorStoreIndexWrapper[source]¶
Create a vectorstore index from documents.
from_loaders(loaders: List[BaseLoader]) → VectorStoreIndexWrapper[source]¶
Create a vectorstore index from loaders.
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using VectorstoreIndexCreator¶
Apify
HuggingFace dataset
Spreedly
Image captions
Figma
Apify Dataset
Iugu
Stripe
Modern Treasury
Question Answering
Multiple Retrieval Sources
langchain_experimental.data_anonymizer.base.AnonymizerBase¶
class langchain_experimental.data_anonymizer.base.AnonymizerBase[source]¶
Base abstract class for anonymizers.
It is public and non-virtual because it allows
wrapping the behavior for all methods in a base class.
Methods
__init__()
anonymize(text[, language, allow_list])
Anonymize text
__init__()¶
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str[source]¶
Anonymize text
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.exact_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.exact_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]]) → str[source]¶
Exact matching strategy for deanonymization.
It replaces all the anonymized entities with the original ones.
Parameters
text – text to deanonymize
deanonymizer_mapping – mapping between anonymized entities and original ones
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], max_l_dist: int = 3) → str[source]¶
Fuzzy matching strategy for deanonymization.
It uses fuzzy matching to find the position of the anonymized entity in the text.
It replaces all the anonymized entities with the original ones.
Parameters
text – text to deanonymize
deanonymizer_mapping – mapping between anonymized entities and original ones
max_l_dist – maximum Levenshtein distance between the anonymized entity and the
text segment to consider it a match
Examples of matching:Kaenu Reves -> Keanu Reeves
John F. Kennedy -> John Kennedy
langchain_experimental.data_anonymizer.presidio.PresidioReversibleAnonymizer¶
class langchain_experimental.data_anonymizer.presidio.PresidioReversibleAnonymizer(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
Attributes
anonymizer_mapping
Return the anonymizer mapping This is just the reverse version of the deanonymizer mapping.
deanonymizer_mapping
Return the deanonymizer mapping
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text
deanonymize(text_to_deanonymize[, ...])
Deanonymize text
load_deanonymizer_mapping(file_path)
Load the deanonymizer mapping from a JSON or YAML file.
reset_deanonymizer_mapping()
Reset the deanonymizer mapping
save_deanonymizer_mapping(file_path)
Save the deanonymizer mapping to a JSON or YAML file.
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_operators(operators: Dict[str, OperatorConfig]) → None¶
Add operators to the anonymizer
Parameters
Add operators to the anonymizer
Parameters
operators – Operators to add to the anonymizer.
add_recognizer(recognizer: EntityRecognizer) → None¶
Add a recognizer to the analyzer
Parameters
recognizer – Recognizer to add to the analyzer.
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text
deanonymize(text_to_deanonymize: str, deanonymizer_matching_strategy: ~typing.Callable[[str, ~typing.Dict[str, ~typing.Dict[str, str]]], str] = <function exact_matching_strategy>) → str¶
Deanonymize text
load_deanonymizer_mapping(file_path: Union[Path, str]) → None[source]¶
Load the deanonymizer mapping from a JSON or YAML file.
Parameters
file_path – Path to file to load the mapping from.
Example:
.. code-block:: python
anonymizer.load_deanonymizer_mapping(file_path=”path/mapping.json”)
reset_deanonymizer_mapping() → None[source]¶
Reset the deanonymizer mapping
save_deanonymizer_mapping(file_path: Union[Path, str]) → None[source]¶
Save the deanonymizer mapping to a JSON or YAML file.
Parameters
file_path – Path to file to save the mapping to.
Example:
.. code-block:: python
anonymizer.save_deanonymizer_mapping(file_path=”path/mapping.json”)
langchain_experimental.data_anonymizer.presidio.PresidioAnonymizer¶
class langchain_experimental.data_anonymizer.presidio.PresidioAnonymizer(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text
anonymize(text[, language, allow_list])
Anonymize text
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_operators(operators: Dict[str, OperatorConfig]) → None¶
Add operators to the anonymizer
Parameters
operators – Operators to add to the anonymizer.
add_recognizer(recognizer: EntityRecognizer) → None¶
Add a recognizer to the analyzer
Parameters
recognizer – Recognizer to add to the analyzer.
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text
langchain_experimental.data_anonymizer.presidio.PresidioAnonymizerBase¶
class langchain_experimental.data_anonymizer.presidio.PresidioAnonymizerBase(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
Methods
__init__([analyzed_fields, operators, ...])
param analyzed_fields
List of fields to detect and then anonymize.
add_operators(operators)
Add operators to the anonymizer
add_recognizer(recognizer)
Add a recognizer to the analyzer
anonymize(text[, language, allow_list])
Anonymize text
anonymize(text[, language, allow_list])
Anonymize text
__init__(analyzed_fields: Optional[List[str]] = None, operators: Optional[Dict[str, OperatorConfig]] = None, languages_config: Dict = {'models': [{'lang_code': 'en', 'model_name': 'en_core_web_lg'}], 'nlp_engine_name': 'spacy'}, add_default_faker_operators: bool = True, faker_seed: Optional[int] = None)[source]¶
Parameters
analyzed_fields – List of fields to detect and then anonymize.
Defaults to all entities supported by Microsoft Presidio.
operators – Operators to use for anonymization.
Operators allow for custom anonymization of detected PII.
Learn more:
https://microsoft.github.io/presidio/tutorial/10_simple_anonymization/
languages_config – Configuration for the NLP engine.
First language in the list will be used as the main language
in self.anonymize(…) when no language is specified.
Learn more:
https://microsoft.github.io/presidio/analyzer/customizing_nlp_models/
faker_seed – Seed used to initialize faker.
Defaults to None, in which case faker will be seeded randomly
and provide random values.
add_operators(operators: Dict[str, OperatorConfig]) → None[source]¶
Add operators to the anonymizer
Parameters
operators – Operators to add to the anonymizer.
add_recognizer(recognizer: EntityRecognizer) → None[source]¶
Add a recognizer to the analyzer
Parameters
recognizer – Recognizer to add to the analyzer.
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.case_insensitive_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.case_insensitive_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]]) → str[source]¶
Case insensitive matching strategy for deanonymization.
It replaces all the anonymized entities with the original ones
irrespective of their letter case.
Parameters
text – text to deanonymize
deanonymizer_mapping – mapping between anonymized entities and original ones
Examples of matching:keanu reeves -> Keanu Reeves
JOHN F. KENNEDY -> John F. Kennedy
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.ngram_fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.ngram_fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], fuzzy_threshold: int = 85, use_variable_length: bool = True) → str[source]¶
N-gram fuzzy matching strategy for deanonymization.
It replaces all the anonymized entities with the original ones.
It uses fuzzy matching to find the position of the anonymized entity in the text.
It generates n-grams of the same length as the anonymized entity from the text and
uses fuzzy matching to find the position of the anonymized entity in the text.
Parameters
text – text to deanonymize
deanonymizer_mapping – mapping between anonymized entities and original ones
fuzzy_threshold – fuzzy matching threshold
use_variable_length – whether to use (n-1, n, n+1)-grams or just n-grams
langchain_experimental.data_anonymizer.deanonymizer_mapping.DeanonymizerMapping¶
class langchain_experimental.data_anonymizer.deanonymizer_mapping.DeanonymizerMapping(mapping: Dict[str, Dict[str, str]] = <factory>)[source]¶
Attributes
data
Return the deanonymizer mapping
mapping
Methods
__init__([mapping])
update(new_mapping)
Update the deanonymizer mapping with new values Duplicated values will not be added If there are multiple entities of the same type, the mapping will include a count to differentiate them.
__init__(mapping: ~typing.Dict[str, ~typing.Dict[str, str]] = <factory>) → None¶
update(new_mapping: Dict[str, Dict[str, str]]) → None[source]¶
Update the deanonymizer mapping with new values
Duplicated values will not be added
If there are multiple entities of the same type, the mapping will
include a count to differentiate them. For example, if there are
two names in the input text, the mapping will include NAME_1 and NAME_2.
langchain_experimental.data_anonymizer.deanonymizer_mapping.create_anonymizer_mapping¶
langchain_experimental.data_anonymizer.deanonymizer_mapping.create_anonymizer_mapping(original_text: str, analyzer_results: List[RecognizerResult], anonymizer_results: EngineResult, is_reversed: bool = False) → Dict[str, Dict[str, str]][source]¶
Creates or updates the mapping used to anonymize and/or deanonymize text.
This method exploits the results returned by the
analysis and anonymization processes.
If is_reversed is True, it constructs a mapping from each original
entity to its anonymized value.
If is_reversed is False, it constructs a mapping from each
anonymized entity back to its original text value.
If there are multiple entities of the same type, the mapping will
include a count to differentiate them. For example, if there are
two names in the input text, the mapping will include NAME_1 and NAME_2.
Example of mapping:
{
“PERSON”: {“<original>”: “<anonymized>”,
“John Doe”: “Slim Shady”
},
“PHONE_NUMBER”: {
“111-111-1111”: “555-555-5555”
}
langchain_experimental.data_anonymizer.base.ReversibleAnonymizerBase¶
class langchain_experimental.data_anonymizer.base.ReversibleAnonymizerBase[source]¶
Base abstract class for reversible anonymizers.
Methods
__init__()
anonymize(text[, language, allow_list])
Anonymize text
deanonymize(text_to_deanonymize[, ...])
Deanonymize text
reset_deanonymizer_mapping()
Abstract method to reset deanonymizer mapping
__init__()¶
anonymize(text: str, language: Optional[str] = None, allow_list: Optional[List[str]] = None) → str¶
Anonymize text
deanonymize(text_to_deanonymize: str, deanonymizer_matching_strategy: ~typing.Callable[[str, ~typing.Dict[str, ~typing.Dict[str, str]]], str] = <function exact_matching_strategy>) → str[source]¶
Deanonymize text
abstract reset_deanonymizer_mapping() → None[source]¶
Abstract method to reset deanonymizer mapping
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.combined_exact_fuzzy_matching_strategy¶
langchain_experimental.data_anonymizer.deanonymizer_matching_strategies.combined_exact_fuzzy_matching_strategy(text: str, deanonymizer_mapping: Dict[str, Dict[str, str]], max_l_dist: int = 3) → str[source]¶
RECOMMENDED STRATEGY.
Combined exact and fuzzy matching strategy for deanonymization.
Parameters
text – text to deanonymize
deanonymizer_mapping – mapping between anonymized entities and original ones
max_l_dist – maximum Levenshtein distance between the anonymized entity and the
text segment to consider it a match
Examples of matching:Kaenu Reves -> Keanu Reeves
John F. Kennedy -> John Kennedy
langchain_community.adapters.openai.convert_message_to_dict¶
langchain_community.adapters.openai.convert_message_to_dict(message: BaseMessage) → dict[source]¶
Convert a LangChain message to a dictionary.
Parameters
message – The LangChain message.
Returns
The dictionary.
Examples using convert_message_to_dict¶
Twitter (via Apify)
langchain_community.adapters.openai.convert_openai_messages¶
langchain_community.adapters.openai.convert_openai_messages(messages: Sequence[Dict[str, Any]]) → List[BaseMessage][source]¶
Convert dictionaries representing OpenAI messages to LangChain format.
Parameters
messages – List of dictionaries representing OpenAI messages
Returns
List of LangChain BaseMessage objects.
langchain_community.adapters.openai.Completions¶
class langchain_community.adapters.openai.Completions[source]¶
Completion.
Methods
__init__()
acreate()
create()
__init__()¶
async static acreate(messages: Sequence[Dict[str, Any]], *, provider: str = "'ChatOpenAI'", stream: Literal[False] = 'False', **kwargs: Any) → ChatCompletions[source]¶
async static acreate(messages: Sequence[Dict[str, Any]], *, provider: str = "'ChatOpenAI'", stream: Literal[True], **kwargs: Any) → AsyncIterator
static create(messages: Sequence[Dict[str, Any]], *, provider: str = "'ChatOpenAI'", stream: Literal[False] = 'False', **kwargs: Any) → ChatCompletions[source]¶
static create(messages: Sequence[Dict[str, Any]], *, provider: str = "'ChatOpenAI'", stream: Literal[True], **kwargs: Any) → Iterable