This notebook shows how to use functionality related to the `Google Cloud Vertex AI Vector Search` vector database.

> [Google Vertex AI Vector Search](https://cloud.google.com/vertex-ai/docs/vector-search/overview), formerly known as Vertex AI Matching Engine, provides the industry’s leading high-scale low latency vector database. These vector databases are commonly referred to as vector similarity-matching or an approximate nearest neighbor (ANN) service.

**Note**: Langchain API expects an endpoint and deployed index already created.Index creation time can take upto one hour.

> To see how to create an index refer to the section [Create Index and deploy it to an Endpoint](#create-index-and-deploy-it-to-an-endpoint)  
> If you already have an index deployed , skip to [Create VectorStore from texts](#create-vector-store-from-texts)

## Create Index and deploy it to an Endpoint[​](#create-index-and-deploy-it-to-an-endpoint "Direct link to Create Index and deploy it to an Endpoint")

*   This section demonstrates creating a new index and deploying it to an endpoint

```
# TODO : Set values as per your requirements# Project and Storage ConstantsPROJECT_ID = "<my_project_id>"REGION = "<my_region>"BUCKET = "<my_gcs_bucket>"BUCKET_URI = f"gs://{BUCKET}"# The number of dimensions for the textembedding-gecko@003 is 768# If other embedder is used, the dimensions would probably need to change.DIMENSIONS = 768# Index ConstantsDISPLAY_NAME = "<my_matching_engine_index_id>"DEPLOYED_INDEX_ID = "<my_matching_engine_endpoint_id>"
```

```
# Create a bucket.! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI
```

### Use [VertexAIEmbeddings](https://python.langchain.com/docs/integrations/text_embedding/google_vertex_ai_palm/) as the embeddings model[​](#use-vertexaiembeddings-as-the-embeddings-model "Direct link to use-vertexaiembeddings-as-the-embeddings-model")

```
from google.cloud import aiplatformfrom langchain_google_vertexai import VertexAIEmbeddings
```

```
aiplatform.init(project=PROJECT_ID, location=REGION, staging_bucket=BUCKET_URI)
```

```
embedding_model = VertexAIEmbeddings(model_name="textembedding-gecko@003")
```

### Create an empty Index[​](#create-an-empty-index "Direct link to Create an empty Index")

**Note :** While creating an index you should specify an “index\_update\_method” from either a “BATCH\_UPDATE” or “STREAM\_UPDATE” \> A batch index is for when you want to update your index in a batch, with data which has been stored over a set amount of time, like systems which are processed weekly or monthly. A streaming index is when you want index data to be updated as new data is added to your datastore, for instance, if you have a bookstore and want to show new inventory online as soon as possible. Which type you choose is important, since setup and requirements are different.

Refer [Official Documentation](https://cloud.google.com/vertex-ai/docs/vector-search/create-manage-index#create-index-batch) for more details on configuring indexes

```
# NOTE : This operation can take upto 30 secondsmy_index = aiplatform.MatchingEngineIndex.create_tree_ah_index(    display_name=DISPLAY_NAME,    dimensions=DIMENSIONS,    approximate_neighbors_count=150,    distance_measure_type="DOT_PRODUCT_DISTANCE",    index_update_method="STREAM_UPDATE",  # allowed values BATCH_UPDATE , STREAM_UPDATE)
```

### Create an Endpoint[​](#create-an-endpoint "Direct link to Create an Endpoint")

```
# Create an endpointmy_index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(    display_name=f"{DISPLAY_NAME}-endpoint", public_endpoint_enabled=True)
```

### Deploy Index to the Endpoint[​](#deploy-index-to-the-endpoint "Direct link to Deploy Index to the Endpoint")

```
# NOTE : This operation can take upto 20 minutesmy_index_endpoint = my_index_endpoint.deploy_index(    index=my_index, deployed_index_id=DEPLOYED_INDEX_ID)my_index_endpoint.deployed_indexes
```

## Create Vector Store from texts[​](#create-vector-store-from-texts "Direct link to Create Vector Store from texts")

```
from langchain_google_vertexai import (    VectorSearchVectorStore,    VectorSearchVectorStoreDatastore,)
```

![Langchainassets.png](https://python.langchain.com/assets/images/b25a61eb-1-Langchainassets-1ce2d60d39467cabd4d5124579bc8ecd.png)

### Create simple vectorstore ( without filters)[​](#create-simple-vectorstore-without-filters "Direct link to Create simple vectorstore ( without filters)")

```
# Input textstexts = [    "The cat sat on",    "the mat.",    "I like to",    "eat pizza for",    "dinner.",    "The sun sets",    "in the west.",]# Create a Vector Storevector_store = VectorSearchVectorStore.from_components(    project_id=PROJECT_ID,    region=REGION,    gcs_bucket_name=BUCKET,    index_id=my_index.name,    endpoint_id=my_index_endpoint.name,    embedding=embedding_model,    stream_update=True,)# Add vectors and mapped text chunks to your vectore storevector_store.add_texts(texts=texts)
```

### OPTIONAL : You can also create vectore and store chunks in a Datastore[​](#optional-you-can-also-create-vectore-and-store-chunks-in-a-datastore "Direct link to OPTIONAL : You can also create vectore and store chunks in a Datastore")

```
# NOTE : This operation can take upto 20 minsvector_store = VectorSearchVectorStoreDatastore.from_components(    project_id=PROJECT_ID,    region=REGION,    index_id=my_index.name,    endpoint_id=my_index_endpoint.name,    embedding=embedding_model,    stream_update=True,)vector_store.add_texts(texts=texts, is_complete_overwrite=True)
```

```
# Try running a simialarity searchvector_store.similarity_search("pizza")
```

### Create vectorstore with metadata filters[​](#create-vectorstore-with-metadata-filters "Direct link to Create vectorstore with metadata filters")

```
# Input text with metadatarecord_data = [    {        "description": "A versatile pair of dark-wash denim jeans."        "Made from durable cotton with a classic straight-leg cut, these jeans"        " transition easily from casual days to dressier occasions.",        "price": 65.00,        "color": "blue",        "season": ["fall", "winter", "spring"],    },    {        "description": "A lightweight linen button-down shirt in a crisp white."        " Perfect for keeping cool with breathable fabric and a relaxed fit.",        "price": 34.99,        "color": "white",        "season": ["summer", "spring"],    },    {        "description": "A soft, chunky knit sweater in a vibrant forest green. "        "The oversized fit and cozy wool blend make this ideal for staying warm "        "when the temperature drops.",        "price": 89.99,        "color": "green",        "season": ["fall", "winter"],    },    {        "description": "A classic crewneck t-shirt in a soft, heathered blue. "        "Made from comfortable cotton jersey, this t-shirt is a wardrobe essential "        "that works for every season.",        "price": 19.99,        "color": "blue",        "season": ["fall", "winter", "summer", "spring"],    },    {        "description": "A flowing midi-skirt in a delicate floral print. "        "Lightweight and airy, this skirt adds a touch of feminine style "        "to warmer days.",        "price": 45.00,        "color": "white",        "season": ["spring", "summer"],    },]
```

```
# Parse and prepare input datatexts = []metadatas = []for record in record_data:    record = record.copy()    page_content = record.pop("description")    texts.append(page_content)    if isinstance(page_content, str):        metadata = {**record}        metadatas.append(metadata)
```

```
# Inspect metadatasmetadatas
```

```
# NOTE : This operation can take more than 20 minsvector_store = VectorSearchVectorStore.from_components(    project_id=PROJECT_ID,    region=REGION,    gcs_bucket_name=BUCKET,    index_id=my_index.name,    endpoint_id=my_index_endpoint.name,    embedding=embedding_model,)vector_store.add_texts(texts=texts, metadatas=metadatas, is_complete_overwrite=True)
```

```
from google.cloud.aiplatform.matching_engine.matching_engine_index_endpoint import (    Namespace,    NumericNamespace,)
```

```
# Try running a simple similarity search# Below code should return 5 resultsvector_store.similarity_search("shirt", k=5)
```

```
# Try running a similarity search with text filterfilters = [Namespace(name="season", allow_tokens=["spring"])]# Below code should return 4 results nowvector_store.similarity_search("shirt", k=5, filter=filters)
```

```
# Try running a similarity search with combination of text and numeric filterfilters = [Namespace(name="season", allow_tokens=["spring"])]numeric_filters = [NumericNamespace(name="price", value_float=40.0, op="LESS")]# Below code should return 2 results nowvector_store.similarity_search(    "shirt", k=5, filter=filters, numeric_filter=numeric_filters)
```

### Use Vector Store as retriever[​](#use-vector-store-as-retriever "Direct link to Use Vector Store as retriever")

```
# Initialize the vectore_store as retrieverretriever = vector_store.as_retriever()
```

```
# perform simple similarity search on retrieverretriever.get_relevant_documents("What are my options in breathable fabric?")
```

```
# Try running a similarity search with text filterfilters = [Namespace(name="season", allow_tokens=["spring"])]retriever.search_kwargs = {"filter": filters}# perform similarity search with filters on retrieverretriever.get_relevant_documents("What are my options in breathable fabric?")
```

```
# Try running a similarity search with combination of text and numeric filterfilters = [Namespace(name="season", allow_tokens=["spring"])]numeric_filters = [NumericNamespace(name="price", value_float=40.0, op="LESS")]retriever.search_kwargs = {"filter": filters, "numeric_filter": numeric_filters}retriever.get_relevant_documents("What are my options in breathable fabric?")
```

### Use filters with retriever in Question Answering Chains[​](#use-filters-with-retriever-in-question-answering-chains "Direct link to Use filters with retriever in Question Answering Chains")

```
from langchain_google_vertexai import VertexAIllm = VertexAI(model_name="gemini-pro")
```

```
from langchain.chains import RetrievalQAfilters = [Namespace(name="season", allow_tokens=["spring"])]numeric_filters = [NumericNamespace(name="price", value_float=40.0, op="LESS")]retriever.search_kwargs = {"k": 2, "filter": filters, "numeric_filter": numeric_filters}retrieval_qa = RetrievalQA.from_chain_type(    llm=llm,    chain_type="stuff",    retriever=retriever,    return_source_documents=True,)question = "What are my options in breathable fabric?"response = retrieval_qa({"query": question})print(f"{response['result']}")print("REFERENCES")print(f"{response['source_documents']}")
```

## Read , Chunk , Vectorise and Index PDFs[​](#read-chunk-vectorise-and-index-pdfs "Direct link to Read , Chunk , Vectorise and Index PDFs")

```
from langchain_community.document_loaders import PyPDFLoaderfrom langchain_text_splitters import RecursiveCharacterTextSplitter
```

```
loader = PyPDFLoader("https://arxiv.org/pdf/1706.03762.pdf")pages = loader.load()
```

```
text_splitter = RecursiveCharacterTextSplitter(    # Set a really small chunk size, just to show.    chunk_size=1000,    chunk_overlap=20,    length_function=len,    is_separator_regex=False,)doc_splits = text_splitter.split_documents(pages)
```

```
texts = [doc.page_content for doc in doc_splits]metadatas = [doc.metadata for doc in doc_splits]
```

```
# Inspect Metadata of 1st pagemetadatas[0]
```

```
vector_store = VectorSearchVectorStore.from_components(    project_id=PROJECT_ID,    region=REGION,    gcs_bucket_name=BUCKET,    index_id=my_index.name,    endpoint_id=my_index_endpoint.name,    embedding=embedding_model,)vector_store.add_texts(texts=texts, metadatas=metadatas, is_complete_overwrite=True)
```

```
my_index = aiplatform.MatchingEngineIndex("5908955807575179264")my_index_endpoint = aiplatform.MatchingEngineIndexEndpoint("7751631742611488768")
```

```
vector_store = VectorSearchVectorStore.from_components(    project_id=PROJECT_ID,    region=REGION,    gcs_bucket_name=BUCKET,    index_id=my_index.name,    endpoint_id=my_index_endpoint.name,    embedding=embedding_model,)
```