## Google Cloud Vertex AI

**Note:** This is separate from the `Google Generative AI` integration, it exposes [Vertex AI Generative API](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overview) on `Google Cloud`.

VertexAI exposes all foundational models available in google cloud: - Gemini (`gemini-pro` and `gemini-pro-vision`) - Palm 2 for Text (`text-bison`) - Codey for Code Generation (`code-bison`)

For a full and updated list of available models visit [VertexAI documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/overview)

## Setup[​](#setup "Direct link to Setup")

By default, Google Cloud [does not use](https://cloud.google.com/vertex-ai/docs/generative-ai/data-governance#foundation_model_development) customer data to train its foundation models as part of Google Cloud’s AI/ML Privacy Commitment. More details about how Google processes data can also be found in [Google’s Customer Data Processing Addendum (CDPA)](https://cloud.google.com/terms/data-processing-addendum).

To use `Vertex AI Generative AI` you must have the `langchain-google-vertexai` Python package installed and either: - Have credentials configured for your environment (gcloud, workload identity, etc…) - Store the path to a service account JSON file as the GOOGLE\_APPLICATION\_CREDENTIALS environment variable

This codebase uses the `google.auth` library which first looks for the application credentials variable mentioned above, and then looks for system-level auth.

For more information, see: - [https://cloud.google.com/docs/authentication/application-default-credentials#GAC](https://cloud.google.com/docs/authentication/application-default-credentials#GAC) - [https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth](https://googleapis.dev/python/google-auth/latest/reference/google.auth.html#module-google.auth)

```
%pip install --upgrade --quiet  langchain-core langchain-google-vertexai
```

```
[notice] A new release of pip is available: 23.2.1 -> 23.3.2[notice] To update, run: pip install --upgrade pipNote: you may need to restart the kernel to use updated packages.
```

## Usage[​](#usage "Direct link to Usage")

VertexAI supports all [LLM](https://python.langchain.com/docs/modules/model_io/llms/) functionality.

```
from langchain_google_vertexai import VertexAI
```

```
model = VertexAI(model_name="gemini-pro")
```

```
message = "What are some of the pros and cons of Python as a programming language?"model.invoke(message)
```

```
'**Pros:**\n\n* **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.\n* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.\n* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.\n* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.\n* **Cross-platform:** Python is available for a'
```

```
await model.ainvoke(message)
```

```
'**Pros:**\n\n* **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.\n* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.\n* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.\n* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.\n* **Cross-platform:** Python is available for a'
```

```
for chunk in model.stream(message):    print(chunk, end="", flush=True)
```

```
**Pros:*** **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.* **Cross-platform:** Python is available for a
```

```
['**Pros:**\n\n* **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.\n* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.\n* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.\n* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.\n* **Cross-platform:** Python is available for a']
```

We can use the `generate` method to get back extra metadata like [safety attributes](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/responsible-ai#safety_attribute_confidence_scoring) and not just text completions.

```
result = model.generate([message])result.generations
```

```
[[GenerationChunk(text='**Pros:**\n\n* **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.\n* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.\n* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.\n* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.\n* **Cross-platform:** Python is available for a')]]
```

```
result = await model.agenerate([message])result.generations
```

```
[[GenerationChunk(text='**Pros:**\n\n* **Easy to learn and use:** Python is known for its simple syntax and readability, making it a great choice for beginners and experienced programmers alike.\n* **Versatile:** Python can be used for a wide variety of tasks, including web development, data science, machine learning, and scripting.\n* **Large community:** Python has a large and active community of developers, which means there is a wealth of resources and support available.\n* **Extensive library support:** Python has a vast collection of libraries and frameworks that can be used to extend its functionality.\n* **Cross-platform:** Python is available for a')]]
```

You can also easily combine with a prompt template for easy structuring of user input. We can do this using [LCEL](https://python.langchain.com/docs/expression_language/)

```
from langchain_core.prompts import PromptTemplatetemplate = """Question: {question}Answer: Let's think step by step."""prompt = PromptTemplate.from_template(template)chain = prompt | modelquestion = """I have five apples. I throw two away. I eat one. How many apples do I have left?"""print(chain.invoke({"question": question}))
```

```
1. You start with 5 apples.2. You throw away 2 apples, so you have 5 - 2 = 3 apples left.3. You eat 1 apple, so you have 3 - 1 = 2 apples left.Therefore, you have 2 apples left.
```

You can use different foundational models for specialized in different tasks. For an updated list of available models visit [VertexAI documentation](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/overview)

```
llm = VertexAI(model_name="code-bison", max_output_tokens=1000, temperature=0.3)question = "Write a python function that checks if a string is a valid email address"print(model.invoke(question))
```

````
```pythonimport redef is_valid_email(email):  """  Checks if a string is a valid email address.  Args:    email: The string to check.  Returns:    True if the string is a valid email address, False otherwise.  """  # Compile the regular expression for an email address.  regex = re.compile(r"[^@]+@[^@]+\.[^@]+")  # Check if the string matches the regular expression.  return regex.match(email) is not None```
````

## Multimodality[​](#multimodality "Direct link to Multimodality")

With Gemini, you can use LLM in a multimodal mode:

```
from langchain_core.messages import HumanMessagefrom langchain_google_vertexai import ChatVertexAIllm = ChatVertexAI(model_name="gemini-pro-vision")image_message = {    "type": "image_url",    "image_url": {"url": "image_example.jpg"},}text_message = {    "type": "text",    "text": "What is shown in this image?",}message = HumanMessage(content=[text_message, image_message])output = llm([message])print(output.content)
```

```
 This is a Yorkshire Terrier.
```

Let’s double-check it’s a cat :)

```
from vertexai.preview.generative_models import Imagei = Image.load_from_file("image_example.jpg")i
```

![](https://python.langchain.com/assets/images/cell-14-output-1-0c7fb8b94ff032d51bfe1880d8370104.png)

You can also pass images as bytes:

```
import base64with open("image_example.jpg", "rb") as image_file:    image_bytes = image_file.read()image_message = {    "type": "image_url",    "image_url": {        "url": f"data:image/jpeg;base64,{base64.b64encode(image_bytes).decode('utf-8')}"    },}text_message = {    "type": "text",    "text": "What is shown in this image?",}message = HumanMessage(content=[text_message, image_message])output = llm([message])print(output.content)
```

```
 This is a Yorkshire Terrier.
```

Please, note that you can also use the image stored in GCS (just point the `url` to the full GCS path, starting with `gs://` instead of a local one).

And you can also pass a history of a previous chat to the LLM:

```
message2 = HumanMessage(content="And where the image is taken?")output2 = llm([message, output, message2])print(output2.content)
```

You can also use the public image URL:

```
image_message = {    "type": "image_url",    "image_url": {        "url": "https://python.langchain.com/assets/images/cell-18-output-1-0c7fb8b94ff032d51bfe1880d8370104.png",    },}text_message = {    "type": "text",    "text": "What is shown in this image?",}message = HumanMessage(content=[text_message, image_message])output = llm([message])print(output.content)
```

## Vertex Model Garden[​](#vertex-model-garden "Direct link to Vertex Model Garden")

Vertex Model Garden [exposes](https://cloud.google.com/vertex-ai/docs/start/explore-models) open-sourced models that can be deployed and served on Vertex AI. If you have successfully deployed a model from Vertex Model Garden, you can find a corresponding Vertex AI [endpoint](https://cloud.google.com/vertex-ai/docs/general/deployment#what_happens_when_you_deploy_a_model) in the console or via API.

```
from langchain_google_vertexai import VertexAIModelGarden
```

```
llm = VertexAIModelGarden(project="YOUR PROJECT", endpoint_id="YOUR ENDPOINT_ID")
```

```
llm.invoke("What is the meaning of life?")
```

Like all LLMs, we can then compose it with other components:

```
prompt = PromptTemplate.from_template("What is the meaning of {thing}?")
```

```
chain = prompt | llmprint(chain.invoke({"thing": "life"}))
```