How to use OpenVINO for inference

🤗 Optimum provides Stable Diffusion pipelines compatible with OpenVINO. You can now easily perform inference with OpenVINO Runtime on a variety of Intel processors (see the full list of supported devices).

Installation

Install 🤗 Optimum Intel with the following command:

pip install --upgrade-strategy eager optimum["openvino"]

The --upgrade-strategy eager option is needed to ensure optimum-intel is upgraded to its latest version.

Stable Diffusion

Inference

To load an OpenVINO model and run inference with OpenVINO Runtime, you need to replace StableDiffusionPipeline with OVStableDiffusionPipeline. In case you want to load a PyTorch model and convert it to the OpenVINO format on-the-fly, you can set export=True.

from optimum.intel import OVStableDiffusionPipeline

model_id = "runwayml/stable-diffusion-v1-5"
pipeline = OVStableDiffusionPipeline.from_pretrained(model_id, export=True)
prompt = "sailing ship in storm by Rembrandt"
image = pipeline(prompt).images[0]

# Don't forget to save the exported model
pipeline.save_pretrained("openvino-sd-v1-5")

To further speed up inference, the model can be statically reshaped :

# Define the shapes related to the inputs and desired outputs
batch_size, num_images, height, width = 1, 1, 512, 512

# Statically reshape the model
pipeline.reshape(batch_size, height, width, num_images)
# Compile the model before inference
pipeline.compile()

image = pipeline(
    prompt,
    height=height,
    width=width,
    num_images_per_prompt=num_images,
).images[0]

In case you want to change any parameters such as the outputs height or width, you’ll need to statically reshape your model once again.

Supported tasks

You can find more examples in the optimum documentation.

Stable Diffusion XL

Inference

Here is an example of how you can load a SDXL OpenVINO model from stabilityai/stable-diffusion-xl-base-1.0 and run inference with OpenVINO Runtime :

from optimum.intel import OVStableDiffusionXLPipeline

model_id = "stabilityai/stable-diffusion-xl-base-1.0"
pipeline = OVStableDiffusionXLPipeline.from_pretrained(model_id)
prompt = "sailing ship in storm by Rembrandt"
image = pipeline(prompt).images[0]

To further speed up inference, the model can be statically reshaped as showed above. You can find more examples in the optimum documentation.

Supported tasks