File size: 2,728 Bytes
b63b892 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
language:
- en
pipeline_tag: text-generation
tags:
- OpenVINO
- meta
- llama
- llama-3
- optimum-intel
license: llama3
extra_gated_prompt: |
Meta Llama 3 Version Release Date: April 18, 2024
library_name: transformers
---
# Llama-3-70B-Instruct-ov-fp16-int4-sym
## Built with Meta Llama 3
## Model Description
This is a version of the original [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) model converted to [OpenVINO™](https://github.com/openvinotoolkit/openvino) IR (Intermediate Representation) format for optimized inference on Intel® hardware. The model is created using the examples shown in [OpenVINO™ Notebooks](https://github.com/openvinotoolkit/openvino_notebooks/tree/latest/notebooks) repository.
## Intended Use
This model is designed for advanced natural language understanding and generation tasks, ideal for academic researchers and developers in commercial settings looking to integrate efficient AI capabilities into their applications. It is not to be used for creating or promoting harmful or illegal content as per the guidelines outlined in the [Meta Llama 3 Acceptable Use Policy](https://llama.meta.com/llama3/use-policy/).
## Licensing and Redistribution
This model is released under the Meta Llama 3 Community License. Redistribution requires inclusion of this license and a citation to the original model. Modifications and derivative works must prominently display "Built with Meta Llama 3" and adhere to the redistribution policies detailed in the original model [license terms](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct/blob/main/LICENSE).
## Weight Compression Parameters
For more information on the parameters, refer to the [OpenVINO™ 2024.1.0 documentation](https://docs.openvino.ai/2024/openvino-workflow/model-optimization-guide/weight-compression.html)
* mode: **INT4_ASYM**
* group_size: **128**
* ratio: **0.8**
## Running Model Inference
Install packages required for using [Optimum Intel](https://huggingface.co/docs/optimum/intel/index) integration with the OpenVINO™ backend:
```sh
pip install --upgrade --upgrade-strategy eager "optimum[openvino]"
```
Run model inference:
```python
from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer
model_id = "nsbendre25/Llama-3-70B-Instruct-ov_fp16-int4_sym"
# Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)
pipeline = transformers.pipeline("text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto")
pipeline("i am in paris, plan me a 2 week trip")
``` |