Llama-3.1-8B-Instruct OpenVINO
model conversion for fast execution on Intel processors
2 min read
This model is a ready-to-use version of the popular Llama-3.1-8B-Instruct language model, converted to the OpenVINO format with INT8 weight quantization (low-precision integer format). The conversion was performed using Optimum Intel and OpenVINO tools, allowing the model to run efficiently on consumer processors and integrated graphics chips without requiring discrete graphics cards. The model maintains a 128k token context window and comes with a built-in chat template. It is useful for developers and enthusiasts who want to run modern text models locally on standard hardware with minimal VRAM requirements.
Key features
- OpenVINO optimization: model weights are quantized to INT8 to reduce memory consumption and accelerate CPU inference.
- Hardware compatibility: runs on standard Intel processors without powerful graphics accelerators.
- Long context: supports sequences up to 128k tokens long.
- Chat readiness: includes a built-in chat template for convenient interactive use.
What you can do
- Local CPU execution: deploy a language assistant on a standard laptop or desktop with an Intel processor.
- Python integration: use the model in scripts and applications via the optimum-intel library.
- C++ deployment: utilize the OpenVINO GenAI pipeline for fast integration into native projects.
Example
from optimum.intel import OVModelForCausalLM
from transformers import AutoTokenizer
model = OVModelForCausalLM.from_pretrained("Nekodeus/llama-3.1-8b-instruct-openvino")
tokenizer = AutoTokenizer.from_pretrained("Nekodeus/llama-3.1-8b-instruct-openvino")
messages = [{"role": "user", "content": "Explain photosynthesis in two sentences."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt")
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=64)[0]))Read more
Details• Base model: meta-llama/Meta-Llama-3.1-8B-Instruct (8B parameters)
• Precision: INT8 weights (NNCF), FP16/FP32 activations
• Conversion tools: optimum-intel 2.2.0, openvino 2026.4.1
• Structure: split into three components (openvino_model, openvino_tokenizer, openvino_detokenizer)










Comments