Hugging Face's logo

mobiuslabsgmbh
/
Mixtral-8x7B-Instruct-v0.1-hf-2bit_g16_s128-HQQ

Text Generation
Transformers
mixtral
Mixture of Experts
conversational
Model card Files Community
4
Mixtral-8x7B-Instruct-v0.1-hf-2bit_g16_s128-HQQ / README.md
mobicham's picture
mobicham
Librarian Bot: Add moe tag to model (#4)
a5176ba
preview code
|
raw
history blame
5.26 kB
metadata
license: apache-2.0
tags:
  - moe
train: false
inference: false
pipeline_tag: text-generation

Mixtral-8x7B-Instruct-v0.1-hf-2bit_g16_s128-HQQ

This is a version of the Mixtral-8x7B-Instruct-v0.1 model (https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1) quantized to 2-bit via Half-Quadratic Quantization (HQQ).

Basic Usage

To run the model, install the HQQ library from https://github.com/mobiusml/hqq and use it as follows:

model_id = 'mobiuslabsgmbh/Mixtral-8x7B-Instruct-v0.1-hf-2bit_g16_s128-HQQ'

#Load the model
from hqq.engine.hf import HQQModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = HQQModelForCausalLM.from_quantized(model_id)

#Optional
from hqq.core.quantize import *
HQQLinear.set_backend(HQQBackend.PYTORCH_COMPILE) 

#Text Generation
prompt = "<s> [INST] How do I build a car? [/INST] "

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
outputs = model.generate(**(inputs.to('cuda')), max_new_tokens=1000)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Output:

  1. Design and Conceptualization: The first step in building a car is to come up with a design and concept for the vehicle. This involves determining the vehicle's intended use, such as a passenger car, a sports car, a truck, or a bus. The design and concept will also include the vehicle's size, shape, and styling, as well as its performance, safety, and fuel efficiency characteristics.

  2. Chassis and Body: The chassis is the vehicle's framework, which is made of steel or aluminum and serves as the load-bearing structure of the vehicle. The body of the car is built around the chassis and is designed to enclose the vehicle's passengers and cargo. The body is usually made of steel, aluminum, or composite materials and is built using a combination of stampings, welds, and mechanical fasteners.

  3. Powertrain: The powertrain is the system that propels the vehicle and includes the engine, transmission, drive shaft, differential, and drive wheels. The engine is the heart of the powertrain and is designed to convert chemical energy into mechanical energy. The engine is connected to the transmission, which is responsible for transmitting the engine's power to the drive wheels.

  4. Suspension and Steering: The suspension system is designed to provide a comfortable ride and to keep the vehicle's tires in contact with the road. The suspension system is made up of a series of springs and shock absorbers that help to absorb road imperfections and keep the vehicle stable. The steering system is designed to allow the driver to control the direction of the vehicle and is usually located at the front of the vehicle.

  5. Electrical Systems: The electrical system is responsible for controlling the vehicle's electrical components, such as the lights, wipers, and starter. The electrical system is also responsible for managing the vehicle's power distribution and ensuring that the vehicle's systems are operating within their specified limits.

  6. Interior: The interior of the vehicle is designed to provide a comfortable and convenient space for the vehicle's passengers. The interior is usually made up of a combination of soft materials, such as leather or fabric, and hard materials, such as plastic or metal. The interior also includes a range of controls and displays, such as the speedometer, fuel gauge, and climate control systems.

  7. Assembly: Once the vehicle's components have been designed and built, they are assembled together at the vehicle manufacturer's assembly plant. The assembly process involves joining the various components together, such as the engine, transmission, and suspension system, and ensuring that they are properly aligned and connected. The vehicle is then subjected to a series of tests and inspections to ensure that it is operating within its specified limits.

  8. Testing and Inspection: Once the vehicle has been assembled, it is subjected to a series of tests and inspections to ensure that it is operating within its specified limits. These tests and inspections include a visual examination of the vehicle's components, a functional test of the vehicle's systems, and a road test to ensure that the vehicle is operating properly.

  9. Delivery: Once the vehicle has been fully assembled, tested, and inspected, it is delivered to the dealership where it will be sold to the end consumer. The dealership will then provide the vehicle with any necessary maintenance and repairs before it is delivered to the end consumer.

Building a car is a complex process that involves the integration of a wide range of components and systems. The vehicle manufacturer must ensure that the vehicle is designed and built to meet the end consumer's needs and expectations while also complying with all relevant safety and environmental regulations. The vehicle manufacturer must also ensure that the vehicle is built to the highest quality standards and that it is delivered to the end consumer in a timely and efficient manner.