poisson-fish/Marcoroni-70B-v1-AWQ

This is AIDC-ai-business/Marcoroni-70B-v1 quantized to LMDeploy 4bit AWQ with the following config:

python3 -m lmdeploy.lite.apis.auto_awq \
  --model ./Marcoroni-70B-v1 \
  --w_bits 4 \
  --w_group_size 128 \
  --work_dir ./quant

Original Model Card:

Marcoroni-70B

Model Details

• Trained by: trained by AIDC AI-Business.
• Model type: Marcoroni-70B is an auto-regressive language model based on the Llama 2 transformer architecture.
• Language(s): English
• License for Marcoroni-70B base weights: Non-Commercial Creative Commons license (CC BY-NC-4.0)

Prompting

Prompt Template for alpaca style

### Instruction:

<prompt> (without the <>)

### Response: