xDAN2099 commited on
Commit
9cc1923
1 Parent(s): 7386fd7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -14
README.md CHANGED
@@ -2,27 +2,22 @@
2
  license: apache-2.0
3
  ---
4
 
5
-
6
-
7
- APUS-xDAN-4.0-MOE
8
-
9
  Introduction
10
- APUS-xDAN-4.0-MOE is a transformer-based MoE decoder-only language model 对齐在 on a large amount of data.
11
 
12
- For more details, please refer to our blog post and GitHub repo.
 
 
13
 
14
  Model Details
15
- APUS-xDAN-4.0-MOE employs Mixture of Experts (MoE) architecture, where the models are upcycled from dense language models. For instance, APUS-xDAN-4.0-MOE is upcycled from xDAN-L2 Series which are high performance alignModels. It has 136B parameters in total and 30B activated parameters during runtime.
16
- 进过先进量化技术优化,我们的开源版本仅仅只有42GB大小,是可以很好的在消费级显卡例如4090,3090上运行。
17
 
 
18
 
19
  Requirements
20
- The code of APUS-xDAN-4.0-MOE has been in the latest Hugging face transformers and we advise you to build from source with command pip install git+https://github.com/huggingface/transformers, or you might encounter the following error:
21
-
22
-
23
- Usage
24
- llama.cpp
25
 
 
26
 
 
 
27
  License
28
- APUS-xDAN-4.0-MOE is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
 
 
2
  license: apache-2.0
3
  ---
4
 
 
 
 
 
5
  Introduction
 
6
 
7
+ APUS-xDAN-4.0-MOE is a transformer-based decoder-only language model, developed on a vast corpus of data to ensure robust performance.
8
+
9
+ For more comprehensive information, please visit our blog post and GitHub repository.
10
 
11
  Model Details
 
 
12
 
13
+ APUS-xDAN-4.0-MOE leverages the innovative Mixture of Experts (MoE) architecture, incorporating components from dense language models. Specifically, it inherits its capabilities from the highly performant xDAN-L2 Series. With a total of 136 billion parameters, of which 30 billion are activated during runtime, APUS-xDAN-4.0-MOE demonstrates unparalleled efficiency. Through advanced quantization techniques, our open-source version occupies a mere 42GB, making it seamlessly compatible with consumer-grade GPUs like the 4090 and 3090.
14
 
15
  Requirements
 
 
 
 
 
16
 
17
+ The codebase for APUS-xDAN-4.0-MOE is integrated into the latest Hugging Face transformers library. We recommend building from source using the command pip install git+https://github.com/huggingface/transformers to ensure compatibility. Failure to do so may result in encountering the following error:
18
 
19
+ Copy code
20
+ Usage llama.cpp
21
  License
22
+
23
+ APUS-xDAN-4.0-MOE is distributed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.