Upload folder using huggingface_hub

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,98 @@
+---
+license: mit
+tags:
+  - onnx
+  - int8
+  - quantized
+  - code
+  - embeddings
+  - justembed
+base_model: microsoft/codebert-base
+library_name: onnxruntime
+pipeline_tag: feature-extraction
+---
+
+# CodeBERT INT8 — ONNX Quantized
+
+ONNX INT8 quantized version of [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) for efficient code and natural language embeddings.
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| Base Model | [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base) |
+| Format | ONNX |
+| Quantization | INT8 (dynamic quantization) |
+| Embedding Dimension | 768 |
+| Quantized by | [JustEmbed](https://pypi.org/project/justembed/) |
+
+## What is this?
+
+This is a quantized ONNX export of CodeBERT, a bimodal pre-trained model for programming and natural language by Microsoft Research. The INT8 quantization reduces model size and improves inference speed while maintaining high accuracy for code-related embeddings.
+
+CodeBERT is trained on both natural language and programming language data (Python, Java, JavaScript, PHP, Ruby, Go).
+
+## Use Cases
+
+- Code search and retrieval
+- Code documentation matching
+- Programming language embeddings
+- Code similarity detection
+- Natural language to code matching
+
+## Files
+
+- `model_quantized.onnx` — INT8 quantized ONNX model
+- `tokenizer.json` — Fast tokenizer
+- `config.json` — Model configuration
+
+## Usage with JustEmbed
+
+```python
+from justembed import Embedder
+
+embedder = Embedder("codebert-int8")
+vectors = embedder.embed(["def sort_list(arr): return sorted(arr)"])
+```
+
+## Usage with ONNX Runtime
+
+```python
+import onnxruntime as ort
+from transformers import AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained(".")
+session = ort.InferenceSession("model_quantized.onnx")
+
+inputs = tokenizer("def sort_list(arr): return sorted(arr)", return_tensors="np")
+outputs = session.run(None, dict(inputs))
+```
+
+## Quantization Details
+
+- Method: Dynamic INT8 quantization via ONNX Runtime
+- Source: Original PyTorch weights converted to ONNX, then quantized
+- Speed: ~2-3x faster inference than FP32
+- Size: ~4x smaller than FP32
+
+## License
+
+This model is a derivative work of [microsoft/codebert-base](https://huggingface.co/microsoft/codebert-base).
+
+The original model is licensed under **MIT License**. This quantized version is distributed under the same license. See the [LICENSE](LICENSE) file for the full text.
+
+## Citation
+
+```bibtex
+@inproceedings{feng2020codebert,
+  title={CodeBERT: A Pre-Trained Model for Programming and Natural Languages},
+  author={Feng, Zhangyin and Guo, Daya and Tang, Duyu and Duan, Nan and Feng, Xiaocheng and Gong, Ming and Shou, Linjun and Qin, Bing and Liu, Ting and Jiang, Daxin and Zhou, Ming},
+  booktitle={Findings of EMNLP},
+  year={2020}
+}
+```
+
+## Acknowledgments
+
+- Original model by [Microsoft Research](https://github.com/microsoft/CodeBERT)
+- Quantization and packaging by [JustEmbed](https://pypi.org/project/justembed/)

config.json

@@ -0,0 +1,22 @@
+{
+  "architectures": [
+    "RobertaModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "type_vocab_size": 1,
+  "vocab_size": 50265
+}

manifest.json

@@ -0,0 +1,17 @@
+{
+  "schema_version": "1.0",
+  "name": "codebert-int8",
+  "version": "1.0.1",
+  "type": "single",
+  "model_class": "BERTEmbedder",
+  "embedding_dim": 768,
+  "min_justembed_version": "0.1.1a9",
+  "description": "Code embeddings for programming and software documentation (INT8 quantized)",
+  "files": null,
+  "metadata": null,
+  "stages": null,
+  "max_sequence_length": 512,
+  "license": "MIT",
+  "source": "microsoft/codebert-base",
+  "quantization": "int8"
+}

model_quantized.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:149eb7ed0e3b5b1cf6a7e0194c56a1b83ba104baecb746b40a6b6c0c4ba1b061
+size 126315271

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "RobertaTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}