Upload 4 files

README.md

@@ -0,0 +1,225 @@
+---
+license: apache-2.0
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+BLING-1b-0.1 is the first model release in the BLING ("Best Little Instruction-following No-GPU-required") model series.
+
+BLING models are designed as custom instruct-following laptop-effective GPT decoder-based models (~1B-2.7B parameters). BLING models are currently built on top of Pythia (GPTNeox architecture) base models and other Apache 2.0-licensed GPT-compatible models with primary focus on 'little' models in the range of 1B, 1.3-1.4B, and 2.7B parameters. (Note: in our testing, we have seen relatively limited success with instruct-following models below <1B parameters.)
+
+BLING models are fine-tuned with distilled high-quality custom instruct datasets, targeted at a specific subset of instruct tasks with the objective of providing a high-quality Instruct model that can be run entirely without a GPU server, with good quality instruct-following capability that can be loaded and run locally on a laptop.
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Developed by:** llmware
+- **Shared by [optional]:** Darren Oberst
+- **Model type:** GPTNeoX instruct-trained decoder
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Finetuned from model [optional]:** EleutherAI/Pythia-1b-deduped
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+The intended use of BLING models is two-fold:
+
+1.  Provide a high-quality Instruct models that can run on a laptop for local testing.  We have found it extremely useful when building a
+   proof-of-concept, or working with sensitive enterprise data that must be closely guarded, especially in RAG use cases.
+
+2.  Push the state of the art for smaller Instruct-following models in the 1B - 7B range.
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+BLING is designed for enterprise automation use cases, especially in knowledge-intensive industries, such as financial services,
+legal and regulatory industries.   BLING is intended to be an experimental series of little instruct models targeted as specific
+RAG automation tasks with complex information sources.   Rather than try to be "all things to all people," BLING models try to focus
+on a narrower set of Instructions more suitable to a ~1B parameter GPT model.
+
+BLING is ideal for rapid prototyping, testing, and the ability to perform an end-to-end workflow locally on a laptop without
+having to send sensitive information over an Internet-based API.
+
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+1.  BLING is not designed for 'chat-bot' or 'consumer-oriented' applications.  
+
+2.  BLING is not optimal for most production applications, other than simple and highly specific use cases.
+
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+BLING has not been designed for end consumer-oriented applications, and there has been any focus in training on important safeguards to
+mitigate potential bias and safety.    We would strongly discourage any use of BLING for any 'chatbot' use case.
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure 
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Data Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+
+

config.json

@@ -0,0 +1,83 @@
+{ "model_name": "bling-1.4b-0.1",
+  "description": "Instruct train fine-tuning using distilled knowledge based critical reading tasks training dataset",
+  "training_timestamp": "Mon Sep 25 20:31:36 2023",
+  "training_comments": "pythia-1.4b-v1.0",
+  "_name_or_path": "EleutherAI/pythia-1.4b-v0",
+  "transformers_version": "4.28.1",
+  "model_type": "gpt_neox",
+  "vocab_size": 50304,
+  "max_position_embeddings": 2048,
+  "hidden_size": 2048,
+  "num_hidden_layers": 24,
+  "num_attention_heads": 16,
+  "intermediate_size": 8192,
+  "hidden_act": "gelu",
+  "rotary_pct": 0.25,
+  "rotary_emb_base": 10000,
+  "initializer_range": 0.02,
+  "layer_norm_eps": 1e-05,
+  "return_dict": true,
+  "output_hidden_states": false,
+  "output_attentions": false,
+  "torchscript": false,
+  "torch_dtype": "float16",
+  "use_bfloat16": false,
+  "tf_legacy_loss": false,
+  "pruned_heads": {},
+  "tie_word_embeddings": false,
+  "is_encoder_decoder": false,
+  "is_decoder": false,
+  "cross_attention_hidden_size": null,
+  "add_cross_attention": false,
+  "tie_encoder_decoder": false,
+  "max_length": 20,
+  "min_length": 0,
+  "do_sample": false,
+  "early_stopping": false,
+  "num_beams": 1,
+  "num_beam_groups": 1,
+  "diversity_penalty": 0.0,
+  "temperature": 1.0,
+  "top_k": 50,
+  "top_p": 1.0,
+  "typical_p": 1.0,
+  "repetition_penalty": 1.0,
+  "length_penalty": 1.0,
+  "no_repeat_ngram_size": 0,
+  "encoder_no_repeat_ngram_size": 0,
+  "bad_words_ids": null,
+  "num_return_sequences": 1,
+  "chunk_size_feed_forward": 0,
+  "output_scores": false,
+  "return_dict_in_generate": false,
+  "forced_bos_token_id": null,
+  "forced_eos_token_id": null,
+  "remove_invalid_values": false,
+  "exponential_decay_length_penalty": null,
+  "suppress_tokens": null,
+  "begin_suppress_tokens": null,
+  "architectures": [
+    "GPTNeoXForCausalLM"
+  ],
+  "finetuning_task": null,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "tokenizer_class": null,
+  "prefix": null,
+  "bos_token_id": 0,
+  "pad_token_id": null,
+  "eos_token_id": 0,
+  "sep_token_id": null,
+  "decoder_start_token_id": null,
+  "task_specific_params": null,
+  "problem_type": null,
+  "use_cache": true,
+  "use_parallel_residual": true,
+  "trained": "custom training"
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3795a5dc929c15d452aff5153107149be43e16f25d23623c8a958287fb8f6ba6
+size 5759395105