Duplicate from awacke1/BigScienceContinualGenerator

Co-authored-by: Aaron C Wacker <awacke1@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,34 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,14 @@
+---
+title: BigScienceContinualGenerator
+emoji: 👀👀👀
+colorFrom: green
+colorTo: indigo
+sdk: gradio
+sdk_version: 3.18.0
+app_file: app.py
+pinned: false
+license: mit
+duplicated_from: awacke1/BigScienceContinualGenerator
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

app.py

@@ -0,0 +1,112 @@
+import gradio as gr
+
+#api = gr.Interface.load("huggingface/EleutherAI/gpt-j-6B")
+api = gr.Interface.load("models/bigscience/bloom")
+
+
+def complete_with_gpt(text):
+    # Use the last 50 characters of the text as context
+    return text[:-50] + api(text[-50:])
+
+
+with gr.Blocks() as demo:
+    with gr.Row():
+        textbox = gr.Textbox(placeholder="Type here and press enter...", lines=21)
+        with gr.Column():
+            btn = gr.Button("Generate")
+
+    btn.click(complete_with_gpt, textbox, textbox)
+
+    with gr.Row():
+        gr.Markdown("""
+# Big Science creates 176 Billion Parameter Large Language Model 
+
+## Bloom Is Setting New Record for Most Performant and Efficient AI Model for Science Ever!
+
+Bloom stands for:
+B: Big Science
+L: Large Language Model
+O: Open Science
+O: Open Access
+M: Multi Lingual Language Model
+
+1. Video Playlist to Check it out:  https://www.youtube.com/playlist?list=PLHgX2IExbFouqnsIqziThlPCX_miiDq14
+2. Summary of Important Models and Sizes:
+
+# Model Sizes to Date
+
+Model Name	|	Model Size (in Parameters)
+----------------|---------------------------------
+BigScience-tr11-176B|176 billion
+GPT-3|175 billion
+OpenAI's DALL-E 2.0|500 million
+NVIDIA's Megatron|8.3 billion
+Google's BERT|340 million
+GPT-2|1.5 billion
+OpenAI's GPT-1|117 million
+ELMo|90 million
+ULMFiT|100 million
+Transformer-XL|250 million
+XLNet|210 million
+RoBERTa|125 million
+ALBERT|12 million
+DistilBERT|66 million
+
+3. Background Information on ChatGPT, Bloom from BigScience on HuggingFace Platform, and RLHF DeepRL and One to Few Shot Learning and Generators:
+
+
+
+# ChatGPT Datasets:
+1. WebText
+2. Common Crawl
+3. BooksCorpus
+4. English Wikipedia
+5. Toronto Books Corpus
+6. OpenWebText
+
+# Comparison to BigScience Model:
+
+# Big Science - How to get started
+
+Big Science is a 176B parameter new ML model that was trained on a set of datasets for Natural Language processing, and many other tasks that are not yet explored..  Below is the set of the papers, models, links, and datasets around big science which promises to be the best, most recent large model of its kind benefitting all science pursuits.
+
+# Model: https://huggingface.co/bigscience/bloom
+
+# Papers: 
+1. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model https://arxiv.org/abs/2211.05100
+2. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism https://arxiv.org/abs/1909.08053
+3. 8-bit Optimizers via Block-wise Quantization https://arxiv.org/abs/2110.02861
+4. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation https://arxiv.org/abs/2108.12409
+5. https://huggingface.co/models?other=doi:10.57967/hf/0003
+6. 217 Other Models optimizing use of bloom via specialization: https://huggingface.co/models?other=bloom
+
+# Datasets
+1. Universal Dependencies: https://paperswithcode.com/dataset/universal-dependencies
+2. WMT 2014: https://paperswithcode.com/dataset/wmt-2014
+3. The Pile: https://paperswithcode.com/dataset/the-pile
+4. HumanEval: https://paperswithcode.com/dataset/humaneval
+5. FLORES-101: https://paperswithcode.com/dataset/flores-101
+6. CrowS-Pairs: https://paperswithcode.com/dataset/crows-pairs
+7. WikiLingua: https://paperswithcode.com/dataset/wikilingua
+8. MTEB: https://paperswithcode.com/dataset/mteb
+9. xP3: https://paperswithcode.com/dataset/xp3
+10. DiaBLa: https://paperswithcode.com/dataset/diabla
+
+# Deep RL ML Strategy
+
+1. Language Model Preparation, Human Augmented with Supervised Fine Tuning
+2. Reward Model Training with Prompts Dataset Multi-Model Generate Data to Rank
+3. Fine Tuning with Reinforcement Reward and Distance Distribution Regret Score
+4. Proximal Policy Optimization Fine Tuning
+
+# Variations - Preference Model Pretraining
+
+1. Use Ranking Datasets Sentiment - Thumbs Up/Down, Distribution
+2. Online Version Getting Feedback
+3. OpenAI - InstructGPT - Humans generate LM Training Text
+4. DeepMind - Advantage Actor Critic Sparrow, GopherCite
+5. Reward Model Human Prefence Feedback
+        
+        """)
+
+demo.launch()