rajatsainisim commited on
Commit
68afbf1
·
2 Parent(s): 97381e8 3861478

🚀 Complete Dwrko-M1.0 setup with HuggingFace upload tools

Browse files

- Added professional upload script (upload_to_hf.py)
- Added one-click upload script (quick_upload.sh)
- Added complete Hindi guide (HUGGINGFACE_UPLOAD.md)
- Updated QUICKSTART.md with upload instructions
- Resolved merge conflicts and integrated with HuggingFace Space
- Ready for Claude-like AI assistant training and deployment

Files changed (3) hide show
  1. HUGGINGFACE_UPLOAD.md +281 -0
  2. QUICKSTART.md +55 -1
  3. quick_upload.sh +95 -0
HUGGINGFACE_UPLOAD.md ADDED
@@ -0,0 +1,281 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 🚀 HuggingFace पर Dwrko-M1.0 Upload करना
2
+
3
+ **Complete Step-by-Step Guide** - हिंदी में समझाया गया!
4
+
5
+ ## 📋 Pre-Requirements (पहले से चाहिए)
6
+
7
+ 1. **HuggingFace Account** - [huggingface.co](https://huggingface.co) पर account बनाएं
8
+ 2. **Trained Model** - आपका `dwrko-m1.0` model ready हो
9
+ 3. **HuggingFace Token** - Access token generate करें
10
+
11
+ ---
12
+
13
+ ## 🔑 Step 1: HuggingFace Token बनाना
14
+
15
+ ### 1.1 HuggingFace पर Login करें
16
+ ```
17
+ https://huggingface.co/login
18
+ ```
19
+
20
+ ### 1.2 Token Generate करें
21
+ 1. Profile → Settings → Access Tokens पर जाएं
22
+ 2. "New token" पर click करें
23
+ 3. Token name: `Dwrko-M1.0-Upload`
24
+ 4. Type: **Write** select करें
25
+ 5. "Generate" पर click करें
26
+ 6. Token को copy करके safe रखें ⚠️
27
+
28
+ ---
29
+
30
+ ## 🛠️ Step 2: Dependencies Install करना
31
+
32
+ ### Virtual Environment बनाएं (Recommended):
33
+ ```bash
34
+ # Virtual environment बनाएं
35
+ python3 -m venv dwrko_env
36
+
37
+ # Activate करें
38
+ source dwrko_env/bin/activate # Linux/Mac
39
+ # या
40
+ dwrko_env\Scripts\activate # Windows
41
+
42
+ # Dependencies install करें
43
+ pip install huggingface_hub transformers peft torch
44
+ ```
45
+
46
+ ### System-wide Install (Alternative):
47
+ ```bash
48
+ pip install --user huggingface_hub transformers peft torch
49
+ ```
50
+
51
+ ---
52
+
53
+ ## 🚀 Step 3: Model Upload करना
54
+
55
+ ### Method 1: Automated Script (आसान तरीका)
56
+
57
+ ```bash
58
+ # Upload script run करें
59
+ python upload_to_hf.py \
60
+ --model_path ./dwrko-m1.0 \
61
+ --username YOUR_USERNAME \
62
+ --repo_name Dwrko-M1.0
63
+
64
+ # Example:
65
+ python upload_to_hf.py \
66
+ --model_path ./dwrko-m1.0 \
67
+ --username rajat123 \
68
+ --repo_name Dwrko-M1.0
69
+ ```
70
+
71
+ ### Method 2: Manual Commands
72
+
73
+ ```bash
74
+ # 1. Login करें
75
+ huggingface-cli login
76
+ # Token paste करें जब पूछे
77
+
78
+ # 2. Repository create करें
79
+ huggingface-cli repo create Dwrko-M1.0 --type model
80
+
81
+ # 3. Model upload करें
82
+ huggingface-cli upload Dwrko-M1.0/ ./dwrko-m1.0 --repo-type model
83
+ ```
84
+
85
+ ### Method 3: Python Script से
86
+
87
+ ```python
88
+ from huggingface_hub import HfApi, login
89
+
90
+ # Login करें
91
+ login(token="YOUR_TOKEN_HERE")
92
+
93
+ # Upload करें
94
+ api = HfApi()
95
+ api.upload_folder(
96
+ folder_path="./dwrko-m1.0",
97
+ repo_id="your-username/Dwrko-M1.0",
98
+ repo_type="model"
99
+ )
100
+ ```
101
+
102
+ ---
103
+
104
+ ## 📁 Step 4: Model Files Check करना
105
+
106
+ Upload करने से पहले ये files होनी चाहिए:
107
+
108
+ ```
109
+ dwrko-m1.0/
110
+ ├── adapter_config.json ✅ LoRA configuration
111
+ ├── adapter_model.safetensors ✅ Trained weights
112
+ ├── tokenizer_config.json ✅ Tokenizer settings
113
+ ├── special_tokens_map.json ✅ Special tokens
114
+ ├── model_info.json ✅ Model information
115
+ └── README.md ✅ Model card (auto-created)
116
+ ```
117
+
118
+ ### Files Check करने के लिए:
119
+ ```bash
120
+ ls -la ./dwrko-m1.0/
121
+ ```
122
+
123
+ ---
124
+
125
+ ## 🎯 Step 5: Upload Process
126
+
127
+ ### Complete Upload Command:
128
+ ```bash
129
+ python upload_to_hf.py \
130
+ --model_path ./dwrko-m1.0 \
131
+ --username YOUR_USERNAME \
132
+ --repo_name Dwrko-M1.0 \
133
+ --token YOUR_TOKEN
134
+ ```
135
+
136
+ ### Upload होने के बाद:
137
+ ```
138
+ 🚀 Uploading Dwrko-M1.0 to HuggingFace Hub...
139
+ ✅ Repository created/updated: https://huggingface.co/YOUR_USERNAME/Dwrko-M1.0
140
+ ✅ Model card created
141
+ 🎉 Successfully uploaded Dwrko-M1.0 to HuggingFace!
142
+ 🔗 Model URL: https://huggingface.co/YOUR_USERNAME/Dwrko-M1.0
143
+ ```
144
+
145
+ ---
146
+
147
+ ## 🔧 Troubleshooting (समस्या समाधान)
148
+
149
+ ### ❌ "externally-managed-environment" Error
150
+ ```bash
151
+ # Solution 1: Virtual environment use करें
152
+ python3 -m venv dwrko_env
153
+ source dwrko_env/bin/activate
154
+ pip install huggingface_hub
155
+
156
+ # Solution 2: User install
157
+ pip install --user huggingface_hub
158
+
159
+ # Solution 3: System packages (Ubuntu/Debian)
160
+ sudo apt install python3-huggingface-hub
161
+ ```
162
+
163
+ ### ❌ "Authentication Error"
164
+ ```bash
165
+ # Token re-generate करें
166
+ # Profile → Settings → Access Tokens
167
+ # Old token delete करके new बनाएं
168
+
169
+ # Login फिर से करें
170
+ huggingface-cli login
171
+ ```
172
+
173
+ ### ❌ "Repository already exists"
174
+ ```bash
175
+ # Force update करें
176
+ python upload_to_hf.py \
177
+ --model_path ./dwrko-m1.0 \
178
+ --username YOUR_USERNAME \
179
+ --repo_name Dwrko-M1.0-v2 # नया नाम दें
180
+ ```
181
+
182
+ ### ❌ "Large file upload failed"
183
+ ```bash
184
+ # Git LFS enable करें
185
+ git lfs install
186
+
187
+ # बड़ी files के लिए:
188
+ huggingface-cli lfs-enable-largefiles ./dwrko-m1.0
189
+ ```
190
+
191
+ ---
192
+
193
+ ## 📊 Post-Upload Checklist
194
+
195
+ Upload के बाद ये check करें:
196
+
197
+ ### ✅ Model Page पर जाकर:
198
+ 1. **Model Card** properly display हो रहा है
199
+ 2. **Files** सभी upload हुई हैं
200
+ 3. **Usage Examples** काम कर रहे हैं
201
+ 4. **Tags** properly set हैं
202
+
203
+ ### ✅ Test करें:
204
+ ```python
205
+ from transformers import pipeline
206
+
207
+ # Your uploaded model test करें
208
+ generator = pipeline(
209
+ "text-generation",
210
+ model="YOUR_USERNAME/Dwrko-M1.0",
211
+ tokenizer="mistralai/Mistral-7B-v0.1"
212
+ )
213
+
214
+ response = generator("### Instruction:\nWrite a hello world program\n\n### Response:\n")
215
+ print(response[0]['generated_text'])
216
+ ```
217
+
218
+ ---
219
+
220
+ ## 🌟 Advanced Options
221
+
222
+ ### Private Repository बनाना:
223
+ ```bash
224
+ python upload_to_hf.py \
225
+ --model_path ./dwrko-m1.0 \
226
+ --username YOUR_USERNAME \
227
+ --repo_name Dwrko-M1.0 \
228
+ --private
229
+ ```
230
+
231
+ ### Custom Model Card:
232
+ ```bash
233
+ # अपना README.md बनाएं
234
+ echo "# My Custom Dwrko-M1.0" > ./dwrko-m1.0/README.md
235
+
236
+ # Upload करें
237
+ python upload_to_hf.py --model_path ./dwrko-m1.0 --username YOUR_USERNAME
238
+ ```
239
+
240
+ ### Multiple Versions:
241
+ ```bash
242
+ # Version tags के साथ
243
+ python upload_to_hf.py \
244
+ --model_path ./dwrko-m1.0 \
245
+ --username YOUR_USERNAME \
246
+ --repo_name Dwrko-M1.0-v1.1
247
+ ```
248
+
249
+ ---
250
+
251
+ ## 🎉 Success! अब आपका Model Live है!
252
+
253
+ ### Model URL:
254
+ ```
255
+ https://huggingface.co/YOUR_USERNAME/Dwrko-M1.0
256
+ ```
257
+
258
+ ### Share करें:
259
+ - Social media पर post करें
260
+ - Community में share करें
261
+ - Documentation में link add करें
262
+ - README में badge लगाएं
263
+
264
+ ### Badge Example:
265
+ ```markdown
266
+ [![HuggingFace](https://img.shields.io/badge/🤗%20HuggingFace-Dwrko--M1.0-blue)](https://huggingface.co/YOUR_USERNAME/Dwrko-M1.0)
267
+ ```
268
+
269
+ ---
270
+
271
+ ## 💡 Pro Tips
272
+
273
+ 1. **Model Name**: Descriptive और unique रखें
274
+ 2. **Tags**: Relevant tags add करें (`coding`, `reasoning`, `mistral`)
275
+ 3. **Documentation**: Complete usage examples दें
276
+ 4. **Version Control**: Regular updates करते रहें
277
+ 5. **Community**: Feedback के लिए discussions enable करें
278
+
279
+ ---
280
+
281
+ **🚀 Congratulations! आपका Dwrko-M1.0 अब HuggingFace पर live है!** 🎉
QUICKSTART.md CHANGED
@@ -33,6 +33,15 @@ python test_dwrko.py --model_path ./my-dwrko-m1.0 --test_suite
33
  python test_dwrko.py --model_path ./my-dwrko-m1.0 --interactive
34
  ```
35
 
 
 
 
 
 
 
 
 
 
36
  ## 🎯 Training Commands
37
 
38
  ### Basic Training
@@ -79,6 +88,36 @@ python test_dwrko.py --model_path ./dwrko-m1.0 --single_test "Write a Python fun
79
  python test_dwrko.py --model_path ./dwrko-m1.0 --interactive
80
  ```
81
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
82
  ## 📚 Data Format
83
 
84
  Your training data should be in JSONL format:
@@ -110,9 +149,23 @@ python train.py --fp16 True --gradient_checkpointing True
110
  python -c "import torch; torch.cuda.empty_cache()"
111
  ```
112
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  ## 🌟 Next Steps
114
 
115
- 1. **Upload to HuggingFace**: `huggingface-cli upload ./dwrko-m1.0/ your-username/Dwrko-M1.0`
116
  2. **Share with Community**: Post your results and get feedback
117
  3. **Improve Training**: Add more data and train longer
118
  4. **Deploy**: Use your model in production applications
@@ -123,6 +176,7 @@ python -c "import torch; torch.cuda.empty_cache()"
123
  - Use **wandb** to monitor training progress
124
  - Save checkpoints frequently during long training runs
125
  - Test your model on diverse tasks to ensure quality
 
126
  - Join our community for support and tips!
127
 
128
  ---
 
33
  python test_dwrko.py --model_path ./my-dwrko-m1.0 --interactive
34
  ```
35
 
36
+ ### 5. 🚀 Upload to HuggingFace
37
+ ```bash
38
+ # Super easy one-click upload!
39
+ ./quick_upload.sh
40
+
41
+ # Or manual upload
42
+ python upload_to_hf.py --model_path ./my-dwrko-m1.0 --username YOUR_USERNAME
43
+ ```
44
+
45
  ## 🎯 Training Commands
46
 
47
  ### Basic Training
 
88
  python test_dwrko.py --model_path ./dwrko-m1.0 --interactive
89
  ```
90
 
91
+ ### 6. 🌐 HuggingFace Upload Commands
92
+
93
+ #### Quick Upload (Automated)
94
+ ```bash
95
+ # One-click upload script
96
+ ./quick_upload.sh
97
+ ```
98
+
99
+ #### Manual Upload
100
+ ```bash
101
+ # Python script
102
+ python upload_to_hf.py \
103
+ --model_path ./dwrko-m1.0 \
104
+ --username YOUR_USERNAME \
105
+ --repo_name Dwrko-M1.0
106
+
107
+ # CLI method
108
+ huggingface-cli login
109
+ huggingface-cli upload ./dwrko-m1.0 YOUR_USERNAME/Dwrko-M1.0 --repo-type model
110
+ ```
111
+
112
+ #### Private Repository
113
+ ```bash
114
+ python upload_to_hf.py \
115
+ --model_path ./dwrko-m1.0 \
116
+ --username YOUR_USERNAME \
117
+ --repo_name Dwrko-M1.0 \
118
+ --private
119
+ ```
120
+
121
  ## 📚 Data Format
122
 
123
  Your training data should be in JSONL format:
 
149
  python -c "import torch; torch.cuda.empty_cache()"
150
  ```
151
 
152
+ ### HuggingFace Upload Issues?
153
+ ```bash
154
+ # Setup virtual environment
155
+ python3 -m venv dwrko_env
156
+ source dwrko_env/bin/activate
157
+ pip install huggingface_hub
158
+
159
+ # Re-login
160
+ huggingface-cli login
161
+
162
+ # Check files
163
+ ls -la ./dwrko-m1.0/
164
+ ```
165
+
166
  ## 🌟 Next Steps
167
 
168
+ 1. **Upload to HuggingFace**: `./quick_upload.sh` या `python upload_to_hf.py`
169
  2. **Share with Community**: Post your results and get feedback
170
  3. **Improve Training**: Add more data and train longer
171
  4. **Deploy**: Use your model in production applications
 
176
  - Use **wandb** to monitor training progress
177
  - Save checkpoints frequently during long training runs
178
  - Test your model on diverse tasks to ensure quality
179
+ - **Upload to HuggingFace** to share with the world! 🌍
180
  - Join our community for support and tips!
181
 
182
  ---
quick_upload.sh ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/bin/bash
2
+
3
+ # 🚀 Dwrko-M1.0 Quick Upload Script
4
+ # HuggingFace पर model upload करने के लिए
5
+
6
+ echo "🤖 Dwrko-M1.0 HuggingFace Upload Script"
7
+ echo "======================================"
8
+
9
+ # Check if virtual environment exists
10
+ if [ ! -d "../dwrko_env" ]; then
11
+ echo "📦 Creating virtual environment..."
12
+ python3 -m venv ../dwrko_env
13
+
14
+ echo "📥 Installing dependencies..."
15
+ source ../dwrko_env/bin/activate
16
+ pip install huggingface_hub transformers peft
17
+ else
18
+ echo "✅ Virtual environment found"
19
+ source ../dwrko_env/bin/activate
20
+ fi
21
+
22
+ # Get user inputs
23
+ echo ""
24
+ echo "📋 Upload Configuration:"
25
+ read -p "🏷️ HuggingFace Username: " USERNAME
26
+ read -p "📁 Model Path (default: ./dwrko-m1.0): " MODEL_PATH
27
+ MODEL_PATH=${MODEL_PATH:-./dwrko-m1.0}
28
+ read -p "🎯 Repository Name (default: Dwrko-M1.0): " REPO_NAME
29
+ REPO_NAME=${REPO_NAME:-Dwrko-M1.0}
30
+
31
+ echo ""
32
+ echo "🔍 Checking model files..."
33
+
34
+ # Check if model directory exists
35
+ if [ ! -d "$MODEL_PATH" ]; then
36
+ echo "❌ Error: Model directory not found: $MODEL_PATH"
37
+ echo "💡 Please train your model first using: python train.py"
38
+ exit 1
39
+ fi
40
+
41
+ # Check required files
42
+ REQUIRED_FILES=("adapter_config.json" "adapter_model.safetensors")
43
+ for file in "${REQUIRED_FILES[@]}"; do
44
+ if [ ! -f "$MODEL_PATH/$file" ]; then
45
+ echo "❌ Missing required file: $file"
46
+ echo "💡 Please complete training first"
47
+ exit 1
48
+ fi
49
+ done
50
+
51
+ echo "✅ All required files found"
52
+
53
+ # Login to HuggingFace
54
+ echo ""
55
+ echo "🔑 Logging into HuggingFace..."
56
+ echo "💡 You'll need to paste your HuggingFace token"
57
+ echo " Get it from: https://huggingface.co/settings/tokens"
58
+
59
+ huggingface-cli login
60
+
61
+ if [ $? -ne 0 ]; then
62
+ echo "❌ Login failed. Please check your token."
63
+ exit 1
64
+ fi
65
+
66
+ echo "✅ Login successful"
67
+
68
+ # Upload model
69
+ echo ""
70
+ echo "🚀 Uploading $REPO_NAME to HuggingFace..."
71
+ echo "📁 From: $MODEL_PATH"
72
+ echo "🎯 To: $USERNAME/$REPO_NAME"
73
+
74
+ # Use Python script for upload
75
+ python upload_to_hf.py \
76
+ --model_path "$MODEL_PATH" \
77
+ --username "$USERNAME" \
78
+ --repo_name "$REPO_NAME"
79
+
80
+ if [ $? -eq 0 ]; then
81
+ echo ""
82
+ echo "🎉 SUCCESS! Your model is now live!"
83
+ echo "🔗 URL: https://huggingface.co/$USERNAME/$REPO_NAME"
84
+ echo ""
85
+ echo "📤 Share your model:"
86
+ echo " • Social media पर post करें"
87
+ echo " • Community में share करें"
88
+ echo " • README में badge add करें"
89
+ echo ""
90
+ echo "Badge:"
91
+ echo "[![HuggingFace](https://img.shields.io/badge/🤗%20HuggingFace-$REPO_NAME-blue)](https://huggingface.co/$USERNAME/$REPO_NAME)"
92
+ else
93
+ echo "❌ Upload failed. Check the error messages above."
94
+ exit 1
95
+ fi