Hugging Face's logo

Spaces:
jeffdshen
/
TorchCode
Sleeping

App Files Community
github-actions[bot] commited on
Commit
08d5cfa
Β·
0 Parent(s):

Deploy prebuilt image 63cae3bca387d68074365385e67b63f976fabbf8

Browse files
Files changed (2) hide show
  1. Dockerfile +1 -0
  2. README.md +401 -0
Dockerfile ADDED
@@ -0,0 +1 @@
 
 
1
+ FROM ghcr.io/duoan/torchcode:latest
README.md ADDED
@@ -0,0 +1,401 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: TorchCode
3
+ emoji: πŸ”₯
4
+ colorFrom: red
5
+ colorTo: yellow
6
+ sdk: docker
7
+ app_port: 7860
8
+ pinned: false
9
+ ---
10
+
11
+ <div align="center">
12
+
13
+ # πŸ”₯ TorchCode
14
+
15
+ **Crack the PyTorch interview.**
16
+
17
+ Practice implementing operators and architectures from scratch β€” the exact skills top ML teams test for.
18
+
19
+ *Like LeetCode, but for tensors. Self-hosted. Jupyter-based. Instant feedback.*
20
+
21
+ [![PyTorch](https://img.shields.io/badge/PyTorch-ee4c2c?style=for-the-badge&logo=pytorch&logoColor=white)](https://pytorch.org)
22
+ [![Jupyter](https://img.shields.io/badge/Jupyter-F37626?style=for-the-badge&logo=jupyter&logoColor=white)](https://jupyter.org)
23
+ [![Docker](https://img.shields.io/badge/Docker-2496ED?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com)
24
+ [![Python](https://img.shields.io/badge/Python_3.11-3776AB?style=for-the-badge&logo=python&logoColor=white)](https://python.org)
25
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow?style=for-the-badge)](LICENSE)
26
+
27
+ [![GitHub stars](https://img.shields.io/github/stars/duoan/TorchCode?style=social)](https://github.com/duoan/TorchCode)
28
+ [![GitHub Container Registry](https://img.shields.io/badge/ghcr.io-TorchCode-blue?style=flat-square&logo=github)](https://ghcr.io/duoan/torchcode)
29
+ [![Hugging Face Spaces](https://img.shields.io/badge/%F0%9F%A4%97%20Spaces-TorchCode-blue?style=flat-square)](https://huggingface.co/spaces/duoan/TorchCode)
30
+ ![Problems](https://img.shields.io/badge/problems-40-orange?style=flat-square)
31
+ ![GPU](https://img.shields.io/badge/GPU-not%20required-brightgreen?style=flat-square)
32
+
33
+ [![Star History Chart](https://api.star-history.com/svg?repos=duoan/TorchCode&type=Date)](https://star-history.com/#duoan/TorchCode&Date)
34
+
35
+ </div>
36
+
37
+ ---
38
+
39
+ ## 🎯 Why TorchCode?
40
+
41
+ Top companies (Meta, Google DeepMind, OpenAI, etc.) expect ML engineers to implement core operations **from memory on a whiteboard**. Reading papers isn't enough β€” you need to write `softmax`, `LayerNorm`, `MultiHeadAttention`, and full Transformer blocks code.
42
+
43
+ TorchCode gives you a **structured practice environment** with:
44
+
45
+ | | Feature | |
46
+ |---|---|---|
47
+ | 🧩 | **40 curated problems** | The most frequently asked PyTorch interview topics |
48
+ | βš–οΈ | **Automated judge** | Correctness checks, gradient verification, and timing |
49
+ | 🎨 | **Instant feedback** | Colored pass/fail per test case, just like competitive programming |
50
+ | πŸ’‘ | **Hints when stuck** | Nudges without full spoilers |
51
+ | πŸ“– | **Reference solutions** | Study optimal implementations after your attempt |
52
+ | πŸ“Š | **Progress tracking** | What you've solved, best times, and attempt counts |
53
+ | πŸ”„ | **One-click reset** | Toolbar button to reset any notebook back to its blank template β€” practice the same problem as many times as you want |
54
+ | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](#) | **Open in Colab** | Every notebook has an "Open in Colab" badge + toolbar button β€” run problems in Google Colab with zero setup |
55
+
56
+ No cloud. No signup. No GPU needed. Just `make run` β€” or try it instantly on Hugging Face.
57
+
58
+ ---
59
+
60
+ ## πŸš€ Quick Start
61
+
62
+ ### Option 0 β€” Try it online (zero install)
63
+
64
+ **[Launch on Hugging Face Spaces](https://huggingface.co/spaces/duoan/TorchCode)** β€” opens a full JupyterLab environment in your browser. Nothing to install.
65
+
66
+ Or open any problem directly in Google Colab β€” every notebook has an [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/01_relu.ipynb) badge.
67
+
68
+ ### Option 0b β€” Use the judge in Colab (pip)
69
+
70
+ In Google Colab, install the judge from PyPI so you can run `check(...)` without cloning the repo:
71
+
72
+ ```bash
73
+ !pip install torch-judge
74
+ ```
75
+
76
+ Then in a notebook cell:
77
+
78
+ ```python
79
+ from torch_judge import check, status, hint, reset_progress
80
+ status() # list all problems and your progress
81
+ check("relu") # run tests for the "relu" task
82
+ hint("relu") # show a hint
83
+ ```
84
+
85
+ ### Option 1 β€” Pull the pre-built image (fastest)
86
+
87
+ ```bash
88
+ docker run -p 8888:8888 -e PORT=8888 ghcr.io/duoan/torchcode:latest
89
+ ```
90
+
91
+ If the registry image is unavailable for your platform, use Option 2 instead. This is the common path on Apple Silicon / `arm64`.
92
+
93
+ ### Option 2 β€” Build locally
94
+
95
+ ```bash
96
+ make run
97
+ ```
98
+
99
+ `make run` will try the prebuilt image first and automatically fall back to a local build when needed.
100
+
101
+ Open **<http://localhost:8888>** β€” that's it. Works with both Docker and Podman (auto-detected).
102
+
103
+ ---
104
+
105
+ ## πŸ“‹ Problem Set
106
+
107
+ > **Frequency**: πŸ”₯ = very likely in interviews, ⭐ = commonly asked, πŸ’‘ = emerging / differentiator
108
+
109
+ ### 🧱 Fundamentals β€” "Implement X from scratch"
110
+
111
+ The bread and butter of ML coding interviews. You'll be asked to write these without `torch.nn`.
112
+
113
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
114
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
115
+ | 1 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/01_relu.ipynb" target="_blank">ReLU</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/01_relu.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `relu(x)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Activation functions, element-wise ops |
116
+ | 2 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/02_softmax.ipynb" target="_blank">Softmax</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/02_softmax.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_softmax(x, dim)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Numerical stability, exp/log tricks |
117
+ | 16 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/16_cross_entropy.ipynb" target="_blank">Cross-Entropy Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/16_cross_entropy.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `cross_entropy_loss(logits, targets)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Log-softmax, logsumexp trick |
118
+ | 17 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/17_dropout.ipynb" target="_blank">Dropout</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/17_dropout.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyDropout` (nn.Module) | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Train/eval mode, inverted scaling |
119
+ | 18 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/18_embedding.ipynb" target="_blank">Embedding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/18_embedding.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyEmbedding` (nn.Module) | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ”₯ | Lookup table, `weight[indices]` |
120
+ | 19 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/19_gelu.ipynb" target="_blank">GELU</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/19_gelu.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_gelu(x)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | Gaussian error linear unit, `torch.erf` |
121
+ | 20 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/20_weight_init.ipynb" target="_blank">Kaiming Init</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/20_weight_init.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `kaiming_init(weight)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | `std = sqrt(2/fan_in)`, variance scaling |
122
+ | 21 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/21_gradient_clipping.ipynb" target="_blank">Gradient Clipping</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/21_gradient_clipping.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `clip_grad_norm(params, max_norm)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | ⭐ | Norm-based clipping, direction preservation |
123
+ | 31 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/31_gradient_accumulation.ipynb" target="_blank">Gradient Accumulation</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/31_gradient_accumulation.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `accumulated_step(model, opt, ...)` | ![Easy](https://img.shields.io/badge/Easy-4CAF50?style=flat-square) | πŸ’‘ | Micro-batching, loss scaling |
124
+ | 40 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/40_linear_regression.ipynb" target="_blank">Linear Regression</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/40_linear_regression.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `LinearRegression` (3 methods) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Normal equation, GD from scratch, nn.Linear |
125
+ | 3 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/03_linear.ipynb" target="_blank">Linear Layer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/03_linear.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SimpleLinear` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | `y = xW^T + b`, Kaiming init, `nn.Parameter` |
126
+ | 4 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/04_layernorm.ipynb" target="_blank">LayerNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/04_layernorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_layer_norm(x, Ξ³, Ξ²)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Normalization, running stats, affine transform |
127
+ | 7 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/07_batchnorm.ipynb" target="_blank">BatchNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/07_batchnorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_batch_norm(x, γ, β)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Batch vs layer statistics, train/eval behavior |
128
+ | 8 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/08_rmsnorm.ipynb" target="_blank">RMSNorm</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/08_rmsnorm.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `rms_norm(x, weight)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | LLaMA-style norm, simpler than LayerNorm |
129
+ | 15 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/15_mlp.ipynb" target="_blank">SwiGLU MLP</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/15_mlp.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SwiGLUMLP` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Gated FFN, `SiLU(gate) * up`, LLaMA/Mistral-style |
130
+ | 22 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/22_conv2d.ipynb" target="_blank">Conv2d</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/22_conv2d.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `my_conv2d(x, weight, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Convolution, unfold, stride/padding |
131
+
132
+ ### 🧠 Attention Mechanisms β€” The heart of modern ML interviews
133
+
134
+ If you're interviewing for any role touching LLMs or Transformers, expect at least one of these.
135
+
136
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
137
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
138
+ | 23 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/23_cross_attention.ipynb" target="_blank">Cross-Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/23_cross_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MultiHeadCrossAttention` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Encoder-decoder, Q from decoder, K/V from encoder |
139
+ | 5 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/05_attention.ipynb" target="_blank">Scaled Dot-Product Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/05_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `scaled_dot_product_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | `softmax(QK^T/√d_k)V`, the foundation of everything |
140
+ | 6 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/06_multihead_attention.ipynb" target="_blank">Multi-Head Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/06_multihead_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MultiHeadAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Parallel heads, split/concat, projection matrices |
141
+ | 9 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/09_causal_attention.ipynb" target="_blank">Causal Self-Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/09_causal_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `causal_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Autoregressive masking with `-inf`, GPT-style |
142
+ | 10 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/10_gqa.ipynb" target="_blank">Grouped Query Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/10_gqa.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `GroupQueryAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | GQA (LLaMA 2), KV sharing across heads |
143
+ | 11 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/11_sliding_window.ipynb" target="_blank">Sliding Window Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/11_sliding_window.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `sliding_window_attention(Q, K, V, w)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Mistral-style local attention, O(n·w) complexity |
144
+ | 12 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/12_linear_attention.ipynb" target="_blank">Linear Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/12_linear_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `linear_attention(Q, K, V)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Kernel trick, `Ο†(Q)(Ο†(K)^TV)`, O(nΒ·dΒ²) |
145
+ | 14 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/14_kv_cache.ipynb" target="_blank">KV Cache Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/14_kv_cache.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `KVCacheAttention` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Incremental decoding, cache K/V, prefill vs decode |
146
+ | 24 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/24_rope.ipynb" target="_blank">RoPE</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/24_rope.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `apply_rope(q, k)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ”₯ | Rotary position embedding, relative position via rotation |
147
+ | 25 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/25_flash_attention.ipynb" target="_blank">Flash Attention</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/25_flash_attention.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `flash_attention(Q, K, V, block_size)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Tiled attention, online softmax, memory-efficient |
148
+
149
+ ### πŸ—οΈ Architecture & Adaptation β€” Put it all together
150
+
151
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
152
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
153
+ | 26 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/26_lora.ipynb" target="_blank">LoRA</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/26_lora.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `LoRALinear` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Low-rank adaptation, frozen base + `BA` update |
154
+ | 27 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/27_vit_patch.ipynb" target="_blank">ViT Patch Embedding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/27_vit_patch.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `PatchEmbedding` (nn.Module) | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ’‘ | Image β†’ patches β†’ linear projection |
155
+ | 13 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/13_gpt2_block.ipynb" target="_blank">GPT-2 Block</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/13_gpt2_block.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `GPT2Block` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Pre-norm, causal MHA + MLP (4x, GELU), residual connections |
156
+ | 28 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/28_moe.ipynb" target="_blank">Mixture of Experts</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/28_moe.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MixtureOfExperts` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | ⭐ | Mixtral-style, top-k routing, expert MLPs |
157
+
158
+ ### βš™οΈ Training & Optimization
159
+
160
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
161
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
162
+ | 29 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/29_adam.ipynb" target="_blank">Adam Optimizer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/29_adam.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `MyAdam` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Momentum + RMSProp, bias correction |
163
+ | 30 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/30_cosine_lr.ipynb" target="_blank">Cosine LR Scheduler</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/30_cosine_lr.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `cosine_lr_schedule(step, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | ⭐ | Linear warmup + cosine annealing |
164
+
165
+ ### 🎯 Inference & Decoding
166
+
167
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
168
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
169
+ | 32 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/32_topk_sampling.ipynb" target="_blank">Top-k / Top-p Sampling</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/32_topk_sampling.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `sample_top_k_top_p(logits, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Nucleus sampling, temperature scaling |
170
+ | 33 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/33_beam_search.ipynb" target="_blank">Beam Search</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/33_beam_search.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `beam_search(log_prob_fn, ...)` | ![Medium](https://img.shields.io/badge/Medium-FF9800?style=flat-square) | πŸ”₯ | Hypothesis expansion, pruning, eos handling |
171
+ | 34 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/34_speculative_decoding.ipynb" target="_blank">Speculative Decoding</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/34_speculative_decoding.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `speculative_decode(target, draft, ...)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Accept/reject, draft model acceleration |
172
+
173
+ ### πŸ”¬ Advanced β€” Differentiators
174
+
175
+ | # | Problem | What You'll Implement | Difficulty | Freq | Key Concepts |
176
+ |:---:|---------|----------------------|:----------:|:----:|--------------|
177
+ | 35 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/35_bpe.ipynb" target="_blank">BPE Tokenizer</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/35_bpe.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `SimpleBPE` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Byte-pair encoding, merge rules, subword splits |
178
+ | 36 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/36_int8_quantization.ipynb" target="_blank">INT8 Quantization</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/36_int8_quantization.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `Int8Linear` (nn.Module) | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Per-channel quantize, scale/zero-point, buffer vs param |
179
+ | 37 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/37_dpo_loss.ipynb" target="_blank">DPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/37_dpo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `dpo_loss(chosen, rejected, ...)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Direct preference optimization, alignment training |
180
+ | 38 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/38_grpo_loss.ipynb" target="_blank">GRPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/38_grpo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `grpo_loss(logps, rewards, group_ids, eps)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | Group relative policy optimization, RLAIF, within-group normalized advantages |
181
+ | 39 | <a href="https://github.com/duoan/TorchCode/blob/master/templates/39_ppo_loss.ipynb" target="_blank">PPO Loss</a> <a href="https://colab.research.google.com/github/duoan/TorchCode/blob/master/templates/39_ppo_loss.ipynb" target="_blank"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="20"></a> | `ppo_loss(new_logps, old_logps, advantages, clip_ratio)` | ![Hard](https://img.shields.io/badge/Hard-F44336?style=flat-square) | πŸ’‘ | PPO clipped surrogate loss, policy gradient, trust region |
182
+
183
+ ---
184
+
185
+ ## βš™οΈ How It Works
186
+
187
+ Each problem has **two** notebooks:
188
+
189
+ | File | Purpose |
190
+ |------|---------|
191
+ | `01_relu.ipynb` | ✏️ Blank template β€” write your code here |
192
+ | `01_relu_solution.ipynb` | πŸ“– Reference solution β€” check when stuck |
193
+
194
+ ### Workflow
195
+
196
+ ```text
197
+ 1. Open a blank notebook β†’ Read the problem description
198
+ 2. Implement your solution β†’ Use only basic PyTorch ops
199
+ 3. Debug freely β†’ print(x.shape), check gradients, etc.
200
+ 4. Run the judge cell β†’ check("relu")
201
+ 5. See instant colored feedback β†’ βœ… pass / ❌ fail per test case
202
+ 6. Stuck? Get a nudge β†’ hint("relu")
203
+ 7. Review the reference solution β†’ 01_relu_solution.ipynb
204
+ 8. Click πŸ”„ Reset in the toolbar β†’ Blank slate β€” practice again!
205
+ ```
206
+
207
+ ### In-Notebook API
208
+
209
+ ```python
210
+ from torch_judge import check, hint, status
211
+
212
+ check("relu") # Judge your implementation
213
+ hint("causal_attention") # Get a hint without full spoiler
214
+ status() # Progress dashboard β€” solved / attempted / todo
215
+ ```
216
+
217
+ ---
218
+
219
+ ## πŸ“… Suggested Study Plan
220
+
221
+ > **Total: ~12–16 hours spread across 3–4 weeks. Perfect for interview prep on a deadline.**
222
+
223
+ | Week | Focus | Problems | Time |
224
+ |:----:|-------|----------|:----:|
225
+ | **1** | 🧱 Foundations | ReLU β†’ Softmax β†’ CE Loss β†’ Dropout β†’ Embedding β†’ GELU β†’ Linear β†’ LayerNorm β†’ BatchNorm β†’ RMSNorm β†’ SwiGLU MLP β†’ Conv2d | 2–3 hrs |
226
+ | **2** | 🧠 Attention Deep Dive | SDPA β†’ MHA β†’ Cross-Attn β†’ Causal β†’ GQA β†’ KV Cache β†’ Sliding Window β†’ RoPE β†’ Linear Attn β†’ Flash Attn | 3–4 hrs |
227
+ | **3** | πŸ—οΈ Architecture + Training | GPT-2 Block β†’ LoRA β†’ MoE β†’ ViT Patch β†’ Adam β†’ Cosine LR β†’ Grad Clip β†’ Grad Accumulation β†’ Kaiming Init | 3–4 hrs |
228
+ | **4** | 🎯 Inference + Advanced | Top-k/p Sampling β†’ Beam Search β†’ Speculative Decoding β†’ BPE β†’ INT8 Quant β†’ DPO Loss β†’ GRPO Loss β†’ PPO Loss + speed run | 3–4 hrs |
229
+
230
+ ---
231
+
232
+ ## πŸ›οΈ Architecture
233
+
234
+ ```text
235
+ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
236
+ β”‚ Docker / Podman Container β”‚
237
+ β”‚ β”‚
238
+ β”‚ JupyterLab (:8888) β”‚
239
+ β”‚ β”œβ”€β”€ templates/ (reset on each run) β”‚
240
+ β”‚ β”œβ”€β”€ solutions/ (reference impl) β”‚
241
+ β”‚ β”œβ”€β”€ torch_judge/ (auto-grading) β”‚
242
+ β”‚ β”œβ”€β”€ torchcode-labext (JLab plugin) β”‚
243
+ β”‚ β”‚ πŸ”„ Reset β€” restore template β”‚
244
+ β”‚ β”‚ πŸ”— Colab β€” open in Colab β”‚
245
+ β”‚ └── PyTorch (CPU), NumPy β”‚
246
+ β”‚ β”‚
247
+ β”‚ Judge checks: β”‚
248
+ β”‚ βœ“ Output correctness (allclose) β”‚
249
+ β”‚ βœ“ Gradient flow (autograd) β”‚
250
+ β”‚ βœ“ Shape consistency β”‚
251
+ β”‚ βœ“ Edge cases & numerical stability β”‚
252
+ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
253
+ ```
254
+
255
+ Single container. Single port. No database. No frontend framework. No GPU.
256
+
257
+ ## πŸ› οΈ Commands
258
+
259
+ ```bash
260
+ make run # Build & start (http://localhost:8888)
261
+ make stop # Stop the container
262
+ make clean # Stop + remove volumes + reset all progress
263
+ ```
264
+
265
+ ## 🧩 Adding Your Own Problems
266
+
267
+ TorchCode uses auto-discovery β€” just drop a new file in `torch_judge/tasks/`:
268
+
269
+ ```python
270
+ TASK = {
271
+ "id": "my_task",
272
+ "title": "My Custom Problem",
273
+ "difficulty": "medium",
274
+ "function_name": "my_function",
275
+ "hint": "Think about broadcasting...",
276
+ "tests": [ ... ],
277
+ }
278
+ ```
279
+
280
+ No registration needed. The judge picks it up automatically.
281
+
282
+ ---
283
+
284
+ ## πŸ“¦ Publishing `torch-judge` to PyPI (maintainers)
285
+
286
+ The judge is published as a separate package so Colab/users can `pip install torch-judge` without cloning the repo.
287
+
288
+ ### Automatic (GitHub Action)
289
+
290
+ Pushing to `master` after changing the package version triggers [`.github/workflows/pypi-publish.yml`](.github/workflows/pypi-publish.yml), which builds and uploads to PyPI. No git tag is required.
291
+
292
+ 1. **Bump version** in `torch_judge/_version.py` (e.g. `__version__ = "0.1.1"`).
293
+ 2. **Configure PyPI Trusted Publisher** (one-time):
294
+ - PyPI β†’ Your project **torch-judge** β†’ **Publishing** β†’ **Add a new pending publisher**
295
+ - Owner: `duoan`, Repository: `TorchCode`, Workflow: `pypi-publish.yml`, Environment: (leave empty)
296
+ - Run the workflow once (push a version bump to `master` or **Actions β†’ Publish torch-judge to PyPI β†’ Run workflow**); PyPI will then link the publisher.
297
+ 3. **Release**: commit the version bump and `git push origin master`.
298
+
299
+ Alternatively, use an API token: add repository secret `PYPI_API_TOKEN` (value = `pypi-...` from PyPI) and set `TWINE_USERNAME=__token__` and `TWINE_PASSWORD` from that secret in the workflow if you prefer not to use Trusted Publishing.
300
+
301
+ ### Manual
302
+
303
+ ```bash
304
+ pip install build twine
305
+ python -m build
306
+ twine upload dist/*
307
+ ```
308
+
309
+ Version is in `torch_judge/_version.py`; bump it before each release.
310
+
311
+ ---
312
+
313
+ ## ❓ FAQ
314
+
315
+ <details>
316
+ <summary><b>Do I need a GPU?</b></summary>
317
+ <br>
318
+ No. Everything runs on CPU. The problems test correctness and understanding, not throughput.
319
+ </details>
320
+
321
+ <details>
322
+ <summary><b>Can I keep my solutions between runs?</b></summary>
323
+ <br>
324
+ Blank templates reset on every <code>make run</code> so you practice from scratch. Save your work under a different filename if you want to keep it. You can also click the <b>πŸ”„ Reset</b> button in the notebook toolbar at any time to restore the blank template without restarting.
325
+ </details>
326
+
327
+ <details>
328
+ <summary><b>Can I use Google Colab instead?</b></summary>
329
+ <br>
330
+ Yes! Every notebook has an <b>Open in Colab</b> badge at the top. Click it to open the problem directly in Google Colab β€” no Docker or local setup needed. You can also use the <b>Colab</b> toolbar button inside JupyterLab.
331
+ </details>
332
+
333
+ <details>
334
+ <summary><b>How are solutions graded?</b></summary>
335
+ <br>
336
+ The judge runs your function against multiple test cases using <code>torch.allclose</code> for numerical correctness, verifies gradients flow properly via autograd, and checks edge cases specific to each operation.
337
+ </details>
338
+
339
+ <details>
340
+ <summary><b>Who is this for?</b></summary>
341
+ <br>
342
+ Anyone preparing for ML/AI engineering interviews at top tech companies, or anyone who wants to deeply understand how PyTorch operations work under the hood.
343
+ </details>
344
+
345
+ ---
346
+
347
+ ## 🀝 Contributors
348
+
349
+ Thanks to everyone who has contributed to TorchCode.
350
+
351
+ <!-- readme: contributors -start -->
352
+ <table>
353
+ <tbody>
354
+ <tr>
355
+ <td align="center">
356
+ <a href="https://github.com/duoan">
357
+ <img src="https://avatars.githubusercontent.com/u/2378740?v=4" width="100;" alt="duoan"/>
358
+ <br />
359
+ <sub><b>duoan</b></sub>
360
+ </a>
361
+ </td>
362
+ <td align="center">
363
+ <a href="https://github.com/Ando233">
364
+ <img src="https://avatars.githubusercontent.com/u/74404658?v=4" width="100;" alt="Ando233"/>
365
+ <br />
366
+ <sub><b>Ando233</b></sub>
367
+ </a>
368
+ </td>
369
+ <td align="center">
370
+ <a href="https://github.com/ThierryHJ">
371
+ <img src="https://avatars.githubusercontent.com/u/51846529?v=4" width="100;" alt="ThierryHJ"/>
372
+ <br />
373
+ <sub><b>ThierryHJ</b></sub>
374
+ </a>
375
+ </td>
376
+ </tr>
377
+ <tbody>
378
+ </table>
379
+ <!-- readme: contributors -end -->
380
+
381
+ Auto-generated from the [GitHub contributors graph](https://github.com/duoan/TorchCode/graphs/contributors) with avatars and GitHub usernames.
382
+
383
+ ---
384
+
385
+ <div align="center">
386
+
387
+ **Built for engineers who want to deeply understand what they build.**
388
+
389
+ If this helped your interview prep, consider giving it a ⭐
390
+
391
+ ---
392
+
393
+ ### β˜• Buy Me a Coffee
394
+
395
+ <a href="https://buymeacoffee.com/duoan" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>
396
+
397
+ <img src="./bmc_qr.png" alt="BMC QR Code" width="150" height="150">
398
+
399
+ *Scan to support*
400
+
401
+ </div>