Update README.md
Browse files
README.md
CHANGED
@@ -102,7 +102,7 @@ In general, the way to think about these (non-orthogonal) "multiplicative-LoRAs"
|
|
102 |
|
103 |
So instead of having just a single vector that we add (in essence we add a bias term and create an [affine transformation](https://en.wikipedia.org/wiki/Affine_transformation)), we now have many different control vectors that can be added (stored in `lora_B`), based on how well they match another set of "directional detection vectors" (stored in `lora_A`).
|
104 |
|
105 |
-
**NOTE**: The [LoRA+](https://arxiv.org/abs/2402.12354) paper uses a similar way of viewing the purpose of `lora_A` and `lora_B`, but where `lora_A` looks at the ***input*** to the `down_proj` transformation (for "additive-LoRAs"); instead of
|
106 |
|
107 |
---
|
108 |
|
|
|
102 |
|
103 |
So instead of having just a single vector that we add (in essence we add a bias term and create an [affine transformation](https://en.wikipedia.org/wiki/Affine_transformation)), we now have many different control vectors that can be added (stored in `lora_B`), based on how well they match another set of "directional detection vectors" (stored in `lora_A`).
|
104 |
|
105 |
+
**NOTE**: The [LoRA+](https://arxiv.org/abs/2402.12354) paper uses a similar way of viewing the purpose of `lora_A` and `lora_B`, but where `lora_A` looks at the ***input*** to the `down_proj` transformation (for "additive-LoRAs"); instead of the ***output*** of the `down_proj` transformation like these new (non-orthogonal) "multiplicative-LoRAs"...
|
106 |
|
107 |
---
|
108 |
|