Update README.md

README.md

@@ -31,7 +31,13 @@ Uses:
 
 `h = (I + lora_B @ lora_A) @ tensor @ x = tensor @ x + lora_B @ lora_A @ tensor @ x`
 
-instead of the normal "addative-LoRA" method of:
+or equivalently:
+
+`h = tensor @ x`
+
+`h' = h + lora_B @ lora_A @ h`
+
+instead of the normal "additive-LoRA" method of:
 
 `h = (tensor + lora_B @ lora_A) @ x = tensor @ x + lora_B @ lora_A @ x`
 
@@ -67,6 +73,33 @@ tensor = tensor.to(old_type)
 
 ---
 
+# The rationale behind the "multiplicative-LoRA" method and the link to control-vectors
+
+There are actually 3 existing "multiplicative-LoRA" methods in [PEFT/tuners](https://github.com/huggingface/peft/tree/main/src/peft/tuners):
+
+- https://github.com/huggingface/peft/tree/main/src/peft/tuners/oft (https://arxiv.org/abs/2306.07280)
+- https://github.com/huggingface/peft/tree/main/src/peft/tuners/boft (https://arxiv.org/abs/2311.06243)
+- https://github.com/huggingface/peft/tree/main/src/peft/tuners/hra (https://arxiv.org/abs/2405.17484)
+
+but all of these deliberately maintain [orthogonality](https://en.wikipedia.org/wiki/Orthogonal_matrix), and thus are more restrictive in the types of transformations they can perform (ie: [Rotations](https://en.wikipedia.org/wiki/Rotation) and/or [Improper Rotations](https://en.wikipedia.org/wiki/Improper_rotation) only; with no scaling and/or sheer possible...).
+
+For example, these can't perform the orthogonal projection performed by [abliteration](https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction):
+
+`h' = h - v @ v^T @ h`
+
+whereas the general (non-orthogonal) "multiplicative-LoRA" method can do this by choosing to set `u = -v` like so:
+
+`h' = h + u @ v^T @ h`
+
+In general, the way to think about these (non-orthogonal) "multiplicative-LoRAs" is as a kind of "conditional control-vector":
+
+- The vectors in `lora_A` look for a certain dirrection, and via the dot-product; generate (signed) weighting factor that measure the similarity between the output of the `down_proj` transformation.
+- The vectors in `lora_B` then get added to the hidden state / residual stream based on the weighting factors.
+
+So instead of having just a single vector that we add (in essence we add a bias term and create an [Affine transformation](https://en.wikipedia.org/wiki/Affine_transformation)), we now have many different control vectors that can be added (in `lora_B`), based on how well they match another set of "directional detection vectors" (in `lora_A`).
+
+---
+
 # Training
 
 - Took just under 4 days using dual-A6000 GPUs connected via NVLink, using [qlora-pipe](https://github.com/tdrussell/qlora-pipe).