Implement encoder layer and attention mechanism functions

This commit introduces essential functions for building an encoder layer and implementing the attention mechanism. The encoder_layer function takes matrices Q, K transpose, and V, along with projection weight matrices Wq, Wk, and Wv as input. It performs projection operations on Q, K transpose, and V using the respective weight matrices and then applies the scaled dot-product attention mechanism using the function scaled_dot_product_attention_2D. The result is the attention scores computed for the input.

The function calcular_pesos_proyeccion(attention_matrix) calculates projection weight matrices Wq, Wk, and Wv based on the dimensions of the attention matrix. These weight matrices are initialized such that each diagonal element is set to 1, ensuring that each letter's representation is projected independently during the attention computation.

The scaled_dot_product_attention_2D function computes the attention scores by performing a scaled dot-product attention operation between Q, K transpose, and V matrices. It iterates over each row of Q, computes the attention scores, applies softmax to obtain attention weights, and then computes the weighted sum of V based on these weights.

These functions are fundamental components of transformer-based architectures used in natural language processing tasks, particularly in sequence-to-sequence models for tasks like machine translation and text summarization. They facilitate effective attention-based information aggregation and feature extraction from input sequences.

attention.py

@@ -0,0 +1,27 @@
+import numpy as np
+
+def encoder_layer(Q, K_transpose, V, Wq, Wk, Wv):
+    Q_proj = np.dot(Q, Wq.T)
+    K_proj = np.dot(K_transpose.T, Wk.T)
+    V_proj = np.dot(V, Wv.T)
+    attention_scores = scaled_dot_product_attention_2D(Q_proj, K_proj, V_proj)
+    return attention_scores
+
+def calcular_pesos_proyeccion(attention_matrix):
+    num_letras = attention_matrix.shape[0]
+    Wq = np.zeros((num_letras, attention_matrix.shape[1]))
+    Wk = np.zeros((num_letras, attention_matrix.shape[1]))
+    Wv = np.zeros((num_letras, attention_matrix.shape[1]))
+    for i in range(num_letras):
+        Wq[i, i] = 1
+        Wk[i, i] = 1
+        Wv[i, i] = 1
+    return Wq, Wk, Wv
+
+def scaled_dot_product_attention_2D(Q, K_transpose, V):
+    out = np.zeros((Q.shape[0], V.shape[1]))
+    for i in range(Q.shape[0]):
+        scores = np.dot(Q[i, :], K_transpose)
+        softmax_scores = np.exp(scores - np.max(scores)) / np.sum(np.exp(scores - np.max(scores)))
+        out[i, :] = np.dot(softmax_scores, V)
+    return out