Upload 6 files

README.md

@@ -1,3 +1,111 @@
----
-license: mit
----
+# Transformer Modeli ile Türkçe Chatbot
+
+Bu depo, Transformer modeli kullanılarak oluşturulmuş bir Türkçe sohbet botunun kodunu içerir. Sohbet robotu, Türkçe konuşmalardan oluşan bir veri kümesi üzerinde eğitilmiştir ve kullanıcı girdilerine yanıtlar üretebilir.
+
+## Dosyalar
+
+* **model.py:** Kodlayıcı, kod çözücü ve dikkat mekanizmaları dahil olmak üzere Transformer model mimarisini tanımlar.
+* **dataset.py:** Konuşma veri kümesinin yüklenmesi, ön işlenmesi ve tokenize edilmesi için fonksiyonlar içerir.
+* **chatbot.py:** Yanıt oluşturma ve kullanıcı geri bildirimi toplama dahil olmak üzere etkileşimli sohbet işlevselliğini yönetir.
+* **data/lines.txt:** Eğitim için kullanılan ham metin satırlarını depolar.
+* **data/conversations.txt:** Eğitim için kullanılan konuşma çiftlerini içerir.
+
+## Kullanım
+
+1. **Depoyu klonlayın:**
+```shell
+git clone https://github.com/EmirhanOzl/transformer-turkish-chatbot.git
+```
+2. **Gerekli bağımlılıkları yükleyin:**
+```shell
+pip install -r requirements.txt
+```
+3. **Modeli eğitin:**
+```shell
+python training.py
+```
+4. **Chatbotu çalıştırın:**
+```shell
+python chatbot.py
+```
+
+## Eğitim Verileri
+
+Chatbot, `data/` dizininde depolanan Türkçe konuşmalardan oluşan bir veri kümesi üzerinde eğitilmiştir. Sağlanan veri setini kullanabilir veya kendi veri setinizi oluşturabilirsiniz. Veri kümesi aşağıdaki formatta olmalıdır:
+
+Veri kümesi iki dosyadan oluşmaktadır:
+
+* **lines.txt:** Bu dosya eğitim için kullanılan ham metin satırlarını içerir. Her satır aşağıdaki formattadır:
+```
+[LINE_ID] +++$+++ [USER_ID] +++$+++ [MOVIE_ID] +++$+++ [CHARACTER_NAME] +++$+++ [TEXT]
+```
+
+* `LINE_ID` hattın kimliğidir.
+* `USER_ID` hattı konuşan kullanıcının kimliğidir.
+* `MOVIE_ID` konuşmanın hakkında olduğu filmin kimliğidir (isteğe bağlı).
+* `CHARACTER_NAME` satırı söyleyen karakterin adıdır (isteğe bağlı).
+* `TEXT` satırın metnidir.
+
+* **conversations.txt:** Bu dosya eğitim için kullanılan konuşma çiftlerini içerir. Her satır aşağıdaki formattadır:
+```
+[USER_ID] +++$+++ [BOT_ID] +++$+++ [MOVIE_ID] +++$+++ [CONVERSATION]
+```
+
+* `USER_ID` kullanıcının kimliğidir.
+* `BOT_ID` sohbet robotunun kimliğidir.
+* `MOVIE_ID` konuşmanın hakkında olduğu filmin kimliğidir (isteğe bağlı).
+* `CONVERSATION`, virgülle ayrılmış, konuşma dönüşlerini temsil eden satır kimliklerinin bir listesidir.
+
+Sağlanan veri kümesini kullanabilir veya aynı formatta kendi veri kümenizi oluşturabilirsiniz.
+
+**Not:** `MOVIE_ID` ve `CHARACTER_NAME` alanları isteğe bağlıdır ve yerine default birşey yazılabilir.
+
+
+
+## Örnek çıktı
+
+Aşağıda listelenen hiperparametreler kullanılarak aşağıdaki sonuçlar elde edilmiştir:
+
+```python
+self.save_model = "model.h5"
+self.max_samples = 25000
+self.max_length = 60
+self.batch_size = 128
+self.num_layers = 3
+self.num_units = 512
+self.d_model = 256
+self.num_heads = 8
+self.dropout = 0.1
+self.activation = "relu"
+self.epochs = 100
+```
+
+Training Results:
+```
+Epoch 100/100
+24/24 [==============================] - 192s 8s/step - loss: 0.0145 - accuracy: 0.3174
+```
+
+Örnek çıktı:
+
+**User:** Merhaba nasılsın?
+**Bot:** Merhaba! Ben iyiyim, teşekkür ederim. Sana nasıl yardımcı olabilirim bugün?
+
+**User:** Sence de gökyüzü çok güzel değil mi?
+**Bot:** O zaman hemen başlayalım, bu filmi izlemek için daha fazla beklemek istemiyorum.
+
+**User:** Sanırım uzaklara gideceğim.
+**Bot:** Nereye gitmek istersin?
+
+**User:** Nereye gitmek istersin?
+**Bot:** Belki de dünya dışına. Sence dünya dışını görebilir miyiz?
+
+**User:** Belki de dünya dışına. Sence dünya dışını görebilir miyiz?
+**Bot:** Belki bir gün teknoloji bize bu imkanı sunar, kim bilir?
+
+**User:** Belki bir gün teknoloji bize bu imkanı sunar, kim bilir?
+**Bot:** Ne zaman istersen yardımcı olmaktan mutluluk duyarım. Yaratıcı bir süreç geçirmeni dilerim!
+
+
+Not: Sohbet robotunun yanıtları, kullanılan eğitim verilerine ve hiperparametrelerine bağlı olarak değişebilir.
+

chatbot.py

@@ -0,0 +1,141 @@
+import argparse
+import tensorflow as tf
+import model
+from dataset import get_dataset, preprocess_sentence
+
+
+def inference(hparams, chatbot, tokenizer, sentence):
+    sentence = preprocess_sentence(sentence)
+
+    sentence = tf.expand_dims(
+        hparams.start_token + tokenizer.encode(sentence) + hparams.end_token, axis=0
+    )
+
+    output = tf.expand_dims(hparams.start_token, 0)
+
+    for _ in range(hparams.max_length):
+        predictions = chatbot(inputs=[sentence, output], training=False)
+
+        predictions = predictions[:, -1:, :]
+        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)
+
+        if tf.equal(predicted_id, hparams.end_token[0]):
+            break
+
+        output = tf.concat([output, predicted_id], axis=-1)
+
+    return tf.squeeze(output, axis=0)
+
+
+def predict(hparams, chatbot, tokenizer, sentence):
+    prediction = inference(hparams, chatbot, tokenizer, sentence)
+    predicted_sentence = tokenizer.decode(
+        [i for i in prediction if i < tokenizer.vocab_size]
+    )
+    return predicted_sentence
+
+def read_file(file_path):
+    with open(file_path, 'r', encoding='utf-8') as file:
+        lines = file.readlines()
+    return lines
+
+def append_to_file(file_path, line):
+    with open(file_path, 'a', encoding='utf-8') as file:
+        file.write(f"{line}\n")
+        
+def get_last_ids(lines_file, conversations_file):
+    lines = read_file(lines_file)
+    conversations = read_file(conversations_file)
+
+    last_line = lines[-1]
+    last_conversation = conversations[-1]
+
+    last_line_id = int(last_line.split(" +++$+++ ")[0][1:])
+    last_user_id = int(last_conversation.split(" +++$+++ ")[1][1:])
+    last_movie_id = int(last_conversation.split(" +++$+++ ")[2][1:])
+
+    return last_line_id, last_user_id, last_movie_id
+
+def update_data_files(user_input, bot_response, lines_file='data/lines.txt', conversations_file='data/conversations.txt'):
+    last_line_id, last_user_id, last_movie_id = get_last_ids(lines_file, conversations_file)
+
+    new_line_id = f"L{last_line_id + 1}"
+    new_bot_line_id = f"L{last_line_id + 2}"
+    new_user_id = f"u{last_user_id + 1}"
+    new_bot_user_id = f"u{last_user_id + 2}"
+    new_movie_id = f"m{last_movie_id + 1}"
+
+    append_to_file(lines_file, f"{new_line_id} +++$+++ {new_user_id} +++$+++ {new_movie_id} +++$+++ Ben +++$+++ {user_input}")
+    append_to_file(lines_file, f"{new_bot_line_id} +++$+++ {new_bot_user_id} +++$+++ {new_movie_id} +++$+++ Bot +++$+++ {bot_response}")
+
+    new_conversation = f"{new_user_id} +++$+++ {new_bot_user_id} +++$+++ {new_movie_id} +++$+++ ['{new_line_id}', '{new_bot_line_id}']"
+    append_to_file(conversations_file, new_conversation)
+
+def get_feedback():
+    feedback = input("Bu cevap yardımcı oldu mu? (Evet/Hayır):  ").lower()
+    return feedback == "Evet"
+
+def chat(hparams, chatbot, tokenizer):
+    print("\nCHATBOT")
+
+    for _ in range(5):
+        sentence = input("Sen: ")
+        output = predict(hparams, chatbot, tokenizer, sentence)
+        print(f"\nBOT: {output}")
+        
+        
+        user_input = sentence
+        bot_response = output
+        
+        feedback = get_feedback()
+        
+        if feedback:
+            update_data_files(user_input, bot_response)
+        else:
+            pass
+        
+        
+def main(hparams):
+    
+    _, token = get_dataset(hparams)
+    
+    tf.keras.backend.clear_session()
+    chatbot = tf.keras.models.load_model(
+        hparams.save_model,
+        custom_objects={
+            "PositionalEncoding": model.PositionalEncoding,
+            "MultiHeadAttention": model.MultiHeadAttention,
+        },
+        compile=False,
+    )
+    
+    
+    chat(hparams, chatbot, token)
+    
+    
+if __name__ == "__main__":
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--save_model", default="model.h5", type=str, help="path save the model"
+    )
+    parser.add_argument(
+        "--max_samples",
+        default=25000,
+        type=int,
+        help="maximum number of conversation pairs to use",
+    )
+    parser.add_argument(
+        "--max_length", default=40, type=int, help="maximum sentence length"
+    )
+    parser.add_argument("--batch_size", default=64, type=int)
+    parser.add_argument("--num_layers", default=2, type=int)
+    parser.add_argument("--num_units", default=512, type=int)
+    parser.add_argument("--d_model", default=256, type=int)
+    parser.add_argument("--num_heads", default=8, type=int)
+    parser.add_argument("--dropout", default=0.1, type=float)
+    parser.add_argument("--activation", default="relu", type=str)
+    parser.add_argument("--epochs", default=80, type=int)
+    
+    main(parser.parse_args())
+        

dataset.py

@@ -0,0 +1,101 @@
+import re
+import tensorflow as tf
+import tensorflow_datasets as tfds
+import nltk
+from nltk.stem import WordNetLemmatizer
+
+nltk.download('wordnet')
+nltk.download('punkt')
+
+lemmatizer = WordNetLemmatizer()
+
+
+def preprocess_sentence(sentence):
+    sentence = sentence.lower().strip()
+    
+    sentence = re.sub(r"([?.!¿])", r" \1 ", sentence)
+    sentence = re.sub(r'[" "]+', " ", sentence)
+    sentence = re.sub(r"[-()\"#/@;:<>{}+=~|.?,]", "", sentence)
+
+    sentence = re.sub(r"[^a-zA-ZğüşöçıİĞÜŞÖÇ?.!,¿]+", " ", sentence)
+    sentence = sentence.strip()  
+    
+    sentence = ' '.join([lemmatizer.lemmatize(w) for w in nltk.word_tokenize(sentence)])
+    
+    return sentence
+
+def load_conversations(hparams, lines_file, conversations_file):
+    id2line = {}
+    
+    with open(lines_file, encoding = "utf-8", errors="ignore") as file:
+        lines = file.readlines()
+    
+    for line in lines:
+        parts = line.replace("\n", "").split(" +++$+++ ")
+        id2line[parts[0]] = parts[4]
+        
+    questions = []
+    answers = []
+    
+    with open(conversations_file, "r") as file:
+        lines = file.readlines()
+    for line in lines:
+        parts = line.replace("\n", "").split(" +++$+++ ")
+        conversation = [line[1:-1] for line in parts[3][1:-1].split(", ")]
+        for i in range(len(conversation) - 1):
+            questions.append(preprocess_sentence(id2line[conversation[i]]))
+            answers.append(preprocess_sentence(id2line[conversation[i + 1]]))
+            if len(questions) >= hparams.max_samples:
+                return questions, answers
+            
+    return questions, answers
+
+
+def tokenize(hparams, tokenizer, questions, answers):
+    tokenized_inputs, tokenized_outputs = [], []
+    
+    for (question, answer) in zip(questions, answers):
+        sentence1 = hparams.start_token + tokenizer.encode(question) + hparams.end_token
+        sentence2 = hparams.start_token + tokenizer.encode(answer) + hparams.end_token
+        
+        if (len(sentence1) <= hparams.max_length and len(sentence2) <= hparams.max_length):
+            tokenized_inputs.append(sentence1)
+            tokenized_outputs.append(sentence2)
+    
+    tokenized_inputs = tf.keras.preprocessing.sequence.pad_sequences(
+        tokenized_inputs, maxlen=hparams.max_length, padding="post")
+    tokenized_outputs = tf.keras.preprocessing.sequence.pad_sequences(
+        tokenized_outputs, maxlen=hparams.max_length, padding="post")
+
+    return tokenized_inputs, tokenized_outputs
+
+
+def get_dataset(hparams):
+    lines_file ="data/lines.txt"
+    conversations_file = "data/conversations.txt"
+    
+    questions, answers = load_conversations(hparams, lines_file, conversations_file)
+    
+    tokenizer = tfds.deprecated.text.SubwordTextEncoder.build_from_corpus(questions + answers, target_vocab_size=2**13)
+    
+    tokenizer.save_to_file('tokenizer')
+    
+    hparams.start_token = [tokenizer.vocab_size]
+    hparams.end_token = [tokenizer.vocab_size + 1]
+    hparams.vocab_size = tokenizer.vocab_size + 2
+    
+    questions, answers = tokenize(hparams, tokenizer, questions, answers)
+    
+    dataset = tf.data.Dataset.from_tensor_slices(
+        ({"inputs": questions, "dec_inputs": answers[:, :-1]}, answers[:, 1:])
+    )
+    
+    dataset = dataset.cache()
+    dataset = dataset.shuffle(len(questions))
+    dataset = dataset.batch(hparams.batch_size)
+    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
+    
+    return dataset, tokenizer
+
+
+

model.py

@@ -0,0 +1,265 @@
+import tensorflow as tf
+
+def sdp_attention(query, key, value, mask):
+    matmul_qk = tf.matmul(query, key, transpose_b=True)
+    depth = tf.cast(tf.shape(key)[-1], tf.float32)
+    logits = matmul_qk / tf.math.sqrt(depth)
+    if mask is not None:
+        logits += mask * -1e9
+    attention_weights = tf.nn.softmax(logits, axis=-1)
+    output = tf.matmul(attention_weights, value)
+    return output
+
+class MultiHeadAttention(tf.keras.layers.Layer):
+    def __init__(self, num_heads, d_model, **kwargs):
+        assert d_model % num_heads == 0
+        super(MultiHeadAttention, self).__init__(**kwargs)
+        self.num_heads = num_heads
+        self.d_model = d_model
+
+        self.depth = self.d_model // self.num_heads
+
+        self.query_dense = tf.keras.layers.Dense(self.d_model)
+        self.key_dense = tf.keras.layers.Dense(self.d_model)
+        self.value_dense = tf.keras.layers.Dense(self.d_model)
+
+        self.dense = tf.keras.layers.Dense(self.d_model)
+
+        
+    def get_config(self):
+        config = super(MultiHeadAttention, self).get_config()
+        config.update({"num_heads": self.num_heads, "d_model": self.d_model})
+        return config
+    
+    def split_heads(self, inputs: tf.Tensor, batch_size: int):
+        inputs = tf.keras.layers.Lambda(
+            lambda inputs: tf.reshape(
+                inputs, shape=(batch_size, -1, self.num_heads, self.depth))
+        )(inputs)
+        return tf.keras.layers.Lambda(
+            lambda inputs: tf.transpose(inputs, perm=[0, 2, 1, 3])
+        )(inputs)
+    
+    def call(self, inputs: tf.Tensor):
+        query, key, value, mask = (
+            inputs["query"],
+            inputs["key"],
+            inputs["value"],
+            inputs["mask"],
+        )
+        batch_size = tf.shape(query)[0]
+
+        query = self.query_dense(query)
+        key = self.key_dense(key)
+        value = self.value_dense(value)
+
+        query = self.split_heads(query, batch_size)
+        key = self.split_heads(key, batch_size)
+        value = self.split_heads(value, batch_size)
+
+        scaled_attention = sdp_attention(query, key, value, mask)
+        scaled_attention = tf.keras.layers.Lambda(
+            lambda scaled_attention: tf.transpose(scaled_attention, perm=[0, 2, 1, 3])
+        )(scaled_attention)
+
+        concat_attention = tf.keras.layers.Lambda(
+            lambda scaled_attention: tf.reshape(
+                scaled_attention, (batch_size, -1, self.d_model)
+            )
+        )(scaled_attention)
+
+        outputs = self.dense(concat_attention)
+
+        return outputs
+    
+    
+def create_padding_mask(x):
+    mask = tf.cast(tf.math.equal(x, 0), dtype=tf.float32)
+    return mask[:, tf.newaxis, tf.newaxis, :]
+
+
+def create_look_ahead_mask(x):
+    seq_len = tf.shape(x)[1]
+    look_ahead_mask = 1 - tf.linalg.band_part(
+        tf.ones((seq_len, seq_len), dtype=tf.float32), -1, 0
+    )
+    padding_mask = create_padding_mask(x)
+    return tf.maximum(look_ahead_mask, padding_mask)
+    
+
+class PositionalEncoding(tf.keras.layers.Layer):
+    def __init__(self, position: int, d_model: int, **kwargs):
+        super(PositionalEncoding, self).__init__(**kwargs)
+        self.position = position
+        self.d_model = d_model
+        self.pos_encoding = self.positional_encoding(position, d_model)
+        
+    def get_config(self):
+        config = super(PositionalEncoding, self).get_config()
+        config.update({"position": self.position, "d_model": self.d_model})
+        return config
+    
+    def get_angles(self, position: tf.Tensor, i: tf.Tensor, d_model: tf.Tensor):
+        angles = 1 / tf.pow(10000, (2 * (i // 2)) / d_model)
+        return position * angles
+    
+    def positional_encoding(self, position: int, d_model: int):
+        angle_rads = self.get_angles(
+            position=tf.cast(tf.range(position)[:, tf.newaxis], dtype=tf.float32),
+            i=tf.cast(tf.range(d_model)[tf.newaxis, :], dtype=tf.float32),
+            d_model=tf.cast(d_model, dtype=tf.float32),
+        )
+        sines = tf.math.sin(angle_rads[:, 0::2])
+        cosines = tf.math.cos(angle_rads[:, 1::2])
+
+        pos_encoding = tf.concat([sines, cosines], axis=-1)
+        pos_encoding = pos_encoding[tf.newaxis, ...]
+        return pos_encoding
+
+    def call(self, inputs: tf.Tensor):
+        return inputs + self.pos_encoding[:, : tf.shape(inputs)[1], :]
+    
+def encoder_layer(hparams, name: str = "encoder_layer"):
+    inputs = tf.keras.Input(shape=(None, hparams.d_model), name="inputs")
+    padding_mask = tf.keras.Input(shape=(1, 1, None), name="padding_mask")
+
+    attention = MultiHeadAttention(
+        num_heads=hparams.num_heads, d_model=hparams.d_model, name="attention"
+    )({"query": inputs, "key": inputs, "value": inputs, "mask": padding_mask})
+    attention = tf.keras.layers.Dropout(hparams.dropout)(attention)
+    attention += tf.cast(inputs, dtype=tf.float32)
+    attention = tf.keras.layers.LayerNormalization(epsilon=1e-6)(attention)
+
+    outputs = tf.keras.layers.Dense(hparams.num_units, activation=hparams.activation)(
+        attention
+    )
+    outputs = tf.keras.layers.Dense(hparams.d_model)(outputs)
+    outputs = tf.keras.layers.Dropout(hparams.dropout)(outputs)
+    outputs += attention
+    outputs = tf.keras.layers.LayerNormalization(epsilon=1e-6)(outputs)
+
+    return tf.keras.Model(inputs=[inputs, padding_mask], outputs=outputs, name=name)
+
+def encoder(hparams, name: str = "encoder"):
+    inputs = tf.keras.Input(shape=(None,), name="inputs")
+    padding_mask = tf.keras.Input(shape=(1, 1, None), name="padding_mask")
+
+    embeddings = tf.keras.layers.Embedding(hparams.vocab_size, hparams.d_model)(inputs)
+    embeddings *= tf.math.sqrt(tf.cast(hparams.d_model, dtype=tf.float32))
+    embeddings = PositionalEncoding(
+        position=hparams.vocab_size, d_model=hparams.d_model
+    )(embeddings)
+
+    outputs = tf.keras.layers.Dropout(hparams.dropout)(embeddings)
+
+    for i in range(hparams.num_layers):
+        outputs = encoder_layer(hparams, name=f"encoder_layer_{i}")(
+            [outputs, padding_mask]
+        )
+
+    return tf.keras.Model(inputs=[inputs, padding_mask], outputs=outputs, name=name)
+
+def decoder_layer(hparams, name: str = "decoder_layer"):
+    inputs = tf.keras.Input(shape=(None, hparams.d_model), name="inputs")
+    enc_outputs = tf.keras.Input(shape=(None, hparams.d_model), name="encoder_outputs")
+    look_ahead_mask = tf.keras.Input(shape=(1, None, None), name="look_ahead_mask")
+    padding_mask = tf.keras.Input(shape=(1, 1, None), name="padding_mask")
+
+    attention1 = MultiHeadAttention(
+        num_heads=hparams.num_heads, d_model=hparams.d_model, name="attention_1"
+    )(
+        inputs={
+            "query": inputs,
+            "key": inputs,
+            "value": inputs,
+            "mask": look_ahead_mask,
+        }
+    )
+    attention1 += tf.cast(inputs, dtype=tf.float32)
+    attention1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)(attention1)
+
+    attention2 = MultiHeadAttention(
+        num_heads=hparams.num_heads, d_model=hparams.d_model, name="attention_2"
+    )(
+        inputs={
+            "query": attention1,
+            "key": enc_outputs,
+            "value": enc_outputs,
+            "mask": padding_mask,
+        }
+    )
+    attention2 = tf.keras.layers.Dropout(hparams.dropout)(attention2)
+    attention2 += attention1
+    attention2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)(
+        attention2 + attention1
+    )
+
+    outputs = tf.keras.layers.Dense(hparams.num_units, activation=hparams.activation)(
+        attention2
+    )
+    outputs = tf.keras.layers.Dense(hparams.d_model)(outputs)
+    outputs = tf.keras.layers.Dropout(hparams.dropout)(outputs)
+    outputs += attention2
+    outputs = tf.keras.layers.LayerNormalization(epsilon=1e-6)(outputs)
+
+    return tf.keras.Model(
+        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],
+        outputs=outputs,
+        name=name,
+    )
+
+def decoder(hparams, name: str = "decoder"):
+    inputs = tf.keras.Input(shape=(None,), name="inputs")
+    enc_outputs = tf.keras.Input(shape=(None, hparams.d_model), name="encoder_outputs")
+    look_ahead_mask = tf.keras.Input(shape=(1, None, None), name="look_ahead_mask")
+    padding_mask = tf.keras.Input(shape=(1, 1, None), name="padding_mask")
+
+    embeddings = tf.keras.layers.Embedding(hparams.vocab_size, hparams.d_model)(inputs)
+    embeddings *= tf.math.sqrt(tf.cast(hparams.d_model, dtype=tf.float32))
+    embeddings = PositionalEncoding(
+        position=hparams.vocab_size, d_model=hparams.d_model
+    )(embeddings)
+
+    outputs = tf.keras.layers.Dropout(hparams.dropout)(embeddings)
+
+    for i in range(hparams.num_layers):
+        outputs = decoder_layer(
+            hparams,
+            name="decoder_layer_{}".format(i),
+        )(inputs=[outputs, enc_outputs, look_ahead_mask, padding_mask])
+
+    return tf.keras.Model(
+        inputs=[inputs, enc_outputs, look_ahead_mask, padding_mask],
+        outputs=outputs,
+        name=name,
+    )
+
+
+
+def transformer(hparams, name: str = "transformer"):
+    inputs = tf.keras.Input(shape=(None,), name="inputs")
+    dec_inputs = tf.keras.Input(shape=(None,), name="dec_inputs")
+
+    enc_padding_mask = tf.keras.layers.Lambda(
+        create_padding_mask, output_shape=(1, 1, None), name="enc_padding_mask"
+    )(inputs)
+
+    look_ahead_mask = tf.keras.layers.Lambda(
+        create_look_ahead_mask, output_shape=(1, None, None), name="look_ahead_mask"
+    )(dec_inputs)
+
+    dec_padding_mask = tf.keras.layers.Lambda(
+        create_padding_mask, output_shape=(1, 1, None), name="dec_padding_mask"
+    )(inputs)
+
+    enc_outputs = encoder(hparams)(inputs=[inputs, enc_padding_mask])
+
+    dec_outputs = decoder(hparams)(
+        inputs=[dec_inputs, enc_outputs, look_ahead_mask, dec_padding_mask]
+    )
+
+    outputs = tf.keras.layers.Dense(hparams.vocab_size, name="outputs")(dec_outputs)
+
+    return tf.keras.Model(inputs=[inputs, dec_inputs], outputs=outputs, name=name)
+
+  

requirements.txt

@@ -0,0 +1,7 @@
+jupyter
+matplotlib
+tensorflow==2.9.1
+tensorflow-addons==0.17.1
+tensorflow-datasets==4.6.0
+protobuf==3.20.3
+nltk

training.py

@@ -0,0 +1,150 @@
+import argparse
+import tensorflow as tf
+import model
+from dataset import get_dataset, preprocess_sentence
+
+
+class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
+    def __init__(self, d_model: int, warmup_steps: int = 4000):
+        super(CustomSchedule, self).__init__()
+        self.d_model = tf.cast(d_model, dtype=tf.float32)
+        self.warmup_steps = warmup_steps
+
+    def __call__(self, step):
+        arg1 = tf.math.rsqrt(step)
+        arg2 = step * self.warmup_steps**-1.5
+        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)
+    
+    
+def inference(hparams, chatbot, tokenizer, sentence):
+    sentence = preprocess_sentence(sentence)
+
+    sentence = tf.expand_dims(
+        hparams.start_token + tokenizer.encode(sentence) + hparams.end_token, axis=0
+    )
+
+    output = tf.expand_dims(hparams.start_token, 0)
+
+    for _ in range(hparams.max_length):
+        predictions = chatbot(inputs=[sentence, output], training=False)
+
+        predictions = predictions[:, -1:, :]
+        predicted_id = tf.cast(tf.argmax(predictions, axis=-1), tf.int32)
+
+        if tf.equal(predicted_id, hparams.end_token[0]):
+            break
+
+        output = tf.concat([output, predicted_id], axis=-1)
+
+    return tf.squeeze(output, axis=0)
+
+
+
+
+def predict(hparams, chatbot, tokenizer, sentence):
+    prediction = inference(hparams, chatbot, tokenizer, sentence)
+    predicted_sentence = tokenizer.decode(
+        [i for i in prediction if i < tokenizer.vocab_size]
+    )
+    return predicted_sentence
+
+
+def evaluate(hparams, chatbot, tokenizer):
+    print("\nDeğerlendir")
+    sentence = "Merhaba nasılsın?"
+    output = predict(hparams, chatbot, tokenizer, sentence)
+    print(f"input: {sentence}\noutput: {output}")
+
+    sentence = "Sence de gökyüzü çok güzel değil mi?"
+    output = predict(hparams, chatbot, tokenizer, sentence)
+    print(f"\ninput: {sentence}\noutput: {output}")
+
+    sentence = "Sanırım uzaklara gideceğim."
+    for _ in range(5):
+        output = predict(hparams, chatbot, tokenizer, sentence)
+        print(f"\ninput: {sentence}\noutput: {output}")
+        sentence = output
+    
+    
+def main(hparams):
+    tf.keras.utils.set_random_seed(1234)
+
+    data, token = get_dataset(hparams)
+
+    chatbot = model.transformer(hparams)
+
+    optimizer = tf.keras.optimizers.Adam(
+        CustomSchedule(d_model=hparams.d_model), beta_1=0.9, beta_2=0.98, epsilon=1e-9
+    )
+
+    cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy(
+        from_logits=True, reduction="none"
+    )
+
+    def loss_function(y_true, y_pred):
+        y_true = tf.reshape(y_true, shape=(-1, hparams.max_length - 1))
+        loss = cross_entropy(y_true, y_pred)
+        mask = tf.cast(tf.not_equal(y_true, 0), dtype=tf.float32)
+        loss = tf.multiply(loss, mask)
+        return tf.reduce_mean(loss)
+
+    def accuracy(y_true, y_pred):
+        y_true = tf.reshape(y_true, shape=(-1, hparams.max_length - 1))
+        return tf.keras.metrics.sparse_categorical_accuracy(y_true, y_pred)
+
+    chatbot.compile(optimizer, loss=loss_function, metrics=[accuracy])
+
+    chatbot.fit(data, epochs=hparams.epochs)
+
+    print(f"\nmodel {hparams.save_model}'a kaydediliyor...")
+    tf.keras.models.save_model(
+        chatbot, filepath=hparams.save_model, include_optimizer=False
+    )
+
+    print(
+        f"\nclear TensorFlow backend session and load model f rom {hparams.save_model}..."
+    )
+    del chatbot
+    tf.keras.backend.clear_session()
+    chatbot = tf.keras.models.load_model(
+        hparams.save_model,
+        custom_objects={
+            "PositionalEncoding": model.PositionalEncoding,
+            "MultiHeadAttention": model.MultiHeadAttention,
+        },
+        compile=False,
+    )
+    evaluate(hparams, chatbot, token)
+
+if __name__ == "__main__":
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--save_model", default="model.h5", type=str, help="path save the model"
+    )
+    parser.add_argument(
+        "--max_samples",
+        default=25000,
+        type=int,
+        help="maximum number of conversation pairs to use",
+    )
+    parser.add_argument(
+        "--max_length", default=40, type=int, help="maximum sentence length"
+    )
+    parser.add_argument("--batch_size", default=128, type=int)
+    parser.add_argument("--num_layers", default=2, type=int)
+    parser.add_argument("--num_units", default=512, type=int)
+    parser.add_argument("--d_model", default=512, type=int)
+    parser.add_argument("--num_heads", default=8, type=int)
+    parser.add_argument("--dropout", default=0.1, type=float)
+    parser.add_argument("--activation", default="relu", type=str)
+    parser.add_argument("--epochs", default=70, type=int)
+    
+    main(parser.parse_args())
+    
+    
+    
+    
+    
+
+