Repository: explodinggradients/ragas
Issue: Generate a Synthetic Test Set & Evaluating Using Your Test Set
Root cause: Documentation Deficiency

- [x] I checked the [documentation](https://docs.ragas.io/) and related resources and couldn't find an answer to my question.

**Your Question**
what is unclear to you? What would you like to know?

I don't quite understand the example of Generate a Synthetic Test Set & Evaluating Using Your Test Set. I only get **question, contexts, and ground_truth** after the first step, but **answer** are missing.

![image](https://github.com/explodinggradients/ragas/assets/101811299/e277c154-2189-40a8-82e9-aced197d546c)
And I think from the example you given, there should be at least four columns.

**Also I am unclear about the definition of critic_llm and generator_llm, is the answer of critic_llm be considered as ground-truth and the answer of generator_llm be considered as answer?**

**Code Examples**
This community speaks code. Share your code snippets to help us understand your question better.

The code corresponding to Generate a Synthetic Test Set
```python
import os
import pandas as pd
from langchain_community.document_loaders import DirectoryLoader
from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
import os

DATA_PATH = "."

def setup_environment():
    """Configure the environment by setting the OpenAI API key."""
    os.environ["OPENAI_API_KEY"] = "MYAPIKEY"
    print("Environment set up with OpenAI API key.")

def load_documents():
    """Load documents from the specified directory using DirectoryLoader."""
    loader = DirectoryLoader(DATA_PATH, glob="**/*.md")
    documents = loader.load()
    print(f"Loaded {len(documents)} documents.")
    return documents

def create_testset_generator():
    """Create and configure the test set generator with OpenAI models."""
    generator_llm = ChatOpenAI(model="gpt-3.5-turbo-16k")
    critic_llm = ChatOpenAI(model="gpt-4")
    embeddings = OpenAIEmbeddings()
    generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings)
    print("Test set generator created.")
    return generator

def generate_test_set(documents, test_size=10):
    """Generate a synthetic test set based on the loaded documents."""
    generator = create_testset_generator()
    distributions = {
        simple: 0.5,
        reasoning: 0.25,
        multi_context: 0.25
    }
    testset = generator.generate_with_langchain_docs(documents, test_size, distributions)
    print(f"Generated test set with {test_size} questions.")
    return testset

def export_to_dataframe(testset):
    """Export the generated test set to a pandas DataFrame."""
    test_df = testset.to_pandas()
    print("Test set exported to DataFrame.")
    return test_df

def save_to_csv(test_df, filename):
    """Save the DataFrame to a CSV file."""
    test_df.to_csv(filename, index=False)
    print(f"Test set saved to {filename}.")

def main():
    setup_environment()
    documents = load_documents()
    testset = generate_test_set(documents)
    test_df = export_to_dataframe(testset)
    save_to_csv(test_df, "test_set.csv")

if __name__ == "__main__":
    main()
```

The code corresponding to evaluating your test set
```python
import os
import pandas as pd
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import context_precision, faithfulness, answer_relevancy, context_recall

# Load your test set
test_df = pd.read_csv('./test_set_new.csv')

# test_df = test_df.rename(columns={"ground_truth": "answer"})

# print(test_df)

# Assuming your test_df has 'question', 'context', and 'ground_truth' columns
# Convert your test set DataFrame to the format expected by RAGAS
test_set = {
    'question': test_df['question'].tolist(),
    'contexts': [eval(c) for c in test_df['contexts'].tolist()], # 'eval' is used to convert string representation of list back to list
    'ground_truth': test_df['ground_truth'].tolist()
}

hf_dataset = Dataset.from_pandas(test_df)

# Run the evaluation
result = evaluate(
    hf_dataset,
    metrics=[
        context_precision,
        faithfulness,
        answer_relevancy,
        context_recall,
    ],
)

# Convert the results to a pandas DataFrame
results_df = pd.DataFrame(result)
results_df.head()

# Export the evaluation results to a new CSV
results_df.to_csv('evaluation_results.csv', index=False)
```

The documents I used
[sentiment-analysis-and-dataset.md](https://github.com/explodinggradients/ragas/files/15002642/sentiment-analysis-and-dataset.md)
[sentiment-analysis-cnn.md](https://github.com/explodinggradients/ragas/files/15002643/sentiment-analysis-cnn.md)
[sentiment-analysis-rnn.md](https://github.com/explodinggradients/ragas/files/15002644/sentiment-analysis-rnn.md)
[finetuning-bert.md](https://github.com/explodinggradients/ragas/files/15002645/finetuning-bert.md)
[index.md](https://github.com/explodinggradients/ragas/files/15002646/index.md)
[natural-language-inference-and-dataset.md](https://github.com/explodinggradients/ragas/files/15002647/natural-language-inference-and-dataset.md)
[natural-language-inference-attention.md](https://github.com/explodinggradients/ragas/files/15002648/natural-language-inference-attention.md)
[natural-language-inference-bert.md](https://github.com/explodinggradients/ragas/files/15002649/natural-language-inference-bert.md)

The csv I get
<html xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">
<link id=Main-File rel=Main-File
href="file:///C:/Users/forev/AppData/Local/Temp/msohtmlclip1/01/clip.htm">
<link rel=File-List
href="file:///C:/Users/forev/AppData/Local/Temp/msohtmlclip1/01/clip_filelist.xml">
<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;
	mso-ruby-visibility:none;}
col
	{mso-width-source:auto;
	mso-ruby-visibility:none;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-number-format:General;
	text-align:general;
	vertical-align:middle;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
ruby
	{ruby-align:left;}
rt
	{color:windowtext;
	font-size:9.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:等线;
	mso-generic-font-family:auto;
	mso-font-charset:134;
	mso-char-type:none;
	display:none;}
-->
</style>
</head>

<body link="#0563C1" vlink="#954F72">


question | contexts | ground_truth | evolution_type | metadata | episode_done
-- | -- | -- | -- | -- | --
What advantage does the   decomposition trick offer in terms of complexity when computing attention   weights? | [' is soft using weighted average, where ideally   large weights are associated with the tokens to be aligned.\n\nThe   decomposition trick leads to a more desirable linear complexity than   quadratic complexity when computing attention weights.\n\nWe can use pretrained   word vectors as the input representation for downstream natural language   processing task such as natural language inference.\n\nExercises\n\nTrain the   model with other combinations of hyperparameters. Can you get better accuracy   on the test set?\n\nWhat are major drawbacks of the decomposable attention   model for natural language inference?\n\nSuppose that we want to get the   level of semantical similarity (e.g., a continuous value between 0 and 1) for   any pair of sentences. How shall we collect and label the dataset? Can you   design a model with attention   mechanisms?\n\n:begin_tab:mxnet\nDiscussions\n:end_tab:\n\n:begin_tab:pytorch\nDiscussions\n:end_tab:'] | nan | simple | [{'source':   'natural-language-inference-attention.md'}] | TRUE
What   is the purpose of the "<pad>" keyphrase in the SNLIDataset   class? | [' s) \n        s = re.sub(\'\\)\', \'\', s)\n        # Substitute two or more consecutive   whitespace with space\n        s =   re.sub(\'\\s{2,}\', \' \', s)\n          return s.strip()\n    label_set   = {\'entailment\': 0, \'contradiction\': 1, \'neutral\': 2}\n    file_name = os.path.join(data_dir,   \'snli_1.0_train.txt\'\n                             if is_train else   \'snli_1.0_test.txt\')\n    with   open(file_name, \'r\') as f:\n          rows = [row.split(\'\\t\') for row in f.readlines()[1:]]\n    premises = [extract_text(row[1]) for row   in rows if row[0] in label_set]\n      hypotheses = [extract_text(row[2]) for row in rows if row[0] in   label_set]\n    labels =   [label_set[row[0]] for row in rows if row[0] in label_set]\n    return premises, hypotheses,   labels\n```\n\nNow let\'s [print the first 3 pairs] of premise and   hypothesis, as well as their labels ("0", "1", and   "2" correspond to "entailment",   "contradiction", and "neutral", respectively   ).\n\n```{.python .input}\n\n@tab all\n\ntrain_data = read_snli(data_dir,   is_train=True)\nfor x0, x1, y in zip(train_data[0][:3], train_data[1][:3],   train_data[2][:3]):\n      print(\'premise:\', x0)\n      print(\'hypothesis:\', x1)\n      print(\'label:\', y)\n```\n\nThe training set has about 550000   pairs,\nand the testing set has about 10000 pairs.\nThe following shows that   \nthe three [labels "entailment", "contradiction", and   "neutral" are balanced] in \nboth the training set and the testing   set.\n\n```{.python .input}\n\n@tab all\n\ntest_data = read_snli(data_dir,   is_train=False)\nfor data in [train_data, test_data]:\n    print([[row for row in data[2]].count(i)   for i in range(3)])\n```\n\n[Defining a Class for Loading the   Dataset]\n\nBelow we define a class for loading the SNLI dataset by   inheriting from the Dataset class in Gluon. The argument num_steps in the   class constructor specifies the length of a text sequence so that each   minibatch of sequences will have the same shape. \nIn other words,\ntokens   after the first num_steps ones in longer sequence are trimmed, while special   tokens “<pad>” will be appended to shorter sequences until their length   becomes num_steps.\nBy implementing the __getitem__ function, we can   arbitrarily access the premise, hypothesis, and label with the index   idx.\n\n```{.python .input}\n\n@tab mxnet\n\n@save\n\nclass   SNLIDataset(gluon.data.Dataset):\n      """A customized dataset to load the SNLI   dataset."""\n    def   init(self, dataset, num_steps, vocab=None):\n        self.num_steps = num_steps\n        all_premise_tokens =   d2l.tokenize(dataset[0])\n          all_hypothesis_tokens = d2l.tokenize(dataset[1])\n        if vocab is None:\n            self.vocab =   d2l.Vocab(all_premise_tokens + all_hypothesis_tokens,\n                      ',   '�包含的信息比我们在实验中真正需要的信息丰富得多。因此，我们定义函数read_snli以仅提取数据集的一部分，然后返回前提、假设及其标签的列表。\n\n```{.python   .input}\n\n@tab all\n\n@save\n\ndef read_snli(data_dir, is_train):\n      """将SNLI数据集解析为前提、假设和标签"""\n    def extract_text(s):\n        # 删除我们不会使用的信息\n        s = re.sub(\'\\(\', \'\', s) \n        s = re.sub(\'\\)\', \'\', s)\n        # 用一个空格替换两个或多个连续的空格\n        s = re.sub(\'\\s{2,}\', \' \', s)\n        return s.strip()\n    label_set = {\'entailment\': 0,   \'contradiction\': 1, \'neutral\': 2}\n      file_name = os.path.join(data_dir, \'snli_1.0_train.txt\'\n                             if is_train else   \'snli_1.0_test.txt\')\n    with   open(file_name, \'r\') as f:\n          rows = [row.split(\'\\t\') for row in f.readlines()[1:]]\n    premises = [extract_text(row[1]) for row   in rows if row[0] in label_set]\n      hypotheses = [extract_text(row[2]) for row in rows if row[0] \\\n                in label_set]\n    labels = [label_set[row[0]] for row in   rows if row[0] in label_set]\n      return premises, hypotheses,   labels\n```\n\n现在让我们[打印前3对]前提和假设，以及它们的标签（“0”“1”和“2”分别对应于“蕴涵”“矛盾”和“中性”）。\n\n```{.python   .input}\n\n@tab all\n\ntrain_data = read_snli(data_dir, is_train=True)\nfor   x0, x1, y in zip(train_data[0][:3], train_data[1][:3],   train_data[2][:3]):\n    print(\'前提：\',   x0)\n    print(\'假设：\', x1)\n    print(\'标签：\',   y)\n```\n\n训练集约有550000对，测试集约有10000对。下面显示了训练集和测试集中的三个[标签“蕴涵”“矛盾”和“中性”是平衡的]。\n\n```{.'] | nan | simple | [{'source':   'natural-language-inference-and-dataset.md'}, {'source':   'natural-language-inference-and-dataset.md'}] | TRUE
How does a one-dimensional   convolution work? | ["Sentiment Analysis: Using Convolutional   Neural Networks\n\n:label:sec_sentiment_cnn\n\nIn :numref:chap_cnn,\nwe   investigated mechanisms\nfor processing\ntwo-dimensional image data\nwith   two-dimensional CNNs,\nwhich were applied to\nlocal features such as adjacent   pixels.\nThough originally \ndesigned for computer vision,\nCNNs are also   widely used\nfor natural language processing.\nSimply put,\njust think of any   text sequence\nas a one-dimensional image.\nIn this way,\none-dimensional   CNNs\ncan process local features\nsuch as $n$-grams in text.\n\nIn this   section,\nwe will use the textCNN model\nto demonstrate\nhow to design a CNN   architecture\nfor representing single text :cite:Kim.2014.\nCompared with   \n:numref:fig_nlp-map-sa-rnn\nthat uses an RNN architecture with GloVe   pretraining\nfor sentiment analysis,\nthe only difference in   :numref:fig_nlp-map-sa-cnn\nlies in\nthe choice of the   architecture.\n\n:label:fig_nlp-map-sa-cnn\n\n```{.python .input}\nfrom d2l   import mxnet as d2l\nfrom mxnet import gluon, init, np, npx\nfrom mxnet.gluon   import nn\nnpx.set_np()\n\nbatch_size = 64\ntrain_iter, test_iter, vocab =   d2l.load_data_imdb(batch_size)\n```\n\n```{.python .input}\n\n@tab   pytorch\n\nfrom d2l import torch as d2l\nimport torch\nfrom torch import   nn\n\nbatch_size = 64\ntrain_iter, test_iter, vocab =   d2l.load_data_imdb(batch_size)\n```\n\nOne-Dimensional Convolutions\n\nBefore   introducing the model, \nlet's see how a one-dimensional convolution   works.\nBear in mind that it is just a special case\nof a two-dimensional   convolution\nbased on the cross-correlation   operation.\n\n:label:fig_conv1d\n\nAs shown in :numref:fig_conv1d,\nin the   one-dimensional case,\nthe convolution window\nslides from left to   right\nacross the input tensor.\nDuring sliding,\nthe input subtensor (e.g.,   $0$ and $1$ in :numref:fig_conv1d) contained in the convolution window\nat a   certain position\nand the kernel tensor (e.g., $1$ and $2$ in   :numref:fig_conv1d) are multiplied elementwise.\nThe sum of these   multiplications\ngives the single scalar value (e.g., $0\\times1+1\\times2=2$   in :numref:fig_conv1d)\nat the corresponding position of the output   tensor.\n\nWe implement one-dimensional cross-correlation in the following   corr1d function. \nGiven an input tensor X\nand a kernel tensor K,\nit   returns the output tensor Y.\n\n```{.python .input}\n\n@tab all\n\ndef   corr1d(X, K):\n    w =   K.shape[0]\n    Y =   d2l.zeros((X.shape[0] - w + 1))\n      for i in range(Y.shape[0]):\n          Y[i] = (X[i: i + w] * K).sum()\n      return Y\n```\n\nWe can construct the input tensor X and the kernel   tensor K from :numref:fig_conv1d to validate the output of the above   one-dimensional cross-correlation implementation.\n\n```{.python   .input}\n\n@tab all\n\nX, K = d2l.tensor([0, 1, 2, 3, 4, 5, 6]),   d2l.tensor([1, 2])\ncorr1d(X, K)\n```\n\nFor any\none-dimensional input with   multiple channels,\nthe convolution kernel\nneeds to have the same number of   input channels.\nThen for each channel, \nperform a cross-correlation   operation on the one-dimensional tensor of the input and the one-dimensional   tensor of the convolution kernel, \nsumming the results over all the   channels\nto produce"] | In the one-dimensional case, the convolution window   slides from left to right across the input tensor. During sliding, the input   subtensor contained in the convolution window at a certain position and the   kernel tensor are multiplied elementwise. The sum of these multiplications   gives the single scalar value at the corresponding position of the output   tensor. | simple | [{'source': 'sentiment-analysis-cnn_origin.md'}] | TRUE
How   can BERT be fine-tuned for question answering? | [',\ntrivial changes can   be applied such as outputting a continuous label value\nand using the mean   squared loss: they are common for regression.\n\nText Tagging\n\nNow let\'s   consider token-level tasks, such as text tagging,\nwhere each token is   assigned a label.\nAmong text tagging tasks,\npart-of-speech tagging assigns   each word a part-of-speech tag (e.g., adjective and determiner)\naccording to   the role of the word in the sentence.\nFor example,\naccording to the Penn   Treebank II tag set,\nthe sentence "John Smith \'s car is   new"\nshould be tagged as\n"NNP (noun, proper singular) NNP POS   (possessive ending) NN (noun, singular or mass) VB (verb, base form) JJ   (adjective)".\n\n:label:fig_bert-tagging\n\nFine-tuning BERT for text tagging   applications\nis illustrated in :numref:fig_bert-tagging.\nComparing with   :numref:fig_bert-one-seq,\nthe only distinction lies in that\nin text   tagging, the BERT representation of every token of the input text\nis fed   into the same extra fully connected layers to output the label of the   token,\nsuch as a part-of-speech tag.\n\nQuestion Answering\n\nAs another   token-level application,\nquestion answering reflects capabilities of reading   comprehension.\nFor example,\nthe Stanford Question Answering Dataset (SQuAD   v1.1)\nconsists of reading passages and questions,\nwhere the answer to every   question\nis just a segment of text (text span) from the passage that the   question is about :cite:Rajpurkar.Zhang.Lopyrev.ea.2016.\nTo   explain,\nconsider a passage\n"Some experts report that a mask\'s   efficacy is inconclusive. However, mask makers insist that their products,   such as N95 respirator masks, can guard against the virus."\nand a   question "Who say that N95 respirator masks can guard against the   virus?".\nThe answer should be the text span "mask makers" in   the passage.\nThus, the goal in SQuAD v1.1 is to predict the start and end of   the text span in the passage given a pair of question and   passage.\n\n:label:fig_bert-qa\n\nTo fine-tune BERT for question   answering,\nthe question and passage are packed as\nthe first and second text   sequence, respectively,\nin the input of BERT.\nTo predict the position of   the start of the text span,\nthe same additional fully connected layer will   transform\nthe BERT representation of any token from the passage of position   $i$\ninto a scalar score $s_i$.\nSuch scores of all the passage tokens\nare   further transformed by the softmax operation\ninto a probability   distribution,\nso that each token position $i$ in the passage is assigned\na   probability $p_i$ of being the start of the text span.\nPredicting the end of   the text span\nis the same as above, except that\nparameters in its   additional fully connected layer\nare independent from those for predicting   the start.\nWhen predicting the end,\nany passage token of position $i$\nis   transformed by the same fully connected layer\ninto a scalar score   $e_i$.\n:numref:fig_bert-qa\ndepicts fine-tuning BERT for question   answering.\n\nFor question answering,\nthe supervised learning\'s training   objective is as straightforward as\nmaximizing the log-likelihoods of the   ground-truth start and end positions.\nWhen predicting the span,\nwe can   compute the score $s_i + e_j$ for a valid span\nfrom position $i$ to position   $j$ ($i \\leq j$),\nand output the span with the highest   score.\n\nSummary\n\nBERT requires minimal architecture changes (extra fully   connected layers) for sequence-level and token-level natural language   processing applications, such as single text classification (e.g., sentiment   analysis and testing linguistic acceptability), text pair classification or   regression (e.g., natural language inference and semantic textual   similarity), text tagging (e.g., part-of-speech tagging), and question   answering.\n\nDuring supervised learning of a downstream application,   parameters of the extra layers are learned from scratch while all the   parameters in the pretrained BERT model are   fine-tuned.\n\nExercises\n\nLet\'s design a search engine algorithm for news   articles. When the system receives an query (e.g., "oil industry during   the coronavirus outbreak"), it should return a ranked list of news   articles that are most relevant to the query. Suppose that'] | To fine-tune BERT for   question answering, the question and passage are packed as the first and   second text sequence, respectively, in the input of BERT. The start and end   positions of the text span in the passage are predicted using additional   fully connected layers. The training objective is to maximize the   log-likelihoods of the ground-truth start and end positions. | simple | [{'source':   'finetuning-bert.md'}] | TRUE
What is the role of the   aggregation step in the decomposable attention model? | ['{v}{A,i} = g([\\mathbf{a}_i,   \\boldsymbol{\\beta}_i]), i = 1, \\ldots, m\\ \\mathbf{v}{B,j} =   g([\\mathbf{b}_j, \\boldsymbol{\\alpha}_j]), j = 1, \\ldots,   n.$$\n\n:eqlabel:eq_nli_v_ab\n\nIn :eqref:eq_nli_v_ab, $\\mathbf{v}{A,i}$ is   the comparison between token $i$ in the premise and all the hypothesis tokens   that are softly aligned with token $i$;\nwhile $\\mathbf{v}{B,j}$ is the   comparison between token $j$ in the hypothesis and all the premise tokens   that are softly aligned with token $j$.\nThe following Compare class defines   such as comparing step.\n\n```{.python .input}\n\n@tab mxnet\n\nclass   Compare(nn.Block):\n    def init(self,   num_hiddens, kwargs):\n          super(Compare, self).init(kwargs)\n        self.g = mlp(num_hiddens=num_hiddens,   flatten=False)\n\n```\n\n```{.python .input}\n\n@tab pytorch\n\nclass   Compare(nn.Module):\n    def init(self,   num_inputs, num_hiddens, kwargs):\n          super(Compare, self).init(kwargs)\n        self.g = mlp(num_inputs, num_hiddens,   flatten=False)\n\n```\n\nAggregating\n\nWith two sets of comparison vectors   $\\mathbf{v}{A,i}$ ($i = 1, \\ldots, m$) and $\\mathbf{v}{B,j}$ ($j = 1,   \\ldots, n$) on hand,\nin the last step we will aggregate such information to   infer the logical relationship.\nWe begin by summing up both   sets:\n\n$$\n\\mathbf{v}A = \\sum{i=1}^{m} \\mathbf{v}{A,i}, \\quad   \\mathbf{v}_B = \\sum{j=1}^{n}\\mathbf{v}_{B,j}.\n$$\n\nNext we feed the   concatenation of both summarization results into function $h$ (an MLP) to   obtain the classification result of the logical relationship:\n\n$$\n\\hat{\\mathbf{y}}   = h([\\mathbf{v}_A, \\mathbf{v}_B]).\n$$\n\nThe aggregation step is defined   in the following Aggregate class.\n\n```{.python .input}\n\n@tab   mxnet\n\nclass Aggregate(nn.Block):\n      def init(self, num_hiddens, num_outputs, kwargs):\n        super(Aggregate,   self).init(kwargs)\n        self.h =   mlp(num_hiddens=num_hiddens, flatten=True)\n          self.h.add(nn.Dense(num_outputs))\n\n```\n\n```{.python   .input}\n\n@tab pytorch\n\nclass Aggregate(nn.Module):\n    def init(self, num_inputs, num_hiddens,   num_outputs, kwargs):\n          super(Aggregate, self).init(kwargs)\n        self.h = mlp(num_inputs, num_hiddens,   flatten=True)\n        self.linear =   nn.Linear(num_hiddens, num_outputs)\n\n```\n\nPutting It All Together\n\nBy   putting the attending, comparing, and aggregating steps together,\nwe define   the decomposable attention model to jointly train these three   steps.\n\n```{.python .input}\n\n@tab mxnet\n\nclass   DecomposableAttention(nn.Block):\n      def init(self, vocab, embed_size, num_hiddens, kwargs):\n        super(DecomposableAttention, self).'] | nan | simple | [{'source':   'natural-language-inference-attention.md'}] | TRUE
What   are the uses of pretrained word vectors in downstream NLP tasks, and what is   the advantage of using the decomposition trick for attention weights? | [' is soft using weighted   average, where ideally large weights are associated with the tokens to be   aligned.\n\nThe decomposition trick leads to a more desirable linear   complexity than quadratic complexity when computing attention weights.\n\nWe   can use pretrained word vectors as the input representation for downstream   natural language processing task such as natural language   inference.\n\nExercises\n\nTrain the model with other combinations of   hyperparameters. Can you get better accuracy on the test set?\n\nWhat are   major drawbacks of the decomposable attention model for natural language   inference?\n\nSuppose that we want to get the level of semantical similarity   (e.g., a continuous value between 0 and 1) for any pair of sentences. How   shall we collect and label the dataset? Can you design a model with attention   mechanisms?\n\n:begin_tab:mxnet\nDiscussions\n:end_tab:\n\n:begin_tab:pytorch\nDiscussions\n:end_tab:'] | We can use pretrained word   vectors as the input representation for downstream natural language   processing tasks such as natural language inference. The advantage of using   the decomposition trick for attention weights is that it leads to a more   desirable linear complexity than quadratic complexity when computing   attention weights. | reasoning | [{'source':   'natural-language-inference-attention.md'}] | TRUE
How can supervised learning be   used for news article search engine algorithms, particularly for text tagging   and question answering? | [' similarity,\ntrivial changes can be applied such   as outputting a continuous label value\nand using the mean squared loss: they   are common for regression.\n\nText Tagging\n\nNow let\'s consider token-level   tasks, such as text tagging,\nwhere each token is assigned a label.\nAmong   text tagging tasks,\npart-of-speech tagging assigns each word a   part-of-speech tag (e.g., adjective and determiner)\naccording to the role of   the word in the sentence.\nFor example,\naccording to the Penn Treebank II   tag set,\nthe sentence "John Smith \'s car is new"\nshould be   tagged as\n"NNP (noun, proper singular) NNP POS (possessive ending) NN   (noun, singular or mass) VB (verb, base form) JJ   (adjective)".\n\n:label:fig_bert-tagging\n\nFine-tuning BERT for text   tagging applications\nis illustrated in :numref:fig_bert-tagging.\nComparing   with :numref:fig_bert-one-seq,\nthe only distinction lies in that\nin text   tagging, the BERT representation of every token of the input text\nis fed   into the same extra fully connected layers to output the label of the   token,\nsuch as a part-of-speech tag.\n\nQuestion Answering\n\nAs another   token-level application,\nquestion answering reflects capabilities of reading   comprehension.\nFor example,\nthe Stanford Question Answering Dataset (SQuAD   v1.1)\nconsists of reading passages and questions,\nwhere the answer to every   question\nis just a segment of text (text span) from the passage that the   question is about :cite:Rajpurkar.Zhang.Lopyrev.ea.2016.\nTo explain,\nconsider   a passage\n"Some experts report that a mask\'s efficacy is inconclusive.   However, mask makers insist that their products, such as N95 respirator   masks, can guard against the virus."\nand a question "Who say that   N95 respirator masks can guard against the virus?".\nThe answer should   be the text span "mask makers" in the passage.\nThus, the goal in   SQuAD v1.1 is to predict the start and end of the text span in the passage   given a pair of question and passage.\n\n:label:fig_bert-qa\n\nTo fine-tune   BERT for question answering,\nthe question and passage are packed as\nthe   first and second text sequence, respectively,\nin the input of BERT.\nTo   predict the position of the start of the text span,\nthe same additional   fully connected layer will transform\nthe BERT representation of any token   from the passage of position $i$\ninto a scalar score $s_i$.\nSuch scores of   all the passage tokens\nare further transformed by the softmax   operation\ninto a probability distribution,\nso that each token position $i$   in the passage is assigned\na probability $p_i$ of being the start of the   text span.\nPredicting the end of the text span\nis the same as above, except   that\nparameters in its additional fully connected layer\nare independent   from those for predicting the start.\nWhen predicting the end,\nany passage   token of position $i$\nis transformed by the same fully connected layer\ninto   a scalar score $e_i$.\n:numref:fig_bert-qa\ndepicts fine-tuning BERT for   question answering.\n\nFor question answering,\nthe supervised learning\'s   training objective is as straightforward as\nmaximizing the log-likelihoods   of the ground-truth start and end positions.\nWhen predicting the span,\nwe   can compute the score $s_i + e_j$ for a valid span\nfrom position $i$ to   position $j$ ($i \\leq j$),\nand output the span with the highest   score.\n\nSummary\n\nBERT requires minimal architecture changes (extra fully   connected layers) for sequence-level and token-level natural language   processing applications, such as single text classification (e.g., sentiment   analysis and testing linguistic acceptability), text pair classification or   regression (e.g., natural language inference and semantic textual   similarity), text tagging (e.g., part-of-speech tagging), and question   answering.\n\nDuring supervised learning of a downstream application,   parameters of the extra layers are learned from scratch while all the   parameters in the pretrained BERT model are   fine-tuned.\n\nExercises\n\nLet\'s design a search engine algorithm for news   articles. When the system receives an query (e.g., "oil industry during   the coronavirus outbreak"), it should return a ranked list of news   articles that are most relevant to the query. Suppose'] | Supervised learning can be used for news article   search engine algorithms by training models to perform text tagging and   question answering tasks. For text tagging, supervised learning can be used   to assign labels to each token in a news article, such as part-of-speech   tags. This allows the search engine to understand the role of each word in   the article and improve search relevance. For question answering, supervised   learning can be used to train models to extract relevant information from   news articles and provide accurate answers to user queries. By fine-tuning   pre-trained models like BERT, the search engine can improve its ability to   understand and respond to user queries. | reasoning | [{'source': 'finetuning-bert_origin.md'}] | TRUE
What   advantage does the decomposition trick offer in terms of computational   complexity when computing attention weights in natural language inference   using pretrained word vectors for downstream tasks? | [' is soft using weighted   average, where ideally large weights are associated with the tokens to be   aligned.\n\nThe decomposition trick leads to a more desirable linear   complexity than quadratic complexity when computing attention weights.\n\nWe   can use pretrained word vectors as the input representation for downstream   natural language processing task such as natural language   inference.\n\nExercises\n\nTrain the model with other combinations of   hyperparameters. Can you get better accuracy on the test set?\n\nWhat are   major drawbacks of the decomposable attention model for natural language   inference?\n\nSuppose that we want to get the level of semantical similarity   (e.g., a continuous value between 0 and 1) for any pair of sentences. How   shall we collect and label the dataset? Can you design a model with attention   mechanisms?\n\n:begin_tab:mxnet\nDiscussions\n:end_tab:\n\n:begin_tab:pytorch\nDiscussions\n:end_tab:',   '�齐，反之亦然。这种对齐是使用加权平均的软对齐，其中理想情况下较大的权重与要对齐的词元相关联。\n\n在计算注意力权重时，分解技巧会带来比二次复杂度更理想的线性复杂度。\n\n我们可以使用预训练好的词向量作为下游自然语言处理任务（如自然语言推断）的输入表示。\n\n练习\n\n使用其他超参数组合训练模型，能在测试集上获得更高的准确度吗？\n\n自然语言推断的可分解注意模型的主要缺点是什么？\n\n假设我们想要获得任何一对句子的语义相似级别（例如，0～1之间的连续值）。我们应该如何收集和标注数据集？请尝试设计一个有注意力机制的模型。\n\n:begin_tab:mxnet\nDiscussions\n:end_tab:\n\n:begin_tab:pytorch\nDiscussions\n:end_tab:\n\n:begin_tab:paddle\nDiscussions\n:end_tab:'] | nan | multi_context | [{'source':   'natural-language-inference-attention.md'}, {'source':   'natural-language-inference-attention.md'}] | TRUE
What is the architecture of the   MLP used in fine-tuning BERT for NLI? | ['自然语言推断：微调BERT\n\n:label:sec_natural-language-inference-bert\n\n在本章的前面几节中，我们已经为SNLI数据集（   :numref:sec_natural-language-inference-and-dataset）上的自然语言推断任务设计了一个基于注意力的结构（   :numref:sec_natural-language-inference-attention）。现在，我们通过微调BERT来重新审视这项任务。正如在   :numref:sec_finetuning-bert中讨论的那样，自然语言推断是一个序列级别的文本对分类问题，而微调BERT只需要一个额外的基于多层感知机的架构，如   :numref:fig_nlp-map-nli-bert中所示。\n\n:label:fig_nlp-map-nli-bert\n\n本节将下载一个预训练好的小版本的BERT，然后对其进行微调，以便在SNLI数据集上进行自然语言推断。\n\n```{.python   .input}\nfrom d2l import mxnet as d2l\nimport json\nimport   multiprocessing\nfrom mxnet import gluon, np, npx\nfrom mxnet.gluon import   nn\nimport os\n\nnpx.set_np()\n```\n\n```{.python .input}\n\n@tab   pytorch\n\nfrom d2l import torch as d2l\nimport json\nimport   multiprocessing\nimport torch\nfrom torch import nn\nimport   os\n```\n\n```{.python .input}\n\n@tab paddle\n\nfrom d2l import paddle as   d2l\nimport warnings\nwarnings.filterwarnings("ignore")\nimport   json\nimport multiprocessing\nimport paddle\nfrom paddle import nn\nimport   os\n```\n\n[加载预训练的BERT]\n\n我们已经在 :numref:sec_bert-dataset和   :numref:sec_bert-pretrainingWikiText-2数据集上预训练BERT（请注意，原始的BERT模型是在更大的语料库上预训练的）。正如在   :numref:sec_bert-pretraining中所讨论的，原始的BERT模型有数以亿计的参数。在下面，我们提供了两个版本的预训练的BERT：“bert.base”与原始的BERT基础模型一样大，需要大量的计算资源才能进行微调，而“bert.small”是一个小版本，以便于演�',   'Natural Language Inference: Fine-Tuning   BERT\n\n:label:sec_natural-language-inference-bert\n\nIn earlier sections of   this chapter,\nwe have designed an attention-based architecture\n(in   :numref:sec_natural-language-inference-attention)\nfor the natural language   inference task\non the SNLI dataset (as described in   :numref:sec_natural-language-inference-and-dataset).\nNow we revisit this   task by fine-tuning BERT.\nAs discussed in   :numref:sec_finetuning-bert,\nnatural language inference is a sequence-level   text pair classification problem,\nand fine-tuning BERT only requires an   additional MLP-based architecture,\nas illustrated in   :numref:fig_nlp-map-nli-bert.\n\n:label:fig_nlp-map-nli-bert\n\nIn this   section,\nwe will download a pretrained small version of BERT,\nthen   fine-tune it\nfor natural language inference on the SNLI   dataset.\n\n```{.python .input}\nfrom d2l import mxnet as d2l\nimport   json\nimport multiprocessing\nfrom mxnet import gluon, np, npx\nfrom   mxnet.gluon import nn\nimport os\n\nnpx.set_np()\n```\n\n```{.python   .input}\n\n@tab pytorch\n\nfrom d2l import torch as d2l\nimport json\nimport   multiprocessing\nimport torch\nfrom torch import nn\nimport   os\n```\n\nLoading Pretrained BERT\n\nWe have explained how to pretrain BERT   on the WikiText-2 dataset in\n:numref:sec_bert-dataset and   :numref:sec_bert-pretraining\n(note that the original BERT model is   pretrained on much bigger corpora).\nAs discussed in   :numref:sec_bert-pretraining,\nthe original BERT model has hundreds of   millions of parameters.\nIn the following,\nwe provide two versions of   pretrained BERT:\n"bert.base" is about as big as the original BERT   base model that requires a lot of computational resources to   fine-tune,\nwhile "bert.small" is a small version to facilitate   demonstration.\n\n{.python .input}\nd2l.DATA_HUB[\'bert.base\'] =   (d2l.DATA_URL + \'bert.base.zip\',\n                               \'7b3820b35da691042e5d34c0971ac3edbd80d3f4\')\nd2l.DATA_HUB[\'bert.small\']   = (d2l.DATA_URL + \'bert.small.zip\',\n                                \'a4e718a47137ccd1809c9107ab4f5edd317bae2c\')\n\n```{.python   .input}\n\n@tab pytorch\n\nd2l.DATA_HUB[\'bert.base\'] = (d2l.DATA_URL +   \'bert.base.torch.zip\',\n                               \'225d66f04cae318b841a13d32af3acc165f253ac\')\nd2l.DATA_HUB[\'bert.small\']   = (d2l.DATA_URL + \'bert.small.torch.zip\',\n                                \'c72329e68a732bef0452e4b96a1c341c8910f81f\')\n```\n\nEither   pretrained BERT model contains a "vocab.json" file that defines the   vocabulary set\nand a "pretrained.params" file of the pretrained   parameters.\nWe implement the following load_pretrained_model function to   load pretrained BERT parameters.\n\n{.python .input}\ndef   load_pretrained_model(pretrained_model, num_hiddens, ffn_num_hiddens,\n                          num_heads'] | nan | multi_context | [{'source': 'natural-language-inference-bert.md'},   {'source': 'natural-language-inference-bert_origin.md'}] | TRUE
What   is the purpose of natural language inference in the SNLI dataset and how is   it used for BERT training and testing? | ["   ffn_num_hiddens=512, num_heads=4,\n      num_layers=2, dropout=0.1, max_len=512, devices=devices)\n```\n\nThe   Dataset for Fine-Tuning BERT\n\nFor the downstream task natural language   inference on the SNLI dataset,\nwe define a customized dataset class   SNLIBERTDataset.\nIn each example,\nthe premise and hypothesis form a pair of   text sequence\nand is packed into one BERT input sequence as depicted in   :numref:fig_bert-two-seqs.\nRecall :numref:subsec_bert_input_rep that segment   IDs\nare used to distinguish the premise and the hypothesis in a BERT input   sequence.\nWith the predefined maximum length of a BERT input sequence   (max_len),\nthe last token of the longer of the input text pair keeps getting   removed until\nmax_len is met.\nTo accelerate generation of the SNLI   dataset\nfor fine-tuning BERT,\nwe use 4 worker processes to generate   training or testing examples in parallel.\n\n```{.python .input}\nclass   SNLIBERTDataset(gluon.data.Dataset):\n      def init(self, dataset, max_len, vocab=None):\n        all_premise_hypothesis_tokens =   [[\n            p_tokens, h_tokens] for   p_tokens, h_tokens in zip(\n              *[d2l.tokenize([s.lower() for s in sentences])\n              for sentences in   dataset[:2]])]\n\n```\n\n```{.python .input}\n\n@tab pytorch\n\nclass   SNLIBERTDataset(torch.utils.data.Dataset):\n    def init(self, dataset, max_len,   vocab=None):\n          all_premise_hypothesis_tokens = [[\n            p_tokens, h_tokens] for p_tokens,   h_tokens in zip(\n              *[d2l.tokenize([s.lower() for s in sentences])\n              for sentences in   dataset[:2]])]\n\n```\n\nAfter downloading the SNLI dataset,\nwe generate   training and testing examples\nby instantiating the SNLIBERTDataset   class.\nSuch examples will be read in minibatches during training and   testing\nof natural language inference.\n\n```{.python .input}\n\nReduce   batch_size if there is an out of memory error. In the original BERT\n\nmodel,   max_len = 512\n\nbatch_size, max_len, num_workers = 512, 128,   d2l.get_dataloader_workers()\ndata_dir =   d2l.download_extract('SNLI')\ntrain_set = SNLIBERTDataset(d2l.read_snli(data_dir,   True), max_len, vocab)\ntest_set = SNLIBERTDataset(d2l.read_snli(data_dir,   False), max_len, vocab)\ntrain_iter = gluon.data.DataLoader(train_set,   batch_size, shuffle=True,\n                                     num_workers=num_workers)\ntest_iter = gluon.data.DataLoader(test_set,   batch_size,\n                                    num_workers=num_workers)\n```\n\n```{.python .input}\n\n@tab   pytorch\n\nReduce batch_size if there is an out of memory error. In the   original BERT\n\nmodel, max_len = 512\n\nbatch_size, max_len, num_workers =   512, 128, d2l.get_dataloader_workers()\ndata_dir =   d2l.download_extract('SNLI')\ntrain_set",   "��入文本对中较长文本的最后一个标记，直到满足max_len。为了加速生成用于微调BERT的SNLI数据集，我们使用4个工作进程并行生成训练或测试样本。\n\n```{.python   .input}\nclass SNLIBERTDataset(gluon.data.Dataset):\n    def init(self, dataset, max_len,   vocab=None):\n        all_premise_hypothesis_tokens   = [[\n            p_tokens, h_tokens]   for p_tokens, h_tokens in zip(\n              *[d2l.tokenize([s.lower() for s in sentences])\n              for sentences in   dataset[:2]])]\n\n```\n\n```{.python .input}\n\n@tab pytorch\n\nclass   SNLIBERTDataset(torch.utils.data.Dataset):\n    def init(self, dataset, max_len,   vocab=None):\n          all_premise_hypothesis_tokens = [[\n            p_tokens, h_tokens] for p_tokens,   h_tokens in zip(\n            *[d2l.tokenize([s.lower()   for s in sentences])\n              for   sentences in dataset[:2]])]\n\n```\n\n```{.python .input}\n\n@tab   paddle\n\nclass SNLIBERTDataset(paddle.io.Dataset):\n    def init(self, dataset, max_len,   vocab=None):\n          all_premise_hypothesis_tokens = [[\n            p_tokens, h_tokens] for p_tokens,   h_tokens in zip(\n              *[d2l.tokenize([s.lower() for s in sentences])\n              for sentences in   dataset[:2]])]\n\n```\n\n下载完SNLI数据集后，我们通过实例化SNLIBERTDataset类来[生成训练和测试样本]。这些样本将在自然语言推断的训练和测试期间进行小批量读取。\n\n```{.python   .input}\n\n如果出现显存不足错误，请减少“batch_size”。在原始的BERT模型中，max_len=512\n\nbatch_size,   max_len, num_workers = 512, 128, d2l.get_dataloader_workers()\ndata_dir =   d2l.download_extract('SNLI')\ntrain_set = SNLIBERTDataset(d2l.read_snli(data_dir,   True), max_len, vocab)\ntest_set = SNLIBERTDataset(d2l.read_snli(data_dir,   False), max_len, vocab)\ntrain_iter = gluon.data.DataLoader(train_set,   batch_size, shuffle=True,\n                               "] | The purpose of natural   language inference in the SNLI dataset is to train and test BERT for   understanding the relationship between two text sequences, namely the premise   and the hypothesis. It is used to generate training and testing examples for   fine-tuning BERT by instantiating the SNLIBERTDataset class and reading the   examples in minibatches during training and testing. | multi_context | [{'source':   'natural-language-inference-bert_origin.md'}, {'source':   'natural-language-inference-bert.md'}] | TRUE



</body>

</html>


**Additional context**
Anything else you want to share with us? 


Thanks for raising this question @YikunHan42 🙂 
this was a issue one our end with not updating the docs. We changes `answer` to `ground_truth` to clear some confusion so what you want is `ground_truth`

I'll update the docs before closing this! Thanks for reporting this

**Great! I think from the example you gave, there should be at least four columns. Could you further explain the definition of critic_llm and generator_llm? Is the answer of critic_llm considered ground truth and the answer of generator_llm considered an answer?**

@jjmachan I've had the same issue when using the Azure OpenAI.

When using evaluating metric (such as **"[Answer semantic similarity](https://docs.ragas.io/en/latest/concepts/metrics/semantic_similarity.html)"**, "Answer Correctness, and Answer Critique" that require **both** `ground_truth` and `answer`, how should we interpret the given output from a synthetic generation - it is the `ground_truth`, the `answer` or both?

Specifically in the case when I combine multiple metrics in an evaluation, I get the following error for `Context Precision` despite `ground_truth` being present, an error is raised of course for `Faithfulness`, missing answer. When evaluating `Context Precision` in isolation it works as expected.

![image](https://github.com/explodinggradients/ragas/assets/6446120/18419bb5-4130-45e9-b6be-332960f05699)


![image](https://github.com/explodinggradients/ragas/assets/6446120/8c793029-59e3-4d78-863a-2df329d7b523)


Hi everyone, is there a workaround for the nan in ground_truth when generating the dataset

I'm also facing same issue im using Azure open AI API, answer cloumn is missing in generated test data
![image](https://github.com/explodinggradients/ragas/assets/37367345/075f6538-2fd9-43a0-ad4c-1cfdb86c634d)


@manojpraba updated the docs for that - we had changed the output format
@ayoubazaouyat that could be due to JSON following capabilities of the model. which model are you using?

the original issue has been fixed with #1176. feel free to comment below if there is anything more 🙂