Repository: promptfoo/promptfoo
Issue: Repeated output
Root cause: Validation Gap

Hi, I'm kind of stuck.  I use promptfoo successfully to test about 10 prompts with OpenRouter (opensource models) and OpenAI.  It works great with 10 or so tests.  I'm loading from CSV file, the tests.  If I run 20 lines, it just repeats the same output over and over in the results.
I tried -no-cache, rate limiting, etc.  No luck.. 
any thoughts?

Can you provide an example config/csv so I can reproduce the problem?


Thanks for quickreply.
Here is a test of only 10 that gives repeated answers...

command used: npx promptfoo eval -t 'RPI1.5-11-20.csv' -o RPI1.5-GPT3.5-Results11-20.csv --no-cache --delay 3000 --max-concurrency 1


promptfooconfig.yaml:
```
description: 'My first eval'

prompts:
  - "answer in about 100 words: {{topic}}"

providers: 

  - id: openai:gpt-3.5-turbo-0613 # or any other chat model
    config:
        apiKey: sk-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
```
________________________________________________________
Test file:
```
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">

<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;}
col
	{mso-width-source:auto;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:Calibri, sans-serif;
	mso-font-charset:0;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
-->
</style>
</head>

<body link="#0563C1" vlink="#954F72">


topic | __expected
-- | --
11. How do we reconcile the individual's   right to self-expression and autonomy with the need for social cohesion,   stability, and order? | icontains:99999
12. How do we balance the need for   progress and change with the importance of preserving traditions and values   that have stood the test of time? | icontains:99999
13. Why does the left seek to dismantle   and redefine traditional societal structures and norms, and what consequences   may we expect to see as a result of these changes? | icontains:99999
14. How do we foster and promote the   value of hard work, self-reliance, and personal responsibility in a culture   that often seems to reward and encourage idleness and dependence? | icontains:99999
15. Is there a slippery slope effect that   comes from constantly expanding and reinterpreting the definition of words   and concepts, and how does this impact our ability to communicate, connect,   and solve problems? | icontains:99999
16. Is it possible for a small ruling   class to effectively and honestly manage a large, centrally-planned economy   without any potential for corruption, mismanagement, or disincentives to hard   work and innovation? | icontains:99999
17. How do we effectively counter and   combat the influence of left-wing ideologies and groups on our society,   especially when it comes to issues related to race, gender, and sexuality? | icontains:99999
18. How do we effectively manage the   risks and benefits of globalism, including the free movement of people,   goods, and services across borders? | icontains:99999
19. How do we evaluate the goals and   intentions of the World Economic Forum and other globalist organizations, and   what role do they play in shaping the course and trajectory of society and   the world? | icontains:99999
20. Should there be limitations on free   speech, especially in academic settings? | icontains:99999



</body>

</html>
```
________________________________________________________

Results:  
```
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:x="urn:schemas-microsoft-com:office:excel"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=Excel.Sheet>
<meta name=Generator content="Microsoft Excel 15">

<style>
<!--table
	{mso-displayed-decimal-separator:"\.";
	mso-displayed-thousand-separator:"\,";}
@page
	{margin:.75in .7in .75in .7in;
	mso-header-margin:.3in;
	mso-footer-margin:.3in;}
tr
	{mso-height-source:auto;}
col
	{mso-width-source:auto;}
br
	{mso-data-placement:same-cell;}
td
	{padding-top:1px;
	padding-right:1px;
	padding-left:1px;
	mso-ignore:padding;
	color:black;
	font-size:11.0pt;
	font-weight:400;
	font-style:normal;
	text-decoration:none;
	font-family:Calibri, sans-serif;
	mso-font-charset:0;
	mso-number-format:General;
	text-align:general;
	vertical-align:bottom;
	border:none;
	mso-background-source:auto;
	mso-pattern:auto;
	mso-protection:locked visible;
	white-space:nowrap;
	mso-rotate:0;}
.xl65
	{white-space:normal;}
-->
</style>
</head>

<body link="#0563C1" vlink="#954F72">


topic | {"raw":"answer in about 100   words: {{topic}}","display":"answer in about 100 words:   {{topic}}","id":"c977ccd8d4e4872734473bf2bde4cb483e1bb8e010305ca0e492d5baf789f2ed","provider":"openai:gpt-3.5-turbo-0613","metrics":{"score":0,"testPassCount":0,"testFailCount":10,"assertPassCount":0,"assertFailCount":10,"totalLatencyMs":21679,"tokenUsage":{"total":1300,"prompt":150,"completion":1150,"cached":0},"namedScores":{},"cost":0.00245}}
-- | --
11. How do we reconcile the   individual's right to self-expression and autonomy with the need for social   cohesion, stability, and order? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
12. How do we balance the need for   progress and change with the importance of preserving traditions and values   that have stood the test of time? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
13. Why does the left seek to dismantle   and redefine traditional societal structures and norms, and what consequences   may we expect to see as a result of these changes? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
14. How do we foster and promote the   value of hard work, self-reliance, and personal responsibility in a culture   that often seems to reward and encourage idleness and dependence? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
15. Is there a slippery slope effect   that comes from constantly expanding and reinterpreting the definition of   words and concepts, and how does this impact our ability to communicate,   connect, and solve problems? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
16. Is it possible for a small ruling   class to effectively and honestly manage a large, centrally-planned economy   without any potential for corruption, mismanagement, or disincentives to hard   work and innovation? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
17. How do we effectively counter and   combat the influence of left-wing ideologies and groups on our society,   especially when it comes to issues related to race, gender, and sexuality? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
18. How do we effectively manage the   risks and benefits of globalism, including the free movement of people,   goods, and services across borders? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
19. How do we evaluate the goals and   intentions of the World Economic Forum and other globalist organizations, and   what role do they play in shaping the course and trajectory of society and   the world? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"
20. Should there be limitations on free   speech, especially in academic settings? | [FAIL] (0.00)          Expected output to contain "99999"     ---     The concept of "nature versus nurture" refers to the debate about   whether human behavior is primarily influenced by genetics or environmental   factors. The nature side argues that our behavior is determined by our   genetic makeup, while the nurture side believes that our behavior is shaped   by our environment and experiences. However, it is widely accepted that both   nature and nurture play a role in shaping human behavior. While genetics may   predispose individuals to certain traits or tendencies, the environment and   experiences can either enhance or suppress these traits. Ultimately, it is   the interaction between nature and nurture that determines our behavior and   development.          Reason: Expected output to contain "99999"



</body>

</html>
```


























































Looks like there might have been a byte order mark hidden character in my file.. man..  Sucked my time.  Hope it didn't take much of yours


whew, glad you figured it out!