Repository: promptfoo/promptfoo
Issue: Sharing Breaks Due To Unicode Character Corruption During Red Team Evaluation
Root cause: Validation Gap

We ran a set of 1200 tests, reviewed them in the local "view" and then tried to share the results to the cloud. 
The sharing failed repeatedly at one specific record.
Claude managed to fix the database records within the local promptfoo SQLite database so that the upload works now.

Below is the error description of that issue.
I also attach a Python script that was used to fix the values (if somebody else runs into this issue and wants to try a quick fix, use on your own risk and backup your db first): [fix_null_unicode_corruption.txt](https://github.com/user-attachments/files/21500820/fix_null_unicode_corruption.txt)

## Bug Report: Unicode Character Corruption During Red Team Evaluation

### Summary
Unicode characters are corrupted during promptfoo red team evaluations, resulting in null bytes that cause database insert failures when sharing evaluation results.

### Environment
- promptfoo version: Latest (as of 2025-07-30)
- Database: SQLite
- Node.js: v21.6.1
- Platform: macOS Darwin 24.5.0
- Red team strategies used: GOAT, crescendo, mischievous-user
- Test configuration: Multi-turn conversations with stateful strategies

### Description
During red team evaluation runs, Unicode characters (specifically the registered trademark symbol ®) are being corrupted by replacement with null bytes. The corruption pattern is consistent:
- Original: `Product® Plus`
- Corrupted: `Product\u0000ae Plus` (where \u0000 is a null byte followed by "ae")

This corruption appears in multiple database fields:
- test_case
- response  
- grading_result
- metadata
- prompt
- named_scores

### Impact
1. **Sharing Failure**: Evaluation sharing fails with HTTP 500 Internal Server Error
2. **Progress Blocking**: Share operation consistently stops at the same record (e.g., "59% | 683/1154 results")
3. **Database Integrity**: SQLite database contains invalid null characters preventing data export

### Error Details
```
Failed to send results chunk to https://www.promptfoo.app/api/v1/results/eval-XXX/results
status code: 500
status text: Internal Server Error
body: {
  "message": "Internal Server Error",
  "details": "Error: Failed query: insert into \"eval_results\" (\"id\", \"created_at\", 
             \"updated_at\", \"eval_id\", \"prompt_idx\", \"test_idx\", \"test_case\", 
             \"prompt\", \"prompt_id\", \"provider\", \"latency_ms\", \"cost\", \"response\", 
             \"error\", \"failure_reason\", \"success\", \"score\", \"grading_result\", 
             \"named_scores\", \"plugin_id\", \"metadata\", \"searchable_vars\", 
             \"searchable_response\", \"searchable_grading_results\") values ($1, default, ...)"
}
```

### Steps to Reproduce
1. Configure red team evaluation with Unicode characters in test data:
   - Product names containing ®, ™ symbols
   - System prompts or metadata with Unicode content
2. Run evaluation: `npx promptfoo eval -c redteam-testgen-XXX.yaml --no-cache`
3. Attempt to share results: `npx promptfoo share eval-XXX`
4. Observe consistent failure at specific test_idx

### Diagnostic Steps
```bash
# Identify failing record
export PROMPTFOO_SHARE_CHUNK_SIZE=1
npx promptfoo share eval-XXX

# Check database for null characters
sqlite3 ~/.promptfoo/promptfoo.db "SELECT test_idx FROM eval_results WHERE eval_id='eval-XXX' AND test_case LIKE '%' || char(0) || '%'"
```

### Observed Patterns
- Corruption occurs in JSON-serialized data structures
- Both literal null bytes (\x00) and escaped sequences (\\u0000) appear
- Affects multi-turn conversation data with complex nested JSON
- Consistent reproduction with specific Unicode characters

### Workaround
Manual database repair script available to fix corrupted records:
```python
# Fix both literal and escaped null patterns
text = text.replace('\x00ae', '®')      # Literal null byte + ae
text = text.replace('\\u0000ae', '®')    # JSON-escaped null + ae
```

### Suggested Investigation
1. Unicode handling in JSON serialization/deserialization pipeline
2. Character encoding during test generation
3. Database driver Unicode support
4. Multi-turn conversation state persistence

### Additional Context
- Issue discovered during large-scale red team evaluation (1154 test cases)
- Affects shareable URL generation for evaluation results
- Local evaluation runs complete successfully despite corruption
- Problem only surfaces during share/upload operation

Thanks for the detailed report.  What kind of target provider are you using?  e.g. HTTP, custom python, etc?

Hi there, its a custom python provider.

Hi @ben-bl! It's great that you were able to fix it. I put together a little script: https://gist.github.com/mldangelo/9880a2cc9e30b854aa95b8812264194b I'm going to see if we can forward fix it in promptfoo for you. Feel free to throw time on my calendar: https://cal.com/michael-dangelo/15min if you'd like chat further. 

Hi @ben-bl,

Sorry for the bug and for not closing this sooner - this was fixed in **PR #5108** on July 30, 2025 (same day you reported it!), released in version 0.118.0.

Unicode characters were getting corrupted in Python providers during red team evaluations. This has been resolved - character encoding now works correctly and sharing functionality preserves Unicode content.

Please pull the latest version and Unicode characters should be preserved correctly. If you encounter any similar issues, feel free to comment or re-open.

See the [Python Provider documentation](https://www.promptfoo.dev/docs/providers/python) for more info.

Thanks for the quick report!