Repository: EleutherAI/lm-evaluation-harness
Issue: TypeError with torch.ones() API in PyTorch 2.7+ - Invalid combination of arguments in huggingface.py
Root cause: External Dependency Breakage

**Bug Description**
When using lm-eval-harness with PyTorch 2.7+, the evaluation fails with a `TypeError` due to incompatible `torch.ones()` API usage in the HuggingFace model backend.

**Error Message**
```
TypeError: ones() received an invalid combination of arguments - got (tuple, device=torch.device), but expected one of:
 * (tuple of ints size, *, tuple of names names, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
 * (tuple of ints size, *, Tensor out = None, torch.dtype dtype = None, torch.layout layout = None, torch.device device = None, bool pin_memory = False, bool requires_grad = False)
```

**Environment**
- **lm-eval version**: 0.4.9.1
- **PyTorch version**: 2.7.1+ (any version 2.7 or higher)
- **Python version**: 3.12
- **Platform**: Any platform with PyTorch 2.7+

**Location**
File: `lm_eval/models/huggingface.py`
Lines: 797-800 and 807-809

**Root Cause**
The current code uses the old PyTorch API pattern:
```python
torch.ones((batch_size, max_length), device=self.device).long()
```

In PyTorch 2.7+, this causes a TypeError because the API no longer accepts both a tuple and device as positional arguments when followed by `.long()`.

**Proposed Fix**
Replace the problematic calls with PyTorch version-agnostic code using `.to()`:
```python
# Current (broken in PyTorch 2.7+)
test_batch = torch.ones((batch_size, max_length), device=self.device).long()

# Fixed - works across all PyTorch versions
test_batch = torch.ones((batch_size, max_length)).to(device=self.device, dtype=torch.long)
```

**Affected Lines**
All three occurrences in the `_detect_batch_size` method's `forward_batch` function:
1. Line 797: `batched_conts = torch.ones((batch_size, length), device=self.device).long()`
2. Line 800: `test_batch = torch.ones((batch_size, length), device=self.device).long()`
3. Line 807-809: `test_batch = torch.ones((batch_size, max_length), device=self.device).long()`

**Impact**
- Affects all evaluations using the HuggingFace backend with PyTorch 2.7+
- Prevents batch size detection from working properly
- Causes evaluation to fail during the `_detect_batch_size` phase

**Backward Compatibility**
The proposed fix using `.to()` is backward compatible with older PyTorch versions and avoids all PyTorch API version conflicts.

## Reproduction Steps
1. Install lm-eval-harness 0.4.9.1
2. Install PyTorch 2.7.1+
3. Run any evaluation using the HuggingFace backend (e.g., `leaderboard_bbh`)
4. Observe the TypeError during batch size detection phase

**Simple reproduction command:**
```bash
lm_eval --model hf --model_args pretrained=any-hf-model --tasks any-task --batch_size auto
```

**Example:**
```bash
lm_eval --model hf --model_args pretrained=meta-llama/Meta-Llama-3-8B-Instruct --tasks hellaswag --batch_size auto
```

## Temporary Workaround
Until this is fixed upstream, users can apply the fix manually:

```bash
# Navigate to your Python environment's lm_eval models directory
cd $VIRTUAL_ENV/lib/python*/site-packages/lm_eval/models/

# Create backup
cp huggingface.py huggingface.py.backup

# Apply fix using .to() method (works across all PyTorch versions)
sed -i 's/torch\.ones((\([^)]*\), device=\([^)]*\))\.long()/torch.ones(\1).to(device=\2, dtype=torch.long)/g' huggingface.py
```

## Additional Context
- This issue affects all PyTorch 2.7+ users attempting to use the HuggingFace backend
- The problem occurs specifically during automatic batch size detection when `batch_size="auto"`
- Manual batch size specification may bypass the issue but limits functionality
- This should be prioritized for the next release to maintain compatibility with current PyTorch versions
- **Status**: Issue confirmed and fix tested successfully

## Fix Verification
The proposed fix has been tested and confirmed working with:
- **PyTorch version**: 2.7.1+  
- **lm-eval version**: 0.4.9.1
- **Python version**: 3.12
- **Test case**: HuggingFace backend evaluations with `batch_size="auto"`
- **Result**: ✅ No more TypeError, evaluation proceeds normally