Repository: pytorch
Issue: `test_triton_bsr_scatter_mm` fails on V100
Labels: module: sparse,triaged

### 🐛 Describe the bug

Tested in a current source build and the latest nightly+cu121.
Repro steps:
```python
python test_sparse_csr.py -v -k test_triton_bsr_scatter_mm
```
Output:
```python
======================================================================
ERROR: test_triton_bsr_scatter_mm_blocksize_16x32_cuda_float16 (__main__.TestSparseCompressedTritonKernelsCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/src/pytorch/test/test_sparse_csr.py", line 3828, in test_triton_bsr_scatter_mm
    result = bsr_scatter_mm(bsr, dense, indices_data=indices_data)
  File "/usr/local/lib/python3.10/dist-packages/torch/sparse/_triton_ops.py", line 781, in bsr_scatter_mm
    scatter_mm(bsr.values(), other, indices_data, accumulators=out)
  File "/usr/local/lib/python3.10/dist-packages/torch/sparse/_triton_ops.py", line 411, in scatter_mm
    _scatter_mm6(blocks, others, c_indices, r_offsets, p_offsets, q_offsets, meta, accumulators)
  File "/usr/local/lib/python3.10/dist-packages/torch/sparse/_triton_ops.py", line 1745, in _scatter_mm6
    _scatter_mm6_kernel[grid](
  File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 532, in run
    self.cache[device][key] = compile(
  File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 543, in compile
    next_module = compile_kernel(module)
  File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 441, in <lambda>
    lambda src: ttgir_to_llir(src, extern_libs, target, tma_infos))
  File "/usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py", line 167, in ttgir_to_llir
    return translate_triton_gpu_to_llvmir(mod, target.capability, tma_infos, runtime.TARGET.NVVM)
IndexError: map::at

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 416, in instantiated_test
    result = test(self, **param_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 1110, in only_fn
    return fn(slf, *args, **kwargs)
  File "/workspace/src/pytorch/test/test_sparse_csr.py", line 3829, in test_triton_bsr_scatter_mm
    except triton.compiler.OutOfResources:
AttributeError: module 'triton.compiler' has no attribute 'OutOfResources'

To execute this test, run the following from the base repo dir:
     python test/test_sparse_csr.py -k test_triton_bsr_scatter_mm_blocksize_16x32_cuda_float16

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_triton_bsr_scatter_mm_blocksize_16_cuda_float16 (__main__.TestSparseCompressedTritonKernelsCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 416, in instantiated_test
    result = test(self, **param_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 1110, in only_fn
    return fn(slf, *args, **kwargs)
  File "/workspace/src/pytorch/test/test_sparse_csr.py", line 3834, in test_triton_bsr_scatter_mm
    self.assertEqual(result, expected)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 3518, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not close!

Mismatched elements: 256 / 512 (50.0%)
Greatest absolute difference: 19.65625 at index (1, 15) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (0, 4) (up to 0.001 allowed)

To execute this test, run the following from the base repo dir:
     python test/test_sparse_csr.py -k test_triton_bsr_scatter_mm_blocksize_16_cuda_float16

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_triton_bsr_scatter_mm_blocksize_32_cuda_float16 (__main__.TestSparseCompressedTritonKernelsCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 416, in instantiated_test
    result = test(self, **param_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 1110, in only_fn
    return fn(slf, *args, **kwargs)
  File "/workspace/src/pytorch/test/test_sparse_csr.py", line 3834, in test_triton_bsr_scatter_mm
    self.assertEqual(result, expected)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 3518, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not close!

Mismatched elements: 512 / 1024 (50.0%)
Greatest absolute difference: 39.125 at index (25, 6) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (0, 4) (up to 0.001 allowed)

To execute this test, run the following from the base repo dir:
     python test/test_sparse_csr.py -k test_triton_bsr_scatter_mm_blocksize_32_cuda_float16

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

======================================================================
FAIL: test_triton_bsr_scatter_mm_blocksize_64_cuda_float16 (__main__.TestSparseCompressedTritonKernelsCUDA)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 2674, in wrapper
    method(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 416, in instantiated_test
    result = test(self, **param_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_device_type.py", line 1110, in only_fn
    return fn(slf, *args, **kwargs)
  File "/workspace/src/pytorch/test/test_sparse_csr.py", line 3834, in test_triton_bsr_scatter_mm
    self.assertEqual(result, expected)
  File "/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_utils.py", line 3518, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not close!

Mismatched elements: 2048 / 4096 (50.0%)
Greatest absolute difference: 73.3125 at index (37, 20) (up to 1e-05 allowed)
Greatest relative difference: 1.0 at index (0, 4) (up to 0.001 allowed)

To execute this test, run the following from the base repo dir:
     python test/test_sparse_csr.py -k test_triton_bsr_scatter_mm_blocksize_64_cuda_float16

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
Ran 30 tests in 121.144s

FAILED (failures=3, errors=1, skipped=18)
```
Unsure, if and how the numerical mismatches are related to the `IndexError` and `AttributeError`.
The tests work fine on e.g. H100s.

### Versions

```
pytorch-triton            2.2.0+e28a256d71
torch                     2.3.0.dev20240110+cu121
```

CC @eqy 

cc @alexsamardzic @nikitaved @pearu @cpuhrsch @amjames @bhosmer @jcaip