Repository: evalplus/evalplus
Issue: MbppPlus has problems in reference solutions or instructions
Root cause: Validation Gap

### Mbpp/459
Problem Description
```
Write a function to remove uppercase substrings from a given string.
assert remove_uppercase('cAstyoUrFavoRitETVshoWs') == 'cstyoravoitshos'
```

Reference solution
```python3
# wrong version (Now)
def remove_uppercase(str1):
  return ''.join(c for c in str1 if c.islower())
# remove_uppercase("Hello World") = "elloorld"
```

Correct reference solution
```python3
# correct version
def remove_uppercase(str1):
    return ''.join(c for c in str1 if not c.isupper()) # just need to remove uppercase characters 
# remove_uppercase("Hello World") = "ello orld"
```

cc @ganler 

Yes this GT is wrong and needs to be fixed.

Also, let's acknowledge https://github.com/crux-eval/eval-arena for the finding. 

### Mbpp/581
Problem Description
```
Write a python function to find the surface area of a square pyramid with a given base edge and height.
assert surface_Area(3,4) == 33
```

The `Height` could also indicate the perpendicular distance from the base to the apex of the pyramid. We could add test logic to accept for both `height` and `slant height` solutions.

- `height` solution from GPT-4:
```python3
def surface_Area(base_edge, height):
    # Calculate the slant height of the pyramid
    slant_height = math.sqrt((base_edge / 2) ** 2 + height ** 2)
    # Calculate the base area
    base_area = base_edge ** 2
    # Calculate the lateral surface area (4 times the area of one triangle)
    lateral_area = 4 * (base_edge * slant_height) / 2
    # Calculate the total surface area
    total_surface_area = base_area + lateral_area
    return round(total_surface_area)
```

- `slant height` solution from GT:
```python3
def surface_Area(b,s): 
    return 2 * b * s + pow(b,2) 
```

### Mbpp/102
Problem Description
```
Write a function to convert a snake case string to camel case string.
assert snake_to_camel('python_program')=='PythonProgram'
```

The example in the description is incorrect. For "Camel Case", the first word does not have its initial letter capitalized.
New Description should be:
```
Write a function to convert a snake case string to camel case string.
assert snake_to_camel('python_program')=='pythonProgram'
```

Also the reference solution
```python3
# wrong version (Now)
def snake_to_camel(word):
    return ''.join(x.capitalize() or '_' for x in word.split('_'))
```

Should be correct as:
```python3
def snake_to_camel(snake_str):
    components = snake_str.split('_')
    return components[0] + ''.join(x.title() for x in components[1:])
```

### Mbpp/558
Description:
```
Write a python function to find the sum of the per-digit difference between two integers.
assert digit_distance_nums(1,2) == 1
```
This description does not specify the program's behavior where the two integers have varying lengths. But the "plus_inputs"  includes some cases like this.

Regarding the various model responses to such scenarios:
- GPT-4 opts to trigger an assertion error when faced with these situations.
- GT disregards the additional digits present in the longer integer.
- A number of other models handle this by padding the shorter integer with leading zeros.

We could add special logic in evalplus or just remove this problem.

### Mbpp/87
Problem Description
```
Write a function to merge three dictionaries into a single dictionary.
assert merge_dictionaries_three({ "R": "Red", "B": "Black", "P": "Pink" }, { "G": "Green", "W": "White" },{ "O": "Orange", "W": "White", "B": "Black" })=={'B': 'Black', 'R': 'Red', 'P': 'Pink', 'G': 'Green', 'W': 'White', 'O': 'Orange'}
```

This description does not specify the order in which values are updated during a key conflict (dict1 -> dict2 -> dict3 or ...).

Reference solution
```python3
# Update order: dict3 -> dict2 -> dict1 
import collections as ct
def merge_dictionaries_three(dict1,dict2, dict3):
    merged_dict = dict(ct.ChainMap({},dict1,dict2,dict3))
    return merged_dict
```

GPT-4's solution:
```python3
# Update order: dict1 -> dict2 -> dict3 
def merge_dictionaries_three(dict1, dict2, dict3):
    merged_dict = {**dict1, **dict2, **dict3}
    return merged_dict
```

We could add special logic in evalplus.

### Mbpp/559
Problem Description
```
Write a function to find the sum of the largest contiguous sublist in the given list.
assert max_sub_array_sum([-2, -3, 4, -1, -2, 1, 5, -3], 8) == 7
```

Reference solution
```python3
def max_sub_array_sum(a, size):
  max_so_far = 0
  max_ending_here = 0
  for i in range(0, size):
    max_ending_here = max_ending_here + a[i]
    if max_ending_here < 0:
      max_ending_here = 0
    elif (max_so_far < max_ending_here):
      max_so_far = max_ending_here
  return max_so_far
```

I think initializing `max_so_far` to `0` is incorrect, even if description doesn't mention about this.

New version is:
```python3
def max_sub_array_sum(a, size):
  max_so_far = a[0] if len(a) else 0 # initialize with the first element 
  max_ending_here = 0
  for i in range(0, size):
    max_ending_here = max_ending_here + a[i]
    if max_ending_here < 0:
      max_ending_here = 0
    elif (max_so_far < max_ending_here):
      max_so_far = max_ending_here
  return max_so_far
```

Looks good! Note that to align with original MBPP we don't want to modify description as much as possible.