Repository: go
Issue: math/big: Is it necessary to limit the value of the prec parameter in some methods, or to add explicit hints?
Labels: NeedsInvestigation

<!--
Please answer these questions before submitting your issue. Thanks!
-->

### What version of Go are you using (`go version`)?

<pre>
$ go version

go version go1.17.5 windows/amd64

</pre>

### Does this issue reproduce with the latest release?

Yes

### What operating system and processor architecture are you using (`go env`)?

<details><summary><code>go env</code> Output</summary><br><pre>
$ go env
</pre></details>

not relevant

### What did you do?

<!--
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on go.dev/play is best.
-->

An error was reported in issue [#50699](https://github.com/golang/go/issues/50699) that ran out of memory due to unexpected integer overflow.

Note that some methods in the `math/big` module may cause memory exhaustion if the user passes an too large `prec` parameter.

For example:

`func (x *Rat) FloatString(prec int) string`. See in [ratconv.go](https://github.com/golang/go/blob/6eb58cdffa1ab334493776a25ccccfa89c2ca7ac/src/math/big/ratconv.go#L325).

`func (x *Float) Text(format byte, prec int) string`. See in [ftoa.go](https://github.com/golang/go/blob/6eb58cdffa1ab334493776a25ccccfa89c2ca7ac/src/math/big/ftoa.go#L47).

But some methods, such as `func (z *Float) SetPrec(prec uint) *Float` (See in [float.go](https://github.com/golang/go/blob/6eb58cdffa1ab334493776a25ccccfa89c2ca7ac/src/math/big/float.go#L164)), can limit the `prec` parameter to [MaxPrec](https://pkg.go.dev/math/big#MaxPrec) . 

Users may mistakenly assume that all these methods already provide some parameter protection internally.

So, do we need explicit maximum protection in these methods as well, or do we need explicit hints in the method documentation? For example, "passing in an excessively large `prec` parameter may cause memory exhaustion".

Note in particular the document `The prec value is ignored for the'b' and'p' formats.` (See in [ftoa.go](https://github.com/golang/go/blob/6eb58cdffa1ab334493776a25ccccfa89c2ca7ac/src/math/big/ftoa.go#L46) ), but in fact an excessively large `prec` parameter causes memory exhaustion, despite the use of `b` or `p` formats.This document may not be very appropriate.

A previous discussion can be found in [this comments](https://go-review.googlesource.com/c/go/+/379537/comments/a3d832e5_94755435).

Some Tests:

```go
func Test1(t *testing.T) {
	f := new(Float).SetInt64(12)
	f.Text('b', math.MaxInt32*100) // too large prec with b format.
	// out of memory
}
```

```go
func Test2(t *testing.T) {
	f := new(Float).SetInt64(12)
	f.Text('f', math.MaxInt32*100) // too large prec with f format.
	// out of memory
}
```

```go
func Test3(t *testing.T) {
	x := new(Rat).SetFloat64(0.5)
	x.FloatString(math.MaxInt32*100) 
	// timeout
}
```



### What did you expect to see?

If this behavior is allowed, it might be better to add documentation to prompt users to protect the parameters.

If this behavior is not allowed, the method should limit the parameter value internally.



### What did you see instead?

```bash
runtime: VirtualAlloc of 214748364800 bytes failed with errno=1455
fatal error: out of memory
```