Repository: go
Issue: proposal: x/text/encoding: handling encoding errors by replacing visually similar unicode characters in ShiftJIS encoding
Labels: Proposal

### Proposal Details

# Summary

When encoding Unicode strings to Shift JIS in Go, certain visually similar characters cannot be directly represented in Shift JIS, leading to encoding errors. This causes confusion because the characters appear similar but result in errors during encoding. This proposal suggests introducing a normalization step that replaces these problematic characters with their Shift JIS-compatible equivalents before encoding. We accept that this transformation is one-way and that the original characters cannot be restored, which is acceptable for our use case.

# Background

Shift JIS is a character encoding for the Japanese language but does not support all Unicode characters. Some visually similar characters have different code points and cannot be encoded in Shift JIS, causing encoding errors and confusion.

# Examples:

The Unicode character "〜" (U+301C) looks similar to "～" (U+FF5E).
The Unicode character "−" (U+2212) resembles the standard hyphen "-" (U+002D).
These visually similar characters are often used interchangeably in text but may cause encoding errors when converting to Shift JIS. In our application, it is acceptable that the transformation is not reversible; we prioritize successful encoding over the ability to revert to the original characters.

# Proposal
Introduce a normalization function that replaces visually similar Unicode characters, which cannot be encoded in Shift JIS, with their equivalent characters that can be encoded. This function can be integrated into the encoding process or provided as a utility in the golang.org/x/text/encoding/japanese package.


https://go.dev/play/p/OtEWoZmxDzb

```
package main

import (
	"fmt"

	"golang.org/x/text/encoding/japanese"
	"golang.org/x/text/transform"
)

func main() {
	replacements := map[string]string{
		"〜": "～", // U+301C (Wave Dash) → U+FF5E (Fullwidth Tilde)
		"−": "-", // U+2212 (Minus Sign) → U+002D (Hyphen-Minus)
		"—": "-", // U+2014 (Em Dash) → U+002D (Hyphen-Minus)
		"•": "*", // U+2022 (Bullet) → U+002A (Asterisk)
	}

	encoder := japanese.ShiftJIS.NewEncoder()

	for orig, replacement := range replacements {
		// Check if the original character can be encoded
		_, _, errOrig := transform.String(encoder, orig)
		// Check if the replacement character can be encoded
		_, _, errReplacement := transform.String(encoder, replacement)

		if errOrig == nil {
			fmt.Printf("Mapping may be unnecessary: Original character %q can be encoded.\n", orig)
		} else {
			fmt.Printf("Mapping necessary: Original character %q cannot be encoded: %v\n", orig, errOrig)
		}
		if errReplacement != nil {
			fmt.Printf("Warning: Replacement character %q cannot be encoded: %v\n", replacement, errReplacement)
		}
	}
}

```

Output

```
Mapping necessary: Original character "•" cannot be encoded: encoding: rune not supported by encoding.
Mapping necessary: Original character "〜" cannot be encoded: encoding: rune not supported by encoding.
Mapping necessary: Original character "−" cannot be encoded: encoding: rune not supported by encoding.
Mapping necessary: Original character "—" cannot be encoded: encoding: rune not supported by encoding.
```