Convert Unicode Symbols & Punctuation to ASCII using ColdFusion/Java
Tags: unicode, ascii, java, coldfusion
Published: 2020-06-06 16:04:03

<!-- # Convert Symbols & Punctuation to ASCII using ColdFusion/Java -->

[symbolsToASCII](https://gist.github.com/JamoCA/6f35220d47caa7fdbf75eb884ff1cec7) is a ColdFusion UDF (user-defined function) to convert Unicode symbols and punctuation to ASCII7.  I was previously using [ConvertSpecialChars](http://www.cflib.org/udf/ConvertSpecialChars) from [CFLib](https://cflib.org/), but it didn't include enough mapped characters.

I found some documentation from NIH's Lexical Systems Group website that documented their approach to "[Map Symbols & Punctuation to ASCII](https://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/2013/docs/designDoc/UDF/unicode/NormOperations/mapSymbolToAscii.html)".  They state that _"converting Unicode punctuation and symbols to ASCII punctuation and symbols is imperative in NLP for preserving the original documents._  Their java code implementation is simply "perform mapping if the character is in the punctuation & [symbols mapping table](#map)".

Their approach makes a lot of sense.  When I'm performing a search using a SQL query or a Verity collection, the HTML5 input field doesn't auto-corrupt "dumb quotes" to &ldquo;smart quotes&rdquo; like Microsoft Word does. If stored content has characters that are HTML-encoded, wouldn't extra logic be required to account for potential substitutions containing high ASCII characters as well as `&lsquo;`, `&rsquo;`, `&ldquo;` and `&rdquo;`?

## Usage: symbolsToASCII(_required string inputString_)

```html
<cfset testString = '#CHR(8220)#I don#CHR(8217)#t know what you mean by #CHR(8216)#glory,#CHR(8217)# #CHR(8221)# Alice said.'>

<cfoutput>
<textarea style="width:95%; height:300px;">
Original: #TestString#

symbolsToASCII: #symbolsToASCII(testString)#
</textarea>
```

## Try it online at TryCF.com

[https://trycf.com/gist/6f35220d47caa7fdbf75eb884ff1cec7](https://trycf.com/gist/6f35220d47caa7fdbf75eb884ff1cec7)

## Source code

{% gist https://gist.github.com/JamoCA/6f35220d47caa7fdbf75eb884ff1cec7 %}

## <a name="map"></a>Symbols and Punctuation Default Mapping Table ([credit](https://lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/2013/docs/designDoc/UDF/unicode/DefaultTables/symbolTable.html))

| Unicode | Mapped String | Char | Unicode Name |
| :-------------: | :---: | :---: | ------------- |
|\u00AB|"|«|LEFT-POINTING DOUBLE ANGLE QUOTATION MARK|
|\u00AD|-|­|SOFT HYPHEN|
|\u00B4|'|´|ACUTE ACCENT|
|\u00BB|"|»|RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK|
|\u00F7|/|÷|DIVISION SIGN|
|\u01C0|&#124;|ǀ|LATIN LETTER DENTAL CLICK|
|\u01C3|!|ǃ|LATIN LETTER RETROFLEX CLICK|
|\u02B9|'|ʹ|MODIFIER LETTER PRIME|
|\u02BA|"|ʺ|MODIFIER LETTER DOUBLE PRIME|
|\u02BC|'|ʼ|MODIFIER LETTER APOSTROPHE|
|\u02C4|^|˄|MODIFIER LETTER UP ARROWHEAD|
|\u02C6|^|ˆ|MODIFIER LETTER CIRCUMFLEX ACCENT|
|\u02C8|'|ˈ|MODIFIER LETTER VERTICAL LINE|
|\u02CB|`|ˋ|MODIFIER LETTER GRAVE ACCENT|
|\u02CD|_|ˍ|MODIFIER LETTER LOW MACRON|
|\u02DC|~|˜|SMALL TILDE|
|\u0300|&#96;|̀|COMBINING GRAVE ACCENT|
|\u0301|'|́|COMBINING ACUTE ACCENT|
|\u0302|^|̂|COMBINING CIRCUMFLEX ACCENT|
|\u0303|~|̃|COMBINING TILDE|
|\u030B|"|̋|COMBINING DOUBLE ACUTE ACCENT|
|\u030E|"|̎|COMBINING DOUBLE VERTICAL LINE ABOVE|
|\u0331|_|̱|COMBINING MACRON BELOW|
|\u0332|_|̲|COMBINING LOW LINE|
|\u0338|/|̸|COMBINING LONG SOLIDUS OVERLAY|
|\u0589|:|։|ARMENIAN FULL STOP|
|\u05C0|&#124;|׀|HEBREW PUNCTUATION PASEQ|
|\u05C3|:|׃|HEBREW PUNCTUATION SOF PASUQ|
|\u066A|%|٪|ARABIC PERCENT SIGN|
|\u066D|*|٭|ARABIC FIVE POINTED STAR|
|\u200B||​|ZERO WIDTH SPACE|
|\u2010|-|‐|HYPHEN|
|\u2011|-|‑|NON-BREAKING HYPHEN|
|\u2012|-|‒|FIGURE DASH|
|\u2013|-|–|EN DASH|
|\u2014|-|—|EM DASH|
|\u2015|--|―|HORIZONTAL BAR|
|\u2016|&#124;|‖|DOUBLE VERTICAL LINE|
|\u2017|_|‗|DOUBLE LOW LINE|
|\u2018|'|‘|LEFT SINGLE QUOTATION MARK|
|\u2019|'|’|RIGHT SINGLE QUOTATION MARK|
|\u201A|,|‚|SINGLE LOW-9 QUOTATION MARK|
|\u201B|'|‛|SINGLE HIGH-REVERSED-9 QUOTATION MARK|
|\u201C|"|“|LEFT DOUBLE QUOTATION MARK|
|\u201D|"|”|RIGHT DOUBLE QUOTATION MARK|
|\u201E|"|„|DOUBLE LOW-9 QUOTATION MARK|
|\u201F|"|‟|DOUBLE HIGH-REVERSED-9 QUOTATION MARK|
|\u2032|'|′|PRIME|
|\u2033|"|″|DOUBLE PRIME|
|\u2034|'''|‴|TRIPLE PRIME|
|\u2035|&#96;|‵|REVERSED PRIME|
|\u2036|"|‶|REVERSED DOUBLE PRIME|
|\u2037|'''|‷|REVERSED TRIPLE PRIME|
|\u2038|^|‸|CARET|
|\u2039|<|‹|SINGLE LEFT-POINTING ANGLE QUOTATION MARK|
|\u203A|>|›|SINGLE RIGHT-POINTING ANGLE QUOTATION MARK|
|\u203D|?|‽|INTERROBANG|
|\u2044|/|⁄|FRACTION SLASH|
|\u204E|*|⁎|LOW ASTERISK|
|\u2052|%|⁒|COMMERCIAL MINUS SIGN|
|\u2053|~|⁓|SWUNG DASH|
|\u2060||⁠|WORD JOINER|
|\u20E5|\|⃥|COMBINING REVERSE SOLIDUS OVERLAY|
|\u2212|-|−|MINUS SIGN|
|\u2215|/|∕|DIVISION SLASH|
|\u2216|\|∖|SET MINUS|
|\u2217|*|∗|ASTERISK OPERATOR|
|\u2223|\||∣|DIVIDES|
|\u2236|:|∶|RATIO|
|\u223C|~|∼|TILDE OPERATOR|
|\u2264|<=|≤|LESS-THAN OR EQUAL TO|
|\u2265|>=|≥|GREATER-THAN OR EQUAL TO|
|\u2266|<=|≦|LESS-THAN OVER EQUAL TO|
|\u2267|>=|≧|GREATER-THAN OVER EQUAL TO|
|\u2303|^|⌃|UP ARROWHEAD|
|\u2329|<|〈|LEFT-POINTING ANGLE BRACKET|
|\u232A|>|〉|RIGHT-POINTING ANGLE BRACKET|
|\u266F|#|♯|MUSIC SHARP SIGN|
|\u2731|*|✱|HEAVY ASTERISK|
|\u2758|&#124;|❘|LIGHT VERTICAL BAR|
|\u2762|!|❢|HEAVY EXCLAMATION MARK ORNAMENT|
|\u27E6|[|⟦|MATHEMATICAL LEFT WHITE SQUARE BRACKET|
|\u27E8|<|⟨|MATHEMATICAL LEFT ANGLE BRACKET|
|\u27E9|>|⟩|MATHEMATICAL RIGHT ANGLE BRACKET|
|\u2983|{|⦃|LEFT WHITE CURLY BRACKET|
|\u2984|}|⦄|RIGHT WHITE CURLY BRACKET|
|\u3003|"|〃|DITTO MARK|
|\u3008|<|〈|LEFT ANGLE BRACKET|
|\u3009|>|〉|RIGHT ANGLE BRACKET|
|\u301B|]|〛|RIGHT WHITE SQUARE BRACKET|
|\u301C|~|〜|WAVE DASH|
|\u301D|"|〝|REVERSED DOUBLE PRIME QUOTATION MARK|
|\u301E|"|〞|DOUBLE PRIME QUOTATION MARK|
|\uFEFF||﻿|ZERO WIDTH NO-BREAK SPACE|