SourceTree - Russian charset support
Asked by haiflive haiflive:

Hello, i found the bug in the program "SourceTree".

The program do not view russian symbols(charset is look like \~cp1251).

![charset bug](http://s11.postimg.org/6hpfpwgg3/image.png),

My system configuration is:

Windows 7 x64, Service Pack 1, language is Russian.

Thanks.


Answers:
author: Vitaliy Timoshenko
body: I have the same shame in SourceTree. As you can see answers is on Unicode and on Russian, but filenames detected like as unknown symbols. Apologize it's ASCII codepage. As a result I can not add file in commit. It's a bug anyway. Apologize it's problem not only for Cyrillic, it should be for any non Latin symbols.

tools-\>options-\>utf-8

```
hg add -y .hgignore src\img\06_????????????_m.jpgskipping unreadable pattern file 'C:\Users\???????\Documents\hgignore_global.txt': No such file or directorysrc\img\06_????????????_m.jpg: ?????????????? ?????? ? ????? ?????,.hgignore already tracked!????????? ? ????????, ??. ????.
```

errors in filename and part of comments that generated by sourcetree (after "m.jpg:") Somebody told that it's all utf-8 =\> no.

tools-\>options-\>windows-1251

```
hg add -y .hgignore src\img\06_????????????_m.jpgskipping unreadable pattern file 'C:\Users\??????????????\Documents\hgignore_global.txt': No such file or directorysrc\img\06_????????????_m.jpg: ?????????????? ?????? ? ????? ?????,.hgignore already tracked!????????? ? ????????, ??. ????.
```

errors in filename and username!?

tools-\>options-\>koi8-r

```
hg add -y .hgignore src\img\06_????????????_m.jpgskipping unreadable pattern file 'C:\Users\??????????????\Documents\hgignore_global.txt': No such file or directorysrc\img\06_????????????_m.jpg: ?????????????? ?????? ? ????? ?????,.hgignore already tracked!????????? ? ????????, ??. ????.
```

filename, username, comments - look like all of them go in different codepage - all wrong

!But found one strange thing those file have Russian letter "?" \[yo\] inside. "CYRILLIC CAPITAL LETTER IO" 0401/0451 codes in Unicode page. when I delete this letter and do commit:

```
hg add -y src\img\06_????????????_m.jpgskipping unreadable pattern file 'C:\Users\???????\Documents\hgignore_global.txt': No such file or directoryhg commit -y --logfile C:\Users\???????\AppData\Local\Temp\tzaur2jz.hwv .hgignore src\img\06_????????????_m.jpghg push --new-branch defaultpushing to bla-bla-blasearching for changesremote: adding changesetsremote: adding manifestsremote: adding file changesremote: added 1 changesets with 2 changes to 2 files??????? ?????????.
```

What is it guys?! What a strange full UTF-8 support in modern instrument? Let's not support letter W (reason can be because it made from two letters V and user can write VV against W)! It will be minimum funny, isn't it?

P.S. Strange that August 2016 on the road but it's not fixed yet.

comments: author: gedeonych
body: P.S. Strange that December 2018 on the road but it's not fixed yet.


author: Max Reshetov
body: P.S. Strange that November 2019 on the road but it's not fixed yet...


author: Sur0vy
body: I have the same problem (I think).

if i select code page win-1251, i see text in commits tree like this:

???????????? ?????? ?????????????????? main ???\>\>?????? - (???????????? ?? ?????????...???????????? ???????????????\>\>???? + ?? ???????????????? ??????????????????????) Sur0vy \<Sur0vy@xxxxx.ru\> daca7cf4d1f55321e3290a2da219e6897eabe5eb 03.06.2021 10:36:46

and name of files looks like normal (even in Russian).

But if I change code page to utf-8 all of my commist will looks like normally, but russian names of files are corrupted.

docs_desc\\????????????_.doc

?ow to solve it?


author: Alexey Makarenya
body: Let's me add some info.  
Current version of SourceTree attempts to detect source file encoding by calling IMultiLanguage2::DetectInputCodepage method. If file contains BOM, this method successfully detects encoding. If the file contains only ASCII chars, detection of codepage turns out to be unnecessary. But if file contains not ASCII chars and it not contains BOM, DetectInputCodepage throw dices. Or maybe it detect moon phase, or temperature on the mars... I don't konw, that actually it do, but it is not a codepage detection. Result of this detection is random. In a same time, some build systems require, that BOM must NOT be included in file. php as the example. And if my system require to exclude BOM, then i can't use any chars, except ASCII.   
Strictly speaking, i can use any codepage, but sourcetree may (and often do) display file with wrong encoding.  
And current version of SourceTree is entirely trust to this foolish DetectInputCodepage. And It gives no way to override it's bad prophecies.   
And this is a bug, not feature.

comments: author: Alexey Makarenya
body: Of course, such decompilation, manual override EncodingTools.dll and substitution in the folder save situation. But it's too dicey decision.


author: stevestreeting
body: Actually from SourceTree For Windows 1.1 we'll support non-UTF8 encoding, specifically old-style Windows codepages. <https://jira.atlassian.com/browse/SRCTREEWIN-169>

We can mostly auto-detect this in the diff view, but you should set your default codepage option in Tools \> Options after release so that other output will be decoded correctly.


author: KieranA
body: Hi haiflive,

We generally don't support anything beyond UTF-8 encoding as this covers most bases. UTF-8 does support Russian character sets, too ([as shown here](http://www.utf8-chartable.de/unicode-utf8-table.pl?start=1024)). If you can use that encoding instead then that'd solve the problem. For us to support all character sets would involve a lot of work so we generally support the most popular ones that have support for the majority of characters used across languages (UTF-8, Latin-1).

Hope that helps