How to detect characters with unknown encoding (like Textmate)?

Question

Textmate does not allow open a file with characters in unknown encoding. For example, when a wrong encoding is selected, the Open button is disabled:

When a proper encoding is selected, the Open button is enabled:

Is there a c or Objective-C library for reporting characters encoded outside current encoding?

Answer 1

create .tm_properties file in your home dir, then put this code in it:

[ attr.file.unknown-encoding ]
encoding=gbk

this will set default encoding to gbk when Textmate 2 don't know file encoding. also you can put this .tm_properties file in project root to use this setting within project only.

Answer 2

The usual approach is to load the text assuming a certain default encoding (like UTF-8/16 or latin 1). If there's a BOM at the start of the text then this is very good indication that the text is some form of Unicode. The BOM also helps to find what transformation format is used (UTF-8 etc.).

If loading the text still fails then it's time to ask the user for a hint (charset selection), because it's difficult to guess what the used encoding is.