0

A file contains non-latin content and is encoded in UTF8. Currently the existing code uses "fopen" to open the file, parses it and calls my validate function with the non-latin content and passes data as char*.

void validate(const char* str)
{
    ....
}

I have to do some validation on passed char array.

The application uses Sun C++ 5.11 and which I think doesn't supports unicode. (I googled for unicode support on Sun C++ 5.11, I didn't get any proper pointers about the unicode support. So I wrote a simple program to check if Sun C++ supports unicode and the program didn't compile).

How do I do the validation on the input char*? Is it possible using wchar_t?

NJMR
  • 1,886
  • 1
  • 27
  • 46
  • It all depends on what you mean by "validation"... How are you supposed to "validate" the text? What requirements do you have for building your function? – Some programmer dude Feb 14 '17 at 10:32
  • I have to validate the presence of special characters(requirement has some non latin chars to compare against such as 'Æ', 'Ð'), consecutive blanks. – NJMR Feb 14 '17 at 10:38
  • Maybe [this old question](http://stackoverflow.com/questions/313555/light-c-unicode-library) can help you? – Some programmer dude Feb 14 '17 at 10:42
  • I can't use third party library for security reasons... – NJMR Feb 14 '17 at 10:43
  • Then you could use [`std::codecvt_utf8`](http://en.cppreference.com/w/cpp/locale/codecvt_utf8) to converse the text to UCS-4 (which is identical to UTF-32). Will that help? – Some programmer dude Feb 14 '17 at 10:56

1 Answers1

1

The application uses <compiler> and which I think doesn't supports unicode

This isn't a problem. You only need compiler support for unicode to embed unicode string literals in the code, or for fixed width character types to represent UTF-16 or UTF-32. Your unicode is UTF-8 and comes from user input, so no unicode compiler support should be needed.

How do I do the validation on the input char*?

The C++ standard library has very few tools for processing unicode. The provided tools primarily consist of conversion between different unicode formats, and even those tools were not available prior to C++11.

Input and output is mostly just copying of bytes, so no significant processing is required to do that. For other processing (which you presumably need for "validation") you will need to implement the tools yourself, or use third party tools. You will need to refer to the ~1000 pages of the unicode standard if you choose to implement yourself: http://www.unicode.org/versions/Unicode9.0.0/UnicodeStandard-9.0.pdf

Is it possible using wchar_t?

wchar_t is the native wide character type used for the native wide character encoding of the system. UTF-8 does not use wide code-units.

eerorika
  • 232,697
  • 12
  • 197
  • 326
  • I am planning to store the list of invalid chars in a text file and the encoding will be utf8. Then read the content to a char array and then do the validation against this array. – NJMR Feb 14 '17 at 12:09