Removing control characters from a UTF-8 string

Question

I found this question but it removes all valid utf-8 characters also (returns me a blank string, while there are valid utf-8 characters plus control characters). As I read about utf-8, there's not a specific range for control characters and each character set has its own control characters.

How can I modify above solution to only remove control characters ?

You know, with a few lines of code (what exactly is a utf-8 string?) and a small sample of the text this would start to look like a real question. — H H, Jul 23 '11 at 09:57
real string is an Arabic `utf-8` string with some semicolon and control characters. I have provided a link to the most similar question: http://stackoverflow.com/questions/20762/how-do-you-remove-invalid-hexadecimal-characters-from-an-xml-based-data-source-pr — Xaqron, Jul 23 '11 at 10:02
How do you define control characters? Those with codepoint `<32`? — CodesInChaos, Jul 23 '11 at 10:22
@CodeInChaos: I found it is not about real control characters. Any special character like `"`, `;` etc makes problem. I'm trying to set the string as a `HttpHeader` but get this exception: `Specified value has invalid Control characters` — Xaqron, Jul 23 '11 at 10:27

score 25 · Answer 1 · answered Apr 02 '14 at 07:12

25

This is how I roll:

Regex.Replace(evilWeirdoText, @"[\u0000-\u001F]", string.Empty)

This strips out all the first 31 control characters. The next hex value up from \u001F is \u0020 AKA the space. Everything before space is all the line feed and null nonsense.

To believe me on the characters: http://donsnotes.com/tech/charsets/ascii.html

answered Apr 02 '14 at 07:12

BritishDeveloper

13,219
9
52
62

it would only filter for ASCII (UTF-8 C0 Controls but without "delete (rubout)") but the question is about UTF-8. There you have more control chars (UTF-8 C1 Controls). – InLaw Jul 29 '18 at 07:07

score 23 · Accepted Answer · edited Aug 17 '14 at 16:55

23

I think the following code will work for you:

public static string RemoveControlCharacters(string inString)
{
    if (inString == null) return null;
    StringBuilder newString = new StringBuilder();
    char ch;
    for (int i = 0; i < inString.Length; i++)
    {
        ch = inString[i];
        if (!char.IsControl(ch))
        {
            newString.Append(ch);
        }
    }
    return newString.ToString();
}

edited Aug 17 '14 at 16:55

Dave Chen

10,887
8
39
67

answered Jul 23 '11 at 10:03

Centro

3,892
2
25
31

Thanks. I still get `Specified value has invalid Control characters.` exception while trying to set the string as a `HttpHeader`. – Xaqron Jul 23 '11 at 10:10
@Xaqon It didn't work for all control characters. I have changed the condition to `!char.IsControl(ch)` and now it should work for you. – Centro Jul 23 '11 at 10:15
I found the problem, it is not about real `control characters`. Any special character like `\"`, `;` etc makes problem. I have no idea how to remove them from a `utf-8` string. – Xaqron Jul 23 '11 at 10:22
@Xaqron What is a _special_ character in your case? Any non alphanumeric? You should define these special characters and change the condition. – Centro Jul 23 '11 at 10:27
For example 1563 is a `;` and makes problem. I want to exclude all not alphabetical characters in all languages. – Xaqron Jul 23 '11 at 10:41
2

@Xaqron: use `char.IsLetter()` method. – Jalal Said Jul 23 '11 at 11:16

score 0 · Answer 3 · edited Jan 04 '13 at 22:35

0

If you plan to use the string as a query string, you should consider using the Uri.EscapeUriString() or Uri.EscapeDataString() before sending it out. Note: You might still need to pull out anything from char.IsControl() first?

edited Jan 04 '13 at 22:35

René Höhle

26,716
22
73
82

answered Jan 04 '13 at 22:17

Plater

105
1
5

Removing control characters from a UTF-8 string

3 Answers3

Linked