UriBuilder().Query will wrongly encode non-ASCII characters

Question

I am working on an asp.net mvc 4 web application. and i am using .net 4.5. now i have the following WebClient() class:

using (var client = new WebClient())
{
    var query = HttpUtility.ParseQueryString(string.Empty);

    query["model"] = Model;
    //code goes here for other parameters....

    string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
    var url = new UriBuilder(apiurl);
    url.Query = query.ToString();

    string xml = client.DownloadString(url.ToString());
    XmlDocument doc = new XmlDocument();
    //code goes here ....

}

now i have noted a problem when one of the parameters contain non-ASCII charterers such as £, ¬, etc....

now the final query will have any non-ASCII characters (such as £) encoded wrongly (as %u00a3). i read about this problem and seems i can replace :-

url.Query = query.ToString();

with

url.Query = ri.EscapeUriString(HttpUtility.UrlDecode(query.ToString()));    

now using the later approach will encode £ as %C2%A3 which is the correct encoded value.

but the problem i am facing with url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString())); in that case one of the parameters contains & then the url will have the following format &operation=AddAsset&assetName=&.... so it will assume that I am passing empty assetName parameter not value =&??

EDIT

Let me summarize my problem again. I want to be able to pass the following 3 things inside my URL to a third part API :

1. Standard characters such as A,B ,a ,b ,1, 2, 3 ...

2. Non-ASCII characters such as £,¬ .

3. and also special characters that are used in url encoding such as & , + .

now i tried the following 2 approaches :

Approach A:

using (var client = new WebClient())
{
    var query = HttpUtility.ParseQueryString(string.Empty);

    query["model"] = Model;
    //code goes here for other parameters....

    string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
    var url = new UriBuilder(apiurl);
    url.Query = query.ToString();

    string xml = client.DownloadString(url.ToString());
    XmlDocument doc = new XmlDocument();
    //code goes here ....

}

In this approach i can pass values such as & ,+ since they are going to be url encoded ,,but if i want to pass non-ASCII characters they will be encoded using ISO-8859-1 ... so if i have £ value , my above code will encoded as %u00a3 and it will be saved inside the 3rd party API as %u00a3 instead of £.

Approach B :

I use :

url.Query = Uri.EscapeUriString(HttpUtility.UrlDecode(query.ToString())); 

instead of

url.Query = query.ToString();

now I can pass non-ASCII characters such as £ since they will be encoded correctly using UTF8 instead of ISO-8859-1. but i can not pass values such as & because my url will be read wrongly by the 3rd party API.. for example if I want to pass assetName=& my url will look as follow:

&operation=Add&assetName=&

so the third part API will assume I am passing empty assetName, while I am trying to pass its value as &...

so not sure how I can pass both non-ASCII characters + characters such as &, + ????

Answer 1

You could use System.Net.Http.FormUrlEncodedContent instead.

This works with a Dictionary for the Name/Value pairing and the Dictionary, unlike the NameValueCollection, does not "incorrectly" map characters such as £ to an unhelpful escaping (%u00a3, in your case).

Instead, the FormUrlEncodedContent can take a dictionary in its constructor. When you read the string out of it, it will have properly urlencoded the dictionary values.

It will correctly and uniformly handle both of the cases you were having trouble with:

• £ (which exceeds the character value range of urlencoding and needs to be encoded into a hexadecimal value in order to transport)
• & (which, as you say, has meaning in the url as a parameter separator, so that values cannot contain it--so that it has to be encoded as well).

Here's a code example, that shows that the various kinds of example items you mentioned (represented by item1, item2 and item3) now end up correctly urlencoded:

String item1 = "£";
String item2 = "&";
String item3 = "xyz";

Dictionary<string,string> queryDictionary = new Dictionary<string, string>()
{
    {"item1", item1},
    {"item2", item2},
    {"item3", item3}
};

var queryString = new System.Net.Http.FormUrlEncodedContent(queryDictionary)
        .ReadAsStringAsync().Result;

queryString will contain item1=%C2%A3&item2=%26&item3=xyz.

Answer 2

Maybe you could try to use an Extension method on the NameValueCollection class. Something like this:

using System.Collections.Specialized;
using System.Text;
using System.Web;

namespace Testing
{
    public static class NameValueCollectionExtension
    {
        public static string ToUtf8UrlEncodedQuery(this NameValueCollection nv)
        {
            StringBuilder sb = new StringBuilder();
            bool firstIteration = true;
            foreach (var key in nv.AllKeys)
            {
                if (!firstIteration)
                    sb.Append("&");
                sb.Append(HttpUtility.UrlEncode(key, Encoding.UTF8))
                    .Append("=")
                    .Append(HttpUtility.UrlEncode(nv[key], Encoding.UTF8));
                firstIteration = false;
            }
            return sb.ToString();
        }
    }
}

Then, in your code you can do this:

url.Query = query.ToUtf8UrlEncodedQuery();

Remember to add a using directive for the namespace where you put the NameValueCollectionExtension class.

Answer 3

The problem here isn't UriBuilder.Query, it's UriBuilder.ToString(). Read the documentation here: https://msdn.microsoft.com/en-us/library/system.uribuilder.tostring(v=vs.110).aspx. The property is defined as returning the "display string" of the builder, not a validly encoded string. Uri.ToString() has a similar problem, in that it doesn't perform proper encoding.

Use the following instead: url.Uri.AbsoluteUri, that will always be a properly encoded string. You shouldn't have to do any encoding on the way into the builder (that's part of it's purpose, after all, to properly encode things).

Answer 4

You need to use:

 System.Web.HttpUtility.UrlEncode(key)

Change your code to this:

using (var client = new WebClient())
{
    var query = HttpUtility.ParseQueryString(string.Empty);

    query["model"] = Model;
    //code goes here for other parameters....

    string apiurl = System.Web.Configuration.WebConfigurationManager.AppSettings["ApiURL"];
    var url = new UriBuilder(apiurl);
    url.Query = HttpUtility.UrlEncode(query.ToString());

    string xml = client.DownloadString(url.ToString());
    XmlDocument doc = new XmlDocument();
    //code goes here ....

}

Answer 5

If nothing helps, then just manually convert those problematic chars inside values of parameters

& to %26
+ to %2B
? to %3F