blocking main thread until all async calls are done

Question

This question had an excellent answer on how to perform an asynchronous HttpWebRequest.

I have a list of 50k+ URLs that I want to asynchronously fetch with this method. The issue I ran into is the main thread would complete and exit before all the async requests could finish.

I found ManualResetEvent could be used to WaitOne(), and to call Set() on the call back method. This would work great if I was only doing one request. Not sure how to handle this for many requests.

What's the point of doing async web requests if you are going to block the main thread during their execution? You're kinda killing the whole purpose of the exercise. — Darin Dimitrov, Oct 15 '12 at 15:17
why don't use use Task-based Asynchronous Pattern and new `HttpClient` ? — tugberk, Oct 15 '12 at 15:17
@DarinDimitrov possibly a Console app and the OP needs to make sure the app doesn't exit before all the emails have been sent. — James, Oct 15 '12 at 15:19
@James, yeah valid point. It's a pity that the OP hasn't provided this kind of information in his question. — Darin Dimitrov, Oct 15 '12 at 15:20
Yea James is correct. I'm doing a console app as a test, and I don't want it to end before all the requests are made. — user17753, Oct 15 '12 at 15:20

Darin Dimitrov · Accepted Answer · 2012-10-15T15:32:01.477

1

You could use TPL:

using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading.Tasks;

class Program
{
    static void Main(string[] args)
    {
        var urls = new[] 
        { 
            "http://www.google.com", 
            "http://www.yahoo.com" 
        };

        var tasks = urls.Select(url =>
        {
            var request = WebRequest.Create(url);
            return Task
                .Factory
                .FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, url)
                .ContinueWith(t =>
                {
                    if (t.IsCompleted)
                    {
                        using (var stream = t.Result.GetResponseStream())
                        using (var reader = new StreamReader(stream))
                        {
                            Console.WriteLine("-- Successfully downloaded {0} --", t.AsyncState);
                            Console.WriteLine(reader.ReadToEnd());
                        }
                    }
                    else if (t.IsFaulted)
                    {
                        Console.WriteLine("There was an error downloading {0} - {1}", t.AsyncState, t.Exception);
                    }
                });
        }).ToArray();

        Task.WaitAll(tasks);
    }
}

edited Oct 15 '12 at 15:32

answered Oct 15 '12 at 15:26

Darin Dimitrov

1,023,142
271
3,287
2,928

This is good, I also found something like this `Parallel.For(0, 100, i => new WebClient().DownloadString(urls[i]));` not sure if this really works the same or not, though. – user17753 Oct 15 '12 at 16:40
@user17753 A `Parallel.For` will attempt to avoid creating a separate asynchronous operation for each iteration of the loop; it will try to chunk it up so that several iterations of the `For` loop will be done sequentially (because for CPU intensive operations this would be faster). Given that each iteration of the loop is a long running IO task, that may not be as efficient. – Servy Oct 15 '12 at 16:44
@user17753, in this example you are not taking advantage of I/O completion ports. The `WebClient.DownloadString` method is a **blocking operation**. So you will be using way too much threads for this. In my example I am using a true asynchronous download without any blocking (except of course the one you've asked for at the end of the program). – Darin Dimitrov Oct 15 '12 at 16:45
As an aside, I found some kind of `AggregateException` is thrown at the `t.Result.GetResponseStream()` if the server gave a `4XX` or `5XX` response code. E.g. 404 – user17753 Oct 15 '12 at 19:42
@user17753, that's why in my answer I test `if (t.IsCompleted)` before calling the `t.Result.GetResponseStream`, otherwise you could check for the exact exception. – Darin Dimitrov Oct 15 '12 at 20:48

blocking main thread until all async calls are done

1 Answers1