Objective-C, NSTask Buffer Limitation

Question

I'm using NSTask to run an external utility which returns a long string of data. The problem is that when the returned string exceeds a large amount of data (around 32759 chars) it becomes null or truncates the returned string. How do I return the full output?

NSTask *myTask = [[NSTask alloc] init];

[myTask setLaunchPath:myExternalCommand];
[myTask setArguments:[NSArray arrayWithObjects: arg1, arg2, nil]];

NSPipe *pipe = [NSPipe pipe];
[myTask setStandardOutput:pipe];

NSFileHandle *taskHandle;
taskHandle = [pipe fileHandleForReading];

[myTask launch];
[myTask waitUntilExit];

NSData *taskData;
taskData = [taskHandle readDataToEndOfFile];

NSString *outputString = [[NSString alloc] initWithData:taskData
                         encoding:NSUTF8StringEncoding];

NSLog(@"Output: \n%@", outputString);
// (null or truncated) when stdout exceeds x amount of stdout

To test the functionality use cat or similar on a large file for the myExternalCommand. The issue seems to happen right after the character length of 32759...

solution? I'm not sure, but what might need to happen is to somehow read the return stdout in chunks, then append the outputString data if possible.

update: I tried moving waitUntilExit after readDataToEndOfFile per suggestion, but it did not affect the outcome.

*please note, I'm looking for an Obj-C solution, thanks.

Answer 1

Found on CocoaDev:

“The data that passes through the pipe is buffered; the size of the buffer is determined by the underlying operating system.”

from: http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSPipe_Class/index.html

The NSPipe buffer limit seems to be 4096 bytes (cf. /usr/include/limits.h: “… #define _POSIX_ARG_MAX 4096 …”)

You can read the output from your NSTask asynchronously, using readabilityHandler. Within the handler, use availableData to read the output piece-by-piece.

Use a terminationHandler to get notified once the task exits, and then set your readabilityHandler to nil to stop it from reading.

It's all async, so you'll need to block and wait until the task exits.

Here is a complete sample that works well enough for me. I used a printf instead of NSLog as it seems that NSLog is truncating the output on the console (not sure if that's a bug or a feature). Error checking is omitted and adds some complexity, you will probably want to read standardError as well in the same way.

dispatch_semaphore_t waitHandle;
NSTask *myTask;
NSMutableData* taskOutput;
        
waitHandle = dispatch_semaphore_create(0);
        
myTask = [[NSTask alloc] init];
[myTask setLaunchPath:@"/bin/cat"];
[myTask setArguments:[NSArray arrayWithObjects: @"/path/to/a/big/file", nil]];
[myTask setStandardOutput:[NSPipe pipe]];
        
taskOutput = [[NSMutableData alloc] init];
        
[[myTask.standardOutput fileHandleForReading] setReadabilityHandler:^(NSFileHandle *file) {
    NSData *data = [file availableData];
    [taskOutput appendData:data];
}];
        
[myTask setTerminationHandler:^(NSTask *task) {
    [task.standardOutput fileHandleForReading].readabilityHandler = nil;
            
    NSString *outputString = [[NSString alloc] initWithData:taskOutput encoding:NSUTF8StringEncoding];
    printf("Output: \n%s\n", [outputString UTF8String]);
            
    dispatch_semaphore_signal(waitHandle);
}];
        
[myTask launch];
        
dispatch_semaphore_wait(waitHandle, DISPATCH_TIME_FOREVER);

Answer 2

I had a task that was throwing a large error immediately, and causing a hang, adding a reader to the stderr solved the issue

[[myTask.standardError fileHandleForReading] setReadabilityHandler:^(NSFileHandle *file) {
    NSData *data = [file availableData];
    [taskError appendData:data];
}];