Why does PowerShell interpret kind/kubectl STDOUT as STDERR and How to Prevent it?

Question

We are moving our DevOps pipelines to a new cluster and while at it, we bumped into a weird behavior when calling kind with PowerShell. This applies to kubectl also.

The below should be taken only as a repro, not a real world application. In other words, I'm not looking to fix the below code but I am searching for an explanation why the error happens:

curl.exe -Lo kind-windows-amd64.exe https://kind.sigs.k8s.io/dl/v0.10.0/kind-windows-amd64
Move-Item .\kind-windows-amd64.exe c:\temp\kind.exe -Force

$job = Start-Job -ScriptBlock { iex "$args" } -ArgumentList c:\temp\kind.exe, get, clusters
$job | Receive-Job -Wait -AutoRemoveJob

Now, if I directly execute the c:\temp\kind.exe get clusters command in the PowerShell window, the error won't happen:

In other words, why does PowerShell (any version) consider the STDOUT of kind/kubectl as STDERR? And how can I prevent this from happening?

There must be an environmental factor to it as the same exact code runs fine in one system while on another it throws an error...

Answer 1

tl;dr

• kind outputs its status messages to stderr, which in the context of PowerShell jobs surface via PowerShell's error output stream, which makes them print in red (and susceptible to $ErrorActionPreference = 'Stop' and -ErrorAction Stop).

• Either:

  • Silence stderr: Use 2>$null as a general mechanism or, as David Kruk suggests, use a program-specific option to achieve the same effect, which in the case of kind is -q (--quiet)

  • Re-route stderr output through PowerShell's success output stream, merged with stdout output, using *>&1.

    • Caveat: The original output sequencing between stdout and stderr lines is not necessarily maintained on output.

• Also, if you want to know whether the external program reported failure or success, you need to include the value of the automatic $LASTEXITCODE variable, which contains the most recently executed external program's process exit code, in the job's output (the exit code is the only reliably success/failure indicator - not the presence or absence of stderr output).

A simplified example with *>&1 (for Windows; on Unix-like platforms, replace cmd and /c with sh and -c):

$job = Start-Job -ScriptBlock {
  param($exe)
  & $exe $args *>&1
  $LASTEXITCODE  # Also output the process exit code.
} -ArgumentList cmd, /c, 'echo data1; echo status >&2; echo data2'
$job | Receive-Job -Wait -AutoRemoveJob

---

As many utilities do, kind apparently reports status messages via stderr.

• Given that stdout is for data, it makes sense to use the only other available output stream, stderr, for anything that isn't data, so as to prevent pollution of the data output. The upshot is that stderr output doesn't necessarily indicate actual errors (success vs. failure should solely be inferred from an external program's process exit code).

• PowerShell (for its own commands only) commendably has a more diverse system of output streams, documented in the conceptual about_Redirection help topic, allowing you to report status messages via Write-Verbose, for instance.

PowerShell maps an external program's output streams to its own streams as follows:

• Stdout output:

  • Stdout output is mapped to PowerShell's success output stream (the stream with number 1, analogous to how stdout can be referred to in cmd.exe and POSIX-compatible shells), allowing it to be captured in a variable ($output = ...) or redirected to a file (> output.txt) or sent through the pipeline to another command.

• Stderr output:

  • In local, foreground processing in a console (terminal), stderr is by default not mapped at all, and is passed through to the display (not colored in red) - unless a 2> redirection is used, which allows you to suppress stderr output (2>$null) or to send it to a file (2>errs.txt)

    • This is appropriate, because PowerShell cannot and should not assume that stderr output represents actual errors, whereas PowerShell's error stream is meant to be used for errors exclusively.

  • Unfortunately, as of PowerShell 7.2, in the context of PowerShell jobs (created with Start-Job or Start-ThreadJob) and remoting (e.g., in Invoke-Command -ComputerName ... calls), stderr output is mapped to PowerShell's error stream (the stream with number 2, analogous to how stdout can be referred to in cmd.exe and POSIX-compatible shells).

    • Caveat: This means that if $ErrorActionPreference = 'Stop' is in effect or -ErrorAction Stop is passed to Receive-Job or Invoke-Command, for instance, any stderr output from external programs will trigger a script-terminating error - even with stderr output comprising status messages only. Due to a bug in PowerShell 7.1 and below this can also happen in local, foreground invocation if a 2> redirection is used.

The upshot:

• To silence stderr output, apply 2>$null - either at the source (inside the job or remote command), or on the receiving end.

• To route stderr output (all streams) via the success output stream / stdout, i.e. to merge all streams, use *>&1

  • To prevent the stderr lines from printing in red (when originating from jobs or remote commands), apply this redirection at the source - which also guards against side effects from $ErrorActionPreference = 'Stop' / -ErrorAction Stop on the caller side.

  • Note: If you merge all streams with *>&1, the order in which stdout and stderr lines are output is not guaranteed to reflect the original output order, as of PowerShell 7.2.

  • If needed, PowerShell still allows you to later separate the output lines based on whether they originated from stdout or stderr - see this answer.