Remove empty lines after simple Format-list command

Question

Suppose I'm running:

some_command_here | fl foo*

and I'm getting:

a fool and his money are soon parted
foot in the door

snafoo


foo-bar

Now, I want to filter out the empty lines. How do I do this?

Answer 1

tl;dr

# Remove empty lines from the fl (Format-List) output
# (Collects all output in memory first.)
@(Get-Date | fl | oss) -ne ''

# Streaming alternative
Get-Item /, $PSHOME | fl | oss | ? { [bool] $_ }

• Get-Date and Get-Item /, $PSHOME are sample commands.

• fl is a built-in alias of the Format-List cmdlet.

• oss is a built-in function that wraps Out-String with the -Stream switch to produce line-by-line output.

• ? is a built-in alias of the Where-Object cmdlet.

Read on for background information.

---

Note:

• The answer below, in line with the question, focuses on filtering command output via its for-display formatted string representations, as produced by PowerShell's formatting system.

  • See the bottom section for a potentially simpler solution based on external utilities such as grep or findstr.exe, but on Windows there are character-encoding pitfalls.

• By contrast, if you're looking to extract data from command output, rely on PowerShell's object-oriented nature and query / return properties of the input objects, using cmdlets such as Where-Object, ForEach-Object, and Select-Object; e.g.:

  # Return those input objects whose .prop value starts with 'foo'
  [pscustomobject] @{ prop = 'foo' }, 
  [pscustomobject] @{ prop = 'other' }, 
  [pscustomobject] @{ prop = 'fool' } |
    Where-Object prop -like foo*
  

---

Format-* cmdlets such as Format-List (fl) output objects whose sole purpose is to provide formatting instructions to PowerShell's output-formatting system - see this answer. In short: only ever use Format-* cmdlets to format data for display, never for subsequent programmatic processing.

If you really want to filter the for-display output-formatting that Format-List emits:

Note:

• What follows are PowerShell-native solutions; for the specific uses case at hand, piping to an external utility such as grep or findstr.exe is actually simpler, although there can be character-encoding issues - see the bottom section.

$sampleInput = [pscustomobject] @{ foo=1 },
               [pscustomobject] @{ other=2 },
               [pscustomobject] @{ fool=3 }

$sampleInput |
  Format-List -Property foo* | 
    Out-String -Stream |  # alternatively, use: oss
      Where-Object { '' -ne $_.Trim() } 

^{Note: Where-Object { [bool] $_ } is enough to remove empty lines (but would keep blank lines, i.e. those composed of intra-line whitespace only), using PowerShell's to-Boolean coercing logic - see the bottom section of this answer for details.}

The above lists only properties whose name starts with foo (as fl foo* does in your question), along with their values, and eliminates empty and blank lines from the formatted output and therefore outputs:

foo : 1
fool : 3

Note:

• Since Format-List, as noted, produces formatting instructions, not text, Out-String -Stream is needed to convert those instructions into a stream of text lines representing the rendering of these formatting instructions, i.e. the output you would normally see in the console.

• The Where-Object then eliminates empty or blank (all-whitespace) lines from the resulting output, by testing if the whitespace-trimmed (.Trim()) line at hand ($_) is non-equal to the empty string (''); note that you could omit the '' -ne part, because PowerShell implicitly treats any non-empty string as $true in a Boolean context (see this answer for a summary of PowerShell's implicit to-Boolean conversion rules)

---

If, by contrast, you want to filter the formatted output produced by Format-List to those lines that contain substring foo anywhere on the line, i.e. irrespective of whether that substring is in the property name, its value, or even in a header line:

$sampleInput |
  Format-List |
    Out-String -Stream | 
      Select-String foo |
        ForEach-Object Line

Note:

• Unfortunately, Out-String -Stream is needed here too, because, as of PowerShell 7.2, Select-String does not automatically search the for-display representations of non-text / formatting-objects input - see GitHub issue #10726.

• To ease the pain somewhat, PowerShell ships with convenience function oss, which is short for Out-String -Stream.

• In PowerShell (Core) 7+, Select-String now supports the -Raw switch to pass only the matching lines (stringified input objects) through, without the default Microsoft.PowerShell.Commands.MatchInfo wrapper; that is, you can then use Select-String -Raw foo, without also needing ForEach-Object Line.

---

Searching through formatted for-display output using native utilities, such as grep and findstr.exe:

You can take advantage of the fact that when PowerShell sends input to external programs, it essentially performs Out-String -Stream implicitly:

• PowerShell only "speaks text" when communicating with external programs, so it must send a string representation when piping (non-string) data to an external program. By default, it uses the same for-display representation you would see in the PowerShell console, sent line by line.

• It follows that extra work is needed if you want to send data that is suitable for programmatic processing: you must then use a structured text format, such as JSON (via ConvertTo-Json) or CSV (via ConvertTo-Csv)

In your case, the implication is that you can simply pipe to a standard string-search utility such as grep on Unix-like platforms, and findstr.exe on Windows:

# Windows
$sampleInput | Format-List | findstr foo

# Unix (macOS, Linux)
$sampleInput | Format-List | grep foo

Note: You don't strictly need a Format-* call; if the default formatting of your command output results in the desired representation, you can pipe directly for grep / findstr.exe.

On Windows, however, if your (formatted) command output comprises non-ASCII-range characters, more work is needed (usually not a concern on Unix-like platforms, where (BOM-less) UTF-8 is pretty much universally used these days):

• You'll have to set the $OutputEncoding preference variable to the character encoding that the target program expects, which in the case of findstr.exe is the system's legacy OEM code page, by default reflected in [Console]::OutputEncoding (and in the output from chcp.com).

  $OutputEncoding = [Console]::OutputEncoding
  

Note that [Console]::OutputEncoding is also what PowerShell uses to decode output received from external programs, into .NET strings.

As a general aside: Given that certain modern programs, such as Node.js (node.exe), invariably use UTF-8 encoding, irrespective of the active OEM code page - in an effort to overcome the limited character sets of the legacy Windows code pages - you may have to (temporarily) set both values in order to both send and receive text properly:

$OutputEncoding = [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new()

For more information on how PowerShell handles character encoding when communicating with external programs, see this answer.