Filtering for Unique Objects in PowerShell

A few weeks ago my friend, Gladys Kravitz, was lamenting about a challenge related to filtering for unique objects. PowerShell has a Get-Unique cmdlet, and Select-Object has a -Unique parameter, but these options are limited. On one hand, I'd say most things we manage with PowerShell are guaranteed to be unique. Objects might have a GUID , ID, or SID, to guarantee uniqueness. But, and this is Gladys' situation, sometimes the things we are managing come from an external source. Such as importing data from a CSV file. In this situation, it is definitely possible to have duplicate objects.

Test Data

Here's a sample CSV source that I'll import into PowerShell.

$Obj = "Animal,Snack,Color
Horse,Quiche,Chartreuse
Cat,Doritos,Red
Cat,Pringles,Yellow
Dog,Doritos,Yellow
Dog,Doritos,Yellow
Rabbit,Pretzels,Green
Rabbit,Popcorn,Green
Marmoset,Cheeseburgers,Black
Dog,Doritos,White
Dog,Doritos,White
Dog,Doritos,White
" | ConvertFrom-Csv

You can plainly see the duplicate objects.

Existing Commands

Using Select-Object doesn't offer any help for this situation, even though you feel like your telling PowerShell, "Select unique objects".

This works fine on a single property.

But that isn't the goal. Then I turned my attention to Get-Unique. But this isn't any better.

So I did what I'm always telling you, I re-read the help. You can get this to work if you tell the command to treat everything as a string.

If you are dealing with flat and simple objects like my demo objects, this should get the job done. And even though the comparison is via a string, the output is the original object. But there is one potential gotcha: the comparison is case-sensitive. I also found that if the object had rich properties, like an array or hashtable, Get-Unique didn't always provide the expected results.

Using Compare-Object

I decided to go down a different rabbit hole and use Compare-Object. This cmdlet lets you specify multiple properties to compare.

Objects 3 and 4 are identical. My plan was to build a list of unique objects and then compare each input object. If the object wasn't already in the list, then add it. I first create a list object.

 $UniqueList = [System.Collections.Generic.list[object]]::new()

Then for each object, I need to test if it exists. The List object has an Exists() method which requires a predicate, which is essentially a scriptblock.

if ($UniqueList.Exists( { -not (Compare-Object $args[0].psobject.properties.value $item.psobject.Properties.value) })) {
    Write-Debug "[$((Get-Date).TimeofDay) PROCESS] Skipping: $($item |Out-String)"
}
else {
    Write-Debug "[$((Get-Date).TimeofDay) PROCESS] Adding as unique: $($item | Out-String)"
    $UniqueList.add($item)
}

The code is saying, "Compare every object in the list, $args[0] with the input object, $item. If there is a match, then the object is already in the list so don't do anything. Otherwise, add it to the list." After processing all objects, $UniqueList should be the unique objects. And by the way, Compare-Object is not case-sensitive.

Get-PSUnique

I put all of this together into a function called Get-PSUnique.

In my testing, this command will handle simple arrays and hashtables for comparisons. I guess this limit is whatever Compare-Object has when it comes to comparing properties. Still, I'd recommend this function for dealing with simple objects.

Want to try yourself? I've added this command to the latest version (2.28.0) of the PSScriptTools module which you can install from the PowerShell Gallery. Or you are welcome to check out the code for Get-PSUnique. I hope you find this as useful as I think Gladys did.