I take a lot of free online coding classes, mainly from Coursera and Udacity, and I\u2019ve picked up a lot of programming tricks in other languages that are easy translated to Windows PowerShell.<\/p>\n
In a Java class on Udacity, I learned a cool way to find duplicates in any collection. It uses the fact that the keys in hash tables must be unique. The parser throw an \u201cItem has already been added\u201d error if you try to add a key that\u2019s already in the hash table.<\/p>\n
In this example, I try to add \u201cDay\u201d to a hash table that already has an \u201cDay\u201d key. The value is arbitrary.<\/p>\n
<\/p>\n
$hash<\/span> =<\/span> @{ Day <\/span>=<\/span> \"Wednesday\"<\/span>; Weather <\/span>=<\/span> \"Sunny\"<\/span> }\r\n <\/span>$hash<\/span>.Add(<\/span>\"Day\"<\/span>, <\/span>\"Friday\"<\/span>)<\/span><\/pre>\n <\/p>\n
ERROR<\/span>:<\/span> Exception calling <\/span>\"Add\"<\/span> with <\/span>\"2\"<\/span> argument(<\/span>s<\/span>)<\/span>:<\/span> \"Item has already been added.<\/strong> Key \r\n in dictionary: 'Day' Key being added: 'Day'\"<\/span>Test.ps1<\/span> (<\/span>15<\/span>)<\/span>:<\/span> ERROR<\/span>:<\/span> At Line<\/span>:<\/span> 15<\/span> char<\/span>:<\/span> 1<\/span>\r\n ERROR<\/span>:<\/span> + <\/span>$hash<\/span>.Add(<\/span>\"Day\"<\/span>, <\/span>\"Friday\"<\/span>)\r\n <\/span>ERROR<\/span>:<\/span> + ~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n <\/span>ERROR<\/span>:<\/span> + CategoryInfo <\/span>:<\/span> NotSpecified<\/span>:<\/span> (<\/span>:<\/span>) [], MethodInvocationException\r\n <\/span>ERROR<\/span>:<\/span> + FullyQualifiedErrorId <\/span>:<\/span> ArgumentException\r\n <\/span>ERROR<\/span>:<\/span><\/pre>\n <\/p>\n
To detect a duplicate in a small collection, create a hash table and add the items to the hash table as keys.<\/p>\n
$hash<\/span> =<\/span> @{ }\r\n <\/span>\"a\"<\/span>, <\/span>\"b\"<\/span>, <\/span>\"a\"<\/span>, <\/span>\"c\"<\/span>, <\/span>\"d\"<\/span> |<\/span> ForEach<\/span> { <\/span>$hash<\/span>.Add(<\/span>$_<\/span>, <\/span>0<\/span>) }<\/span><\/pre>\n <\/p>\n
Use a Try block to add each item as a key with a value of 0. If a MethodInvocationException occurs in the Try block code, instead of erroring out and interrupting the script, it falls in the Catch block. I use the Catch block to save the duplicates in an array.<\/p>\n
$Items<\/span> =<\/span> \"a\"<\/span>, <\/span>\"b\"<\/span>, <\/span>\"a\"<\/span>, <\/span>\"c\"<\/span>, <\/span>\"d\"\r\n<\/span> $hash<\/span> =<\/span> @{ }\r\n <\/span>$duplicates<\/span> =<\/span> @()<\/span><\/pre>\n <\/p>\n
foreach<\/span> (<\/span>$item<\/span> in<\/span> $Items<\/span>)\r\n {\r\n <\/span>try<\/span>\r\n {\r\n <\/span>$hash<\/span>.add(<\/span>$item<\/span>, <\/span>0<\/span>)\r\n }\r\n<\/span> catch<\/span> [<\/span>System.Management.Automation.MethodInvocationException<\/span>]\r\n<\/span> {\r\n <\/span>$duplicates<\/span> +=<\/span> $item<\/span>\r\n }\r\n }<\/span><\/pre>\n <\/p>\n
You can return the duplicates that you saved and\/or the unique items, which are the keys in the hash table.<\/p>\n
$hash<\/span>.keys\r\n <\/span>c<\/span>\r\n a<\/span>\r\n d<\/span>\r\n b<\/span><\/pre>\n <\/p>\n
For the final version of my little script, I convert the hash table to an ordered dictionary, which preserves the order in which the keys were added. I also allow users to pipe the items to the script by adding the ValueFromPipeline parameter attribute and the Process block that supports it.<\/p>\n
<#\r\n .SYNOPSIS\r\n Gets duplicates or unique values in a collection.\r\n\r\n .DESCRIPTION\r\n The Get-Duplicates.ps1 script takes a collection and returns\r\n the duplicates (by default) or unique members (use the Unique\r\n switch parameter).\r\n\r\n .PARAMETER Items\r\n Enter a collection of items. You can also pipe the items to\r\n Get-Duplicates.ps1.\r\n\r\n .PARAMETER Unique\r\n Returns unique items instead of duplicates. By default, Get-Duplicates.ps1\r\n returns only duplicates.\r\n\r\n .EXAMPLE\r\n PS C:\\> .\\Get-Duplicates.ps1 -Items 1,2,3,2,4\r\n 2\r\n\r\n .EXAMPLE\r\n PS C:\\> 1,2,3,2,4 | .\\Get-Duplicates.ps1\r\n 2\r\n\r\n .EXAMPLE\r\n PS C:\\> .\\Get-Duplicates.ps1 -Items 1,2,3,2,4 -Unique\r\n 1\r\n 2\r\n 3\r\n 4\r\n\r\n .INPUTS\r\n System.Object[]\r\n\r\n .OUTPUTS\r\n System.Object[]\r\n\r\n .NOTES\r\n ===========================================================================\r\n Created with: SAPIEN Technologies, Inc., PowerShell Studio 2014 v4.1.72\r\n Created on: 10\/15\/2014 9:34 AM\r\n Created by: June Blender (juneb)\r\n#><\/span>\r\n\r\nparam<\/span>\r\n(\r\n [<\/span>Parameter<\/span>(Mandatory <\/span>=<\/span> $true<\/span>,\r\n ValueFromPipeline <\/span>=<\/span> $true<\/span>)]\r\n [<\/span>Object<\/span>[]]\r\n <\/span>$Items<\/span>,\r\n \r\n [<\/span>Parameter<\/span>(Mandatory <\/span>=<\/span> $false<\/span>)]\r\n [<\/span>Switch<\/span>]\r\n <\/span>$Unique<\/span>\r\n)\r\n<\/span>Begin<\/span>\r\n{\r\n <\/span>$hash<\/span> =<\/span> [<\/span>ordered<\/span>]@{ }\r\n <\/span>$duplicates<\/span> =<\/span> @()\r\n}\r\n<\/span>Process<\/span>\r\n{\r\n <\/span>foreach<\/span> (<\/span>$item<\/span> in<\/span> $Items<\/span>)\r\n {\r\n <\/span>try<\/span>\r\n {\r\n <\/span>$hash<\/span>.add(<\/span>$item<\/span>, <\/span>0<\/span>)\r\n }\r\n <\/span>catch<\/span> [<\/span>System.Management.Automation.MethodInvocationException<\/span>]\r\n {\r\n <\/span>$duplicates<\/span> +=<\/span> $item<\/span>\r\n }\r\n }\r\n}\r\n<\/span>End<\/span>\r\n{\r\n <\/span>if<\/span> (<\/span>$unique<\/span>)\r\n {\r\n <\/span>return<\/span> $hash<\/span>.keys\r\n \r\n }\r\n <\/span>elseif<\/span> (<\/span>$duplicates<\/span>)\r\n {\r\n <\/span>return<\/span> $duplicates<\/span>\r\n }\r\n}<\/span><\/pre>\n <\/p>\n
Remember that qualification about small<\/em> collections of items? This strategy is a little programming trick that is not optimized for large data sets. For those, stick with Microsoft.PowerShell.Utility\\Get-Unique and other optimized methods.<\/p>\nJune Blender is a technology evangelist at SAPIEN Technologies, Inc. You can reach her at juneb@sapien.com<\/a> or follow her on Twitter at @juneb_get_help<\/a>.<\/i><\/p>\n","protected":false},"excerpt":{"rendered":"I take a lot of free online coding classes, mainly from Coursera and Udacity, and I\u2019ve picked up a lot of programming tricks in other languages that are easy translated to Windows PowerShell. In a Java class on Udacity, I learned a cool way to find duplicates in any collection. It uses the fact that […]<\/p>\n","protected":false},"author":31,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[25],"tags":[934,28],"class_list":["post-8008","post","type-post","status-publish","format-standard","hentry","category-windows-powershell","tag-juneb","tag-powershell"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/posts\/8008","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/users\/31"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/comments?post=8008"}],"version-history":[{"count":6,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/posts\/8008\/revisions"}],"predecessor-version":[{"id":8082,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/posts\/8008\/revisions\/8082"}],"wp:attachment":[{"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/media?parent=8008"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/categories?post=8008"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.sapien.com\/blog\/wp-json\/wp\/v2\/tags?post=8008"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}