From 8c463a90e3a2972a766caeb9f4c8890e667609f1 Mon Sep 17 00:00:00 2001 From: jbranchaud Date: Sun, 5 Jul 2026 17:55:06 -0500 Subject: [PATCH] Add Deduplicate List While Preserving Original Order as a Unix TIL --- README.md | 3 +- ...te-list-while-preserving-original-order.md | 71 +++++++++++++++++++ 2 files changed, 73 insertions(+), 1 deletion(-) create mode 100644 unix/deduplicate-list-while-preserving-original-order.md diff --git a/README.md b/README.md index 81b9a7c..c36b7da 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ working across different projects via [VisualMode](https://www.visualmode.dev/). For a steady stream of TILs, [sign up for my newsletter](https://visualmode.kit.com/newsletter). -_1809 TILs and counting..._ +_1810 TILs and counting..._ See some of the other learning resources I work on: @@ -1724,6 +1724,7 @@ If you've learned something here, support my efforts writing daily TILs by - [Curl With Cookies](unix/curl-with-cookies.md) - [Curling For Headers](unix/curling-for-headers.md) - [Curling With Basic Auth Credentials](unix/curling-with-basic-auth-credentials.md) +- [Deduplicate List While Preserving Original Order](unix/deduplicate-list-while-preserving-original-order.md) - [Determine ipv4 And ipv6 Public IP Addresses](unix/determine-ipv4-and-ipv6-public-ip-addresses.md) - [Diff Two Files In Unified Format](unix/diff-two-files-in-unified-format.md) - [Different Ways To Generate A v4 UUID](unix/different-ways-to-generate-a-v4-uuid.md) diff --git a/unix/deduplicate-list-while-preserving-original-order.md b/unix/deduplicate-list-while-preserving-original-order.md new file mode 100644 index 0000000..b5cac04 --- /dev/null +++ b/unix/deduplicate-list-while-preserving-original-order.md @@ -0,0 +1,71 @@ +# Deduplicate List While Preserving Original Order + +Usually when I want to deduplicate a list coming out of some command, I'll reach +for `sort | uniq`. This is a nice Unix trick where `uniq` removes consecutive +duplicate lines which relies on `sort` first reorganizing all lines in +alphabetically sorted order, bringing all duplicate lines together. + +The caveat to using `sort | uniq` (or even `sort -u`) is that it will reorder +entries alphabetically. That means you'll lose the original order, which may +have been important. + +```bash +❯ echo "red green blue red yellow green blue red green" | tr ' ' '\n' | sort -u +blue +green +red +yellow +``` + +Another approach is to use `awk` which can deduplicate while preserving the +order of entries as they first appear. This can be done with a pattern that +records the count of each line in an associative array. + +```bash +❯ echo "red green blue red yellow green blue red green" | tr ' ' '\n' | awk '!seen[$0]++' +red +green +blue +yellow +``` + +The above pattern accepts on the first occurrence of each line and rejects on +any subsequent occurrences. That is done by adding `$0` (the current line) to +`seen` (associative array that auto-initializes inline). If it doesn't exist in +`seen` yet, then `0` is returned which is negated to a truthy value with `!`. +That entry is then incremented from `0` to `1` via the `++`. As `awk` continues +to process each line, `seen` is continually added to and incremented. The +default _action_ for `awk` is to print the line. Those truthy lines are the ones +that are printed. + +An example of where this might be useful is when creating a unique listing of +all authors of a git repository while maintaining the order that they become +committers. I wanted to show this with a high-contribution public repo that I +worked on, so I referenced the [`egghead-next` +repo](https://github.com/skillrecordings/egghead-next). + +```bash +❯ git log --reverse --format='%an <%ae>' | awk '!seen[$0]++' +Joel Hooks +johnlindquist +John Lindquist +William Johnson +depfu[bot] <23717796+depfu[bot]@users.noreply.github.com> +Evgeniy Nagalskiy +Taylor Bell +Maggie Appleton +John Lindquist +Vojta Holik +Daniel Miller +jh3y +Jhey Tompkins +Josh Branchaud +Lauro Silva <57044804+laurosilvacom@users.noreply.github.com> +LB +kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com> +dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> +samuelhulick +Ian Jones +Zac Jones +... +```