til/unix/track-line-occurrences-from-input-with-awk.md

Track Line Occurrences From Input With AWK

In Deduplicate List While Preserving Original Order, I showed a terse AWK pattern that allows for sifting out all duplicate lines as they are encountered. This looks like !seen[$0]++.

I thought it would be useful to look at a less dense version of this where I break out the conditional check, make the print explicit, and add lines to the associative array in the action block.

❯ echo "red green blue red yellow green blue red green" | tr ' ' '\n' | awk '!($0 in seen) { print; seen[$0] = 1 }'
red
green
blue
yellow

Let's take a look at that. The first part is the pattern that determines whether the action(s) runs.

!($0 in seen)

Here we check if the current line from the input being processed ($0) has already been added to the associative array we declared with a name of seen. If it is the first time we've seen that exact line, then it won't be in, so a false which gets negated (!) to true, so the actions are triggered.

The second part in curly braces is a sequence of actions separated by semicolons.

{ print; seen[$0] = 1 }

The first action is print which will print the current line to stdout. The second action adds the current line to the associative array (seen) with a value of 1. Now any time we encounter a recurring line it will be present in seen and the pattern will evaluate to false, preventing these actions from running.

The whole thing then is:

awk '!($0 in seen) { print; seen[$0] = 1 }'

Again this is an expanded, easier-to-understand version of awk '!seen[$0]++ which has the same behavior.