This is great Matt. I do a lot of competitive analysis myself.

There are some things, as you know, that you cannot tease out with a competitive analysis.

Say you see what you think is a really good feature on a site. What you don't know is:

1. Is anyone actually using this feature?
2. If they do use it, how do they like it from a usability perspective?
3. Is this feature adding to, or hurting conversion (if it’s a commerce site).

These things can only be found in the metrics from that site unless you want to run usability tests on competitive sites.

So I take everything I see with a grain of salt because as an observer you never know the actual value or usage of a particular pattern or feature.

P.S. I just bought your book!