As a designer I only speak for myself because I know my view is quite different from other designers. Imagine this scenario: you bring four aliens to a table, you give them a deck of cards, and you explain the mechanics of poker (but not the actual rules). You get them to play, remain silent throughout the game, and at the end you announce the winner.

On the first game, an alien wins with a pair of sevens. So the aliens think “hey, it’s the sevens that have value!”, and pretty soon the aliens are fighting over the sevens, and they call it the game of sevens.

A|B testing gives you a tiny glimpse of a wicked system that happens to work under certain circumstances. I love A|B testing, but I know you can’t make all encompassing announcements that state that buttons should always be beveled (as Jakob Nielsen says) because it depends on the specific circumstances of the design. Just look at gmail’s new design, I’m sure it’s pretty well tested and there’s no trace of beveled buttons.

I remember an interesting piece you wrote on changing the logo for Bingo Card Creator, and you announced that logos are worth nothing. But then comes GAP with their horrendous logo and the internet goes batshit over it.

I think we are in the alchemy ages of design, and there is a lot of resistance in adopting scientific methods for validating our work. But we must understand that every test is a tiny glimpse of the overall rules of the game. Every little piece of evidence is precious in understanding the whole system.