Aemond has literally never had anyone dare to do such a thing to him and especially not when he was armed with live steel so it takes him back just enough that he doesn't immediately follow. When he gets his bearings, however, he goes after the man and his face is angry.

"What do you mean the cloak was playing hard to get? I nearly had it before you came up. And never touch my nose again."