What your mother and daughter say doesn't matter, because my mother and everyone I know says the same about me. And I'm a cuddly teddy bear, you know that! But a very insecure, defensive one. Just like you say. ;)
The difference between you and I is that I've never been very affectionate, not even as a young child and I know that because my mother told me. I never bonded with my mother because I just never had the desire to. Where as, you seem to have a desire for affection, yet hide it as a type of defense mechanism. Which then makes me wonder why.
What happened to you to want to hide your affection from your own family?