...i doubt it would destroy yours... though it does seem like there would be a lot of drama and potential social ramifications if you got caught giving into your desires for a married man...
if you really thought you could do this and not get caught, and more importantly not get caught up in the inevitable emotions that your type always gets caught up in then go for it... best case, you have exciting naughty sex a few times, worst case you destroy a marriage, end up pregnant, traumatise his kids, and end up on the jerry springer show...
you jus gotta know where to draw the line.