Have you ever fallen for someone you know isn't that attractive?

That's your soul recognizing that other soul from a past lifetime, thus, it matters not what the physique looks like.