Personally, I disagree. I think you do owe it other people to not make their situation even worse and acting like it doesn't concern you at all despite actively and knowilingly participating in doing, regardless if you know them or not. I am pretty sure that if you were in their shoes, you would not just look at the person your partner cheated on you with and say "Oh, I don't blame them a bit. Not their problem." I don't believe that.
Sometimes, we should look beyond only what concerns us directly, and also consider how our actions make others feel. I think engaging in sex with a married person is selfish and immoral.