Don't worry, it wasn't directed at you or yours. They were saying "you" in Mandarin, which happens to be pronounced exactly like "n.i.g-ga".
There you learned a Chinese word!
Edit: oops, my bad! I forgot it doesn't mean "you" but "that", eg; "that" dish is tasty or "that" person is pretty, or I want "that" one.
To the person below, wouldn't the Japanese pronounciation of nigai be something close to three syllables...prounounce as ni-ga-ee (last syllable almost silent)? The mandarin ni-ga is two syllables and open ended, pronounced exactly as a southerner would say the N word.
There's a Russell Peters video on Youtube where he talks about a black person ordering fried chicken and thinking the Chinese boy in line with her is calling her the N word, when he's actually telling his mom which menu item he wants when he said "ni-ga", "ni-ga" (that,that).