I would say it's an age and regional thing. 5-10 years ago, everyone I knew and talked to watched at least one anime. I remember everyone had to draw a picture of their interests and hobbies in my 8th grade class, so mine had a bunch of references to different animes. I was amazed at how many people would catch at least one reference, some being people I would never talk to in a million years if they didn't come up to me.
As I got older the people who watched Anime were less vocal about it, but you would still catch snippets here and there in the halls or someone would jump in to a conversation about Anime out of the blue.
Now in college it's gotten to be a huge thing again, with a big anime club and people discussing their favorite animes around campus. It's like 8th grade all over again.
Now, my college is a big game major and tech school, so obviously anime will be big here, but I hear similar stories from my sister who goes to a big state school. I feel my age is close to the upper limit, maybe 3-5 years below the limit, of where Anime is a norm and accepted thing. Then my youngest sister is where Anime starts to become weird again.
Now I can't speak for everywhere, which is why I think it's a regional thing. I know my family members in the Southern US have probably never heard of some mainstream animes, nevermind watched them.