The thing that bothers me about smokers is the general inconsiderateness regarding the smoke. Too many smokers acting like cockbags, mostly without even realizing it.
If you're not doing that I don't have anything against you, and anyone getting uppity just because you have an unlit cigarette on your person should probably stuff it.