I work about 38 hours a week as a delivery driver @ one of my local Domino's Pizza places. I still get two days off (back to back) and I've begun raiding with a friend whose guild only raids once a week (Sunday evenings, which I have off due to opening Sunday mornings). It works and I still have fun.
It really just comes down to what the times are you're working and the times that you're raiding. I used to work at Lockheed Martin before I started going to college and that was 40+ hours a week. Raids started late and ended late, which killed me at work (I had to be there @6AM). If you can be to bed at a decent time, it won't really bother you. Otherwise you'll possibly start feeling like it's a second job.