I can understand the athletics argument since you don't need to be an athlete to compete necessarily, but if you guys don't think the top pro golfers are athletes just compare the likes of Tiger Woods or Rory Mcilroy when they started playing to them in their prime. It may not be an athletic sport in itself, but they are absolutely athletes. They train to be in peak physical condition for the sport they play.
As for whether it's a sport? It is by definition a sport, there is no denying it.