I would say it depends what you want to see and how much you wish to travel. Tokyo, Kyoto and Nara are must seen of course.
In Tokyo, Asakusa, Imperial Gardens, Shibuya and Shinjuku are the typical places to see. Akihabara, the Otaku Paradise is also a nice place is you are in that kind of stuff. I would anyways advice everyone going to Japan to go and see the big Sega Arcade places there, just the noise is something worth going there
If you want to visit the Ghibli museum, you need to book your tickets in advance, directly on their website (they open the booking one month before).
Next to Tokyo you have Kamakura which have a lot of temples. Another thing you can do is Nikko Tosho-gu, a shrine dedicated to Tokugawa Ieyasu, the first Shogun of the Tokugawa dynasty. It is a one-day trip from Tokyo, leaving early and coming back in the evening.
Kyoto and Nara are the more historical places of Japan. Nara is quite small, and is a must see: a town in a forest with countless shrines and temples and a lot (A LOT) of deers in total freedom roaming there. Kyoto, there is the several temples and the Imperial Palace, as well as the Golden Pavillon and the Silver Pavillon.
Not too far from Kyoto you have Himeji, with its castle which is one of the few originals ones remaining in Japan. Because don't be fool, most castles aren't originals and had been rebuild after WW2. And if the outside is like the old times, the inside are... different (modern building type).
On the way there, you can stop at Nagoya. There is a Castle that had been rebuild using the old technics, and it's kinda beautiful to see how these places looked like at the time.
Going further South you can stop in Okayama, where there is a very beautiful Japanese Garden and the Black Castle (not original).
Hiroshima is okay I guess, but be aware that the presentation of what happenned is partial.
If you have time, you can go off-track and visit the island of Shikoku. It's easy to get there by train, starting from Okayama. Takamatsu is a must go, with the remains of its castle but also its garden. You can also go to Kochi to visit another castle there (but there isn't much more there).
Mount Fuji is nice, but be aware ! You will need good weather to see it from afar like in the pictures. I myself spent a night in Fuji and couldn't see it because of that.
In the overall there is ton of things to do in Japan. Miscelleaneous advices I could give you would be to try to eat at least in a Yoshinoya (a "fast-food" chain which serve delicious beef on rice bowl for 500 Yens), find some food in a mart (there is so many strange things there), definitively go to an Onsen and not be too shy with that...
Also be aware that if you go in summer, Japan is really really hot and you can get thirsty really easily. However Japan being a perfect place, there is automat everywhere selling you drinks. If you are a smoker, you can't smoke on the streets, you can get a hefty fine for that. Think to always have an umbrella with you, rain comes and goes quickly there.
And in the overall, don't worry to get lost. Japanese people are super helpful is you ask them nicely, but be aware that English isn't that widely spread in the population. However, they will try to help you.
Hope you will enjoy your trip in Japan