One factor to take into account is also the typical size of your personal writing, or the scale of the drawings/sketches you plan to create.
Take a normal page of ruled paper and/or checkered paper (the distance between the lines ought to be 10mm or 1cm; the side of the squares ought to be 5mm, or 1/2cm; I’m using metric measurements as they’re the standard here in Europe), try to copy a passage from any source you might like — but make it so it spans a few lines, write on every line, no skipping — and then inspect the result.
If on the one hand the lines are jammed together, there is a lot of crossing between ascenders and descenders, and you feel that your writing needs more space to become clear (e.g. by skipping very other line), then if I were you I thought that probably a 0.7 or even a 0.9 would be a better choice, especially for note-taking. If, on the other hand, your lines are well separated on the page, and hence the calligraphy is tiny, I would go with a 0.5, a 0.4, or even a 0.3mm, so that the thickness of the pencil lines do not interfere with the structure and form of your handwriting.
Same story with the drawings, but in that case you may need different sizes to achieve different results in the drawing, shifting from minuscule details, to wide areas just blocked out quickly.
I usually work with 0.5 pencils (also because the range of available lead hardness is much larger than what can be found for any other diameter: everything between 4B to 6H can be found, at least in principle), but since I write tiny letters, I can take notes even with a 0.2mm pencil (especially if I want that nobody understands what I’m scribbling about), without any particular fuss.
Apart from that, I back Cytherian’s advice: buy the whole Pentel P20X series if you can, add up even a nice 2mm leadholder, select a wide range of lead hardness, and have fun!