Be very careful about the graphics you use on your site. How many and how big depends a lot on the content. If you want to show of pictures then a page will take longer to download.
A site that contains mainly textural information really doesn't need them although a logo or a title graphic would be OK as long as it wasn't too big.
I'd say I would wait 20 seconds at the most for a page to download unless it contained information I was really interested in. If a page takes too long to download I will invariably hit the back button and go elsewhere.
Jakob Nielson's website contains a wealth of information on usable web design, in an essay entitled The Need For Speed he states;
"Currently, the minimum goal for response times should therefore be to get pages to users in no more than ten seconds, since that's the limit of people's ability to keep their attention focused while waiting"
This means cut the clutter, which generally is graphical on web pages.
So how can you use graphics without badly affecting the download speed of your site?
Use jpeg or gif format for your graphics, jpeg being better for photos
Use animated gifs carefully. Although they can look good in the right context (perhaps on a Christmas site for example). They can also be distracting.
Keep graphics small and optimize them, take a look at this page on using graphics for some really useful information.
Don't use unnecessary graphics, think "does the page really need it or can you convey your message with text?"
Keep the graphics on your Homepage or Index page minimal, it is the first thing that visitors see but if it takes too long to download they're going to hit the back button and not see it at all.