Oversized images can be a major culprit, and that's because your website may be sprawled with, say, 3-4 images which are more than 15MB. You can save precious loading seconds by compressing them before uploading to your site, and you can do this by compressing your images using Photoshop (Save for Web option), or using PNG as file format for lossless image compression.
A fast-loading site is good for SEO
Google prefers fast-loading website, which then is a factor in page rankings. The more time a user spends on your website, the more relevant your website is to the user, which means more conversions.
Enabling website caching can improve website loading, and if you use a CMS you can install cache plugins which will let you enable either site-wide caching or caching of selected site elements.
Less is more
Sometimes your website's source code may contain unnecessary elements which will add up to loading times. If you use a CMS there are plugins which may be provided by your host, or if you're on the detail-oriented side you can always manually edit the source on your own.