I did some experiments with one of my WordPress sites that is hosted on Hostgator.
The site in question is relatively simple. It's using the twenty-ten theme with a bunch of usual plugins (all-in-one-seo, google xml sitemap etc) and has 3 posts, each of which are around 1200-1400 words.
I conducted my tests via Alertra Website Monitoring Service
Alertra tested the speed of your site from various locations around the world, so you get a much more realistic picture of your overall site performance.
The caching plugin I am using is W3 Total Cache. With the caching enabled, I just selected and enabled the standard caching options in W3 Total Cache.
I ran a series of tests with and without caching and grabbed a screenshot of the results that were representivie of an average test.
Test With Caching Disabled
Test With Caching Enabled
As you can see, with caching enabled, the load times of the page have significantly improved.
Of all the WordPress caching plugins, W3 Total Cache is certainly the most complex, but I also suspect (I'll need to do further testing) that it also gives you the best performance increase.
If you don't want to use W3 Total Cache, then my other suggestion is "Quick Cache".
Quick Cache is what I use on most of my sites at present, but I think I'll seriously look at using W3 Total Cache in future.