How to Optimize for Performance in WP?

I don't think you are likely to do better than 2-3 seconds on WordPress page load, no matter what you do.

You have to overcome the latency of connecting between the webserver and the mysql database.

Just to see the inefficiency, compare the difference if you force the canonical using .htaccess as opposed to using WordPress to deliver the 301 redirect.

In my case on a hostgator shared account, .htaccess issues the 301 from www to without the www in 319ms. If I remove the .htaccess directive and leave it up to WordPress, it takes 1761ms just for wordpress and the server to tell the browser not to use the www. that's just over 1 full second just communicating between apache and the database.

Hosting provider makes major role in site speed.If you are not using a good hosting provider then you will always have this type of trouble.Shared hosting providers can itself handle few thousands of visitors in their servers.If you have much budget and visitors, consider purchasing VPS or Dedi (Some VPS confg are even less poweful than shared hosting!!)

Minimize the number of plugins and the java script you use in blog.Also use a cache plugin like w3 total cache, wp super cache etc,
Try the plugin WP minify to manage images, You can also use some CDN for a better performance!!

I agree with MoreTricks. My site loading speed was too high and some of my IM'r friends suggested me to change the hosting.

One little trick that has helped on my websites with a lot of images is to use a plugin called smush.it. It shrinks all image files without losing quality and seems to help with load time. The other one that can help (other than a good caching plugin) is wp-optimize.

2-3 seconds isn't bad.

How much have you managed to do with the caching?

I did some tweaking on one site and got is really fast by changing the resolution of images, and minifying javascript and css. Your CDN will partially account for that, but you add the extra DNS lookup for a CDN.

If you're using pingdom tools, it will tell you where the bottleneck is on the full page load tab... See which elements are taking longer to load, and try to reduce them...

Having big media files is going to affect performance - the only way you can change that is to find ways to reduce the footprint. So load up an image of a video rather than the video, then the click opens it in a new window, for example. That might be annoying to your readers though, you'd have to test it.

Definitely reduce the image sizes / resolutions to something small as you can, look at the different sorts of imagine types (jpg, png, etc) JPEG is usually smallest.

Honestly for me, I would leave it as is, as you're not going to massively impact rankings with speeds faster than that...