Website speed not consistent

Is there a specific reason why you chose a dedicated server? If the problem lies within the site itself, a dedicated server isn't going to fix the problem.... At most, you may end up wasting a bunch of money on unused resources..

To answer your question: Yes themes can cause delays. I'm not sure which speed test you're using, but the one I use is the PageSpeed one by Google. It helps you optimize those areas. If you're using WP Total Cache, you need to tweak and find the right settings for your server. Do you have APC enabled on your dedicated? If not, you can enable APC. Make sure you bump up the memory limit on it, otherwise you'll see a whole bunch of errors flying over your WP page.

Everytime the index.php page is loaded from WP, it loads all the active plugins theme and wp-cron. You should disable wp-cron and set it up in a manual cron yourself. There's a whole bunch of wordpress optimization tips, but as someone else mentioned, there's way too many variables as to why your site is slow.

If you're not familiar with SSH, I'd recommend getting familiar with it. You will need root access to run proper tests on it. Alternatively, you can watch your server load, memory usage and swap in WHM. We've had a client with similar issues as you. Your fastest and best bet if you have a budget is to hire a server admin. If not, then the big G is your greatest resource!

I have to agree in part with Nitish. Ping times will fluctuate between hops to your site showing various levels of server activity of those hops.

The CDN is a good idea, and Google does have a free one, but I cannot remember it off the top of my head.

You also might want to look into optimizing your CSS, HTML, and PHP for whites space and ref notes that you don't need. If you go this route always make a back up copy.

That traffic doesn't warrant a dedicated server. In any case, you're probably over-paying for a bunch of resources you don't need. What host are you with? Could you give us a link to your site?

Actually I am not so concerned about the ranking but rather the impatience of visitors. If the page doesn't load in 5 seconds there is a good chance some will just hit the back tab.

No doubt speed is important. You need to tackle the issue though by first identifying it then finding the help for that specific area. Then you can ask in programming forums (ie. stackoverflow) to get the help you need. Since no one has access to your server let alone your URL, it's difficult to give you proper advice. I gave you a few possible things to investigate in my above post. Check those out. Sounds like a load issue since it's an issue of inconsistency. Either hire a server admin or your hosting company to fix it for you. You're right..People hate waiting