Penguin update is great example of over optimization in link building. Google penalized so many websites in Penguin update.

Here are few examples:
-using your KW too much in your content
-having your KW in all your H tags
-bolding, underlining or italicizing your KW in articles on your site
-having your KW in every article on your site
-using your KW for all your anchor text when creating links

Over-optimization, mostly related with link-building, it is a term imposed by Google, when it find that you are using the same keyword, to target, for to get a link, or for the anchor text also, and you are doing this practice, for all the link building activities for a page. example:

Suppose you have website: example.com/abc
and you want to get rank for the keyword abc, the you are using the keyword abc, the anchor text is also abc, you are point to the page abc, and you are also using the title for the anchor text abc, and you are doing this suppose 10 out of 10 times, then it is over-optimization.

I use the variations of 5:5, 5 same text and 5 alternate text, to balance it.

Basically, if it looks like unnatural link building, then chances are you have over-optimised your site/keywords/links etc.

Just make sure you have a nice mix of incoming anchors, and also don't point all the links you build to the same page(s)

Anchor text is the main thing as many have said. Most people don't keyword stuff or use their keyword too often in tags or on a page. Anchor text is generally where people overdo it and when you couple too much targeted anchor text with large amounts of link building; it's pretty easy to detect and penalize you without much further investigation.

Google cares about natural link earning, meaning those other webmasters give to you because of your good content. And when another webmaster links to your page, if they even use a hyperlink then odds are they're not going to be using your keyword as the anchor text, that's why it understandably looks so suspicious to Google.

Therefore it's a smart idea to vary your anchor text. Use your URL/branded keywords, generic keywords, and every now and then throw in your keyword or an LSI variant for the best results.

[DELETED]

Over Optimization is having too many keywords and links that only point to a specific page. Google will penalize this if they detect it. It's better to differentiate the pages you link to and don't use too many keyword in a page (no more than 4 percent of total content).

[DELETED]

Over optimization can be defined as any activity which is done in excess. Not only link building but keyword stuffing or other on page done in excess is also categorized as over optimization.

Your website may be over optimized in many ways. So it is not one thing that we can say that your website is over optimized due to 1 reason. Find below few points that count over optimized
1. Keywords stuffing
2. Single Keywords base anchor text(External)
3. Content Keywords Stuffing
4. Too many internal linking
5. Adding Single Keywords in website in image, alt tag, hidden text to increase Keywords density.

i hope it will help you.

Well, your query is already answer correctly by the warriors, But still I can write a line over optimising can be said as optimising in an excess way ! I can be said as keword stuffing, too many internal linking etc.