Need advaice: Penalty or geo-targeting issue?

Hi there,
My site seemed to have been affected by Penguin (At least I thought it was Penguin) when it first rolled out. The site was ranked #1-#3 but went below #1000 and I abandoned it. (I tried to rescue it for more than 3 months to no avail).

However recently I noticed somehow it was back to #80. I was not sure how I should make of it. Whatever the reason is it is much better than below #1000. I was wondering if I should try using this site/domain one more time.

Then I noticed that my site is actually NOT affected at all on Google.com (OK, I skipped the very important part, my site is in Japanese targeted to Japanese market. So all of the ranking changes are on Google Japan.)

Actually, my server is in the US and I did a lot of link building using websites outside of Japan (including links on English websites in Japanese).

Now I'm wondering if it is more about algorithm change on geo-targetting - that affected my ranking meaning my server location and a lot of incoming links are from the US and that's why I still rank high on Google.com?

If it was because of Penguin (meaning unnatural backlinks) then shouldn't the site be affected by it on Google.com even though it is in the Japanese language?

Any thoughts would be appreciated.
Thanks!