(this is my backlink count now, it actually was something around 6000 on the day I found out)
Next thing I did was jump on the WF and searched for possible solutions on how to counter a negative SEO attack... and apart from trying to manually delete every backlink, some people suggested to just keep doing what you're doing and keep building links, don't stop and don't accelerate your link building, just work as if nothing had happened.
And because manually deleting those backlinks seemed impossible I just continued to build backlinks the way I always did... and sure enough the backlink count went down slowly and I regained my first page position!
And yes, new sites are more susceptible to negative SEO but my case clearly proves that you can eliminate the effects, so DON'T throw those sites away just because someone decided to give you a link blast...
The fun part of the whole thing is knowing that someone just wasted money on trying to eliminate my site