Backlinks fallen off a cliff

I have a relatively new site 3 months old, competing in a difficult niche. Over the last couple of weeks, a few keywords have been coming through - showing in webmaster tools between position 70 and 200. Good progress - or so I thought.

On the 11th of July, it seems that backlinks relating to any "payday" keyword fell off a cliff. Search queries identified via webmaster tools simply stopped that day, although some relating to my site name still rank.

I looked at my backlink data and discovered that a number of links were discovered on the 11th July - all to one site. I recently subscribed to what I thought was a legit business directory, however it seems that the site has spawned over 10,000 backlinks to my site.

Obviously I can do things to have them removed, but the question I have as a relative SEO newbie is simply, would Google penalise my site straight away on discovering these links (i.e. found on the 11th July, penalised on the 11th July), or would it take longer for the penalty to kick in?

If the answer to the above is no, it would take longer, it could be to do with the recent payday reshuffle, however I'm sure that many SEO experts here will also tell me to remove the site links anyway.

Any advice appreciated.