This is all from my own experiences and observations, I could of course be wrong, so don't take this as gospel, but I've seen a lot of evidence to suggest otherwise.
Also it is only in extreme circumstances, so don't get worried just yet... I am certainly not saying that a few dodgy links will harm you and it is very very difficult to attack a competitor in this way. But a site not playing by the rules could get in trouble.
So Why Can Bad Backlinks Hurt?
It is true for the most part that bad backlinks cannot hurt you, but from what I have seen, Google does take into account the overall backlink quality when issuing a penalty.
From what I can gather Google typically won't do anything negative to your site based on a bad backlink portfolio alone, but in conjunction with something else (such as keyword spamming) it can result in a penalty. You need a lot of bad backlinks though - in the thousands.
One thing to note is that the more authoritative a site is, the less effect bad backlinks will have, but in that case their overall backlink portfolio will be healthy. This does mean that you can't try and de-rank a big site by linking to them with dodgy backlinks.
Smaller sites are more vulnerable, but like I said the site typically seems to need something else wrong with it to get hurt, again making it much harder for someone to damage a site through dodgy backlinks.
On the whole it is not something you really need to worry about.
Also there are human moderators which will look at backlinks and decide on intent - i.e whether they were from the site owner trying to game Google. Human moderators seem to be getting more common so this is something to be concerned of, and also there re-inclusion request team at Google will look through your overall backlink portfolio.
How Have I Come to These Conclusions?
Most people won't agree with me that bad badlinks can potentially hurt but I figured this out from speaking with experienced 7 figures a year or more webmasters and from assisting several people with recovering from penalties which helped me spot various patterns in how Google acts.
Also Matt Cutts has said Google only tries to ensure that site A can't hurt site B. He also says that penalized sites with large numbers of bad backlinks should start on a new domain rather than trying to get reincluded as their reinclusion team won't go through and manually devalue all the bad backlinks. Both comments from Matt are a sign that bad backlinks do upset Google in some way.
There's also a backlink over optimization penalty that Google issues on a one-page basis and it can pretty much effect any site (with bigger sites still being less vulnerable). Get a lot of links to the same page with similar link text and that page will drop down for that keyword. I've seen this happen a number of times over the past 2 years.