Site ranking going down inspite of having more links

Yes. Easily - if their quality and site-relevance are good enough.

Rankings aren't primarily determined by numbers of backlinks. They're mostly determined by quality and relevance.

Even as long ago as 2009, when I started building my business seriously and first became interested in SEO, the authors of the standard, recognized SEO textbooks were explaining that in "linkjuice terms", you'd need something between 50,000 and 100,000 self-submitted backlinks on pages of non-context-relevant sites to confer the same benefit as that from one quality backlink on a relevant site. It was true then and it's even more true now.

We can all see for ourselves, by looking at Google's SERP's and checking out the backlinks of the listings, that "numbers of backlinks" and "page ranks" are not what determines the SERP's rankings.

Just like most income-determining, traffic-attraction-determining aspects of internet marketing, good SEO is all about quality and relevance. Unfortunately for the backlink-sellers, whose trade is getting harder and harder to justify all the time. "Building more backlinks" is almost never, in itself, the answer - exactly as your own analysis of the SERP's has already shown you.

Welcome to the Warrior Forum.

Raking usually depends upon the quality and relevancy of the website you got links from not just the number of links. A few links from authority sites may be worth more than 1000's of low quality irrelevant links.

Quality backlinks are the backbone but there are also other factors involving in the sites ranking. Social signals from Facebook, Twitter and GooglePlus also affects the google ranking, and it may be your competitor's positive point.