Twitter Strategy: Followers?

Hi All,

I have a sort of a technical question to ask, would appreciate it if a Twitter expert can help me answer it...

So I have a Twitter account, I only follow people I like to hear from, brands or public figures, etc...

On the other hand, I also tweet a lot and many people add me... However I do NOT add them back (as most people who are looking to grow they follower base do), because I figured out a lot of them are just "spammers" or bots, just looking for people who will follow them back... They have no interest in what I tweet about (stuff related to my work), and I have no interest in hearing from them either...

Not adding them back results in them unfollowing me after a while... So over time, my follower base increases and them decreases constantly, resulting in almost no net increase...

Now what if I did decide to follow them back?
Would this affect the impression of my tweet negatively?

Meaning, most of those people I will follow back won't engage with my tweets, which means that Twitter won't help my tweets get more impressions (although I'm always using relevant hashtags)...

Am I correct with this assumption?