About to Drop My First Client- How to Do It Right?

I've posted before the problems I've had with my first client as we began to work together and now am at the point where I'm exhausted my budget for them and they are draining my energy.

There's been a couple of on-going issues, nickel and diming my quote, refusing to work with contract (currently don't have a contract with them), trying to add on tasks not wanting to pay extra fees for them, his business is failing and most of my outsourcers have walked away from projects related to him and advised me to do the same... so I'm dropping him after thinking about how he's costing me time from getting new, better clients.

I'm a bit worried though since I'm mostly done with a last "request"of his to recreate their site in Wordpress (he only wants Wordpress as a CMS and are instant about it look the same) which he's paid 2 payments of the total cost for already. If asked I would return the money obviously since that's the right thing to do since it's taken 2 months due to my team gradually dropping out after not being paid for his constant reversions to their design... A learning experience for sure!

Should I tell him the project has now gone beyond my scope and it's better if he finds someone else to finish it? Any advice of how to drop a client without them getting irate or threatening to sue you? Which is a worry also

I now know I should have never taken on this client and should have heeded the warning signals during our first meeting; his old web designer "disappeared on him" Refusing to sign a reasonable contract is a huge red flag.