There are cheap alternatives like PHP Protection - License Your PHP Scripts Today ! or PHPLicengine - Professional License Solutions but these can ALWAYS be decoded (by someone who knows their way around code) and thus your script is at risk to be handed out for free.

Really the best licensing would be Zend or ionCube. Yes it's more expensive, but they really are much harder to circumvent.

I've tried php-protection but I'm looking at things like:

1) User install plugin
2) User needs to enter a license code
3) The plugin will authenticate the code, upon authentication, it will activate the plugin.

Is there a way to do so?

ActivateWP WordPress Plugin Licensing System

Some people may not want to deliver their plugins via a WP based site.

We are in the process of developing an easy to integrate licensing system in a box.
Its already active and has been used, with improvements, for the past 5 years to protect our membership site software.

I was wondering what kind of features would you look for in a licensing system?

I'd have to say that ProEFI definitely offers one of the better licensing plugins with ActivateWP.
Yes, there are many out there, but the "pickins" is slim for quality and the right features.

EasyDigitalDownloads licensing version is good, but I've found that by time you end up buying all the addons required for a member area, licensing, and a few more essential addons (for EDD which is free), it may end up costing you more. Don't get me wrong, Pippins coding is exceptional. (EDD creator).

That's my 2 cents as a long time WP dev who sees tons of garbage beneath the surface.