Your explanation is somewhat vague so I'm going to assume that you want something like Axe's ultimate ability from DotA where it either damages or instantly kills the target unit.
Well, to me it sounds more like the Execute ability in WoW, i.e. an ability can only be used on / target units below a certain health percentage.
This is going to be tricky - as far as I know, there's no easy way to have custom targetting (i.e. the game showing you an error message and blocking you from using the ability, like it does for instance when you try to target a unit with full life with a simple healing ability).
The closest you can get, I think, is trying to cancel the ability if it's used on an invalid target, i.e. the game will let you "click" the target with the ability, but you'll use triggers to stop the caster from actually using it. Technically that could work, but... It's messy:
1. You won't be blocked from clicking on the target and you won't get the standard error sound & message.
2. There might be some visible effects, e.g. the caster might move to the target or play the spell cast animation.
You could technically play the sound yourself and have some custom error message, but it might be difficult to simulate the default effects to such a degree that there will be no noticeable difference. And as for the "blocked from clicking" part... well, perhaps you could do some janky workaround with the Game - Force UI Key or something to make it appear as if the player was blocked from clicking on the target, but it might be impossible to detect clicking on the unit with the targeting cursor of a specific ability and by extension triggering your workaround at the right moment.
Like, maybe some smarter people than me will know ways to do this, but as far as I know - there's no good way to solve these issues, which means that if you attempt to cancel the ability, you won't catch it at the correct moment, so it might be difficult if not impossible to cancel the visible effects mentioned above. Again, perhaps there is a way to do this, but I've spent a solid amount of time and couldn't do it.
---
Overall, most likely you'll a) be forced to use some elaborate workarounds and / or b) not achieve the desired result of blocking the player from using the ability in a way that's indistinguishable from the way Warcraft 3 does it. Thus, your ability will probably look and feel rather weird and awkward to the player (e.g. you'll be able to click on a target only to see the caster move, play the spell cast animation and have nothing happen).
My recommendation would be to slightly tweak how the ability works, i.e. have it deal a small amount of damage regardless of the target's health and then add the extra damage on top of that if the target has the required health percentage (using methods show by
@Uncle).
It's not exactly what you asked for, but it's pretty close thematically and it would be an easy way to circumvent all of the targetting issues.
