Voting is subjective. While there are guidelines for how to vote (as seen in Samthere's answer), it is ultimately up to you to decide if an answer is helpful, unhelpful, or neither. Ultimately, how you vote as an individual matters very little as long as a majority of users are following the voting guidelines. If you disagree with an answer on principle, even if the conclusion is the same as what you provided in your own competing answer, it's okay for you to downvote if you think it isn't helpful in explaining why it was answered that way.
In general, as long as you are voting in good faith -- that is, you genuinely think an answer is or is not helpful and aren't voting out of spite or simply to boost your own answer -- then you shouldn't worry about the outcome, no matter how you vote. And the truth is, even if a few users vote spitefully the margin for error is small enough that votes from good faith users will override those negative votes and they'll get lost in the static anyway.
Stack Exchange is a democracy, so even if you think an answer is not helpful, there may be more than enough other votes to offset your not helpful vote with helpful votes. It's not up to each of us as individuals to decide if an answer is helpful or not; voting results are in fact greater than the sum of their parts.
Also, don't forget that you also have the option not to vote on an answer at all if you're torn. If an answer is good but needs that last little push to make it great, you can withhold an upvote until you feel the problem has been addressed.