Remember that the standard is backing it up
Experience is a form of backing up answers—but not the only one. So is a book citation, but again not the only one. We don’t arbitrarily require that answers be backed up in some certain way—we require them to simply be backed up, one way or another.
This is, necessarily, a personal, subjective judgment call on the part of any given voter. But remember also that moderation tools—closing questions, deleting answers—are not the only option we have, or even necessarily the first one we should reach for. Downvotes are much more appropriate for indicating personal opinion of given question or answer. Moderating them should be reserved for truly egregious cases—generally speaking, moderation decisions should ideally be uncontroversial.¹
Deletion/closure—the bar is intentionally low
So very often, the question for a downvote might be “do I personally find this answer convincing?” while for closing or deleting it might be “does this question/answer make an attempt to back up what they’re saying, convincingly or otherwise?” Just as we don’t delete wrong answers, or even largely-tangential answers, as Not An Answer—that being reserved for things that make zero attempt to interface with the question at all—we don’t delete answers with attempts at backing points up that we find unconvincing, whether that is because we don’t trust the source they reference, or because their sources only cover some of the points made, or whatever.
The standard for a question remaining on the site is, intentionally, low. They attempted to interface with the question in any fashion, they made any attempt to back up their points, there’s nothing rude or otherwise problematic in the answer? It stays. It might stay heavily-downvoted, but that’s up to the individual voters. Deletion is reserved really for those cases where no attempt is made at all.²
Downvote—the standard is intentionally very personal
Ultimately, Meta doesn’t decide how you up- and downvote. You do. Literally no one on the site can check how you vote, and that’s on purpose—because no one should be checking up on how you vote.
You should upvote if you find something convincing, and downvote when you find something doubtful and insufficiently backed up to convince you, personally. But you shouldn’t—in my (strong) opinion—downvote “just because” there’s something un-backed-up, if you don’t actually doubt that those things are true. Everything un-backed-up? Yeah, sure, I can see it, though remember to consider how material may be backed up. But some innocuous statements that don’t specifically get cited? Adding in citations of books or experience for every single statement does not necessarily make an answer better—and adding citations for every single thing across every single answer is not going to improve the site, but also simply never going to happen. Harping on it too much just drives would-be answerers away. If much of an answer is backed-up, and the answer has convinced you that it is coming from expertise, and they make some un-backed-up, but not terribly dubious, statements—I really don’t think you should be downvoting that.
But, as I said, your votes are personal. There is simply never going to be any kind of policy about how you’re supposed to use them, beyond outright abuses like serial voting vendettas (which are handled automatically by a network-wide algorithm, not by any of us here).
Remember also that voting and moderating are separate
By talking about the relative “bars” for voting and moderation, and pointing to the higher bar for moderation, it suggests that the two are much the same—that moderation is just “downvoting but harder.” That is not the case. Very often, it is appropriate to not downvote something that you think needs moderation—a downvote shouldn’t be thought of as automatic for every thing that needs to be closed, or even deleted. For instance, there are many, many questions where I think “great question! unfortunately, we can’t answer it,” and upvote but also vote to close.
But I have seen comments suggesting that people “like” something, but due to some policy, they “must” downvote. That’s not how it’s supposed to work.
If you think something might be in violation of some policy, flagging it or voting to close or whatever is appropriate—it allows more of the community to see it and make decisions about it. But if the community looks into it, and decides not to moderate it, then you ought to accept that—downvoting as a way to try to impose your policy preference despite the community as a whole not agreeing isn’t really appropriate. Beyond being an inappropriate use of your privileges, to the extent people “like” something but “must” downvote, that couldn’t be farther from how votes are intended to be used!
Obviously not every moderation action is or should be uncontroversial, nor is this principle necessarily the highest priority—moderating in the face of controversy is necessary, of course. But all that means is that sometimes there are more important ideals than the one that says we’d prefer moderation actions to be largely agreed upon by a large majority of the site. The purpose of this maxim is not to say that controversy should prevent moderation, so much as it is to say that when considering whether or not to moderate, stepping away from personal preferences and trying to match the expectations of the broader community is a responsibility we all take on.
There’s sometimes an argument made for deleting answers that are being subject to “bad voting,” that is, that have limited or poor backing up but still receive positive voting. As I discuss here, the problem with that is that positive votes on the material in question, effectively, means that the community has decided that it does meet our requirements. That matters way more than any policy we have, so it’s hard to imagine any situation beyond outright abuse that would justify countermanding the community in that fashion.