We have only three diamond mods here on RPG.SE, @BrianBallsunStanton, @CRoss, and myself, but we have a lot of moderators - you, the rest of the community. As long as everyone understands that it's their job to help moderate the site as well, and understand the abilities they get along with their rep, I think we have enough manpower - if one of us calls for help in mod chat the others come to their rescue.
The way the site works takes care of most items automatically, as long as people use close votes, flags, delete votes, edit privs, are responsible about not having discussions in comments, etc. The main thing we have to intervene in nowadays that requires the diamond is primarily comment war deletion. A lot of the other things we do are because we can't get 5 close votes on a bad question, for example, even with all these users... More users and higher rep users actually should cause us to require less diamond moderation, as you all become empowered to help out yourselves! If everyone sees it as their job to do all that as well, then we're kinda superfluous. Edit that comment into that post... Crack down on spurious comments... Engage in meta...
In fact, as I write this I see that Brian was active 1 hour ago and CRoss 4 hours ago, and for the month Brian and I are neck and neck on all the mod activity stats. I think a lot of the perception is that we tend to be 'quiet' about our interventions. I tend to be the voice of the mods on issues for a couple reasons.
One, none of us are all that fond of site chat. In fact, we tend to see it as contravening the otherwise helpful and democratic paradigm of the site - see The role of chat at RPG Stack Exchange for details on that. As a result, you don't "see" any of us hovering about in chat even when we are active on the site. You see us talk in comments or meta, but that's off the radar of a lot of casual site users.
Two, I've got the thickest skin. It's not really easy to have the same four or so usual characters scream about how you are an awful person and abusive mod and every time you take (or don't take) an action on the site. Every mod deals with that in different ways. I tend to be the kind of person who takes the action and lays it out there and can deal with it when people attack me, so I tend to step up and take the public communication more, and that's fine. The other mods are involved and we discuss all the major policy statements and whatnot ahead of time, and ask for another set of eyes on things that get contentious, but we prefer to let the already engaged mod do the talking, to avoid the "Mom said no, I'll ask Dad" syndrome. Unified front.
I wouldn't mind more mods, but I am guessing there's sites much larger than ours getting along fine with three, so I wouldn't say it's really needed. In the last month between all three of us we've only done 80 edits, 20 or so answer closes and 20 or so post deletes, which is only a little more than one activity per mod per day (though we've done more like 500 comment deletes, which is a PITA, we wish people would abide by the comment guidance).