It appears that after a bad question/answer removal the reputation points are not returned to the downvoters. Why is it so? Shouldn't it be different?
Downvotes on questions don't take rep from the voter.
Deleted answers do restore rep lost from downvotes for both the poster and the voter, though it happens rather quietly. You should see a green +1 on your Achievements button but probably not any note about where it came from on the Achievements dropdown.
Your profile reputation tab will list the events where you've been refunded that 1 reputation, along with a reason of "removed".
If something gets deleted and you don't see a change in rep instantly, be patient. The Stack often doesn't update this sort of thing immediately--either to give time for an accidental click to be undone, or because running the checks at regular intervals is more efficient.