Why are the survivors in The Walking Dead (TV show) so keen on killing each other?

One thing that is bugging me with The Walking Dead is that it is implied (and often shown) that the biggest foes are not the zombies, but the other survivors. Throughout the series, we see groups of survivors meet with substantial reservation and open hostility. I can understand why this would be, given that more often than not other groups of survivors actually end up being thugs, but why is that?

Currently, I don't have the impression that most people in the world around me are just waiting for a good opportunity to kill and rob me. Have most of the regular people died in the apocalypse, and only the thugs and murderers survived?

Besides, even aside from ethics, killing other healthy survivors for a very short-term profit (like some gun rounds) or no reason at all seems like a decidedly stupid move in the zombie apocalypse. You would assume that in a world with an omni-present common foe, which you can't negotiate with, and which can creep up to you at any time, you would want to maximize the number of friendly people around you (if for no other reason than that they can keep watch while you are sleeping). Yet, we generally see Rick et al. try to minimize the size of their group if at all possible. I see that food might be an issue, but given enough people, there really shouldn't be a problem just growing your food yourself.

So why is everybody trying to fight for themselves or even kill each other, instead of forming a more stable (and much more safe) community, which would arguably be able to fend off the walking dead quite handily for an indefinite amount of time?