How could the Giver society thrive?

I don't understand how the society in The Giver could survive, based on a few key facts. Am I missing something?

1. A "birth mother" only gave birth to 3 children in her life. it seems as though there are not many of them.


2. A family unit could only have 2 children, one boy and one girl. Again, not all people could even have these children.

Given these facts, I can't help but believe the society must be constantly shrinking each generation. Am I missing something?