Why did they had to remove "someone" in Star Trek Voyager?

I've read some interview (though I don't remember its source), that between certain seasons of Star Trek Voyager producers have decided to remove Garrett Wang (Harry Kim) from the play. But then it turned out, that Garrett won place on 50 Most Handsome Man-like list and this forced producers to change their decission. Instead, they decided to remove Jennifer Lien (Kes).

Why did the had to remove anyone? Are there any sources, that proves, that current Voyager cast is too big? Was there any reasons behind this? Removing / replacing characters doesn't happen too often in Star Trek franchise, if I'm not mistaken.