How come Annuminas (and Arnor in general) got depopulated?

Before Angmar was formed, it seems as though Arnor should have prospered: No external threats, a prosperous ally to the South, Elvish advisors to the East and West... why was there a depopulation? Specifically, of Annuminas? This, while Gondor fared relatively well for quite some time.