Why wasn't Voldemort already hunted by the Aurors by the time he went back to Dumbledore to request a job?

In Half-Blood Prince when Dumbledore shows Harry the memory of Voldermort requesting for the D.A.D.A post we know that his followers had began calling themselves death-eaters and people were already scared to use Voldermort's name. If people refuse to speak this man’s name and there are disturbing rumours circulating about him, how come he’s free to make appointments with Dumbledore and then join his pals in the Hog’s Head? There is a gap of more than twenty years between when Voldemort worked at Borgin and Burke's and the first war (it began when he was in his mid-forties). More than twenty years, people. Years during which he killed he tortured and raised an army of creatures and was, apparently, free to chat with Dumbledore and spend the evenings in the Hog's Head along with his henchmen, just like any law-abiding wizard citizen. Sirius told Harry that “lots of people thought Voldemort had the right ideas about things before he showed his true colours.” Judging by that scene in Dumbledore’s office, to me it looks like he showed his true colours pretty much immediately. Why did he remain a free man long enough to what, make anti-Muggle speeches? J. K. Rowling said in the interview that by the time Voldemort came back to request that job he was "already killing and doing some evil stuff." Why didn’t Dumbledore alert the Aurors? I know the Ministry of Magic is notoriously incompetent, but it seems to me like its employees truly sat around and did nothing and pretty much invited this horrible Voldemort to start a war against their community. I would really love to know the true, logical reason why there wasn’t a battalion of Aurors ready to arrest him the moment he set foot inside Hogwarts the night he came back for that job.