Why was the elder wand not conflicted?

Harry has part of Voldemort living within himself, when Voldemort's spell rebounded when he first tried to kill Harry, part of Voldemort's soul latched on to Harry, which basically makes him and Voldemort the same person(that's why Voldemort didn't kill Harry in the forest, he essentially killed part of himself). We know that the wands choose the wizard. Draco killed Dumbledore and became owner of the wand, then Harry 'defeated' Draco and won the allegiance of the Elder Wand. However, he did this when there was still a part of Voldemort's soul living within him. So when Voldemort killed Harry in the forest and the two had their final battle, why was the Elder Wand not conflicted on who its master was? Did it not recognize the part of Voldemort that lived in Harry? Why would the wand not answer to the both of them since Harry was still part Voldemort when he 'defeated' Draco?