Learning to Read the World Through Other Eyes