The former prince has faced calls to have his name removed from road names and places