The prime minister has said that he is prepared to scrap plans to allow citizens from EU accession countries to have free access to the UK.