Assuming that you will stay in the US, this change of field shouldn't be a problem. However, limiting your applications to the top 20 grad schools will be a problem and you risk finding no open slot in so narrow a field. You are naming at a very small target that has a lot of very highly qualified applicants.
If you start with a bachelors it will be easier to switch as you have more time to align with the new field. With a masters would be a bit tight, since you still need to pass qualifying exams in most places and these are usually broad based requiring some advanced coursework for most students.
But a lot of people starting with a bachelors make much broader jumps to the final research area.
Cast a broad net, top 50 at least. There are a lot of good R1 universities in US. Have a few backups at minimum.