I first would recommend that you talk to the writing support services people at your institution, such as the writing center or the center for teaching and learning. They will have some suggestions for you about how to approach this topic in a way that is constructive and useful for students. I'm not in math, but I do have students who have challenges with standard written English, both immigrant and native born.
Here are some ideas I would suggest.
I always say that one of my learning outcomes is that you will improve your skill in writing. This is not a grammar class, and writing mechanics will not be an important part of the grading, but what will be important is that you get a lot of practice in professional writing in our discipline.
You can also add oral expression to this, and your example of explaining on a walk is a nice way of expressing this. I would even consider adding something about that in your learning outcomes.
All the writing pedagogy people will tell you two things. One is that to get better at writing you have to write a lot. The second is that revision is essential. Create opportunities for your students to revise their writing.
Don't try to teach a grammar or composition course. You're not actually qualified for that besides the fact that you have math content to cover. If you can help students make small steps in their writing be happy with that.
Make sure you encourage students to take advantage of tutoring and other academic supports. Remind them every week that this is an option so that using them becomes de-stigmatized.
Since you are a fellow CUNY person I would strongly suggest that you see if you can get a WAC fellow to work with you. They are generally excellent and have good training.