The level of the TOEFL or IELTS speaking score is important as a hard criteria for admission to graduate schools in the US because it is an effective first step for the university, the department, and the advisor to police against allowing specific situations to go wrong.
You may be supported by the department as a teaching assistant. In this case, you will have to interact with undergraduate students in labs, in recitations, or as they visit you to discuss their homework assignment or grade. Your ability to speak coherently and clearly is critical in all of these cases, but especially in the first two. Otherwise, the students will evaluate you as being unable to carry out your duties as a teaching assistant because you are unable to speak English.
You may be supported by a faculty member as a research assistant. In this case, you will be required to provide your advisor with verbal reports on your research progress and likely also to give oral presentations about your work. Here again, you ability to communicate coherently and clearly is critical. Otherwise, your advisor may evaluate the situation and decide not to continue to support you because the extra effort he or she requires to communicate effectively with you in English will delay achievements on the project beyond reason.
As a graduate student doing research in the US, you will be expected if not required to be able to communicate the results of your work to the community of researchers in the US in English. This means you will be asked if not required to give oral presentations at local, regional, or national conferences. Your ability to speak coherently and clearly will reflect positively on you. The inverse case is often more dire. When you cannot speak well in English, the net result will reflect poorly on you but can also be taken to reflect badly on your work or the quality of your project.
Graduate programs in the US also require students to take courses. The teaching approach in US graduate courses often if not always demands that graduate students interact verbally with the instructor and with their peers. Your ability to communicate effectively in English can and often absolutely will determine your level of success to complete homework, do project assignments, and pass quizzes or exams.
In summary, you must meet the required verbal score because not doing so is a sign that you can and likely will fail at one or another of your required duties to communicate effectively in English as a graduate student in a US degree program. Universities, departments, and faculty are not interested to support such cases of potential failure. They have enough applications that meet or exceed the speaking requirements that they do not need to waste their time on those that do not.
One possible exception to the above must be acknowledged. In some cases, the admission criteria to the university could state that verbal scores below the cutoff may be admitted but that, in such a case, the applicant (you) will be required in the first year to pass courses offered at the university in English as a second language. In such cases, university policy may disallow you to be supported as a teaching assistant until you pass the courses. Alternatively, faculty may not wish to support you on their research funding until you have passed the courses. The English language courses may run for a full academic year.
In conclusion, as one who has reviewed many applications to graduate programs in science and engineering, I offer you little to no hope to continue your application process until you bring your TOEFL or IELTS verbal score to the levels that are demanded in the admission requirements of the universities where you intend to apply. The best alternative is to have a faculty member in the department who will make the case to the department and the university to support you as an exception to the rules.