All graduate schools want to admit candidates who will succeed in their program, which is why most require an undergraduate degree.
Here is a quote about a Physics Master's Admission from the University of Tennessee.
A student who enrolls in graduate study with the intention of attaining an advanced degree in physics will have completed an undergraduate major in physics or its equivalent.
Nevertheless since there is such a high overlap in math, physics, and computer science/engineering; you will likely be able to find a graduate program that will accept you -- especially if you are willing to take a year of prerequisite classes. Here is another quote about a master's degree in Applied Math
The following prerequisites must be satisfied in our program or in course work passed elsewhere and judged equivalent by the
- Numerical Analysis, Numerical Algebra, or Numerical Algorithms
- Methods in Applied Mathematics or both Differential Eqns. II and Partial Differential Eqs.
- Advanced Calculus
- Matrix Algebra II