Sydney University accepts students into a Bachelor of Science (Physics) and a Bachelor of Advanced Science (Physics) even when they have not done HSC physics. It's not a requirement, after all, there are
no prerequisites in this day and age (at least in New South Wales).
For a Bachelor of Science, whatever you major in, unis just recommend advanced (2 Unit) maths. An HSC science course gives you background knowledge, but you don't need it and it may slow you down in the HSC...esp. if you don't like it ...
Edit: oh and not to forget, maths IS a science course!
A very "specialised" and prestigious science at that.