Both are streunuous in different ways. The content in chemistry is hard for some people in Year 11 but most find that it gets easier in Year 12... that is if you've understood the majority of the content in Year 11.
Physics is just a rote learning course. There are barely any mathematical question when it comes to the HSC, but most are simple and just plugging in values into formulas. The rest of the course is memorisation for me, and I don't really like it. I personally find the content in chemistry more fascinating.