I know this thread is old but this is why I quit recently. I find it odd some people complain about the work itself taking a toll on your body while others seem to only focus on the harassments, mind games and office politics (or "union politics" if you want to be that cringe.) When I quit there were so many trucks not getting loaded on a regular basis and my sup had to get people to load it at the very end of the day. I scrambled every second of the day and had nothing to show for it. Nobody wants to work at this job it seems.
I suggest you quit as well. I can't tell you how many loaders also felt the same way you did. You'll regret not leaving sooner.