They are giving you crap because you have you hands in your pockets playing pocket pool... At least pretend to go outside to check the trailers to go at it...
How hard is it to deal with the teamsters? If you walk around with your hands in your pocket and tell people what to do you'll be just fine. Stop touching the packages.