If you have a wonderful contractor who hires horrible subcontractors, the sum of the work starts to look a little less rosy. Like a business manager, part of a contractor's skill should be in hiring good people, and the contractor who shrugs his shoulders in apology for bad subcontractors is passing blame that is really his.
Similarly, a good contractor should manage these subcontractors and other workers well. Screaming slave drivers rarely hire the right workers or get the best work from them. So, if you see your contractor acting cruelly to his crew, it's not only the right thing for the workers, but in the best interest of your project, to replace the contractor with a better people person.
Also, it's your contractor's job to hire people on the site that work well with you, the homeowner. If you can't get along with a contractor's employee, it's the contractor's job, not yours, to resolve the dispute. If your contractor can't get rid of a bad egg, you should consider getting rid of the contractor.