What does the job site look like at 10 a.m. on a Monday morning or 2 p.m. on a Friday afternoon? If the answer is "quiet," you need to reevaluate your contractor. Simply put, a contractor sets his or her own hours and these hours not only determine how quickly your project is completed, but they reflect a contractor's overall professionalism and his commitment to the project. A 30-hour week, shaved at both ends, doesn't imply professionalism.
That said, when you notice your contractor absent, it's your job to ask about it instead of silently stewing until you can't stand it anymore. There may be a perfectly legitimate reason your contractor never shows up on Mondays until noon. Is he buying materials? Is it standard for your contractor to take Monday mornings to tie up paperwork and permits? What looks like slacking to you may in fact be necessary work. So ask.