The basic difference between a hammer drill and an impact driver is the direction from which they each exert additional force on their twisting action. A hammer drill exerts greater force directly into the bit as it hits the material being drilled, while an impact driver increases the force being delivered perpendicular to the bit.
If you're using a hammer drill, picture someone slamming the back of the drill harder into surface being drilled. That's the direct force; it's why a hammer drill can feel almost like a jackhammer in your hands. In contrast, an impact driver has a special mechanism inside of it that pushes a small anvil against the rotating mechanism (on the inside) from the side. This perpendicular pressure has been described as increasing the force on a wrench around the screw, as opposed to just pounding the screw in with more force directly on its head.
Specifically, an impact driver has three components a hammer drill doesn't: strong compression spring, weight (also called an impact mass) and the T-shaped anvil. When you first start drilling, the spring rotates at the same speed as the weight, which abuts the anvil. However, as resistance increases the weight begins to rotate more slowly. While this happens, the motor is rotating at its original speed, which keeps the spring rotating at the same original speed. At this moment, the spring, which sits just behind the weight, is rotating more quickly than the weight. The disparity of these rotating speeds allows the faster rotating spring to exert greater pressure against the slower rotating weight, which in turns pushes on the anvil. The anvil then pushes against the drill's bit and fastener from the side. The increased perpendicular pressure increases the torque and provides you with greater control when you use an impact driver.