You wouldn't clean your teeth with a dirty gym sock, so why store your toothbrush in a remarkably germy place? According to NSF International, an organization specializing in sanitation and public health standards, the toothbrush holder ranks third in germ infestations, just behind the kitchen sponge and sink!
Most germs on your actual toothbrush are anaerobic, which means they'll die when exposed to oxygen, so simply letting your toothbrush air-dry will help. Cleaning your toothbrush holder will require more diligence. Once a week, wash the holder in hot, soapy water (or wipe it clean if it's stuck to the wall.) For added protection, wipe it down with a disinfecting wipe in between [sources: BreathMD, NSF International].