Soap scum may be unsightly, but it's not going to hurt you. Mildew, on the other hand, just might. It's that usually black (but sometimes pink or green) growth you sometimes see around the edges of shower doors, where the glass meets the frame, and it's a fungus. It can cause illness, especially in people allergic to mold. And it's ugly.
"Mold" and "mildew" actually refer to the same growth. "Mold" is the fungus, and "mildew" is what you get when that fungus builds up.
Whatever you call it, you can get rid of it using a solution with either chlorine or hydrogen peroxide in it. Both will kill the mold. Just spray or wipe it on -- you don't need to scrub. If you have a bigger mold problem, you might need to repeat the application a couple of times.
If, however, the mildew creeps behind the caulk, you may have to replace it. Once it gets back there (or into the grout anywhere else in the bathroom), it's very hard to get rid of.
To prevent mildew's return, leave the shower door open between uses so air can circulate, and consider using an anti-mildew spray after each shower.
Up next: The spots that are "hard" to wipe off.