The reason you have to sand your walls before you paint is to get rid of any imperfections; otherwise, they'll be clearly visible after you paint. Since your goal is to remove any bumps, flaws, gouges, brush marks and paint drippings, you certainly don't want to add any scratches from the sandpaper to the imperfections that are already adorning the wall. Although using a coarse-grit of sandpaper will make the job go faster, you're more likely to produce scratches, which then have to be removed by finer sandpaper. If you continue to go over the wall using a finer and finer grit of sandpaper, eventually you'll have walls that are close to flawlessly smooth.
If you’re sanding drywall and you have holes in the walls or deep gouges, you're better off first filling them in and then sanding over them to get rid of excess joint compound. On drywalls, 120- or 150-grit sandpaper is probably your best bet, and sand using only light-to-moderate pressure. A very important tip is to close the door of the room where you're working, so the dust and debris from the drywall doesn’t settle throughout the rest of the house. Always remember to wear a mask so you won't be inhaling any dust.
Aside from the varying grits of sandpaper, you can buy specialty sandpapers such as for sanding in between coats of paint and surface-removal sandpaper, which is fine-grit sandpaper that does a good job on latex paint. However, if you're sanding the wall to get rid of paint, you can use a coarser grit to begin with.