Use a dry cleaning cloth to wipe away dust and oil. Don't use aerosol sprays, solvents, or abrasives because it could damage the headsets.
Mix a few drops of mild dish detergent with warm water. Then, using a slightly dampened cloth, wipe down the headphones.
If your earbuds need extensive cleaning, apply rubbing alcohol to the surface with a Q-tip. Just be sure not to use too much solution, as mixing electronics and liquid can be damaging (to say the least).