Cileena, with your red hair and warm-toned skin, I'm sure olive and rust really are fantastic colors on you. However, I bet that there are some pastels you can wear (granted, maybe only a few). I'm thinking that if you look for pastels that have yellow/warm tones, you might be able to find something that works--not all pastels have to be cool in tone. I'm thinking peach, buttermilk, and maybe even a light, mossy green?
I do sometimes buy colors that are probably not my best colors, but if I like the outfit and I don't look horrible, I will purchase it anyway. I find that a lot of very deep colors wash me out, as do very pale pastels, but I have been known to purchase them. I probably don't wear them as much as colors I think are truly flattering, but there aren't a lot of colors that I absolutely refuse to wear.