If someone's casual outfit catches my eye as looking particularly "smart", it's usually because of the little details. Jeans and a white t-shirt is casual, but when the jeans are the exact right length for the shoes (which they usually aren't :P) and the shoes are a step up from sneakers (like ballet flats), and the shirt is a bit fitted instead of baggy and has a flattering neckline, and there's a nice handbag involved - I think that elevates it from "casual" to "smart." Also, accessories. And grooming! Not necessarily a full face of make-up, but a little bit of polish and tidy hair.
If you took all that and then added something like a blazer and/or dressier pants, I think it would be moving into business casual.
The other day I saw a lady in her 70s (at least!) wearing plain, slim black pants, red ballet flats, a plain black T and a red wool coat. She also had fantastic snow-white hair. I'm kicking myself I didn't ask to take her picture, she really did look fab.
