I am having a similar problem - there's a 'dressy' component to my closet, and then there's casual, and in my case, the casual is pretty bad (very gear-ish). Your casual is very stylish, and you're in good shape, despite feeling like the casual MOTG stuff isn't quite where you'd like it to be.
In terms of ideas - the only thing I can suggest is to take a close look at Angie's outfits where she wears her Converse shoes for a casual outing on the weekend - you'll notice that even though the jeans are faded, and she's wearing Converse shoes, there is always a crisp element to the ensemble.
So, if you're wearing casual faded jeans, converse and a graphic tee - you might want to add a structured (albeit casual) jacket to add the element of 'crisp.' Likewise, if you're wearing casual jeans, a slouchy sweater and something like Converse shoes, you might add a button down shirt underneath that sweater to give the whole thing an element of 'crisp', or structure, if you will. In some cases, just adding a structured handbag might do the trick.
I guess what I'm suggesting is that you add an element of 'crisp' to your most casual outfits, and that this might make you feel a lot more comfortable and more like 'you'. Does that make sense?
Further, those Converse that you're thinking look too junior at times? They aren't - but if you're at all concerned about this - I think if you're able to make sure that your most casual outfits have some element of structure to them, then this won't be a concern. It might be a button down shirt under a slouchy sweater, it might be switching out a jacket in place of a soft/casual-ish cardigan. It might be that your handbag is more structured etc.
Leaving aside what the other MOTG's are wearing, perhaps adding an element of 'crisp' might make these casual outfits feel more like you?
What do you think of this idea? Adding just one element that is crisp or structured to your most casual weekend ensembles? Would that help? Scan the whole outfit, and determine if at least one piece is crisp or structured, even if it's meant to be the most casual of casual OTG or MOTG outfits?
I realize these suggestions don't exactly address the issue of what other MOTG's might be wearing, but I'm hoping that adding an element of either 'crisp' or 'structure' to your casual outfits might make you feel a little bit more like yourself, and make the whole wardrobe and style thing a bit more cohesive for you.
Just thinking (writing) aloud here really.