I don't think you're crazy at all! Actually, that was my favorite item in the Boden catalog for fall. I already have a bunch of sweater vests, but I think that could add a lot of pizazz to plain button ups. Plus, look it has all the colors that look great on you--turquoise, red, etc. You could pull those colors out in accessories, too.
You have white clamdiggers, right? You could wear a tank, layer this sweater vest over it and wear your white clams and red or turquoise flats and it would be a great piece to transition into fall with.