The concept of 'pyjamas' is, I'm learning, rather culturally dependent. When we first moved to the UK, people would make fun of my mother for wearing 'pyjamas' outside because she was wearing shalwar kameez. A decade later though, dresses over trousers became a 'trend', ugh lol...
Anyway, my point is that it really depends on how you *feel* in clothing as to whether it counts as PJ's or not. I wear elasticated waistbands to fit over both my waist & hips when I'm in a rush to get out of the house lol. And whilst I feel comfy in them, I wouldn't go to sleep in them. But those linen wide legged pants that are popular over the summer? *Those* feel like pajamas to me, because of the wrinkled & slightly sheer look. My actual pyjamas, however, are probably more than what most folk here would wear for loungewear (I posted it in the forum during COVID WFH - a long cardigan, elasticated cuffed trousers, thin top & a scarf).
If you're still worried, pair the trousers with slightly more dressed up accessories (like patent brogues or a metallic bag).