Only infront of Allah

A Muslim is never poor except in front of his Lord, because being poor is a mindset which no one deserves except his creator.
Broke and penniless, yes as this is the nature of finances 
But being poor is a mindset.

نحن الفقراء و الله هو الغني الحميد

It’s worse of a sin..

قال عبد الله بن عباس رضي الله عنه وَخَوْفُكَ مِنَ الرِّيحِ إِذَا حَرَّكَتْ سِتْرَ بَابِكَ وَأَنْتَ عَلَى الذَّنْبِ وَلا يَضْطَرِبُ فُؤَادُكَ مِنْ نَظَرِ اللَّهِ إِلَيْكَ أَعْظَمُ مِنَ الذَّنْبِ إِذَا عَلِمْتَهُ 

حلية الأولياء

يا سلام

Abdullah Ibn Abbas رحمة الله عليه said:

Your fearing the movement of wind against your door whilst you are engaged in a sin and your heart is not disturbed from the fact Allah is looking at you, is more of a sin then the sin you are involved in, if you only knew.

Ya Allah!

Meaning whatever sin you are involved in, maybe a minor or major sin but fearing the shame of the people over the shame of Allah is a type of shirk.
Allah protect us