All perfect praise be to Allah, The Lord of the Worlds. I testify that there is none worthy of worship except Allah, and that Muhammad is His slave and Messenger.
What this person has said to you is wrong. It is more correct for a person to continue his act of worship and not interrupt it out of fear of showing off because this is one of the tricks of the devil to prevent the people from acts of worship.
However, if he stops his act of worship in this case, then he is not sinful, let alone for it to be an act of Shirk.
Based on this, your fast is valid and you are not sinful in the case that you mentioned.
Allah knows best.