Hazrat Shaikh Moulana Muhammad Zakariyya (rahmatullahi ‘alaih) once mentioned:

My beloved friends! As long as you are treading upon the Sunnah, do not be concerned and worried about what people have to say; whether they have a low opinion of you or regard you to be insane. The Ambiyaa were not spared by the people and similarly our pious predecessors were also labelled with many names. Hence if you become a target of people’s criticism (due to treading on deen), then why do you let this worry you? Keep your pious predecessors before you as your example – those pious predecessors who remained devoted and committed to the mubaarak sunnah of Rasulullah (sallallahu ‘alaihi wasallam) under all circumstances. (Malfoozaate Hazrat Shaikh 1/136)

Source: Ihyaauddeen.co.za