Ah, looks like it was only on the French side, I was thinking it was from the Dutch side. From Wikipedia:

French Huguenots During 1688–1689, the colony was greatly strengthened by the arrival of nearly two hundred French Huguenots, who were political refugees from the religious wars in France following the revocation of the Edict of Nantes. They joined colonies at Stellenbosch, Drakenstein, Franschhoek and Paarl.[13] The influence of the Huguenots on the character of the colonists was marked, leading to the VOC directing in 1701 that only Dutch should be taught in schools. This resulted in the Huguenots assimilating by the middle of the 18th century.

Forgive my novicery in 17th century Suidlander history! I am just an amateur!