The Duffs are descended from the original Gaels who lived in the Scottish Highlands well before the Romans arrived and before the Christian era. Their Gaelic name, Dhuibh, pronounced Duff, means a dark-complexioned man with dark hair. The first Scottish Highlanders came from ancient German tribes that crossed the sea to settle in Caledonia, later moving inland. These tribes, including the Chauci, Cimbri, Suevi, and Catti, had light skin and either red or brown hair. They were known for their height and stamina.

The people of Britain and the lowlands of Scotland originally came from France and southern Europe, but Highlanders kept to themselves and did not mix with lowlanders.
The Duffs are of German Catti ancestry and settled in Caithness early on. They were once part of the Kournaovioi Tribe before moving to Moray, where they became Mormaers of the Kanteai Tribe for many years. Moray once included all of the north-central Highlands. Historians agree that the famous Thane of Fife came from Moray before significant historical events made him known.

The Duffs fought alongside other Caledonian tribes against Roman invaders to protect Scotland. An old manuscript states that they were Mormaers of Moray during the Pictish Kings' time and were influential in Fife and Fothriff. Strath Avon was one of their historical neighborhoods near the Cairngorm Mountains.