He was the son of poor Greek parents who had Russified the family name during years spent in exile in Russia. As a young man he worked for his uncle in the cloth trade in Istanbul. In 1866 he was sent to England for further schooling, and in 1870 he became the representative in London for his uncle's firm. Two years later his uncle accused him of embezzlement, but he was acquitted of the charge.

Zaharoff left England for the eastern Mediterranean area under an assumed name. For a time he lived in Athens, where he met Stefanos Skoulodis, a financier and diplomat. On the recommendation of Skoulodis he was named agent of Thorsten Nordenfelt, the Swedish gun designer, for the Balkans area. In 1888 Hiram Stevens Maxim, inventor of the machine gun, joined Nordenfelt in business, and Zaharoff became the firm's representative for all of eastern Europe and Russia. When the Vickers Company of England purchased Maxim-Nordenfelt in 1895, Zaharoff's area of operations was extended even farther. Zaharoff became a millionaire from arms sales, and in 1913 he took out French citizenship.

During World War I Zaharoff became an Allied agent working on the highest levels. Following the war, France recognized his services by making him a grand officer of the Legion of Honour, and Britain honoured him with a knight grand cross of the Order of the Bath.

Zaharoff's first wife, deserted, died in London in the 1890s. Meanwhile, he carried on a liaison for a number of years with the Duchess de Villafranca, whose husband, Francisco de Borbón, Duke de Marchena, a member of the Spanish royal house, was mentally deranged. When the duke died in 1923, Zaharoff married the widow. Following her death in 1926, he retired to Monte Carlo, where he controlled the casino (but never gambled).