Hertzberg studied constitutional law and history at Halle. In 1745 he entered the Prussian civil service, serving in the state record office and holding a minor foreign office post. By 1750, he had become head of the secret cabinet archives. Two years later, he was nominated to the Berlin Academy of Sciences, where his annual speeches on Prussian administration became notable events.

Hertzberg's knowledge of the Hohenzollern dynasty's genealogies and hereditary claims proved of considerable value to Frederick II the Great's politics. A regular attendant at the secret cabinet meetings from 1754, Hertzberg was the author of the famous Mémoire raisonné (“Reasoned Memorandum”) that justified Prussia's attack on Saxony at the beginning of the Seven Years' War (1756). Elevated to undersecretary of state in 1757, he negotiated Prussia's peace treaties with Russia (1762) and Austria and Saxony (1763). As minister for foreign affairs from 1763, he vainly sought both the exclusion of Austria from the First Partition of Poland (1772) and the acquisition of territory after the War of the Bavarian Succession (1778–79), Prussia's victorious campaign which prevented Bavaria's cession to Austria. Hertzberg was never able to influence Frederick the Great to any substantial degree and finally joined the opposition party around the crown prince, Frederick William II.

With the accession of Frederick William (1786), Hertzberg came to exert a dominant influence. He created Prussia's Fürstenbund (League of Princes) but failed to forge a powerful northern league despite alliances with Britain and the Netherlands, for Russia never joined it. When Austria and Russia declared war on Turkey (1787), Hertzberg attempted a settlement favourable to Prussia. He envisioned Russian acquisition of Bessarabia; Austria was to receive Moldavia and Walachia while restoring Galicia to Poland, which in turn would cede West Prussia, Danzig, and Thorn to Prussia. Britain refused to back Hertzberg and relations with Austria deteriorated almost to the point of war, when Frederick William II disavowed his foreign minister and signed the Convention of Reichenbach with Austria (1790), by which the latter renounced any territorial acquisitions in the Turkish war. Retiring from the ministry in 1791, Hertzberg nevertheless continued to provide unsolicited advice until his death in 1795.