Grandson of the 2nd duke, Charles Fitzroy (1683–1757), and great-grandson of the 1st, he was educated at Westminster School and Peterhouse, Cambridge. He was secretary of state in 1765 under the Marquess of Rockingham but retired the following year. William Pitt formed a ministry in which Grafton was first lord of the Treasury (1766). He led the ministry after Chatham's resignation in 1768. As a politician Grafton was diffident and ineffective. Political differences and the attacks of the press led to his resignation in January 1770. He was lord privy seal (1771–75) in Lord North's ministry but resigned because he was in favour of conciliatory action toward the American colonists. In the ministries (1782–83) of Lord Rockingham and Lord Shelburne, he was again lord privy seal. In later years he was a prominent Unitarian.