Bruce had been elected to the British House of Commons for Southampton as a liberal Tory in 1841, but later that year he inherited his father's title (Scottish peerage) and left the Commons. In 1842 he was appointed governor of Jamaica. In 1846 he was named governor general of British North America and given the task of implementing the policy of responsible government recommended by his father-in-law, John George Lambton, 1st earl of Durham (Durham, John George Lambton, 1st earl of). He worked with the existing government of the Province of Canada until its defeat in the 1848 general election, when he supported the next administration's Rebellion Losses Act (1849), which compensated all Canadians for losses incurred during an 1837 rebellion in Lower Canada. His stand attracted strong Tory opposition; Elgin himself was stoned (though uninjured) by a mob, and the Parliament buildings in Montreal were burned.

Elgin maintained good relations with the two subsequent administrations. In 1849 he was created Baron Elgin (United Kingdom peerage) and was made a privy councillor. He negotiated the Reciprocity Treaty (1854) between the Canadian colonies and the U.S. He also worked on the Canadian educational system and abolished seigneurial tenure. In 1857–59 and 1860–61 he served as special commissioner to China, and in 1858 he made an official visit to Japan. In England he served as postmaster general (1859–60) in Lord Palmerston's Cabinet, before undertaking his last post as viceroy of India in 1862.