Originally named Fox Maule, he became 2nd Baron Panmure in 1852 and Earl of Dalhousie in 1860. In 1861 he assumed the Dalhousie family surname, Ramsay. After retiring from the army in 1832 with the rank of captain, he sat in the House of Commons as a Liberal from 1835 to 1837 and from 1838 to 1852. He was an under secretary of state (1835–41) and vice president of the Board of Trade (1841) in the cabinet of Lord Melbourne and then served as secretary at war (1846–52) under Lord John Russell. On the formation of the first Palmerston government in February 1855, Panmure was appointed to the new office of secretary of state for war.

Nicknamed “the Bison” for his brawny physique and resolute temperament, Panmure was a forceful administrator but often proved clumsy and insensitive. His attempt to secure preferential treatment for a young relative stationed in the Crimea provoked severe criticism and typified the aristocratic favouritism that hindered the British war effort.