After his father, a Prussian (Prussia) army officer, died during the French occupation following Prussia's disastrous war with France in 1806, Roon was reared mainly by his maternal grandmother. He received a commission in 1821 and served at the Berlin War Academy from 1824 to 1827. In 1832, after joining the headquarters of the army corps at Krefeld, Roon became aware of the inefficiency of the Prussian army and of the need for reorganization. He published his three-volume Grundzüge der Erd-, Völker- und Staatenkunde (1832; 3rd ed., 1847–55; “Principles of Physical, National and Political Geography”), which was widely read in Prussia and abroad.

Roon's rise to power in the Prussian army began after his aid to Crown Prince William (later Emperor William I) in suppressing the insurrection in Baden during the revolutions of 1848. He became a major general in 1850, a lieutenant general and member of the commission to reorganize the army, minister of war in 1859, and minister of the navy in 1861.

As war minister, Roon reorganized the Prussian army, thus contributing to its victories of 1866 and 1870–71. As part of the army reorganization commission, Roon succeeded, with the support of Gen. Edwin von Manteuffel, chief of the royal military cabinet, and of Moltke, chief of the general staff, in getting his plan accepted. Roon's aim was an extension of General Gerhard von Scharnhorst's system: a “nation-in-arms” maintained through universal three-year service and a permanent reserve (Landwehr) to defend the country when the army was actively engaged. Roon's system made him the most hated man in Prussia until the quick victory over Austria in the Seven Weeks' War (1866) proved the worth of the remodeled army. Roon rectified the remaining weaknesses exposed during the war with Austria, and by 1871, with the defeat of France, Germany became the leading power in Europe.

Roon, made a count in 1871, resigned as war minister in 1872 because of poor health. A practical military administrator rather than a fighting soldier, he liked to be known by his nickname of “the king's sergeant”; his political opponents called him “ruffian Roon.”