Gordon accomplished little of note during his first 29 years. He attended but did not graduate from the University of Georgia. He became a lawyer but abandoned his practice to develop coal mines in Georgia's northwestern tip. Then came the Civil War. Although lacking any military education or experience, Gordon was elected captain of a company of mountaineers and displayed remarkable capabilities. He quickly climbed from captain to brigadier general (1862) to major general (1864) to lieutenant general (1865). He was at many major Civil War battles—Seven Pines, Malvern Hills, Chancellorsville, Gettysburg, Spotsylvania, and Petersburg—and he commanded one wing of General Robert E. Lee's army just prior to Appomattox.

A hero to Georgians at the age of just 33, Gordon returned to his home state and began to practice law once again. He vigorously opposed federal Reconstruction policies, but, when he ran for governorship as a Democrat in 1868, he was defeated by his Republican opponent. Unquestionably a symbol of the age of white supremacy to his Georgian constituents, Gordon was rumoured to be a Grand Dragon in the Ku Klux Klan.

Gordon was elected to the U.S. Senate (1873–79). Though he was reelected, he resigned in 1880 to take an important position with a railroad company, thereby leading the shift of the New South to commercialism and industrialism. He returned to politics in 1886 for one term as governor and, at the conclusion of that term in 1890, was sent back to the U.S. Senate, where he served until 1897.

When the United Confederate Veterans organization was formed in 1890, Gordon was made commander in chief, a position he occupied until his death. He published memoirs of his military exploits in Reminiscences of the Civil War (1903).