Wheeler was the son of Almon Wheeler, a lawyer, and Eliza Woodworth. He was a successful lawyer and held several positions in New York state government in the 1840s and '50s before serving as a member of the United States House of Representatives (Representatives, House of) (1861–63). Returned to Congress in 1875, he was appointed to a committee investigating a disputed election in Louisiana and devised the “Wheeler compromise,” by which governmental control of the state was shared between the Democratic (Democratic Party) and Republican (Republican Party) parties.

He was nominated as vice president in order to lend sectional balance to the ticket, and in his acceptance letter he alluded to the need to end Reconstruction, as Hayes subsequently did. Wheeler ran on a platform favouring administrative integrity, civil service reform, and aid to education in the South. Distracted by health and personal problems—his wife died just prior to the Republican convention in 1876—he retired from public life after his term in office was over.