A younger brother of General William Tecumseh Sherman, he practiced law in Ohio before entering politics. He served in the U.S. House of Representatives (1855–61) and in the U.S. Senate (1861–77, 1881–97) and was secretary of the Treasury under President Rutherford B. Hayes (1877–81) and secretary of state under President William McKinley (1897–98).

Early in his congressional career Sherman gained a reputation as a fiscal expert. He was chairman of the House Ways and Means Committee (1859–61) and of the Senate Finance Committee (1867–77). He consistently preferred conservative financial policies but was often forced to balance his own convictions with the preferences of his constituents, many of whom favoured inflationary measures. He played a leading role in the establishment of the national banking system (1863), in the enactment of the bill (1873) that discontinued the coinage of silver dollars (denounced by critics as the “the crime of '73”) and of the Specie Payment Resumption Act (Resumption Act of 1875) (1875), which provided for the redemption of Civil War greenbacks in gold. It was thus largely through his efforts that the United States returned to the gold standard. During the administration of President Benjamin Harrison, the Antitrust Act (Sherman Antitrust Act) of 1890 and the Silver Purchase Act of the same year bore his name, but both represented compromises that had only his qualified approval.

Sherman's name was presented as a presidential consideration to three Republican national conventions (1880, 1884, and 1888). His lack of popular appeal, however, and his middle-of-the-road course on monetary policies, which suited neither the inflationist West nor the conservative East, prevented his winning the nomination.

In 1897 President William McKinley appointed Sherman secretary of state, but partly for reasons of health and partly for reasons of principle, he resigned on April 25, 1898, the day Congress declared war against Spain.