Sherman was the son of Richard Updike Sherman, a newspaper editor and Democratic Party politician, and Mary Frances Sherman. Admitted to the New York bar in 1879, Sherman practiced law in Utica, N.Y., and was active in Republican Party affairs. In 1884 he was elected mayor of Utica. Sherman served 10 terms (1887–91; 1893–1909) in the United States House of Representatives (Representatives, House of), rising to the post of chairman of the House Committee on Indian Affairs. He gained a reputation as a deft parliamentarian and loyal Republican; his affability earned him the nickname “Smiling Jim.”

In 1908 Sherman was elected vice president on the Republican ticket with Taft. Nominated for reelection in 1912, Sherman, suffering from Bright's disease (Bright disease), was unable to campaign. He died less than a week before election day and thus did not live to see the party's loss to Woodrow Wilson (Wilson, Woodrow).