In 1897 Baker began to practice law in his hometown, moving later to Cleveland, where he served two terms (1912–16) as mayor. Baker, who had played an important role in Woodrow Wilson's nomination in the Democratic National Convention of 1912, was appointed secretary of war by President Wilson and remained in the Cabinet to the end of Wilson's term of office. Although he was, as he himself said, so much of a pacifist that “he would fight for peace,” he soon submitted to Congress a plan for universal military conscription, and he efficiently presided over the mobilization of more than 4 million men during World War I.

In 1928 he was appointed by President Calvin Coolidge to the Permanent Court of Arbitration at The Hague, and in 1929 President Herbert Hoover named him to the Law Enforcement Commission. His book, Why We Went to War, appeared in 1936.