Smith began his military career as an enlisted man in the Indiana National Guard (1910–15) and in 1917 was commissioned a second lieutenant of infantry in the U.S. Army. He fought briefly in World War I, and, advancing through grades, he served in the United States and the Philippines and taught in the U.S. Army Infantry School, Fort Benning, Georgia. In February 1942 he was named secretary of the U.S. Joint Chiefs of Staff and U.S. secretary of the Anglo-American Combined Chiefs of Staff, with the rank of brigadier general. The following September he became chief of staff of the European theatre of operations and chief of staff to General Dwight D. Eisenhower (Eisenhower, Dwight D.), serving in those posts until Eisenhower's departure from Europe after the war. He negotiated and accepted for the Allies the surrender of Italy (1943) and of Germany (1945).

On returning to the United States in 1945, Smith became chief of the operations and planning division of the War Department general staff. Shortly afterward he was appointed U.S. ambassador to the Soviet Union, a post he held from 1946 to 1949. Later he commanded the U.S. First Army (1949–50) and was director of central intelligence (1950–53), becoming general in 1951. He retired from the army in 1953 to become undersecretary of state. In October 1954 he resigned from government service and entered private business. He was the author of My Three Years in Moscow (1950) and Eisenhower's Six Great Decisions (1956).