Carmichael's father was an itinerant electrician, and his mother earned extra money for the family as a pianist for dances and silent movies; from her, Carmichael absorbed the basics of playing the piano. He was strongly influenced by ragtime music and by the music he heard from black families and churches in his neighbourhood. As a teenager, he made every effort to hear and play as much jazz as possible, studying in Indianapolis, Indiana, with pianist Reginald DuValle and traveling to Chicago to hear Louis Armstrong (Armstrong, Louis). While studying at Indiana University in Bloomington (LL.B., 1926), Carmichael led a small jazz band that had some success playing for college dances throughout the Midwest. In the spring of 1924, Carmichael became friends with Bix Beiderbecke (Beiderbecke, Bix) after engaging the young cornetist to play for several fraternity parties. Carmichael's first composition, "Free Wheeling," was retitled "Riverboat Shuffle" when recorded by Beiderbecke and his band, the Wolverines, later the same year; the recording subsequently became a jazz classic.

Carmichael moved to Hollywood, California, in 1936. There he composed songs for films and found additional success as a character actor, often playing the role of a philosophical and world-weary piano player, as in To Have and Have Not (1944). His hit songs for movies include "Two Sleepy People," "Small Fry," "Heart and Soul," "Ole Buttermilk Sky," "The Nearness of You," and "In the Cool, Cool, Cool of the Evening," which won an Oscar for the best film song of 1951. One of his best-known compositions of the 1940s was "Skylark," another collaboration with Mercer, and a song that reflected Carmichael's jazz influences in that, according to one music scholar, it “seemed to have the improvisations built right into the melody.”

As the golden age of American popular song waned during the advent of rock and roll in the 1950s, Carmichael continued to write songs—including such minor hits as "My Resistance Is Low" and "Winter Moon" —but had no more major successes as a songwriter. He also acted in a variety of television roles, such as his recurring dramatic part on the western series Laramie during the 1959–60 season. He never stopped composing, although most of his later songs were never recorded. One notable exception was a collection of children's music released in 1971, Hoagy Carmichael's Music Shop. Mostly, he devoted his later years to his hobbies of golf and coin collecting.

Carmichael wrote two well-received volumes of memoirs, The Stardust Road (1946) and Sometimes I Wonder (1965). After Carmichael's death, his family donated his archives and personal effects to his alma mater, Indiana University, which opened the Hoagy Carmichael Room in his honour in 1986.