Coleman began playing alto, then tenor saxophone as a teenager and soon became a working musician in dance bands and rhythm-and-blues groups. Early in his career, his approach to harmony was already unorthodox and led to his rejection by established musicians in Los Angeles, where he lived for most of the 1950s. While working as an elevator operator, he studied harmony and played an inexpensive plastic alto saxophone at obscure nightclubs. Until then, all jazz improvisation had been based on fixed harmonic patterns. In the “harmolodic theory” that Coleman developed in the 1950s, however, improvisers abandoned harmonic patterns (“chord changes”) in order to improvise more extensively and directly upon melodic and expressive elements. Because the tonal centres of such music changed at the improvisers' will, it became known as “free jazz.”

In the 1960s Coleman taught himself to play the violin and trumpet, using unorthodox techniques. By the 1970s he was performing only irregularly, preferring instead to compose. His most notable extended composition is the suite Skies of America, which was recorded in 1972 by the London Symphony Orchestra joined by Coleman on alto saxophone. Influenced by his experience of improvising with native musicians in the Rif Mountains of Morocco in 1973, Coleman formed an electric band called Prime Time, whose music was a fusion of rock rhythms with harmonically free collective improvisations; this band remained his primary performance vehicle until the 1990s. In 2005, with a quartet made up of two acoustic double bass players (one bowing his instrument, the other plucking), a drummer, and Coleman himself (playing alto saxophone, trumpet, and violin), he recorded Sound Grammar during a live performance in Italy; the work, which was said to hearken back to his music of the 1960s, was awarded the Pulitzer Prize for music in 2007. Coleman's early style influenced not only fellow saxophonists but also players of all other instruments in jazz.