Johnson received early training as a pianist, and at age 14 he began to study the trombone. He became a professional musician in 1941 and during the decade worked in the orchestras of Benny Carter (Carter, Benny) and Count Basie (Basie, Count). He became widely recognized as a dexterous soloist (to the extent that many listeners believed he was playing a valve, rather than slide, trombone) who had assimilated the techniques of the bebop movement of the 1940s. He was in great demand among jazz musicians and performed with Charlie Parker (Parker, Charlie), Dizzy Gillespie (Gillespie, Dizzy), and Miles Davis (Davis, Miles), among others. After a temporary retirement (1952–54), he returned to tour with fellow trombonist Kai Winding; their duets have been recognized as watersheds in the evolution of jazz trombone technique.

In the late 1950s and the 1960s, Johnson composed steadily, including the large-scale works El Camino Real (1959), Sketch for Trombone and Orchestra (1959), and Perceptions (1961). He also worked as a composer and arranger for commercials, films (including Shaft, 1971, with Isaac Hayes; Across 110th Street, 1972; and Cleopatra Jones, 1973), and television (including Barefoot in the Park, 1970–71, The Mod Squad, 1970–73, and Starsky and Hutch, 1975).

In 1977 Johnson undertook a tour of Japan, and he eventually returned to performing full-time, and at full technical capacity, until he retired in 1997.