Instructed by his mother, a piano teacher, Jones began playing drums as a child. During the 1940s he accompanied visiting artists such as Dexter Gordon (Gordon, Dexter) and Fats Navarro (Navarro, Fats) in local clubs and toured with Lionel Hampton (Hampton, Lionel) and Joe Morris. Moving to New York, he worked with composer-bandleader Tadd Dameron (Dameron, Tadd) (1953–54) and enjoyed a busy freelance career before the most important association of his career, with the Miles Davis (Davis, Miles) quintet (1955–58).

By the 1960s Jones was also leading recording groups. After living and teaching in London and Paris (1967–72), he returned to Philadelphia. He went on to tour with pianist Bill Evans and with his own groups and in the 1980s led Dameronia, a band whose repertoire consisted of compositions by Dameron.

Jones's aggressive style, characteristic fourth-beat rim taps, and unique coordination with bassist Paul Chambers—Jones played on or slightly ahead of the beat, Chambers played slightly behind the beat—created a current of tension that distinguished the Davis quintet, one of the era's most popular jazz units. Jones's nervous interplay, with artful arhythms, lent an inflammable atmosphere to other groups as well, from swing to free-jazz bands. For all his explosive qualities as accompanist, Jones was uncommonly sensitive to the sounds of his drum kit, and his solos were thoughtfully constructed.