Brown first made his mark at 19 when he went to New York City to join Dizzy Gillespie (Gillespie, Dizzy)'s band at a time when the modern jazz revolution, spearheaded by saxophonist Charlie Parker (Parker, Charlie), was just getting under way. Brown began reaching wider audiences in the late 1940s when he started working with singer Ella Fitzgerald (Fitzgerald, Ella) (the couple married in 1947 and divorced in 1952), and from 1951 to 1966 he was a member of the Oscar Peterson (Peterson, Oscar) Trio, considered by many to be the finest small group in jazz history. Beginning in 1952 he was a frequent performer on Norman Granz's popular Jazz at the Philharmonic concert tours. Brown's playing was characterized by magnificent tonal resonance, extremely subtle grasp of harmony, and astonishing technique.

After leaving Peterson's trio in 1966, Brown settled in California, where he became a freelance and studio musician. He played on more than 2,000 recordings and worked with such musicians as Duke Ellington (Ellington, Duke), Louis Armstrong (Armstrong, Louis), and Frank Sinatra (Sinatra, Frank). Brown continued to perform until his death.