Both of Oliver's parents were music teachers in Ohio, where he grew up. He played the trumpet as a boy and at the age of 17 took a job (1927–30) with Zack Whyte and his Chocolate Beau Brummels. He joined the Jimmie Lunceford (Lunceford, Jimmie) orchestra in 1933. There he established his reputation for innovative arranging characterized by imaginative instrumentation and a full-bodied sound. He also developed a distinctive sound, sometimes called “growl” trumpet, in his own playing. In 1939 he joined the orchestra of Tommy Dorsey (Dorsey, Thomas Andrew) as a singer and arranger. He led a band while in the army during World War II and returned to Dorsey's orchestra after the war. From the late 1940s to the early '70s Oliver held a variety of jobs, including a decade as musical director of Decca Records. In the early '70s he formed a nine-piece orchestra that continued to perform until 1984.