Williams moved from Georgia to Chicago at the age of three. As a youth he sang with a gospel group. In 1937 he joined clarinetist Jimmie Noone (Noone, Jimmie)'s band, which was broadcast nationally. Subsequently Williams worked with the big bands of Coleman Hawkins (Hawkins, Coleman), Lionel Hampton (Hampton, Lionel), Andy Kirk, and Red Saunders; he made his recording debut in 1950. Williams was unable at this point to work full time in music; he held a job as backstage doorman at Chicago's Regal Theater, where he met some of the era's great jazz musicians.

Williams's breakthrough came when he joined the Count Basie Orchestra in 1954. His recording of "Every Day I Have the Blues" with Basie in 1955 made him famous and was a factor in the Basie band's comeback. Staying with Basie until 1961, Williams also had hits with "Alright, Okay, You Win," "Going to Chicago," and "The Comeback." The rich timbre of Williams's baritone voice, his smooth delivery, his wit, and his style were widely appreciated by critics, audiences, and other performers.

Williams launched a solo career in 1961. He began this period as coleader of a group with trumpeter Harry Edison, and in later years he fronted small combos. For recording sessions Williams was teamed with Cannonball Adderley (Adderley, Cannonball), George Shearing, and the Thad Jones–Mel Lewis Orchestra, among others. Williams spent many weeks of each year on the road, appearing with big bands or as a solo act in nightclubs, at jazz festivals, on cruises, and on television (he was a favourite of Johnny Carson). He also acted in the movie The Moonshine War (1970) and on such television series as The Cosby Show (1985–92) and Lou Grant (1982). He won the 1984 Grammy Award for best jazz vocal performance with the album Nothin' but the Blues.