Rivers early developed an interest in jazz, and after briefly serving in the army during World War II he studied composition at the Juilliard School of Music. One of his classmates there was Miles Davis (Davis, Miles), who introduced him to other jazz musicians, and Rivers was soon touring the United States with different groups as a jazz saxophonist. In 1945, however, he was given a book on modern art and quickly discovered he had a natural talent for painting. From 1947 to 1948 he studied in the New York City and Provincetown, Massachusetts, school of the prominent Abstract Expressionist Hans Hofmann (Hofmann, Hans). Rivers later studied at New York University College, graduating in 1951. His early paintings were exhibited in New York City in 1949.

Rivers's first major work was The Burial (1951), a grim depiction of his grandmother's funeral, based on the Burial at Ornans by Gustave Courbet (Courbet, Gustave). His Washington Crossing the Delaware (1953) was based on the familiar work by a 19th-century American painter, Emanuel Leutze (Leutze, Emanuel Gottlieb). Though criticized for its banal subject matter and mixture of styles, the painting nonetheless attracted widespread attention. From 1951 to 1957 he made a series of portraits of his mother-in-law, of which the harshly realistic Double Portrait of Berdie (1955) is perhaps best known.

Rivers's works were characterized by competent draftsmanship, a fine sense of colour, and the frequent use of complex, fragmentary, and multiple views. Beginning in 1961, commercial images, such as cigarette packages, figured prominently in his pictures, which, after 1963, frequently had elements of collage, construction, and sculpture. A particularly elaborate example of such mixed-media works was The History of the Russian Revolution: From Marx to Mayakovsky (1965), which had some 30 individual paintings and included, among other objects, a machine gun. His autobiography, What Did I Do? (cowritten with Arnold Weinstein), was published in 1992.