Fain was a self-taught pianist who played by ear. He began working as a staff pianist and composer for music publisher Jack Mills in 1925 and in the late 1920s appeared in a vaudeville act with Artie Dunn entitled “Fain and Dunn.” Fain achieved his first success as a songwriter after teaming up with lyricist Irving Kahal in 1927, and the pair collaborated until Kahal's death in 1942. In the 1930s Fain moved to Hollywood and contributed songs to more than 20 films, among them The Big Pond (1930), Footlight Parade (1933), Anchors Aweigh (1945), Three Sailors and a Girl (1953), April Love (1957), and the Walt Disney feature cartoons Alice in Wonderland (1951) and Peter Pan (1953). He was nominated 10 times for an Academy Award for best song and, with lyricist Paul Francis Webster, won twice: the first in 1953 for “Secret Love,” from the film Calamity Jane, and the second in 1955 for “Love is a Many-Splendored Thing,” from the film of the same name.

Fain returned frequently to Broadway, where his songs were included in such shows as Hellzapoppin (1938), George White's Scandals (1939), Flahooley (1951), and Christine (1960). “I'll Be Seeing You,” introduced in the show Right This Way in 1938, became one of the most popular songs of World War II.