Adler's family was not particularly musical, but their observance of Orthodox Judaism provided access to religious music. By age 10 Adler was the youngest cantor in Baltimore, although unhappy in school. By feigning a nervous breakdown, he convinced his parents to let him enroll in the Peabody Conservatory of Music, but he was soon dismissed as untalented. The rebuff made him even more determined. At age 11, without his parents' consent, Adler ordered a piano for their home, which he then persuaded them to accept. He also began playing the mouth organ. Unable to read music, he listened assiduously, buying records and concert tickets with money earned by selling magazines.

In 1927 Adler won the Maryland National Harmonica Championship, playing a Beethoven minuet. The following year in New York City he accompanied early motion-picture cartoons and performed vaudeville routines dressed as a vagabond. He had played in both musicals and motion pictures before he was invited to play with an orchestra. His solo debut took place in 1939 with the symphony orchestra of Sydney, Australia. Adler did not learn to read music until 1940, when the French composer Jean Berger wrote a harmonica concerto for him. Ralph Vaughan Williams, Darius Milhaud, and others also wrote musical scores for Adler. Accused of communist sympathies and blacklisted during the ascendancy of the U.S. Senator Joseph R. McCarthy, Adler was unable to find work and took up residence in England.

Adler wrote musical scores for motion pictures and television and also wrote several books, including How I Play (1937) and Larry Adler's Own Arrangements (1960). His autobiography, It Ain't Necessarily So, was published in 1987.