Although Allen announced other sporting events, he is best known for his work in baseball. The owner of one of the most recognizable voices in radio, he was the play-by-play announcer for the New York Yankees from 1940 through 1964. Initially, Allen broadcast all the Yankee home games and studio re-creations of road games. After serving three years in the U.S. Army, he returned to the Yankees in 1946 and became the first announcer to call all of his team's away games live on site. In the early 1950s, Allen added work on Yankee telecasts to his radio duties. He rose to national prominence as the Yankees appeared in 15 of the 18 World Series between 1947 and 1964, with Allen calling a majority of the series games that were broadcast on network television. In an event that astonished the baseball world, Allen was fired by the Yankees without any explanation in 1964.

After an extended absence from baseball broadcasting, Allen returned as the host of Major League Baseball's televised weekly highlight show, This Week in Baseball, in 1977. He was hired the same year to work on the cable television transmissions of Yankees games, a job he held until 1985. In 1978 Allen and Red Barber (Barber, Red)—his longtime friendly rival in the New York media market—were the first recipients of the Ford C. Frick Award, which is given each year to a broadcaster who has made a major contribution to baseball and results in enshrinement in a special exhibit in the Baseball Hall of Fame in Cooperstown, N.Y.