Only 5 feet 7 inches (1.7 metres) and 150 pounds (68 kg), Pollard won the grudging acceptance of his teammates at Brown University in Rhode Island in 1915, leading the team to a victory over Yale and an invitation to the Tournament of Roses game in Pasadena, California. Pollard had a subpar game in a 14–0 defeat to Washington State, but he became the first African American to play in the Rose Bowl game. In 1916 Pollard's outstanding play led Brown to a season of eight victories and one defeat, including wins over both Yale and Harvard.

After service in World War I, Pollard became head football coach at Lincoln University (Pennsylvania) and began playing professional football for Akron in the informal Ohio League in 1919. The following year Pollard was the star player for the Akron Pros, who won the first National Football League (NFL) championship. Pollard continued to play and coach in the NFL until 1926. In 1923, while playing for the Hammond Pros, he became the first African American quarterback in the league. Pollard also facilitated integration in the NFL by recruiting other African American players such as Paul Robeson, Jay Mayo Williams, and John Shelbourne and by organizing the first interracial all-star game featuring NFL players in 1922.

After he was let go by Akron (which had changed its name to the Indians) in 1926, Pollard continued to promote integration in professional football as a coach of the barnstorming Chicago Black Hawks (1928–32) and the New York Brown Bombers (1935–37). In 1954 Pollard became the second African American selected to the College Football Hall of Fame. He was posthumously inducted into the Pro Football Hall of Fame in 2005.