In high school and at Syracuse University in New York, Brown displayed exceptional all-around athletic ability, excelling in basketball, baseball, track, and lacrosse as well as football. In his final year at Syracuse, Brown earned All-America (All-America team) honours in both football and lacrosse. Many considered Brown's best sport to be lacrosse, and he was inducted into both the Pro Football Hall of Fame and the U.S. Lacrosse National Hall of Fame.

From 1957 through 1965, Brown played for the Cleveland Browns of the NFL, and he led the league in rushing yardage every year except 1962. Standing 6.2 feet (1.88 metres) and weighing 232 pounds (105 kg), Brown was a bruising runner who possessed the speed to outrun opponents as well as the strength to run over them. He rushed for more than 1,000 yards in seven seasons and established NFL single-season records by rushing for 1,527 yards in 1958 (12-game schedule) and 1,863 yards in 1963 (14-game schedule), a record broken by O.J. Simpson (Simpson, O.J.) in 1973. On November 24, 1957, he set an NFL record by rushing for 237 yards in a single game, and he equaled that total on November 19, 1961. At the close of his career, he had scored 126 touchdowns, 106 by rushing, had gained a record 12,312 yards in 2,359 rushing attempts for an average of 5.22 yards, and had a record combined yardage (rushing along with pass receptions) of 15,459 yards. Brown's rushing and combined yardage records stood until 1984, when both were surpassed by Walter Payton (Payton, Walter) of the Chicago Bears.

At 30 years of age and seemingly at the height of his athletic abilities, Brown retired from football to pursue a career in motion pictures. He appeared in many action and adventure films, among them The Dirty Dozen (1967) and 100 Rifles (1969). Brown was also active in issues facing African Americans, forming groups to assist black-owned businesses and to rehabilitate gang members.