Torres was a member of the 1956 U.S. Olympic boxing team and a silver medalist in the light middleweight (71 kg, or 156.5 pounds) division before turning professional in 1958. He won the light heavyweight title by knocking out American Wilfred (“Willie”) Pastrano in the ninth round on March 30, 1965. In 1966 Torres defended his title four times, losing the last bout on December 16 in a 15-round decision (a fight whose outcome is determined by judges' scoring) to Dick Tiger (Tiger, Dick) of Nigeria. On May 16, 1967, Torres lost a championship rematch with Tiger on another 15-round decision. Torres retired in 1969 with a record of 41 wins (29 by knockouts), 3 losses, and 1 draw.

Following his retirement, Torres stayed active in the sport, working as a journalist for boxing publications. He wrote biographies of Muhammad Ali (Ali, Muhammad) (Sting Like a Bee: The Muhammad Ali Story; 1971) and Mike Tyson (Tyson, Mike) (Fire and Fear; 1987). Torres was a member from 1983 and chairman from 1984 of the New York State Athletic Commission, from which he retired in 1988. He was president of the World Boxing Organization (WBO) from 1990 to 1995. Torres was inducted into the International Boxing Hall of Fame in 1997.