Johnson competed in his first decathlon in 1954 as a sophomore at the University of California, Los Angeles (UCLA), and in 1955 he won the gold medal at the Pan American Games. Injuries forced him to settle for a silver medal in the 1956 Olympic decathlon in Melbourne, Australia, but he set a world record in 1958.

At the 1960 Games the decathlon competition became a duel between Johnson and Taiwan's Yang Chuan-kwang, who was Johnson's friend and teammate at UCLA. After the first day, Johnson led Yang by 55 points, despite the fact that Yang had finished ahead of Johnson in four of the five competitions. On the second day, Johnson fell from the lead when he hit the first hurdle in the 110-metre hurdles and finished 0.7 sec behind Yang. The two traded positions in the standings again after the discus throw, and Johnson increased his lead with a career-best performance in the pole vault and a better throw than Yang in the javelin. Yet victory for Johnson was far from certain at the start of the final event, the 1,500 metres. He led by only 67 points, and Yang, favoured in this event, needed to beat Johnson by 10 sec to win the decathlon. Johnson ran a personal best of 4 min 49.7 sec and finished only 1.2 sec behind Yang. Johnson won the gold medal, and Yang took the silver medal—the first medal of any kind won by a Taiwanese athlete.

Johnson received the James E. Sullivan Memorial Award as the outstanding amateur athlete of 1960. In 1984 he lit the torch signaling the opening of the Olympic Games in Los Angeles.