Sheppard was a member of the Irish American Athletic Club of New York City. In 1906 and 1907 he set records for the 880-yard and 1,000-yard races. At the 1908 Olympic Games in London, he won gold medals in the 800-metre and 1,500-metre races and was also a member of the relay team in the 4 × 400-metre race that won a gold medal. Sheppard is important in the history of middle-distance racing because he was one of the first runners to set a fast pace early in the race. At the 1912 Olympic Games in Stockholm, he won the silver medal for the 800-metre race and was a member of the gold-medal-winning 4 × 400-metre relay team that set a world record of 3 min 16.6 sec, unbroken until 1924. He was also a member of a team that set a 4 × 440-yard relay world record (1911–15) of 3 min 18.2 sec and of another team that set a 4 × 880-yard relay world record of 7 min 53 sec (1910–20). He was the Amateur Athletic Union 800-yard champion (1906–08, 1911–12).