Smythe was educated at the University of Toronto, receiving his engineering degree in 1920. Both before and after World War I, in which he served in the artillery (1915–17) and the air force (1917), he played hockey at the university and coached the varsity (1927) team. He also coached the gold-medal-winning Canadian team at the 1928 Olympic Games in St. Moritz, Switzerland.

In 1926 he helped to organize the New York Rangers in the NHL, and, when the club was taken over by Lester Patrick (Patrick family), he acquired the franchise of the Toronto St. Patricks, which he renamed the Maple Leafs. The team won seven Stanley Cups (1932, 1942, 1945, 1947–49, and 1951) before he sold the club in 1961 to a group led by his son Stafford. In 1931 he built the Maple Leaf Gardens, in which the Leafs played. He introduced coast-to-coast Canadian broadcasting of Maple Leaf games. In 1964 the Maple Leaf Gardens created the Conn Smythe Trophy, which is awarded annually for the best player in the Stanley Cup play-offs. When the NHL was expanded, the Conn Smythe Division (1974–94) was named in his honour.

Smythe commanded an antiaircraft battalion in World War II. After retirement from hockey, Smythe built a Thoroughbred breeding and racing operation; his horses won the Queen's Plate and the Canadian Oaks three times each.