During the 1890s McGraw was a star infielder for the Baltimore National League club. (Both the American and the National League Baltimore teams of this era were named the Orioles; neither team, however, was affiliated with the current American League Orioles, who took that name upon moving from St. Louis, Missouri, in 1954.) His .391 mark of 1899 remains the highest batting average attained by any major league third baseman.

In 1901 McGraw was appointed manager of the Baltimore club in the new American League. In that first year McGraw bought the contract of African American player Charlie Grant from the Negro league Columbia Giants. Because of the segregation that existed in baseball, McGraw tried to pass Grant off as a Cherokee Indian. The ruse was unsuccessful, and the colour bar would not be breached until Branch Rickey signed Jackie Robinson in 1947. The attempt to sign Grant was typical of McGraw, who was always on the lookout for talent and was sometimes willing to bend the rules if it enabled his team to win.

McGraw and American League president Ban Johnson had been in conflict for some time, and on July 19, 1902, McGraw returned to the National League as manager of the New York team. The enmity between McGraw and Johnson was so great that, when the Giants won the National League championship in 1904, they refused to play against the American League team in the newly organized World Series.

Until McGraw's retirement in June 1932, the Giants were generally the most feared team in the league. McGraw was a tyrant of a manger; he was abusive and difficult with umpires and put winning ahead of nearly all other considerations. His tactics succeeded, as the Giants won league championships in 1904, 1905, 1911–13, 1917, and 1921–24, taking World Series titles in 1905, 1921, and 1922. McGraw retired in 1932; in his 33 years of managing, his teams won 2,840 games, a total exceeded only by that of one other manager, Connie Mack (Mack, Connie). McGraw returned to baseball the year after his retirement to manage the National League team in the first All-Star game. He was voted into the Baseball Hall of Fame in Cooperstown, New York, in 1937.