McKim was educated at Harvard University and at the École des Beaux-Arts (“School of Fine Arts”) in Paris. He was trained as a draftsman by the architect Henry Hobson Richardson (Richardson, Henry Hobson) while the latter was completing Trinity Church in Boston. In 1879 McKim joined William Rutherford Mead and Stanford White to found McKim, Mead & White, which became the most successful and influential American architectural firm of its time. Until 1887 the firm excelled at informal summer houses built of shingles, and McKim designed one of the most significant of these, the residence at Bristol, R.I., of W.G. Low (1887); other examples of the firm's work cluster at Newport, R.I., on the New Jersey shore, and on Long Island, N.Y.

In later years the firm was famous for championing the formal tradition of the Italian Renaissance and its Classical antecedents, particularly because the styles embodied a view of the increasingly powerful United States that placed it in this grand lineage. Among the celebrated examples of the formal planning of McKim are the Boston Public Library (1887) and in New York City the Columbia University Library (1893), the University Club (1899), the Morgan Library (1903), and Pennsylvania Station (1904–10; demolished). The railway station was the largest of these buildings; its enormous hall, with its vaulted ceiling, was explicitly based on the baths of Caracalla (Caracalla, Baths of) in Rome. With Daniel H. Burnham (Burnham, Daniel H.) and Richard Morris Hunt (Hunt, Richard Morris), McKim developed and oversaw the building program of the World's Columbian Exposition at Chicago in 1893, which was inspired by Classical styles. McKim designed the Agricultural Building. He also aided Burnham (Burnham, Daniel H.) in reviving Pierre L'Enfant (L'Enfant, Pierre-Charles)'s plan for Washington, D.C., and was the originator of the American Academy in Rome.