Low was graduated as valedictorian of Columbia College in 1870, thereupon entering his father's prosperous importing business and becoming a leader in the tea and silk trades. Low was twice elected mayor of Brooklyn, and his administration (1881–85) was noted for its civil service reform, administrative efficiency, public school reorganization, and advocacy of home rule.

Low served as president of Columbia from 1890 to 1901. By absorption or alliance he brought the College of Physicians and Surgeons, Barnard College, Teachers College, and other Columbia affiliates more closely into the university system. He also organized the University Council and the graduate faculties of philosophy and pure science. To attract leading scholars, Low offered incentives such as faculty sabbaticals and pension plans. His recruiting approach was copied by other major American universities.

His philanthropy was marked by his gift of an imposing central building on the new campus, by his support of professorships, and by his aid, through Frederick Barnard, of education for women. In 1899 he was a delegate to the first Hague Peace Conference.

Elected mayor of New York City in 1901 on an anti-Tammany fusion ticket, Low eschewed a paternalistic governing style and conducted municipal administration on business principles. Failing reelection in 1903, he nonetheless continued outstanding public service, furthering education for blacks as chairman of the trustees of Tuskegee Institute (Alabama) from 1907 until his death.