Admitted to the bar in 1851, Gray practiced law in Massachusetts and was active in Free-Soil and, later, Republican party affairs. In 1860 he ran unsuccessfully for state attorney general. He served with distinction for many years at the Massachusetts Supreme Court, first as court reporter (1854–61) and later as associate justice (1864–73) and chief justice (1873–81).

In 1881 he was appointed to the U.S. Supreme Court by Pres. Chester A. Arthur. During his 21 years on the bench, Gray was distinguished for his broad knowledge of the court's previous decisions and the deft application of precedent to cases before him. In his most notable opinion, Juilliard v. Greenman (1884), Gray upheld the right of the federal government to make paper money legal tender for the payment of private debt even in times of peace, a procedure previously considered constitutional only as an emergency war measure.