Frelinghuysen was born into a family that had long been prominent in politics. Left an orphan at the age of three, he was adopted by his uncle, Theodore Frelinghuysen. He graduated from Rutgers College in 1836, studied law in his uncle's law office, and succeeded to the latter's large practice in 1839. Frelinghuysen was one of the founders of the Republican Party in New Jersey and served as the state's attorney general from 1861 to 1866.

In 1866 he was appointed to the U.S. Senate to fill a vacancy and was elected the following year to fill the unexpired term. Defeated for the Senate in 1869, he was elected for a full six-year term beginning in 1871. He was one of the Senate's ablest debaters and did important work as chairman of the committee on foreign affairs during the negotiations over the Alabama claims. President Chester A. Arthur appointed Frelinghuysen secretary of state in 1881 to succeed James G. Blaine. Patient and firm in his handling of diplomatic affairs, he favoured closer commercial relations with Latin America and negotiated a change in the treaty with Hawaii to secure a U.S. naval base at Pearl Harbor. He also opened treaty relations with Korea (1882) and mediated several international disputes.