An attorney, Sullivan was elected to the New Hampshire provincial congress (1774) and served at the First Continental Congress, in Philadelphia, the same year. In June 1775 he was appointed brigadier general in the Continental Army and aided in the siege of Boston. The following year he was ordered to Canada to command the retreating American troops after the death of their commander at the disastrous Battle of Quebec (Quebec, Battle of) (December 31, 1775). Sullivan shortly rejoined General George Washington (Washington, George) and, after being promoted to major general, participated in the Battle of Long Island (Long Island, Battle of) (August 1776), where he was taken prisoner. Exchanged in December, he led the right column in Washington's successful attack on Trenton, New Jersey (December 1776), but a night attack on Staten Island in August was unsuccessful.

In 1779 Sullivan was commissioned to lead an expedition in retaliation for British-inspired Indian raids in the Mohawk Valley of New York. With 4,000 troops he routed the Iroquois and their loyalist supporters at Newtown, New York (near present Elmira), burning their villages and destroying their crops. He thus earned the thanks of Congress (October 1779), but ill health forced him to resign from military service soon afterward.

Sullivan continued in public service for 15 years, however: as a delegate to the Continental Congress (1780–81), state's attorney general (1782–86), New Hampshire governor (1786–87, 1789), presiding officer of the state convention that ratified the federal Constitution (1788), and U.S. district judge (1789–95).