Until 1765 Carroll attended Jesuit colleges in Maryland and France and studied law in France and England. Before and during the American Revolution, he served on committees of correspondence and in the Continental Congress (1776–78), where he was an important member of the board of war. In 1776, with Benjamin Franklin, Samuel Chase, and his cousin, the Reverend John Carroll, he was sent to Canada in a fruitless effort to persuade Canadians to join the cause of the 13 colonies.

Carroll was a state senator in Maryland (1777–1800) and concurrently a U.S. Senator (1789–92). When political parties were formed in the United States, he became a Federalist.