A descendant of notable colonial families of Virginia as well as of the Indian princess Pocahontas, Randolph distinguished himself from a distant relative by assuming the title John Randolph of Roanoke, where he established his home in 1810.

In 1799 Randolph was elected to the U.S. House of Representatives, and he served in that legislative body almost continuously until 1829. His political rise was so rapid that by 1801 he was chairman of the House Ways and Means Committee and leader of the Jeffersonian Republicans in Congress. His debating skill and biting sarcasm made him a feared opponent through the years, and he anticipated the states'-rights theories of John C. Calhoun by passionately defending state sovereignty on every occasion. He thus opposed a national bank, protective tariffs, federally financed internal improvements (such as roads and canals), and federal interference with the institution of slavery—though he freed his own bondsmen in his will.

After his failure as manager of the impeachment trial of Supreme Court justice Samuel Chase in 1804–05, in addition to his opposition to President Thomas Jefferson's efforts to acquire Florida, Randolph drifted away from the Jeffersonian Republican Party. He returned to national prominence in 1820 when he represented Southern planters in resisting the Missouri Compromise, which outlawed slavery in new western territory north of the 36°30′ parallel. During those years, when party feelings ran high, Randolph's denunciation of Henry Clay's (Clay, Henry) support of John Quincy Adams for the presidency in the disputed election of 1824–25 led him into a duel with Clay from which both emerged unscathed.

He served briefly in the Senate (1825–26) and three years later was a prominent member of the convention that drafted a new Virginia constitution. In 1830 President Andrew Jackson sent him on a special mission to Russia, but ill health forced him to return to the United States after only a few weeks at his post.