His mother was a house slave and his father an unknown white man. Smalls was taken by his master in 1851 to Charleston, South Carolina, where he worked as a hotel waiter, hack driver, and rigger. In 1861, at the outbreak of the war, he was hired to work aboard the steamship Planter, which operated as an armed transport and dispatch vessel, carrying guns and ammunition for the Confederate army. On May 13, 1862, he and the other blacks on board seized control of the ship in Charleston Harbor, succeeded in passing through Confederate checkpoints, and turned the ship, its cargo of weapons, and several important documents over to a Union naval squadron blockading the city. This exploit brought Smalls great fame throughout the North. In 1863, when he was piloting the ironclad Keokuk in the battle for Fort Sumter, the vessel took many hits and was eventually sunk. Smalls's bravery was rewarded with command of the Planter later that year. He was the first African American captain of a vessel in U.S. service.

After the war, Smalls rose rapidly in politics, despite his limited education. From 1868 to 1870 he served in the South Carolina House of Representatives and from 1870 to 1874 in the state Senate. He was elected to the U.S. House of Representatives (1875–79, 1882–83, 1884–87), where his outstanding political action was support of a bill that would have required equal accommodations for both races on interstate conveyances. In 1877, however, he was convicted of having taken a $5,000 bribe while in the state Senate; sentenced to three years in prison, he was pardoned by the governor. The case against him was clearly politically motivated. In 1895 he delivered a moving speech before the South Carolina constitutional convention in a gallant but futile attempt to prevent the virtual disfranchisement of blacks.

A political moderate, Smalls spent his last years in Beaufort, where he served as port collector (1889–93, 1897–1913).