Smith was born into a wealthy family. In about 1828 he became an active worker in the cause of temperance, and in his home village, Peterboro, he built one of the first temperance hotels in the country. He became an abolitionist (abolitionism) in 1835, after witnessing the disruption of an antislavery meeting by a mob in Utica. In 1840 he took a leading part in the organization of the antislavery Liberty Party, and in 1848 and 1852 he was nominated for the presidency by the remnant of this organization that had not been absorbed by the Free Soil Party. In 1852 he was elected to the U.S. House of Representatives as an independent. At the end of the first session in 1854 he resigned his seat.

After becoming an opponent of land monopoly, Smith gave numerous farms of 50 acres each to indigent families, and he also attempted to colonize tracts in northern New York with free blacks, but this experiment was a failure. Peterboro became a station on the Underground Railroad, and after 1850 Smith furnished money for the legal expenses of persons charged with infractions of the Fugitive Slave Law (Fugitive Slave Acts). He became very intimate with John Brown (Brown, John), to whom he gave a farm in Essex County (New York), and from time to time supplied him with funds, though it seems without knowing that any of the money would be employed in an attempt to incite a slave insurrection. Under the excitement following the raid on Harpers Ferry in October 1859, he became temporarily insane, and for several weeks was confined in an asylum in Utica.

Smith favoured a vigorous prosecution of the Civil War (American Civil War) but at its close advocated a mild policy toward the late Confederate States, declaring that part of the guilt of slavery lay upon the North. He even became one of the securities for Jefferson Davis, thereby incurring the resentment of Northern Radical leaders.