Trained in music and other subjects by his mother, a schoolteacher, Johnson graduated from Atlanta University with A.B. (1894) and M.A. (1904) degrees and later studied at Columbia University. For several years he was principal of the black high school in Jacksonville, Fla. He read law at the same time, was admitted to the Florida bar in 1897, and began practicing there. During this period, he and his brother, John Rosamond Johnson (1873–1954), a composer, began writing songs, including “Lift Every Voice and Sing,” based on James's 1900 poem of the same name, which became something of a national anthem to many African Americans. In 1901 the two went to New York, where they wrote some 200 songs for the Broadway musical stage.

In 1906 President Theodore Roosevelt appointed him U.S. consul to Puerto Cabello, Venezuela, and in 1909 he became consul in Corinto, Nicaragua, where he served until 1914. He later taught at Fisk University. Meanwhile, he began writing a novel, The Autobiography of an Ex-Colored Man (published anonymously, 1912), which attracted little attention until it was reissued under his own name in 1927. From 1916 Johnson was a leader in the National Association for the Advancement of Colored People (NAACP) (National Association for the Advancement of Colored People).