After finishing high school, Rustin held odd jobs, traveled widely, and obtained five years of university schooling at the City College of New York and other institutions without taking a degree. Rustin became a foe of racial segregation and a lifelong believer in pacifist agitation. He worked for the Fellowship of Reconciliation, a nondenominational religious organization, from 1941 to 1953, and he organized the New York branch of another reformist group, the Congress on Racial Equality, in 1941.

In the 1950s Rustin became a close adviser to the civil-rights leader Martin Luther King, Jr. (King, Martin Luther, Jr.), and he was the chief organizer of King's Southern Christian Leadership Conference. Rustin was the chief organizer of the March on Washington (August 1963), a massive demonstration to rally support for civil-rights legislation that was pending in Congress. In 1964 he directed a one-day student boycott of New York City's public schools in protest against racial imbalances in that system. Rustin subsequently served as president of the A. Philip Randolph Institute, a civil-rights organization in New York City, from 1966 to 1979.