A Boston attorney who had served only one term in the Massachusetts legislature (1873–74), Olney was suddenly thrust into national prominence when President Grover Cleveland appointed him U.S. attorney general in 1893. In this position, during the strike of railway employees against the Pullman Company (Pullman Strike) in Chicago (1894), he obtained a court-ordered injunction to restrain the strikers from acts of violence, thus setting a precedent for the use of such injunctions to help break labour strikes. Olney sent federal troops to the scene, arrested Eugene Debs and other strike leaders, and saw his use of injunctions sustained by the Supreme Court the following year.

Becoming secretary of state in June 1895, Olney was almost immediately faced with the problem of appeals by Venezuela for U.S. support in its dispute with Great Britain over the Venezuela-British Guiana boundary. With Cleveland's support, Olney issued (July 20) an aggressive note demanding that Britain, in conformity with the Monroe Doctrine, arbitrate the controversy to avoid war and asserting the sovereignty of the United States in the Western Hemisphere. The matter was in fact arbitrated in 1899, after Olney retired in 1897 to the private practice of law.