Of peasant birth, Konev was drafted into the tsarist army in 1916. After the Russian Revolution, he joined (1918) the Communist Party and the Red Army. During the Civil War, he fought against the White Russian Army of Aleksandr V. Kolchak (Kolchak, Aleksandr Vasilyevich), organized guerrilla bands, and fought the Japanese. Konev participated in subduing the Kronshtadt Rebellion (1921) against Bolshevik rule. He finished his military training at the Frunze Military Academy, graduating in 1926.

Konev was a general in the army when the Germans invaded the Soviet Union in June 1941; it was he who led the first real counterattack of the war. He defeated the German tank expert General Heinz Guderian (Guderian, Heinz)'s advance on Moscow (December 1941) by employing the “Konev ambush”—a planned retreat of troops in the centre, with the flanks then snapping shut across the breach to trap the pursuing enemy. In the critical summer of 1942 he halted a large German force that had been sent to reinforce the German army pounding at Stalingrad. He was one of the generals involved in repelling Germany's third summer offensive (1943). After his entrapment of 100,000 Nazi troops in the Korsun salient, Konev was made a marshal of the Soviet Union (March 1944). In August 1944 his army was the first to carry the fighting beyond Soviet frontiers, crossing the Vistula River, and, after sweeping across Poland, it was the first to march onto German soil. Konev's 1st Ukrainian Army advanced to the Oder River and, together with Marshal Georgy K. Zhukov (Zhukov, Georgy Konstantinovich)'s forces, entered and captured Berlin. His army linked with the U.S. forces under Lieutenant General Courtney H. Hodges at Torgau.

After the defeat of Germany, Konev became supreme commissar for Austria (1945), succeeded Zhukov as commander in chief of Soviet ground forces (1946–50), and thereafter held various military posts on the ministerial level, including commander in chief of the Warsaw Pact forces (1955–60). In 1961 he was temporarily recalled to act as commander in chief of Soviet forces in East Germany, stepping down in 1962 although still retaining a post in the defense ministry.