Graun was a chorister in Dresden, where as a youth he composed several cantatas for church services and worked under the Neapolitan-opera composer Antonio Lotti. In 1725 he made his debut in opera as a tenor at Brunswick. But he was dissatisfied with the arias given him and rewrote them; he then began composing entire operas. At Brunswick he also composed six operas and two Passions.

He became music director to Frederick the Great (Frederick II) (then crown prince) in 1735 and in 1740 recruited singers for Frederick's Italian opera company. While in royal service, Graun composed about 30 operas to Italian words, two of them, Montezuma and Merope, to librettos by Frederick. For about 150 years his Der Tod Jesu (1755), like Handel's Messiah in England, was performed annually in Germany during Holy Week. His Te Deum (1757) was written to celebrate the Prussian victory at Prague. Graun's compositions also include harpsichord concerti, trio sonatas, and other chamber works.

As a composer, Graun was a leading exponent of the preclassical Berlin school, which also included C.P.E. Bach, Graun's elder brother, Johann Gottlieb (1703–71), and Frederick the Great himself. His music shows a combination of old and new melodic and formal concepts. His operas are highly Italianized in the predominant Neapolitan style.