The oldest part of the canal lies between the Yangtze and the city of Huaiyin (formerly called Qingjiang) in Jiangsu (Kiangsu) province, which was originally on the Huang He (Yellow River) when that river followed a course much farther to the south. This section, traditionally known as the Shanyang Canal, in recent centuries has been called the Southern Grand Canal (Nan Yunhe). This ancient waterway was first constructed as early as the 4th century BC, was rebuilt in AD 607, and has been used ever since.

China's first great canal system, which created a northeast-southwest link from the Huang He (when the Huang had a northern course) to the Huai River, was built beginning in 605 during the Sui dynasty (581–618). Known as the New Bian Canal, it remained the chief waterway throughout the Tang (Tang dynasty) period (618–907) and the Northern Song (Song dynasty) period (960–1125/26).

The need for a major transport link again arose during the Yuan (Yuan dynasty) (Mongol) dynasty (1279–1368), because its capital at Dadu (Beijing) required a grain-supply system. In 1282–83 the decision was made to build a new canal from the Huang He—which since 1195 had changed its course southward and taken over the former mouth of the Huai below Huaiyin—to the Daqing River in northern Shandong (Shantung) province, which was dredged to give an outlet to the sea. The mouth of the Daqing, however, silted up almost immediately. An alternative canal, cut across the neck of the Shandong Peninsula from the harbour of Qingdao (Tsingtao) to Yixian, also proved impracticable and was abandoned. Eventually another stretch of canal, the Huitong Canal, was built to join Dong'e Zhen on the Huang He with the Wei River at Linqing. In this way, the modern Grand Canal came into being. During the Yuan period, however, canal transport was expensive and inefficient, and most grain went by sea.

At the beginning of the Ming dynasty (1368–1644), the capital was at Nanjing (Nanking). After Beijing again became the seat of government in 1403, the whole canal—including the section from Linqing on the Wei to its junction with the Huang He, which was dredged and repaired—remained in operation until the 19th century. It comprised six main sections: (1) a short canal from the outskirts of Beijing to Tongzhou, (2) a canalized river joining the Hai River (Hai River system) to Tianjin and then joining the Wei River as far as Linqing, (3) a section in Shandong rising over comparatively high ground from Linqing to its highest point near Jining and then falling again to a point near Xuzhou, a difficult stretch using a series of dams, sluices, and locks supplied with water from a number of small rivers flowing off the Mount Tai (Tai, Mount) massif and from the string of lakes southeast of Jining, (4) a stretch from Xuzhou that followed the southern course of the Huang He as far as Huaiyin, (5) a section from Huaiyin following the ancient Shanyang Canal south to Zhenjiang on the Yangtze, and (6) a section south of the Yangtze where the canal, there called the Jiangnan Yunhe, ran southeast and then southwest for some 200 miles (320 km) via Suzhou to Hangzhou.

In the 19th century a series of disastrous floods broke the dikes of the Huang He (which began to shift to its present northern course), caused great problems in the section of the canal between Xuzhou and Huaiyin, and cut across the canal between Linqing and Jining. After the Taiping Rebellion (1850–64) and the Nian Rebellion (1853–68), the use of the canal as the major supply line to Beijing was abandoned, and the canal gradually fell into disrepair in its northern sections. After 1934 the Chinese government carried out extensive works on the canal between Huaiyin and the Yangtze; ship locks were constructed to allow medium-sized steamers to use this section, which was dredged and largely rebuilt.

New work was begun in 1958 to restore the whole system as a trunk waterway able to carry ships of up to 600 tons. Between 1958 and 1964 it was straightened, widened, and dredged; one new section 40 miles (64 km) long was constructed, and modern locks were added. The canal can now accommodate medium-sized barge traffic throughout its length. The main traffic, however, is concentrated in the southern half. The canal is also used to divert water from the Yangtze to northern Jiangsu province for irrigation, making possible double cropping of rice.