Maskelyne was ordained a minister in 1755, but his interest in astronomy had been aroused by the eclipse of July 25, 1748. In 1758 he was admitted to the Royal Society of London, which in 1761 sent him to the island of St. Helena to observe a transit of Venus. During the voyage he experimented with the determination of longitude by observations of the Moon's position and introduced this method into navigation by publishing The British Mariner's Guide (1763). Succeeding Nathaniel Bliss as astronomer royal in 1765, he published the first volume of the Nautical Almanac in 1766 and continued the supervision of the almanac until his death.

Maskelyne suggested to the Royal Society an experiment for determining the Earth's density with the use of a plumb line. He carried out the experiment two years later in Scotland on Schiehallion Mountain, North Perthshire. From his observations, it was found that the Earth's density is approximately 4.5 times that of water. He was also the first to make time measurements that were accurate to the nearest tenth of a second.