pali vs. sanskrit: what's the deal?

People new to Buddhism sometimes ask how it came to be that the teachings of Theravada were transmitted in Pali, but the teachings of Mahayana were primarily transmitted in Sanskrit. I'm certainly not an expert on the subject, so take whatever I say with a grain of salt; but from what I understand, it's mainly due to the combination of when and where each set of texts were initially composed, and when and where they were eventually transmitted.

Pali, an early form of Prakrit related to Hindi and Sanskrit, is thought to be a composite of several dialectal forms and expressions most likely based on the language the Buddha himself taught in, which is generally held to be a dialect of Magadhi Prakrit; although there's still a great deal of debate among scholars as to the exact dates and place of origin of Pali itself.

The commentarial tradition of Theravada holds that Pali is identical to Magadhi; but as the introduction to A New Course in Reading Pali: Entering the Word of the Buddha notes, it doesn't share many of the distinctive characteristics found in Magadhan inscriptions, primarily from the time of Asoka (approximately 300–232 BCE). Nevertheless, it's considered by many scholars, such as Wilhelm Geiger and Walpola Rahula, to at least be closely related to Magadhi, especially in the sense of being a type of popular speech.

Whatever the case, it's believe that at the time of the Buddha (approximately 400 BCE), many of the great wandering ascetics (samana) in the northern area of India known as Magadha, like the Buddha and his contemporary Mahavira (Nigantha Nataputta), taught in the popular vernacular of the people used for general communication and commerce, as opposed to Vedic Sanskrit, the sacred language of Vedas used by brahmins. This was not only done because they rejected the authority of the Vedas, but because they wanted to make their teachings more widely available. The use of Vedic Sanskrit also appears to be in decline by this time.

Not long afterwards, however, Sanskrit seems to have made a serious comeback as a literary and religious language thanks to the great Indian scholar and grammarian, Panini (scholarly dates vary from 500 to 300 BCE), and texts were starting to be written down as well as passed on orally. By the time the early Mahayana sutras were being composed, mostly in the south but also the north, Panini's Sanskrit had already become the standard, most likely starting in Gandhara, his home, and spreading south and east throughout the rest of India.

Early Buddhist texts underwent various degrees of Sanskritization, while newer texts were being composed in what's now termed Buddhist Hybrid Sanskrit or even classical Sanskrit itself. Much of the former, making the scriptural basis of Theravada, were transmitted via Asoka to places like Sri Lanka, where they survived the decline of Buddhism in India and Central Asia, while much of the latter, making the scriptural basis of Mahayana, found a safe home in places like China.