Principles of Message-Passing Programming

There are two key attributes that characterize the message-passing programming paradigm.

1. the first is that it assumes a partitioned address space,
2. the second is that it supports only explicit parallelization.

• Each data element must belong to one of the partitions of the space;
  • hence, data must be explicitly partitioned and placed.
  • Adds complexity, encourages data locality, NUMA architecture.
• All interactions (read-only or read/write) require cooperation of two processes (the process that has the data and the process that wants to access the data).
  • process that has the data must participate in the interaction,
  • for dynamic and/or unstructured interactions, the complexity of the code can be very high,
  • primary advantage of explicit two-way interactions is that the programmer is fully aware of all the costs of non-local interactions
  • more likely to think about algorithms (and mappings) that minimize interactions.

• The programmer is responsible for analyzing the underlying serial algorithm/application.
• Identifying ways by which he or she can decompose the computations and extract concurrency.
• As a result, programming using the message-passing paradigm tends to be hard and intellectually demanding.
• However, on the other hand, properly written message-passing programs can often achieve very high performance and scale to a very large number of processes.