Non-Blocking Communication Operations

In order to overlap communication with computation, MPI provides a pair of functions for performing non-blocking send and receive operations.
- MPI_Isend $\Longrightarrow$ starts a send operation but does not complete, that is, it returns before the data is copied out of the buffer.
- MPI_Irecv $\Longrightarrow$ starts a receive operation but returns before the data has been received and copied into the buffer.
With the support of appropriate hardware, the transmission and reception of messages can proceed concurrently with the computations.
At a later point in the program, a process that has started a non-blocking send or receive operation must make sure that this operation has completed before it proceeds with its computations.
This is because a process that has started a non-blocking send operation may want to
- overwrite the buffer that stores the data that are being sent,
- or a process that has started a non-blocking receive operation may want to use the data.

To check the completion of non-blocking send and receive operations, MPI provides a pair of functions
1. MPI_Test $\Longrightarrow$ tests whether or not a non-blocking operation has finished
2. MPI_Wait $\Longrightarrow$ waits (i.e., gets blocked) until a non-blocking operation actually finishes.
```
int MPI_Isend(void *buf, int count, MPI_Datatype datatype, 
    int dest, int tag, MPI_Comm comm, MPI_Request *request) 
int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, 
  int source, int tag, MPI_Comm comm, MPI_Request *request)
```
MPI_Isend and MPI_Irecv functions allocate a request object and return a pointer to it in the request variable.
This request object is used as an argument in the MPI_Test and MPI_Wait functions to identify the operation whose status we want to query or to wait for its completion.

The MPI_Irecv function does not take a status argument similar to the blocking receive function,

but the status information associated with the receive operation is returned by the MPI_Test and MPI_Wait functions.

int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status) 
int MPI_Wait(MPI_Request *request, MPI_Status *status)

MPI_Test tests whether or not the non-blocking send or receive operation identified by its request has finished.
It returns flag = true (non-zero value in C) if it is completed.
The request object pointed to by request is deallocated and request is set to MPI_REQUEST_NULL.
Also the status object is set to contain information about the operation.

It returns flag = false (a zero value in C) if it is not completed.
The request is not modified and the value of the status object is undefined.
The MPI_Wait function blocks until the non-blocking operation identified by request completes.
For the cases that the programmer wants to explicitly deallocate a request object, MPI provides the following function.
```
int MPI_Request_free(MPI_Request *request)
```
Note that the deallocation of the request object does not have any effect on the associated non-blocking send or receive operation.
That is, if it has not yet completed it will proceed until its completion.

Hence, one must be careful before explicitly deallocating a request object,
since without it, we cannot check whether or not the non-blocking operation has completed.
A non-blocking communication operation can be matched with a corresponding blocking operation.
For example, a process can send a message using a non-blocking send operation and this message can be received by the other process using a blocking receive operation.

Avoiding Deadlocks; by using non-blocking communication operations we can remove most of the deadlocks associated with their blocking counterparts.

For example, the following piece of code is not safe.

 1   int a[10], b[10], myrank; 
 2   MPI_Status status; 
 3   ... 
 4   MPI_Comm_rank(MPI_COMM_WORLD, &myrank); 
 5   if (myrank == 0) { 
 6     MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); 
 7     MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); 
 8   } 
 9   else if (myrank == 1) { 
10     MPI_Recv(b, 10, MPI_INT, 0, 2, &status, 
                                      MPI_COMM_WORLD); 
11     MPI_Recv(a, 10, MPI_INT, 0, 1, &status, 
                                      MPI_COMM_WORLD); 
12   } 
13   ...

However, if we replace either the send or receive operations with their non-blocking counterparts, then the code will be safe, and will correctly run on any MPI implementation.

Safe with non-blocking communication operations;

 1   int a[10], b[10], myrank; 
 2   MPI_Status status; 
 3   MPI_Request requests[2]; 
 4   ... 
 5   MPI_Comm_rank(MPI_COMM_WORLD, &myrank); 
 6   if (myrank == 0) { 
 7     MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); 
 8     MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); 
 9   } 
10   else if (myrank == 1) { 
11     MPI_Irecv(b, 10, MPI_INT, 0, 2, &requests[0],
                                      MPI_COMM_WORLD); 
12     MPI_Irecv(a, 10, MPI_INT, 0, 1, &requests[1], 
                                      MPI_COMM_WORLD); 
13   } //Non-Blocking Communication Operations
14   ...

This example also illustrates that the non-blocking operations started by any process can finish in any order depending on the transmission or reception of the corresponding messages.
For example, the second receive operation will finish before the first does.

Cem Ozdogan 2011-09-28