Non-Blocking Communication Operations

• In order to overlap communication with computation, MPI provides a pair of functions for performing non-blocking send and receive operations.
  • MPI_Isend $\Longrightarrow$ starts a send operation but does not complete, that is, it returns before the data is copied out of the buffer.
  • MPI_Irecv $\Longrightarrow$ starts a receive operation but returns before the data has been received and copied into the buffer.

    int MPI_Isend(void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm, MPI_Request *request) 
    int MPI_Irecv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Request *request)
    

• MPI_Isend and MPI_Irecv functions allocate a request object and return a pointer to it in the request variable.
• At a later point in the program, a process that has started a non-blocking send or receive operation must make sure that this operation has completed before it proceeds with its computations.

• This is because a process that has started a non-blocking send operation may want to
  • overwrite the buffer that stores the data that are being sent,
  • or a process that has started a non-blocking receive operation may want to use the data.
• To check the completion of non-blocking send and receive operations, MPI provides a pair of functions
  1. MPI_Test $\Longrightarrow$ tests whether or not a non-blocking operation has finished
  2. MPI_Wait $\Longrightarrow$ waits (i.e., gets blocked) until a non-blocking operation actually finishes.

     int MPI_Test(MPI_Request *request, int *flag, MPI_Status *status) 
     int MPI_Wait(MPI_Request *request, MPI_Status *status)
     

• The request object is used as an argument in the MPI_Test and MPI_Wait functions to identify the operation whose status we want to query or to wait for its completion.

MPI_Test tests whether or not the non-blocking send or receive operation identified by its request has finished.

True

It returns flag = true (non-zero value in C) if it is completed. The request object pointed to by request is deallocated and request is set to MPI_REQUEST_NULL. Also the status object is set to contain information about the operation.

False

It returns flag = false (a zero value in C) if it is not completed. The request is not modified and the value of the status object is undefined. The MPI_Wait function blocks until the non-blocking operation identified by request completes. A non-blocking communication operation can be matched with a corresponding blocking operation. For example, a process can send a message using a non-blocking send operation and this message can be received by the other process using a blocking receive operation.

• Avoiding Deadlocks; by using non-blocking communication operations we can remove most of the deadlocks associated with their blocking counterparts.
• For example, the following piece of code is not safe.

      1 int a[10], b[10], myrank; 
      2 MPI_Status status; 
      3  ... 
      4 MPI_Comm_rank(MPI_COMM_WORLD, &myrank); 
      5 if (myrank == 0) { 
      6   MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); 
      7   MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); 
      8 } 
      9 else if (myrank == 1) { 
     10   MPI_Recv(b, 10, MPI_INT, 0, 2, &status, MPI_COMM_WORLD); 
     11   MPI_Recv(a, 10, MPI_INT, 0, 1, &status, MPI_COMM_WORLD); 
     12 } 
     13 ...
  

• However, if we replace either the send or receive operations with their non-blocking counterparts, then the code will be safe, and will correctly run on any MPI implementation.

• Safe with non-blocking communication operations;

      1 int a[10], b[10], myrank; 
      2 MPI_Status status; 
      3 MPI_Request requests[2]; 
      4 ... 
      5 MPI_Comm_rank(MPI_COMM_WORLD, &myrank); 
      6 if (myrank == 0) { 
      7   MPI_Send(a, 10, MPI_INT, 1, 1, MPI_COMM_WORLD); 
      8   MPI_Send(b, 10, MPI_INT, 1, 2, MPI_COMM_WORLD); 
      9 } 
     10 else if (myrank == 1) { 
     11   MPI_Irecv(b, 10, MPI_INT, 0, 2, &requests[0], MPI_COMM_WORLD); 
     12   MPI_Irecv(a, 10, MPI_INT, 0, 1, &requests[1], MPI_COMM_WORLD); 
     13 } //Non-Blocking Communication Operations
     14 ...
  

• This example also illustrates that the non-blocking operations started by any process can finish in any order depending on the transmission or reception of the corresponding messages.
• For example, the second receive operation will finish before the first does.