Sending and Receiving Messages

• The basic functions for sending and receiving messages in MPI are the MPI_Send and MPI_Recv, respectively. The calling sequences of these routines are as follows:

  int MPI_Send(void *buf, int count, MPI_Datatype datatype, 
          int dest, int tag, MPI_Comm comm) 
  int MPI_Recv(void *buf, int count, MPI_Datatype datatype, 
          int source, int tag, MPI_Comm comm, MPI_Status *status)
  

• MPI_Send sends the data stored in the buffer pointed by buf. This buffer consists of consecutive entries of the type specified by the parameter datatype.
• The number of entries in the buffer is given by the parameter count.
• The correspondence between MPI datatypes and those provided by C is shown in Table 7.2. Note that for all C datatypes, an equivalent MPI datatype is provided. However, MPI allows two additional datatypes that are not part of the C language. These are MPI_BYTE and MPI_PACKED.
  • MPI_BYTE corresponds to a byte (8 bits) and MPI_PACKED corresponds to a collection of data items that has been created by packing non-contiguous data.
  • Note that the length of the message in MPI_Send, as well as in other MPI routines, is specified in terms of the number of entries being sent and not in terms of the number of bytes.
  • Specifying the length in terms of the number of entries has the advantage of making the MPI code portable, since the number of bytes used to store various datatypes can be different for different architectures.
  
  

  Table 7.2: Correspondence between the datatypes supported by MPI and those supported by C.

  MPI Datatype	C Datatype
  MPI_CHAR	signed char
  MPI_SHORT	signed short int
  MPI_INT	signed int
  MPI_LONG	signed long int
  MPI_UNSIGNED_CHAR	unsigned char
  MPI_UNSIGNED_SHORT	unsigned short int
  MPI_UNSIGNED	unsigned int
  MPI_UNSIGNED_LONG	unsigned long int
  MPI_FLOAT	float
  MPI_DOUBLE	double
  MPI_LONG_DOUBLE	long double
  MPI_BYTE
  MPI_PACKED

  
  
• The destination of the message sent by MPI_Send is uniquely specified by the dest and comm arguments.
• The dest argument is the rank of the destination process in the communication domain specified by the communicator comm.
• Each message has an integer-valued tag associated with it. This is used to distinguish different types of messages. The message-tag can take values ranging from zero up to the MPI defined constant MPI_TAG_UB. Even though the value of MPI_TAG_UB is implementation specific, it is at least 32,767.
• MPI_Recv receives a message sent by a process whose rank is given by the source in the communication domain specified by the comm argument.
• The tag of the sent message must be that specified by the tag argument. If there are many messages with identical tag from the same process, then any one of these messages is received.
• MPI allows specification of wildcard arguments for both source and tag.
  • If source is set to MPI_ANY_SOURCE, then any process of the communication domain can be the source of the message.
  • Similarly, if tag is set to MPI_ANY_TAG, then messages with any tag are accepted.
• The received message is stored in continuous locations in the buffer pointed to by buf. The count and datatype arguments of MPI_Recv are used to specify the length of the supplied buffer. The received message should be of length equal to or less than this length.
• This allows the receiving process to not know the exact size of the message being sent. If the received message is larger than the supplied buffer, then an overflow error will occur, and the routine will return the error MPI_ERR_TRUNCATE.
• After a message has been received, the status variable can be used to get information about the MPI_Recv operation. In C, status is stored using the MPI_Status data-structure. This is implemented as a structure with three fields, as follows:

  typedef struct MPI_Status { 
    int MPI_SOURCE; 
    int MPI_TAG; 
    int MPI_ERROR; 
  };
  

• MPI_SOURCE and MPI_TAG store the source and the tag of the received message. They are particularly useful when MPI_ANY_SOURCE and MPI_ANY_TAG are used for the source and tag arguments. MPI_ERROR stores the error-code of the received message.
• The status argument also returns information about the length of the received message. This information is not directly accessible from the status variable, but it can be retrieved by calling the MPI_Get_count function. The calling sequence of this function is as follows:

  int MPI_Get_count(MPI_Status *status, MPI_Datatype datatype, 
          int *count)
  

  MPI_Get_count takes as arguments the status returned by MPI_Recv and the type of the received data in datatype, and returns the number of entries that were actually received in the count variable.
• The MPI_Recv returns only after the requested message has been received and copied into the buffer. That is, MPI_Recv is a blocking receive operation.
• However, MPI allows two different implementations for MPI_Send.
  1. MPI_Send returns only after the corresponding MPI_Recv have been issued and the message has been sent to the receiver.
  2. MPI_Send first copies the message into a buffer and then returns, without waiting for the corresponding MPI_Recv to be executed.
• In either implementation, the buffer that is pointed by the buf argument of MPI_Send can be safely reused and overwritten.
• MPI programs must be able to run correctly regardless of which of the two methods is used for implementing MPI_Send. Such programs are called safe.
• In writing safe MPI programs, sometimes it is helpful to forget about the alternate implementation of MPI_Send and just think of it as being a blocking send operation.