MPI comes with a built-in interface for performing measurements and other actions during MPI calls. Here’s a quick tutorial on how it works:

Create a file called wrap.c:

#include <mpi.h>
#include <stdio.h>
double start, end;
int ret;

int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) {
   start = MPI_Wtime();
   ret = PMPI_Send(buf, count, datatype, dest, tag, comm);
   end = MPI_Wtime();
   printf("MPI_Send took %f seconds.\n", (end - start));
   return ret;
}

This function will determine how long it takes to perform MPI_Send and print it out. Each rank will perform this operation independently, so if you run this on even a few ranks, it’s likely the printf messages will clobber each other, but we’ll ignore that for our purposes.

You can either compile your wrapper function with your application, or you can create a shared library that is called dynamically using LD_PRELOAD. Let’s do the latter:

$ mpicc -c  wrap.c -fPIC
$ gcc -shared -o libwrapmpi.so wrap.o

To test our code, let’s create a simple program: mpi_hello_world.c

#include <mpi.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char ** argv) {
   int rank;
   char *msg = "Hello World!";
   char buffer[strlen(msg)];

   MPI_Init(NULL,NULL);
   MPI_Comm_rank(MPI_COMM_WORLD, &rank);
   if (rank == 0) {
      MPI_Send((void *)msg, strlen(msg), MPI_CHAR, 1, 0, MPI_COMM_WORLD);
      printf("Rank 0 sent [%s] to rank 1\n", msg);
   }
   else if (rank == 1) {
      MPI_Recv((void *) &buffer, strlen(msg), MPI_CHAR, 0, MPI_ANY_TAG, MPI_COMM_WORLD, NULL);
      printf("Rank 1 received [%s] from rank 0\n", buffer);
   else {
      printf("Rank [%d] did nothing.\n", rank);
   }
   MPI_Finalize();
   return 0;
}

We compile it without explicitly linking our library:

$ mpicc -o hello mpi_hello_world.c

First, let’s run it without our wrapper to make sure that it works by itself:

mpirun -n 4 ./hello
Rank [2] did nothing.
Rank 0 sent [Hello World!] to rank 1
Rank [3] did nothing.
Rank 1 received [Hello World!] from rank 0

To add the wrapper dynamically, use LD_PRELOAD:

export LD_PRELOAD=<path/to/lib>/libwrapmpi.so

And, finally, run our program:

$ mpirun -n 4 ./hello
MPI_Send took 0.000033 seconds.
Rank 0 sent [Hello World!] to rank 1
Rank 1 received [Hello World!] from rank 0
Rank [2] did nothing.
Rank [3] did nothing.

Almost all of the MPI functions can be wrapped in this fashion. Just construct a wrapper for the function you want to wrap, add any other code you want to perform, and return the result to the PMPI equivalent of the function.