[SOLVED] CS CS402/922 High Performance Computing

MPI and Types
aka What if everything isnt together? https://warwick.ac.uk/fac/sci/dcs/teaching/material/cs402/ 01/02/2022 CS402/922 High Performance Computing

01/02/2022

Copyright By Assignmentchef assignmentchef

Recap on Message Passing Interface (MPI)
In case it got lost in the post!
MPI sends data from one processor to anothera MPI_Send and MPI_Recv
Base version is blocking (function waits for a response)
Required info includes:
Number of elements of data
The type of data to be sent
Source/Destination processor
Communicator (the group of processors that could be accessed) Tag (to distinguish between different messages)
A status message (for MPI_Recv only)

01/02/2022
Different modes for MPI_Send and MPI_Recv Plenty of options to choose from
Standard version are blocking functions
We can have non-blocking versions
The way in which data is sent and processed can be altered:
Buffered
Synchronous Ready
(Standard)
Synchronous
Blocking (Send)
Blocking (Recv)
Non-blocking (Send)
MPI_IBsend
MPI_ISsend
MPI_IRsend
Non-blocking (Recv)
MPI_IBrecv
MPI_ISrecv
MPI_IRrecv

01/02/2022
Whoops, wrong typing
MPI messages need to be common between different
processors
They may require different sizes, so need a unified interface
Some are only relevant to:
FORTRANaMPI_COMPLEX,MPI_LOGICAL
CaMPI_C_BOOL,MPI_INT64_T
C++aMPI_CXX_BOOL,MPI_CXX_DOUBLE_COMPLEX
Many more that can be used, make sure to use the right type!
MPI Spec (Data TypesaPage 34)
https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf

01/02/2022
What MPI_Send is actually doing
I write, I compile, I run, thats it!
Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)

01/02/2022
What MPI_Send is actually doing
I write, I compile, I run, thats it!
Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)

01/02/2022
What MPI_Send is actually doing
I write, I compile, I run, thats it!
Say we wanted to send 4 floats from processor 0 to 1
float * data = {1.0f, 2.0f, 3.0f, 4.0f}; MPI_Send(data, 4, MPI_FLOAT, 1, 0, MPI_COMM_WORLD);
(MPI Envelope)

01/02/2022
Non-contiguous memory
In case it got lost in the post!
Data shown in the previous example is con$guous All the data is uniform with no gaps
SomeOmes our data is not in conOguous memory (An array of) struct or custom classes
Block of memory manually addressed
If we ran the same thing, you would get garbled data
16 bits per float 4 bits per char

01/02/2022
Dealing with Non-contiguous memory
Not all data is the same size!
3 different options to deal with this:
1. Call a MPI_Send for each data block occupying contiguous blocks Lots of messages means lots of congestion
Lots of messages means lots of confusion if not managed well
Lots of messages means lots of overhead for sending multiple messages
2. Copy all the data into a contiguous block, and send that
Better than the first option as one message is being sent
Lots of time needs to be spent copying memory
Not great if dealing with large amounts of data, where memory is limited
3. Derived datatypes

01/02/2022
Derived Datatypes
Thats the thing from the previous slide; he said the thing!
We can generate our own datatypes
Useful as we can send a single message with different types of data
and/or non-contiguous memory
This can improve performance (but is not guaranteed)
Derived types need to be committed (using MPI_Type_commit)
5 key derived types
MPI_Type_contiguous MPI_Type_vector MPI_Type_create_indexed_block MPI_Type_indexed MPI_Type_create_struct
More complex

01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
Useful to be specify a collection of elements with a variable amount of counts
Allows for better reuse and less calculations
int MPI_Type_contiguous(int count, MPI_Datatype oldtype, MPI_Datatype *newtype)
Returns 0 if success, otherwise error number
Number of The old data type The new data type blocks
(not negative)

01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
int MPI_Type_contiguous(int c3ount, MPI_DaMtPaIt_yFpLeOAoTldtype, MPI&_fDlaotaattCyopneti*gnueowutsype)
Returns 0 if success, otherwise error number
Number of The old data type The new data type blocks
(not negative)

01/02/2022
MPI_Type_contiguous
A bit like MPI_Type++!
MPI_Datatype floatContiguous;
MPI_Type_contiguous(3, MPI_FLOAT, &floatContiguous); MPI_Type_commit(&floatContiguous);
/* floatContiguous =
{(MPI_FLOAT, 0), (MPI_FLOAT, 16), (MPI_FLOAT, 32)} */
MPI_Type_free(&floatContiguous);

01/02/2022
MPI_Type_vector
A vector called Victor!
Consecutive blocks of contiguous data
Each block between contiguous blocks need to be equally spaced,
the same size, and all the same type
int MPI_Type_vector(int count, int blocklength, int stride, MPI_Datatype oldtype, MPI_Datatype *newtype)
Number of blocks (not negative)
Number of elements in each block
(not negative)
Number The old data type The new data type of
elements between blocks

01/02/2022 Slide 15
MPI_Type_vector
A vector called Victor!
int MPI_Type_vector(int 3count, int blo2cklength, int s4tride, MPI_DaMtPaIt_yFpLeOAoTldtype, MPI_D&aftlaotaytpVeec*tnoerwtype)
Number of blocks (not negative)
Number of elements in each block
(not negative)
Number of elements between blocks
The old data type
The new data type
blocklength

01/02/2022
MPI_Type_vector
A vector called Victor!
MPI_Datatype floatVector;
MPI_Type_vector(3, 2, 4, MPI_FLOAT, &floatVector); MPI_Type_commit(&floatVector);
/* floatVector =
{(MPI_FLOAT, 0), (MPI_FLOAT, 16),
(MPI_FLOAT, 64),(MPI_FLOAT, 80),
(MPI_FLOAT, 128), (MPI_FLOAT, 144)} */
MPI_Type_free(&floatVector);

01/02/2022
MPI_Type_create_indexed_blockA block called Blocky!
Sometimes the data doesnt start at the beginning, or might have
multiple offsets
Requires a the same number of elements in each block, and them all to be the same type
int MPI_Type_create_indexed_block(int count, int blocklength, const int offsets[], MPI_Datatype oldtype, MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of offsets)
Number of elements in each block
(not negative)
Array of offsets, in mulXples of oldtype
The old data type

01/02/2022 Slide 18
MPI_Type_create_indexed_block
A block called Blocky!
int MPI_Type_create_indexed_block(int c3ount, int bloc2klength, const{1i,nt5,of1f0s}ets[], MPI_DaMtPaIt_yFpLeOAoTldtype, MPI_Datatype *newtype);
&floatCreateIndexBlock
The new data type
Number of blocks (not negative, size of offsets)
Number of elements in each block
(not negative)
Array of offsets, in multiples of oldtype
The old data type
blocklength

01/02/2022
MPI_Type_create_indexed_blockA block called Blocky!
MPI_Datatype floatCreateIndexBlock;
MPI_Type_create_indexed_block(3, 2, {1, 5, 10}, MPI_FLOAT, &floatCreateIndexBlock); MPI_Type_commit(&floatCreateIndexBlock);
/* floatCreateIndexBlock =
{(MPI_FLOAT, 16),(MPI_FLOAT, 32),
(MPI_FLOAT, 80),(MPI_FLOAT, 96),
(MPI_FLOAT, 160), (MPI_FLOAT, 176)} */
MPI_Type_free(&floatCreateIndexBlock);

01/02/2022 Slide 20
MPI_Type_indexed
An index called Indexy?
Sometimes the data blocks arent the same size
Requires all to be the elements to be the same type
int MPI_Type_indexed(int count, const int blocklengths[], const int offsets[], MPI_Datatype oldtype, MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of blocklengths and offsets)
Array of block lengths (all elements cannot be negative)
Array of offsets, in mulXples of oldtype
The old data type

01/02/2022 Slide 21
MPI_Type_indexed
An index called Indexy?
3 {3, 1, 2} {1, 6, 10} MPI_FLOAT
int MPI_Type_indexed(int count, const int blocklengths[], const int offsets[], MPI_Datatype oldtype,
MPI_Datatype *newtype); &floatIndexed
Number of blocks (not negative, size of blocklengths and offsets)
Array of block lengths (all elements cannot be negative)
Array of offsets, in multiples of oldtype
The old data type
The new data type
blocklength[0]
blocklength[1] blocklength[2]

01/02/2022
MPI_Type_indexed
An index called Indexy?
MPI_Datatype floatIndexed;
MPI_Type_indexed(3, {3, 1, 2}, {1, 6, 10}, MPI_FLOAT, &floatIndexed); MPI_Type_commit(&floatIndexed);
/* floatIndexed =
{(MPI_FLOAT, 16), (MPI_FLOAT, 32), (MPI_FLOAT, 48), (MPI_FLOAT, 96),
(MPI_FLOAT, 160), (MPI_FLOAT, 176)} */
MPI_Type_free(&floatIndexed);

01/02/2022
MPI_Type_create_struct
An struct called Structy?
Sometimes even the blocks are different types
Requires the elements within each individual block need to be
the same type
int MPI_Type_create_struct(int count, const int blocklengths[], const int offsets[], const MPI_Datatype oldtypes[], MPI_Datatype *newtype);
The new data type
Number of blocks (not negative, size of blocklengths , offsets and oldtypes)
Array of block lengths (all elements cannot be negative)
Array of offsets,
Array of data types for each block

oldtypes[]
01/02/2022 Slide 24
MPI_Type_create_struct
An struct called Structy?
int MPI_Type_create_struct(int 3count, const in{t3,bl1o,ck2l}engths[], cons{t4,in1t8,of3f1s}ets[], c{oMnPsIt_FMLPOIA_TD,atMaPtIy_pCeHAR, , MPI_Datatype *newtype);
The new data type
Number of blocks (not negaXve, size of blocklengths , offsets and oldtypes)
Array of block lengths (all elements cannot be negative)
Array of offsets,
Array of data types for each block
oldTypes[0]=MPI_FLOAT oldTypes[1]=MPI_CHAR oldTypes[2]=MPI_INT
&structType
blocklength[0]
blocklength[1] blocklength[2]

01/02/2022
MPI_Type_create_struct
An struct called Structy?
MPI_Datatype structType;
MPI_Type_create_struct (3, {3, 1, 2}, {4, 18, 31}, {MPI_FLOAT, MPI_CHAR, MPI_INT}, &structType); MPI_Type_commit(&structType);
/* structType =
{(MPI_FLOAT, 16), (MPI_FLOAT, 32), (MPI_FLOAT, 48), (MPI_CHAR, 66),
(MPI_INT, 83), (MPI_INT, 91)} */
MPI_Type_free(&structType);

01/02/2022 Slide 26
Overview of Complex Derived Datatypes
That that was too much info!
Same type in each block
Same number of elements in each block
Same distance between consecutive block
MPI_Type_vector
MPI_Type_create_indexed_block
MPI_Type_indexed
MPI_Type_create_struct

01/02/2022
Interesting related reads
Some of this might even be fun
Zero (cost) copy with MPI Derived Datatypes
T. Hoefler and S. Gottlieb. Parallel Zero-Copy Algorithms for Fast Fourier Transform and Conjugate Gradient Using MPI Datatypes. In
Recent Advances in the Message Passing Interface, pages 132141, 2010, Springer, Berlin, Heidelberg MPI Derived Datatypes performance improvements
W. Gropp, E. Lusk and D. Swider. Improving the Performance of MPI Derived Datatypes. In Proceedings of the Third MPI Developers and Users Conference, pages 2530, March 1999, Springer, Berlin, Heidelberg
J. L. Traff, R. Hempel, H. Ritzdorf and F. Zimmermann. Flattening on the Fly: Efficient Handelling of MPI Derived Datatypes. In European Parallel Virtual Machine / Message Passing Interface Users Group Meeting (EuroPVM/MPI 1999): Recent Advances in Parallel Virtual Machine and Message Passing Interface, pages 109116, September 1999, Springer, Berlin, Heidelberg
Next lecture: Advanced MPI

CS: assignmentchef QQ: 1823890830 Email: [email protected]