lectures

LECTURES

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
Brian Kernighan

One can develop parallel programs with MPI either in Fortran or C/C++ and launch them on clusters or supercomputers under Linux or AIX. Because scientific computations usually have very complicated logical structure and their debugging is typically very demanding, what turns out to be essentially helpful is an Integrated Development Environment (IDE). Because of this, all sample programs of my lectures can be debugged in IDE of Visual Studio and then can be copied to a cluster or a supercomputer without any changes and launched under Windows, Linux, macOS, or AIX. It is achieved with the help of blocks that are bordered by preprocessor directives #ifdef and #endif in my codes. Nevertheless, beginners can ignore these blocks during the first reading.

See a guide for installation of C++ in the paragraph I3 for Linux and in I4 for macOS in MPI section below.

I1. Instalation of Visual Studio in Windows: an installation file of free Visual Studio Community 2022 can be downloaded here. After its installation, open Visual Studio, select Create a new project ➜ C++ Console App, push Next, and select Project name, e.g., hello_world_simplest, and Location. Then you can: 1) run a default Hello World program provided by Visual Studio pushing a green arrow, 2) copy the code of my project hello_world_simplest (PDF) to hello_world_simplest.cpp and run it, or 3) download my project hello_world_simplest (Visual_Studio), unzip it, open Visual Studio, push Open a project or solution, locate and double click hello_world_simplest.sln, unfold Source Files in Solution Explorer on the left, double click on hello_world_simplest.cpp in Solution Explorer, and push the button with a green arrow. Alternatively, you can select Build ➜ Build Solution or push F7 to compile the program. If there were errors or warnings in your program, they would be listed in Errors resp. Warnings at the bottom of your IDE. You can double click on them to locate their positions in your program or double click on a code in front of them to see a help. If you get into problems during debugging, don't forget to step through your program line by line and observe the current values of your program's variables. When you correct all errors, you can compile and run the program.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The basics of C/C++

Recommended textbooks: Herout, P.: Učebnice jazyka C, 1. díl (buy) or Deitel, P. J. & Deitel, H. M., 2012. C: How to Program (buy).

Hello world simplest runs only in Visual Studio. If you wanted to make it running under various operating systems, it would be sufficient to use preprocessor directives as in the next sample.

hello_world_basics (Visual Studio, Linux&Mac, PDF):
You can compile this code in Linux by
g++ hello_world_basics.cpp -o hello_world_basics
and launch it by
./hello_world_basics

Hereinafter, the most important samples are emphasized by !!! and the most demanding ones by *, ** or ***. The samples indicated by + only comprise programs that can appear to be useful sometime in the future: almost all of them will be skipped during the lecture.

Basic types and operators: Data_type_ranges, Data_type_constants, Standard_types, and Operators in C and C++.

Sample programs (flowcharts, Visual Studio, Linux&Mac):

1) Control structures: loops for (flowchart), while (flowchart), and do-while (flowchart):
- A) (if, if-else): if_quadratic_equation.cpp (cout, cin, iostream) (Visual Studio, Linux&Mac, PDF, flowchart)
- B) for_cycle_max.cpp (Visual Studio, Linux&Mac, PDF, bad_flowchart, flowchart)
- C) do_while_max.cpp (Visual Studio, Linux&Mac, PDF, flowchart)
- D) do_while_prime_number.cpp (modulo operators) (Visual Studio, Linux&Mac, PDF)
- E) control_structures.cpp (Boolean and conditional expressions, comma operator, and statements break, goto, continue, switch, and return) (Visual Studio, Linux&Mac, PDF)
2) Fields and the sorting of a field:
- A) strings_strcpy.cpp (strcpy, strcat) (Visual Studio, Linux&Mac, PDF)
- B) while_field.cpp (switch, printf, strings) (Visual Studio, Linux&Mac, PDF, flowchart)
- C) sorting_of_field.cpp (buble method) (Visual Studio, Linux&Mac, PDF, flowchart)
- D) sorting_strings (Visual Studio, PDF)
3) Matrices and arithmetics of pointers:
- A) multiply_matrices.cpp (Visual Studio, Linux&Mac, PDF, flowchart)
4) Input-output operations:
- A) matrices_txt_ansi.cpp (fopen, fscanf, fprintf, fclose) (Visual Studio, Linux&Mac, PDF, MATICA.TXT)
- B)* matrices_txt_stream.cpp (fstream.h) (Visual Studio, Linux&Mac, PDF, MATICA.TXT)
5) Functions:
- A)!!! parameter_called_by_value.cpp (Visual Studio, Linux&Mac, PDF)
- B)!!! parameter_called_by_reference.cpp (Visual Studio, Linux&Mac, PDF)
- C)!!! parameter_called_by_dereference.cpp (Visual Studio, Linux&Mac, PDF)
- D)!!! fields_as_parameters.cpp (Visual Studio, Linux&Mac, PDF)
- E) exponentiate_matrix.cpp (Visual Studio, Linux&Mac, PDF)
6) The dynamic allocation of memory (2 × pdf):
- A) sum_of_fields_new.cpp (new, delete) (Visual Studio, Linux&Mac, PDF)
- B) dynamic_alocation_of_memory_malloc.cpp (malloc, free) (Visual Studio, Linux&Mac, PDF)
- C)!!! dynamic_alocation_of_memory_new.cpp (new, delete) (Visual Studio, Linux&Mac, PDF, diagram)
- D) dynamic_alocation_of_tensor_new.cpp (new, delete) (Visual Studio, Linux&Mac, PDF)
7) Strings and classes:
- A) hello_world_classes.cpp (Visual Studio, Linux&Mac, PDF (hello_class.h, hello_class.cpp, hello_world_classes.cpp ))
- B) int_to_string (Visual Studio, Linux&Mac, PDF )
- C) process_text_ansi.cpp (Visual Studio, Linux&Mac, PDF , VETA.TXT)
- D) process_text_stream.cpp (Visual Studio, Linux&Mac, PDF (Veta.h, Veta.cpp, process_text_stream.cpp, VETY.TXT))

hello_world_classes: Because the sample 7A consists of 3 files - hello_world_classes.cpp, hello_class.cpp, and hello_class.h - you have to compile it in Linux by
g++ hello_class.cpp hello_world_classes.cpp -o hello_world_classes
and launch it by
./hello_world_classes

Useful tables: Keywords.

Object-oriented programming in C++

Recommended textbooks: Eckel, B.: Thinking in C++, Volume 1 (download TICPP-2nd-ed-Vol-one-html.zip from https://archive.org/details/TICPP2ndEdVolOne). The sample programs mentioned below whose names begin with C are taken from this textbook. For example, C11_HowMany.cpp means that the attendees can double click on Contents.html, select Chapter 11, and find HowMany.cpp to see its code and further commentaries in case of a need.

Sample programs (Visual Studio, Linux&Mac):

1) Classes:
- A) complex_numbers.cpp (Visual Studio, Linux&Mac, PDF)
- The following sample is the same as the preceding one, but a class TCisloKomplexne is transposed to a separate module of files (i.e., a module of CisloKomplexne.h and CisloKomplexne.cpp):

B)!!! complex_numbers_classes.cpp (Visual Studio, Linux&Mac, PDF (CisloKomplexne.h, CisloKomplexne.cpp, complex_numbers_classes.cpp))
The following sample is almost the same as the sample 5E of C/C++ for beginners, but most of the functions is nested into a class TMatica:
C)!!! exponentiate_matrix_classes.cpp (Visual Studio, Linux&Mac, PDF (Matica.h, Matica.cpp, exponentiate_matrix_classes.cpp, MATICA.TXT))

2) Data structures (linked lists):
- A)!!** linked_list_simple.cpp (Visual Studio, Linux&Mac, PDF (LinkedList1.h, LinkedList1.cpp, linked_list_simple.cpp))
- B)!!** linked_list.cpp (Visual Studio, Linux&Mac, PDF (LinkedList2.h, LinkedList2.cpp, linked_list.cpp), and diagrams (insertAtFront, insertAtBack, removeFromFront, removeFromBack, sortElements))
3) Inheritance:
- A) Why should we have the constructors, destructors, copy constructors, and the operator = under control? A wrong sample: C11_HowMany.cpp
- B) A corrected sample: C11_HowMany2.cpp

C)!!! inheritance_simple.cpp (Visual Studio, Linux&Mac, PDF (MaticaObdlznikova.h, MaticaObdlznikova.cpp, MaticaStvorcova.h, MaticaStvorcova.cpp, inheritance_simple.cpp, MATICA.TXT), and diagrams (inheritance_diagram))
D)!!!* inheritance_multiple.cpp (Visual Studio, Linux&Mac, PDF (MaticaObdlznikova.h, MaticaObdlznikova.cpp, MaticaStvorcova.h, MaticaStvorcova.cpp, CisloKomplexne.h, CisloKomplexne.cpp, MaticaStvorcovaKomplexna.h, MaticaStvorcovaKomplexna.cpp, inheritance_multiple.cpp, MATICE.TXT), and a diagram (inheritance_diagram))
E)! C14_Combined.cpp
F)! C14_Order.cpp
G) C14_NameHiding.cpp
H)+ C14_SynthesizedFunctions.cpp
I) C14_Car.cpp
J)! C14_PrivateInheritance.cpp
K)! C14_Protected.cpp
L) C14_CopyConstructor.cpp

4) Overloading:
- A) functions_overloading.cpp (Visual Studio, Linux&Mac, PDF)
- B)!!!** inheritance_multiple_new.cpp (Visual Studio, Linux&Mac, PDF (MaticaObdlznikova.h, MaticaObdlznikova.cpp, MaticaStvorcova.h, MaticaStvorcova.cpp, CisloKomplexne.h, CisloKomplexne.cpp, MaticaStvorcovaKomplexna.h, MaticaStvorcovaKomplexna.cpp, inheritance_multiple_new.cpp, MATICE.TXT), and a diagram (inheritance_diagram))
- C) C12_OperatorOverloadingSyntax.cpp
- D)+++ complex_numbers.cpp (Visual Studio, Linux&Mac, PDF)
- E)+++ prefix_and_postfix.cpp (Visual Studio, Linux&Mac, PDF)
- F)+++ random_generator.cpp (Visual Studio, Linux&Mac, PDF)
- G)+++ field1.cpp (Visual Studio, Linux&Mac, PDF)
- H)+++ C12_ExplicitKeyword.cpp
- I)+++ arrow.cpp (Visual Studio, Linux&Mac, PDF)
- J)+++ field2.cpp (Visual Studio, Linux&Mac, PDF)
- K)+++ casting.cpp (Visual Studio, Linux&Mac, PDF)
- L)+++ new1.cpp (Visual Studio, Linux&Mac, PDF)
- M)+++ new2.cpp (Visual Studio, Linux&Mac, PDF)
5) Polymorphism:
- A)!!! C15_Instrument2.cpp
- B)!!! C15_Instrument3.cpp
- C)!!! C15_Instrument4.cpp
- D)+ C15_Early.cpp
- E)+ C15_ObjectSlicing.cpp
- F) C15_PureVirtualDefinitions.cpp
- G) C15_AddingVirtuals.cpp
- H) C15_VirtualDestructors.cpp
- I)+ C15_DynamicCast.cpp
- J)+ C15_StaticHierarchyNavigation.cpp
- K)+ C15_OperatorPolymorphism.cpp
6) Containers are useful for parallel programming. You can first gather intermediate results, which are consecutively calculated on a given slave process, into a container and when they get all completed, you can perform some operations on them, e.g., sort them, select an appropriate number of the best ones, and throw the rest out, and, finally, you can send the entire container from a given slave process to the master process. Containers and templates:
- A) C16_IntStack.cpp
- B) C16_Array.cpp
- C) C16_Array2.cpp
- D)** C16_StackTemplateTest.cpp
- E)** C16_TStackTest.cpp
- F)** C06_Stash2Test.cpp
- G)** C16_IterIntStack.cpp
- H)** C16_NestedIterator.cpp
7) Containers and iterators:
- A)*** C16_TStack2Test.cpp
- B)*** C16_TPStash2Test.cpp
- C)!!!*** C16_Drawing.cpp

Useful tables: Inheritance and Overloading.

MPI for C/C++

Recommended textbooks: Quinn, M.J.: Parallel Programming in C with MPI and OpenMP (buy).

I2. Instalation of MS MPI in Windows:

The installation files msmpisetup.exe and msmpisdk.msi of free MS MPI can be downloaded here. It is necessary to restart your computer after the installation of MS MPI. Then you can test its functionality in a Command Promt. Write (Settings for Visual Studio):

C:\Windows\System32> set MSMPI

If the echo of the system is

MSMPI_BENCHMARKS=C:\Program Files\Microsoft MPI\Benchmarks\
MSMPI_BIN=C:\Program Files\Microsoft MPI\Bin\
MSMPI_INC=C:\Program Files (x86)\Microsoft SDKs\MPI\Include\
MSMPI_LIB32=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x86\
MSMPI_LIB64=C:\Program Files (x86)\Microsoft SDKs\MPI\Lib\x64\

C:\Windows\System32>

it means that your MS MPI has been installed successfully. Then:

1. Open Visual Studio, click on Create a new project, write C++ instead of Search for templates, select Console App, and push Next. Let Project name is, e.g., myMPI. Select Location and push Create.
2. Do not forget to specify Solution Platform, e.g., x64 in a selector list of a small upper left window.
3. In the main menu, open Project ➜ myMPI Properties ➜ C/C++. In the upper central edit memo named Platform, set your operating system, e.g., Active(x64), and, depending on it, add $(MSMPI_INC);$(MSMPI_INC)\x64 or $(MSMPI_INC);$(MSMPI_INC)\x86 into Additional Include Directories.
4. Open Project ➜ myMPI Properties ➜ Linker ➜ Input and add msmpi.lib into Additional Dependencies.
5. Open Project ➜ myMPI Properties ➜ Linker ➜ General and, depending on your operating system, add $(MSMPI_LIB64) or $(MSMPI_LIB32) into Additional Library Directories.

Afterwards, you can copy the code from a PDF hyperlink of 1A_Hello_world.cpp to your myMPI. Try compiling it. If the compiler has announced errors for mpi.h, MPI_Init, etc., your MPI has not started to collaborate with Visual Studio yet, and so try to wait to the next day and repeat the compilation. Having installed MS MPI, you can parallelly launch 1A_Hello_world as follows:

1. Download a Visual Studio project 1A_Hello_world. Let a full path to the directory where you have saved this project be C:\path\1A_Hello_world.

2. Double click on a file C:\path\1A_Hello_world\1A_Hello_world.sln.
3. Open Developer Command Prompt for VS 2022, but ordinary Command Prompt would also work. Write a command:

C:\> cd C:\path\1A_Hello_world\x64\Debug

4. Write a command:

C:\path\1A_Hello_world\x64\Debug> mpiexec -n 2 1A_Hello_world.exe

provided that you have a 64-bit computer and you want to launch 1A_Hello_world.exe parallelly on 2 cores. The rest of the sample programs can be launched in analogy, but you can set -n higher than 2 starting from 2C_Sum_of_Numbers.

I3. Instalation of gcc and Open MPI in Linux:

1. Provided that you are working on a personal computer, not on a Linux server, check if you have gcc among your installed programs:

~$ gcc --version

If you do not have gcc installed yet, write:

~$ sudo apt update

~$ sudo apt install build-essential

Check if you have mpi installed:

~$ mpic++ --version

If you do not it yet, write:

~$ sudo apt-get update -y

~$ sudo apt-get install -y mpi

2. Provided that you have an account on a Linux server, check if you have brew installed:

~$ brew --version

If you do not it yet, copy a hyperlink for its installation from https://brew.sh. After the installation starts, press Control-D to install it locally and wait for the installation to complete. Then write:

~$ brew install gcc

~$ brew install open-mpi

and check if you have gcc and open-mpi among your installed programs:

~$ brew list

3. Finally, in both cases, download a Linux&Mac source code 1A_Hello_world.cpp. Let a directory where you have saved this source code be hello_world. Write:

~$ cd hello_world

~/hello_world$ mpic++ -o 1A_Hello_world 1A_Hello_world.cpp
~/hello_world$ mpiexec -n 2 ./1A_Hello_world

to launch a parallel program. You could also use a command:

~/hello_world$ mpirun -n 2 ./1A_Hello_world

I4. Instalation of gcc and Open MPI in macOS:

Check if you have gcc among your installed programs:

~$ brew list

If you do not have gcc installed yet, write:

~$ sudo brew install gcc

If you do not have open-mpi installed yet, write:

~$ sudo brew install open-mpi

Finally, download a Linux&Mac source code 1A_Hello_world.cpp. Let a directory where you have saved this source code be hello_world. Write:

~/$ cd hello_world

~/hello_world$ mpic++ -o 1A_Hello_world 1A_Hello_world.cpp
~/hello_world$ mpiexec -n 2 ./1A_Hello_world

to launch a parallel program. You could also use a command:

~/hello_world$ mpirun -n 2 ./1A_Hello_world

MPI_Routines (taken from www.mpich.org/static/docs/v3.2).
MPI_Constants (taken from www.mpich.org/static/docs/latest/www3/Constants.html).

Sample programs (Visual Studio, Linux&Mac):

1) Hello World:
- 1A_Hello_world.cpp (Visual Studio, Linux&Mac, PDF)
2) Point-to-point Blocking Communication:
- ! 2A_Ping_pong_Ssend.cpp (Visual Studio, Linux&Mac, PDF)
- + 2B_Ping_pong_various_sends (Visual Studio, Linux&Mac, PDF)
- 2C_Sum_of_Numbers.cpp (Visual Studio, Linux&Mac, PDF)
- 2D_pi_Monte_Carlo.cpp (Visual Studio, Linux&Mac, PDF, diagram , text_of_the_task)
3) Point-to-point Non-blocking Communication:
- 3A_Ring_Issend.cpp (Visual Studio, Linux&Mac, PDF)
- 3B_Ring_Send.cpp (Visual Studio, Linux&Mac, PDF)
- 3C_Ring_Ssend.cpp (Visual Studio, Linux&Mac, PDF) (DEADLOCK)
4) User-defined Types:
- 4A_Structure.cpp (Visual Studio, Linux&Mac, PDF)
- ! 4B_Vector_1.cpp (Visual Studio, Linux&Mac, PDF, diagram)
- ! 4C_Vector_2.cpp (Visual Studio, Linux&Mac, PDF, diagram)
- 4D_Vector_3.cpp (Visual Studio, Linux&Mac, PDF, diagram)
- !!** 4E_Jacobi_parallel_strips.cpp (Visual Studio, Linux&Mac, PDF, diagram)
5) Topology:
- ! 5A_Topology_1.cpp (Visual Studio, Linux&Mac, PDF)
- ! 5B_Topology_2.cpp (Visual Studio, Linux&Mac, PDF)
- !!!*** 5C_Jacobi_parallel_submatrices.cpp (Visual Studio, Linux&Mac, PDF, diagram_1 , diagram_2 , diagram_3 , diagram_4)
6) Collective Communication (collective communication routines):
- 6AMPI_Bcast.cpp (Visual Studio, Linux&Mac, PDF)
- 6BMPI_Reduce.cpp (Visual Studio, Linux&Mac, PDF)
- 6CMPI_Scan.cpp (Visual Studio, Linux&Mac, PDF)
- 6DMPI_Op_create.cpp (Visual Studio, Linux&Mac, PDF)
7) Some more complex samples:
- !!** The problem of a bussines traveler: 7A_Floyd_shortest_path.cpp (Visual Studio, Linux&Mac, PDF (7A_Floyd_shortest_path.cpp, distances.txt , text_of_the_task))
- !!** The sieve of Eratosthenes: 7B_Sieve_for_prime_numbers.cpp (Visual Studio, Linux&Mac, PDF)

A video of my Slovak lecture on parallel programming is here.

Bioinformatics

Based on Chipster and their tutorial videos.

Sample analyses:

1) RNAseq ENCODE ( PDF)
2) RNAseq Drosophila (PDF)
3) RNAseq parathyroid (PDF)