IKC-MH.57 Introduction to High Performance and Parallel Computing

2023-2024 Fall

Most recent announcement: February03

Announcements

Watch this space for the latest updates (If the characters do not show properly, please try viewing this page with Unicode (UTF-8) encoding). Last updated:

Saturday, February 03, 2024 Resit Examination grades are available.

Friday, January 26, 2024 Final Examination grades are available.

Friday, January 12, 2024 Midterm Examination grades are available.

Tuesday, December 19, 2023 Lecture notes for week13 is published, see Course Schedule section.

Wednesday, December 13, 2023 Lecture notes for week12 is published, see Course Schedule section.

Tuesday, December 12, 2023 Lecture notes for week11 is published, see Course Schedule section.

Friday, December 8, 2023 Lecture notes for week10 is published, see Course Schedule section.

Tuesday, December 5, 2023 Lecture notes for week9 is published, see Course Schedule section.

Thursday, November 23, 2023 Take-home Midterm Examination will be announced at "Lecture Hour" November 24, 2023, see Course Schedule section.

Tuesday, November 21, 2023 Lecture notes for week7 is published, see Course Schedule section.

Wednesday, November 15, 2023 Lecture notes for week6 is published, see Course Schedule section.

Friday, November 10, 2023 Lecture notes for week5 is published, see Course Schedule section.

Monday, November 06, 2023 Lecture notes for week4 is published, see Course Schedule section.

Thursday, October 26, 2023 Lecture notes for week3 is published, see Course Schedule section.

Sunday, October 22, 2023 Lecture notes for week2 is published, see Course Schedule section.

Sunday, October 22, 2023 Video for Installation of Kubuntu & Parallel Tools under VirtualBox is available, see Course Schedule section.

Monday, October 9, 2023 Lecture notes for week1 is published, see Course Schedule section.

Monday, October 9, 2023 In the first lecture, there will be first meeting. The lecture notes will be published soon, see Course Schedule section.

Back to top

Lecture Information

Almost all computer systems today are multi-core processors systems. Parallel programming must be used to take benefit of the full performance of such systems. Parallel programming also describes the processes and instructions for dividing a larger problem into smaller steps. The instructions are passed to multiple processors for required calculations to be executed in parallel. A practical approach to parallel program design and development will be presented in the course content. Awareness of potential design and performance concepts in heterogeneous computer architectures will be gained. Important announcements will be posted to the Announcements section of this web page, so please check this page frequently. You are responsible for all such announcements, as well as announcements made in lecture.

Back to top

Course Overview

IKC-MH.57 is intended to provide students an introduction to parallel programming by using C and/or similar programming languages. In this course, the basics of high performance and parallel computing will be given. OpenMP (Open Multi-Processing) in multi-core systems and MPI (Message Passing Interface) message passing in distributed memory systems will be taught for parallel programming. MPI is the industry standardized parallelization paradigm in high-performance computing and enables programs to be written that run on distributed memory machines. OpenMP is a thread-based approach to parallelize a program over a single shared memory machine. An introduction to the basic concepts of hybrid and accelerated paradigms as Cuda OpenCL programming will be given. The course consists of theoretical topics and hands-on practical exercises on parallel programming.

Upon completion of this course the students will be able to understand/explain/apply;

• Learn how to work in a scientific computing environment.

• Gain awareness of Parallel and High Performance Computing concepts for systems with shared/distributed memory.

• Can write parallel programs both for systems with shared memory using threading (OpenMP ) and systems with distributed memory using message passing (MPI).

• May define the MPI message passing standard’s control of communication between processes, subroutines, or functions within the program.

• Gains basic knowledge of Cuda OpenCL hybrid and accelerated paradigms.

• Gains the ability and understanding to develop parallel programs to solve a given big numerical/engineering/scientific problem.

Back to top

Text Book

Lecture material will be based on them. It is strongly advised that student should read textbooks rather than only content with the lecture material supplied from the lecturer.

Back to top

Online Resourcess

The following resources are available online. Please inform me about the usefullness of the materials. Check this place for updates.

• https://www.cs.purdue.edu/homes/ayg/book/Slides/

• https://sites.cs.ucsb.edu/~tyang/class/240a17/

• https://hpc-tutorials.llnl.gov/

Back to top

Grading Criteria

• Midterms & Final Exams: There will be one take-home midterm and one take-home final exam, will count 40% each and 60% of your grade, respectively.

• Homeworks/Assignments (or Term Project): ??

Back to top

Policies

• Attendance is not compulsory (30%), but you are responsible for everything said in class.

• You can use ideas from the literature (with proper citation).

• You can use anything from the textbook/notes.

• I encourage you to ask questions in class. You are supposed to ask questions. Don't guess, ask a question!

Back to top

Tentative Course Outline (pdf)

• Contents

• List of Tables

• List of Figures

• Preliminaries

  • First Meeting

    • Lecture Information

    • Course Overview

    • Text Book

    • Online Resourcess

    • Grading Criteria

    • Policies

  • Installation of Required Tools/Programs

    • Linux System

    • Others

• Introduction

  • View of the Field

  • Four Decades of Computing

  • Flynn's Taxonomy of Computer Architecture

  • Parallel and Distributed Computers

  • MPI Hands-On; Performance Analysis

    • Analysis of Parallel Summation with Point-to-Point Communications

  • SIMD Architecture

  • MIMD Architecture

  • Shared Memory Organization

  • Message Passing Organization

  • MPI Hands-On - Introduction to MPI

    • Parallel Computing

    • Communicating with other processes

    • What is MPI?

    • MPI Implementations

    • Is MPI Large or Small?

    • Where to use MPI?

    • How To Use MPI? Essential!!

    • Getting started

      • Writing MPI programs I

      • Writing MPI programs II

      • Writing MPI programs III

      • Exercise - Getting Started

• Performance Metrics, Postulates

  • Performance Analysis

    • Computational Models

      • Equal Duration Model

      • Parallel Computation with Serial Sections Model

    • Skeptic Postulates For Parallel Architectures

      • Amdahl's Law

  • MPI Hands-On - Sending and Receiving Messages I

    • Current Message-Passing

    • The Buffer

    • MPI Basic Send/Receive

    • Exercises/Examples

• Message-Passing Paradigm

  • Programming Using the Message-Passing Paradigm

    • Principles of Message-Passing Programming

    • Structure of Message-Passing Programs

    • The Building Blocks: Send and Receive Operations

      • Blocking Message Passing Operations

      • Non-Blocking Message Passing Operations

  • MPI Hands-On; Sending and Receiving Messages II

  • MPI: the Message Passing Interface

    • Starting and Terminating the MPI Library

    • Communicators

    • Getting Information

    • Sending and Receiving Messages

    • Avoiding Deadlocks

    • Sending and Receiving Messages Simultaneously

  • MPI Hands-On; Sending and Receiving Messages III

  • Parallelization Application Example

    • Pi Computation

  • Overlapping Communication with Computation

    • Non-Blocking Communication Operations

  • Collective Communication and Computation Operations

    • Broadcast

    • Reduction

    • Gather

    • Scatter

    • All-to-All

  • MPI Hands-On; Collective Communications I

• Shared Memory Paradigm

  • Programming Shared Memory

    • What is a Thread?

    • Threads Model

    • Why Threads?

    • Thread Basics: Creation and Termination

      • Thread Creation

      • Thread Termination

  • Hands-on; Shared Memory I; Threads

  • OpenMP: a Standard for Directive Based Parallel Programming

    • The OpenMP Programming Model

    • The OpenMP Design Concepts

  • Hands-on; Shared Memory II; OpenMP

  • Parallelization Application Example-OpenMP

    • Computing

• GPU parallelization

  • Exploring the GPU Architecture

  • Execution and Programming Models

  • Hands-on; GPU parallelization

• References:

• About this document ...

Back to top

Tentative Course Schedule & Lecture Notes

The following schedule is tentative; it may be updated later in the semester, so check back here frequently.

Back to top

References (other than text books)

Back to top