CENG 471 Parallel Computing
Fall 2007

Most recent announcement: Feb13

Announcements

Watch this space for the latest updates. Last updated: 13.Şubat.2008 11:45 Term projects are accesible.

15.Ocak.2008 14:33 The final date for submission of term project report is Jan 17, 2008 Thursday. I will submit your grades by Friday. I will not be available the following week.

January 4, 2008 19:06 You should bring calculator (preferably scientific) to the exam hall. It will be open book exam.

December 27, 2007 15:16 Project presentations will be held on December 28 , 2007 at 14.40 (A-319). You have 15 or 20 minutes.

December 25, 2007 10:43 The lecture notes and lab notes for the twelfth week is available, see Course Schedule section.

December 12, 2007 15:49 There will be no lecture and lab hours for next week. The research project presentation date is changed as on Dec 28, 2007 at 16.30. Project topics:

• Erdem Bozoğlu: Advantages and Disadvantages of Parallel Computing

• Yağmur Şengez: Deep Blue Chess Playing Supercomputer

• Burcu Taşkan: Parallel File Systems

• SenaErkaya: Parallel Programming Languages

• Rabia Mallı : Grid Computing

• Efe Çiftci: Google as an example

• Sebile Aydın: Parallel Password Cracking

• Gökhan Demireğen: Linux Clustering

• Mesut Ateser: Cluster Computing

• Duygu Yilmaz: Fibonacci numbers as an example

• Serdar Çetinkaya: Deadlock-free parallel processing

• Ali Yılmaz: Message passing in Java

• Ferdi Tekin: Motivation Parallel Computing and Compare 3 Parallel Computing System

December 4, 2007 09:46 The lecture notes and lab notes for the eleventh week is available, see Course Schedule section.

November 29, 2007 14:31 The graduate students also have to do a research project and make a presentation at the given dates below (should contact with me as soon as possible).

November 29, 2007 14:11 The lab project is given as follows and due to the final exam date (the submission date may subject to change).

1. Parallel Summation

• General Solution

• with point-to-point communications

• with collective communicatins

• performance analysis (Speed-Up, efficiency, (Amdahl and Gustafson laws : optional))

2. Parallel matrix-vector multiplication

• General Solution

• with point-to-point communications

• with collective communicatins

• performance analysis (Speed-Up, efficiency, (Amdahl and Gustafson laws : optional))

3. Parallel matrix-matrix multiplication (Optional)

• General Solution

• with point-to-point communications

• with collective communicatins

• performance analysis (Speed-Up, efficiency, (Amdahl and Gustafson laws : optional))

November 29, 2007 13:47 The research project presentations will be on Dec 25-28, 2007 at lecture hours.

November 20, 2007 10:26 The lecture will be held at INT1 lab.

November 11, 2007 15:47 There will be no lecture at Nov 13, 2007. I will not be available all the day.

November 6, 2007 10:16 The lecture notes and lab notes for the seventh week is available, see Course Schedule section.

October 30, 2007 00:41 The lecture notes and lab notes for the sixth week is available, see Course Schedule section.

October 22, 2007 13:20 The lecture notes and lab notes for the fifth week is available, see Course Schedule section.

October 19, 2007 16:55 The Lab hour is rearragened as Friday at 13.00.

October 15, 2007 23:30 The lecture notes and lab notes for the fourth week is available, see Course Schedule section.

October 9, 2007 01:39 The lecture notes for the third week is available, see Course Schedule section.

October 9, 2007 01:39 The lecture notes for the second week is updated, see Course Schedule section.

October 5, 2007 12:40 THIS WEB PAGE IS FINALLY AVAILABLE. In the first lecture, there will be first metting. The lecture notes for the second week will be published soon, see Course Schedule section.

Overview

This course provides an introduction to parallel and distributed computing and practical experiences in writing parallel programs on a cluster of computers. You will learn about the following topics:

• Parallel Computers

• Message Passing Computing

• Embarrassingly Parallel Computations

• Partitioning and Divide-and-Conquer Strategies

• Pipelined Computations

• Synchronous Computations

• Load Balancing

• Programming with Shared Memory

Topics might be classifed into two main parts as;

• Parallel computers: architectural types, shared memory, message passing, interconnection networks, potential for increased speed.

• Basic techniques: embarrassingly parallel computations, partitioning and divide and conquer, pipelined computations, synchronous computations, load balancing, shared memory programming.

Lecture Information

There is one group for lecturing. You will be expected to do significant programming assignments, as well as run programs we supply and analyze the output. Since we will program in C on a UNIX environment, some experience using C on UNIX will be important. We will provide tutorials for basic C on UNIX during the first few class periods.

In lab sessions, we will concentrate upon the message-passing method of parallel computing and use the standard parallel computing environment called MPI (Message Passing Interface). Thread-based programming will also be outlined, and the distributed shared memory (DSM) approach (If we have enough time). Each student will complete a project based on parallel computing for the laboratory study.

Also, each student will complete a project based on parallel computing, (distributed computing, cluster computing) for the midterm exam.

Important announcements will be posted to the Announcements section of this web page above, so please check this page frequently. You are responsible for all such announcements, as well as announcements made in lecture.

Course Outline

• Contents

• List of Tables

• List of Figures

• Introduction

  • Four Decades of Computing

  • Flynn's Taxonomy of Computer Architecture

  • SIMD Architecture

  • MIMD Architecture

    • Shared Memory Organization

    • Message Passing Organization

• Introduction to MPI

  • Parallel Computing

  • Types of parallel computing

  • Communicating with other processes

  • Hardware models

  • What is MPI?

  • Who Designed MPI?

  • MPI Implementations

  • Is MPI Large or Small?

  • Where to use MPI?

  • How To Use MPI?

  • Getting started

    • Writing MPI programs

    • Compiling and linking

    • Running MPI programs

    • Finding out about the environment

    • A simple program

    • Exercise - Getting Started

• Performance Analysis

  • Computational Models

    • Equal Duration Model

    • Parallel Computation with Serial Sections Model

  • Skeptic Postulates For Parallel Architectures

    • Grosch's Law

    • Amdahl's Law

    • Gustafson-Barsis's Law

• Message Passing Architecture

  • Introduction to Message Passing

  • Routing for Broadcasting and Multicasting

    • Routing Potential Problems

  • Message Passing vs Shared Memory

  • MPI Hands-On; Sending and Receiving messages

    • Current Message-Passing

    • The Buffer

    • Sample Program using Library Calls

    • MPI Basic Send/Receive

    • Exercise/Example: Translate this Fortran http://siber.cankaya.edu.tr/ParallelComputing/cfiles/sendreceive.fcode to C

    • Computation of PI as an example

    • Exercise

• Programming Using the Message-Passing Paradigm

  • Principles of Message-Passing Programming

    • Structure of Message-Passing Programs

  • The Building Blocks: Send and Receive Operations

    • Blocking Message Passing Operations

    • Non-Blocking Message Passing Operations

  • MPI Hands-On; Examples

  • MPI: the Message Passing Interface

    • Starting and Terminating the MPI Library

    • Communicators

    • Getting Information

    • Sending and Receiving Messages

      • Avoiding Deadlocks

      • Sending and Receiving Messages Simultaneously

  • Performance tuning MPI 1

    • Overlapping Communication with Computation

      • Non-Blocking Communication Operations

    • Collective Communication and Computation Operations

      • Reduction

      • Gather

      • Scatter

      • All-to-All

    • Groups and Communicators

  • Performance tuning MPI 2

  • Parallel Summation

• Shared Memory Architecture

  • Classification of Shared Memory Systems

    • Uniform Memory Access (UMA)

    • Nonuniform Memory Access (NUMA)

    • Cache-Only Memory Architecture (COMA)

  • Basic Cache Coherency Methods

  • Shared Memory Programming

    • Task Creation

    • Communication

    • Synchronization

    • Thread Basics

  • Examples; Buffer Limit, Barrier, Collective Communications.

• Programming Shared Memory

  • Why Threads?

  • Thread Basics: Creation and Termination

  • Synchronization Primitives in Pthreads

    • Mutual Exclusion for Shared Variables

  • Shared Memory I; Processes, Threads

    • Contd: Mutual Exclusion for Shared Variables

    • Condition Variables for Synchronization

  • Controlling Thread and Synchronization Attributes

    • Attributes Objects for Threads

  • Thread Cancellation

  • Composite Synchronization Constructs

    • Barriers

  • Tips for Designing Asynchronous Programs

  • OpenMP: a Standard for Directive Based Parallel Programming

    • The OpenMP Programming Model

  • Shared Memory II; Threads, OpenMP Examples

• References:

• vita

• About this document ...

Text Book

Required

Recommended

• Readings will be assigned in Parallel Programming: Techniques and Application Using Networked Workstations and Parallel Computers, 2^nd edition, by B. Wilkinson and M. Allen, Prentice Hall Inc., 2005, ISBN 0-13-140563-2.

• Beowulf Cluster Computing with Linux, 2nd edition, edited by William Gropp, Ewing Lusk, Thomas Sterling, MIT Press, 2003, ISBN 0-262-69292-9.

• Beowulf Cluster Computing with Windows, Thomas Sterling , MIT Press, 2001, ISBN 0-262-69275-9.

• Using MPI , Portable Parallel Programming with the Message Passing Interface, William Gropp, Ewing Lusk and Anthony Skjellum, The MIT Press, 1999, ISBN 0-262-57132-3.

• Using MPI-2, Advanced Features of the Message Passing Interface, William Gropp, Ewing Lusk, Rajeev Thakur, The MIT Press, 1999, ISBN 0-262-57133-1.

• MPI: The Complete Reference (Vol. 1) - The MPI Core, Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker and Jack Dongarra, The MIT Press, 1998, ISBN 0-262-69215-5.

• MPI: The Complete Reference (Vol. 2) - The MPI-2 Extensions, William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William Saphir and Marc Snir, The MIT Press, 1998, ISBN 0-262-57123-4.

• In Search of Clusters: The ongoing battle in lowly parallel computing, Second Edition, by Gregory F. Pfister, Prentice Hall Publishing Company, 1998, ISBN: 0-13-899709-8.

• How to Build a Beowulf – A Guide to the Implementation and Application of PC Clusters, by Thomas Sterling, John Salmon, Donald J. Becker and Daniel F. Savarese, MIT Press, 1999, ISBN 0-262-69218-X.

• PVM: Parallel Virtual Machine, A Users' Guide and Tutorial for Network Parallel Computing, Al Geist, Adam Beguelin, Jack Dongarra, Weicheng Jiang, Robert Manchek and Vaidyalingam S. Sunderam, MIT Press, 1994, ISBN 0-262-57108-0.

This texts are only recommended rather than required. This books are useful for reference, for an alternative point of view.

Online References

Some materials are given. Please inform me about the usefullness of the materials. Check this place for updates.

The following references are available online

• Take a look at the TOP500 list

• MPI: The Complete Reference

• Message Passing Interface Forum

• MPICH Home Page

• MPI Man pages (Web format)

• ANSI C for Programmers on UNIX Systems

• Greg Wilson's Parallel Computing Timeline (1955 to 1993)

• I. Foster. Designing and Building Parallel Programs, Addison-Wesley, 1995 available on-line

• Glossary of Terms

Grading Criteria

• There will be a final exam: 40%

• Term Project as Midterm exam: 25%

• Term Project as Lab. exam: 25%

• Attendance is required and constitutes part of your course grade; 10%

Policies

• Attendance is not compulsory, but you are responsible for everything said in class.

• I encourage you to ask questions in class. You are supposed to ask questions. Don't guess, ask a question!

• You may discuss homework problems with classmates (although it is not to your advantage to do so).

• You can use ideas from the literature (with proper citation).

• You can use anything from the textbook/notes.

• The code you submit must be written completely by you.

Course Schedule

The following schedule is tentative; it may be updated later in the semester, so check back here frequently.