Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.1
Lecture 9
Programming Shared Memory I
Why Threads?
IKC-MH.57 Introduction to High Performance and Parallel
Computing at December 15, 2023
Dr. Cem Özdo
˘
gan
Engineering Sciences Department
˙
Izmir Kâtip Çelebi University
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.2
Contents
1 Programming Shared Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation and Termination
Thread Creation
Thread Termination
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.3
What is Threads?
•
Technically, a thread is defined as an independent
stream of instructions that can be scheduled to run by
the operating system.
• Suppose that a main p r ogram contains a number of
procedur es (functions, subroutines, ...).
• Then suppose all of these procedures being able to be
scheduled to run simultaneously and/or independently.
• That would d escribe a "multi-threaded" program.
•
Before unders tanding a thread, one first needs to
understand a UNIX process.
•
Processes contain information about program resources
and program execution state.
• Thread s use and exist within these process resources,
• To be scheduled by the OS,
• R un as independent entities.
• A thread has its own independ ent flow of co ntrol as long
as its parent process exists (dies if the parent d ies!).
• A thread duplicates only the essential resources it needs.
•
A thread is "lightweight" because most of the overhead
has already been accomplished through the creation of its
process.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.4
Threads Model I
•
In shared memory multiprocessor arc hitectures, such as
SMPs, threads can be used to implement parallelism.
•
In the threads model of parallel programming, a single
process can have
•
multiple concurrent,
•
execution paths.
•
Most simple analogy for threads is the concept of a single
program that includes a number of subroutines:
Figure: Thread s model.
•
Main program loads and
acquires all of the
necessary system and user
resources to run.
•
Main program performs
some serial work,
•
and then creates a number
of tasks (threads) that can
be scheduled and run by
the OS concurrently.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.5
Threads Model II
•
Each thread has local data, but also, shares the entire
resources of main program.
Figure: Thread sha r ed memory model.
•
This saves the overhead associated with r eplicating a
program’s resources for each thread.
•
Each thread also benefits from a global memory view
because it shares the memory space of program.
•
Any thread can execute any subroutine at the same time
as other threads.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.6
Threads Model III
•
Threads communicate with each other through global
memory (updating address locations).
•
Changes made by one thread to shared system resources
(such as closing a file) will be seen by all other threads.
•
This requires synchron ization constructs to insur e that
more than one thread is not updating
the same global
address at any time.
Figure: Thread s Unsafe! Pointer s having the same value point
to the same d ata.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.7
Why Threads? I
•
The primary motivation for using threads is to realize
potential program performance gains
.
•
When compared to the cost of creating and managing a
process, a thread can be created with much less OS
over head.
•
Managing threads requires fewer system resources than
managing processes.
•
Threaded programming models offer significant
advantages over message-passing programming models
along with some disadvantages
as well.
•
Software Portability;
• Thread ed applications can be developed on serial
machines and run on parall el machines without any
changes.
• This abilit y to migrate programs between diverse
architectu ral platforms is a very significant advantage of
threaded APIs.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.8
Why Threads? II
•
Latency Hiding;
• One of t he major overheads in programs (both serial and
parallel) is the access latency for memory a ccess, I/O, and
communication
.
• By allowing multip le threads to execute on the same
processor, threaded APIs enable this latency to be hidden
.
• In effect, while one thread is waiting for a communication
operation, other threads ca n utilize the CPU, thus masking
associated ove r head.
•
Scheduling and L oad Balancing;
• While in many structured a pplications the task of
allocating equal work to processors is easily accomplished,
• In unstru ctured and dynamic application s (such as game
playing and discrete optimization) this task is more difficult.
• Thread ed APIs allow the programmer
• to specify a large number of concurrent tasks
• and support system-level dynamic mapping of tasks to
processors with a view to minimizing idling overheads
.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.9
Why Threads? III
•
Ease of Programming, Widespread Use
• D ue to t he mentioned advantages, threaded pro grams are
significantly easier to write (!)
than corresponding programs
using message passing APIs.
• With widespread acceptance of t he POSIX thread API,
development tools for POSIX threads are more widely
available and stable.
•
Overlapping CPU work with I/O: For example, a program
may have sections where it is performing a long I/O
operation. While one thread is waiting for an I/O system
call to complete, CPU intensive work can be performed by
other threads.
•
Priority/real-time scheduling: tasks which are more
important c an be scheduled to supersede or interrupt
lower priority tasks.
•
Asynchronous event handlin g: tasks which service
events of indeterminate frequency and duration can be
interleaved. For example, a web server can both transfer
data from previous requests and manage the arrival of
new requests.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.10
Why Threads? IV
•
A number of vendors provide vendor-specific thread APIs.
Standardization efforts have r esulted in two very different
implementations of threads.
•
Microsoft has its own implementation for threads, which is
not related to the UN IX POSIX standard or OpenMP.
1 POSIX Threads. Library based; requires parallel coding.
•
C Language only. Ver y explicit parallelism; requires
significant programmer attention to detail.
•
Commonly referred to as Pthreads.
•
POSIX has emerged as the standard threads API,
supported by most vendors.
2 OpenMP. Compiler directive based; can use serial code.
•
Jointly defined by a group o f major computer hardware and
software vendors.
•
The OpenMP C/C++ API was released in late 1998.
•
Portable / multi-platform, inclu ding Unix and Windows
platforms
•
Can be very easy and simp le to use - provides for
“increment al parallelism“.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.11
Why Threads? V
•
MPI =⇒ on-node communications,
•
MPI libraries usually impleme nt on-node task
communication via shared memory, which involves at least
one memory copy operatio n
(process to process).
•
Threads =⇒ on-node data transfer.
• For Pthreads there is no intermediate memory copy
required because threads share the same address space
within a single process.
• There is no data transfer.
• It becomes more of a cache-to-CPU or memory-to-CPU
bandwidth (worst case) situation.
• These speeds are much higher.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.12
Thread Basics: Creation and Termination I
•
The Pthreads API subroutines can be informally grouped
into four major groups:
1 Thread management: Routi nes that work directly on
threads - creating, detaching, joining, se t/query thread
attributes (joinable, scheduling etc.), etc.
2 Mutexes: Routines that deal with synchronization. Mutex
functions provide for creati ng, destroying, locking and
unlocking mutexes, setting or modifying attributes
associated with mutexes.
3 Condition variables: Routines that address
communications b etween threads that share a mutex.
Functions to create, destroy, wait and signal based upon
specified variable values, set/query condition variable
attributes.
4 Synchronization: Routines that manag e read/write locks
and barriers.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.13
Thread Basics: Creation and Termination II
Creating Threads:
•
Initially, main program contains a single, default thread.
•
pthread_create creates a new thread and makes it
executable.
1 # i ncl ude <pthread . h>
2 i n t
3 pthread_cr e at e ( pthread _ t
*
thread_handle ,
4 const p thr e ad_ at t r _t
*
a t t ri b u t e ,
5 void
*
(
*
th r ead _f un c ti o n ) (
void
*
) ,
6 void
*
arg ) ;
•
Creates a single thread that corresponds to the invocation
of the function thread_function (and any other functions
called by thread_function).
•
Once created, threads are peers, and may create other
threads.
•
On successful creation of a thread, a unique identifier is
associated with the thread and assigned to the location
pointed to by thread_handle.
•
On successful creation of a thread, pthread_create
returns 0; else it returns an error code.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.14
Thread Basics: Creation and Termination III
•
The thr ead has the attributes described by the attribute
argument.
•
The arg field s pecifies a pointer to the argument to
function thread_function.
•
This argument is typically used to pass the wor kspace and
other thread-specific data to a thread.
•
There is no implied hierarchy or dependency between
threads.
•
Unless you are using the Pthreads scheduling
mechanism, it is up to the implementation and/or OS to
decide where and when threads will execute.
•
Robust programs should not depend upon threads
executing in a specific order.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.15
Thread Basics: Creation and Termination IV
Terminating Th reads.
•
There are several ways in which a Pthread may be
ter minated:
a The thread returns from its starting routine (the main
routine for the initial thread).
b The thread makes a call to the pthread_exit subroutine.
c The thread is cancelled by another thread via the
pthread_cancel routine.
d The entire process is terminated due to a call to either the
exec or exit subroutines.
•
If main finishes before the threads and exits with
pthread_exit(), the other threads will continue to execute
(join function!).
•
If main finishes af ter the threads and exits, the thr eads wil l
be automatically terminated.
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.16
Thread Basics: Creation and Termination V
Example Code:
•
This example code creates 5 threads with the
pthread_create() routine.
•
Each thread prints a ’Hello World!’ message, and then
ter minates with a call to p thread_exit().
1 # in c lude <pthread . h>
2 # i ncl ude < st d i o . h>
3 # i ncl ude < s t d l i b . h>
4 # i ncl ude < uni std . h>
5
6 # d e f i ne NUM_THREADS 5
7
8 void
*
P ri n t H el l o (
void
*
th r ead id )
9 {
10 sleep (1 0 ) ;
11 long t i d ;
12 t i d = ( long ) th r ead id ;
13 p r i n t f ( " Hello World ! I t ’ s me, thread #%l d ! \ n " , t i d ) ;
14 p t hre ad _ex it (NULL) ;
15 }
Programming Shared
Memory I
Dr. Cem Özdo
˘
gan
LOGIK
Programming Shared
Memory
What is a Thread?
Threads Model
Why Threads?
Thread Basics: Creation
and Termination
Thread Creation
Thread Termination
9.17
Thread Basics: Creation and Termination VI
1 i n t main ( i n t argc , char
*
argv [ ] )
2 {
3 pthre a d _ t threads [NUM_THREADS ] ;
4 i n t rc ;
5 long t ;
6 f o r ( t =0; t <NUM_THREADS; t ++) {
7 p r i n t f ( " I n main : cr e ati ng thread %ld \ n" , t ) ;
8 rc = pthread_cr e a te (& threads [ t ] , NULL , Pr i nt H el l o , ( void
*
) t )
;
9 i f ( rc ) {
10 p r i n t f ( "ERROR; re t urn code from pthread_create ( ) i s %d \ n " ,
rc ) ;
11 ex i t (− 1) ;
12 }
13 }
14
15 /
*
Last thi ng t h a t main ( ) should do
*
/
16 p t hre ad _ex it (NULL) ;
17 }