[SOLVED] C shell parallel operating system computer architecture network

informatics
HOME
MAP
CONTACT
FOR EXISTING
Students
Staff
Alumni
CO2017 CW1 Operating Systems & Networks Shell scripting
Preamble
DEADLINE:This assessed work should be handed in by the 08:59:59 on 2th February.Instructions on how to submit it will appear here in due course.
You have several weeks to complete this work, including at least 2 TA supported sessions in the labs. The expectation is that you will spend an adequate amount of independent study time to complete this assessment. You should look through the whole assessment, and make sure that as far as possible you ask any questions you have during the TA supported sessions. You can also ask questions on the CO2017 discussion board.
This assessment will be marked anonymously by programmatically obfuscating user names before the marking process.
This assignment constitutes 10% of your CO2017 grade.
This assignment assesses the following learning outcomes:
Explain files, directories, memory, the command line and fundamental structures. Write scripts and useful but simple command line programs.
Discuss detailed overviews of operating systems; computer architecture.
In this page (and in other similar):
Pale pink boxes indicate interactive use, with the input you should type in RED; other text is expected output from a program.
Hints and asides are written in pink italics.
Pay close attention to the details. For example where file names are specified, make sure you use exactly the specified name. Similarly, where output is specified, make sure that yours matches as closely as possible. You will lose marks in the assessment if your submissions do not match precisely.
Before you start, you need to have attempted and understood the material from the first lab session.
Most of the questions ask you to produce a shell script. In each case you should first familiarise youself with the commands involved by practicing with them on the command line; once you have an idea of how the commands work, you can continue with creating a shell script.
You might find it helpful to work through this help sheet on bash shell, these hints and/or the main bash manual. You can download the example script from the help sheet; remember to make it executable after downloading it so that you can invoke it.

Q1 Simple conditional scripts and variables
1.Write a shell script called q1-msg.sh that displays a message like the one shown here: $ ./q1-msg.sh
2.Hello! This is module CO2017.
3.The current directory is /the/current/directoryExcept that /the/current/directory should be replaced with the path of the actual directory where the script is executed. Will be assessed by running your script and checking that the output is exactly as described. Any deviation will result in loss of marks; for example missing or extra spaces; variations in the capitalisation or spelling, etc.
4.Write a shell script called q1-home.sh that displays a message saying whether or not the working directory is your home directory (by referring to the environment variable $HOME). $ ./q1-home.sh
5.Not in HOME directory
6.$ cd $HOME
7.$ /path/to/CO2017/scripts/q1-home.sh
8.In HOME directoryWill be assessed by running your script twice: once within the home directory, once elsewhere.
Assessed: 15 marks

Q2 Files, searching and finding
1.The grep command can be used to search for patterns within files. Write a shell script called q2-grep.sh that takes a list of file names as command-line arguments and searches all of them when it is invoked for the username of the current user. As output it should display the line of each occurence plus the one line before and after. For example, invoked by user gtl1 would produce something like this: $ ~/CO2017/q2-grep.sh test1 test2
2.test1-On the other hand, we denounce with righteous indignation and dislike
3.test1:gtl1 who are so beguiled and demoralized by the charms of pleasure of
4.test1-the moment, so blinded by desire, that they cannot foresee the pain
5.
6.test1-saying through shrinking from toil and pain. These cases are perfectly
7.test1:simple and easy to distinguish. In a free hour, when gtl1s power of
8.test1-choice is untrammelled and when nothing prevents our being able to do
9.
10.test1-have to be repudiated and annoyances accepted. The wise man therefore
11.test1:always holds in these matters to this principle of selection: gtl1
12.test1-rejects pleasures to secure other greater pleasures, or else he
13.
14.test2-Far far away, behind the word mountains, far from the countries
15.test2:Vokalia and Consonantia, there live the gtl1. Separated they
16.test2-live in Bookmarksgrove right at the coast of the Semantics, a large
17.test2:language ocean. A small river named gtl1 flows by their place and
18.test2-supplies it with the necessary regelialia. It is a paradisematic
19.
20.test2-text by the name of Lorem Ipsum decided to leave for the far World of
21.test2:Grammar. gtl1 advised her not to do so, because there were
22.test2-thousands of bad Commas, wild Question Marks and devious Semikoli, but
23.Will be assessed by running your script using multiple test files similar to test1 and test2, etc, but which will not necessarily have the target string contained. (If you use these test files, you will need to modify them so that your own username appears.) Assessed: 15 marks
24.The find command can be used to search for files (or directories, etc) that match certain conditions. Write a shell script called q2-find.sh that takes a directory path as a command line argument and displays the following for the contents of that directory:
the plain files in the specified directory (one level only, do NOT descend into sub-directories) which were most recently modified more than 72 hours before the script is run;
the directories (not plain files) in sub-directories of the specified directory where the directory name begins with the string dir (not including quote marks).
25.For example: $ ./q2-find.sh /s_home/gtl1/CO2017
26.Plain files modified more than 72 hours ago in /s_home/gtl1/mydir:
27./s_home/gtl1/CO2017/myhome1.txt
28./s_home/gtl1/CO2017/fileA.txt
29.Directories in sub-directories of /s_home/gtl1/CO2017 starting with dir:
30./s_home/gtl1/CO2017/abc/dir1
31./s_home/gtl1/CO2017/xyz/dir6A major part of this exercise is constructing a test directory structure that allows you to verify that your script is working. Note that the touch command can be used to arbitrarily set the modification date on a file for test purposes. Assessed by running the script in a directory with a mixture of files and sub-directories; some of the files will have been modified within the past 72 hours, and some will be older; some directories will have names beginning with dir and some will not.
Assessed: 15 marks

Q3 Shell scripting
Here is a data file. The file contains a list of names and figures in four columns: Name (alpha-numeric), size (in human readable format), month (3 letter month), value (numeric value). The columns are semi-colon (;) separated. You may assume that the file contains correctly formatted data.
Write a shell script called q3-sort.sh that takes a file path to a data file (of the form described above) as its single argument. The script should use the sort command to produce two reverse sorted versions of the file based on column 1 (name) and column 3 (date) respectively. The two output files will contain all of the original data, but arranged in the reverse order of the relevant column.
If the original file is called ffff.txt then the two output files should be called:
alpha-ffff.txt
reverse alphabetic sort based on the names in the first column;
date-ffff.txt
sort based on the 3 letter month in the third column, but in reverse order; so Dec comes before Nov etc to Jan; and
If the original file is called something other than ffff, the output file names will change accordingly.
The output files should be written to the current working directory, regardless of the actual path of the supplied argument. If the output files already exist, they should simply be over-written without further warning.
In addition, your script should exit with a suitable error message if either:
the number of arguments is not precisely 1; or
the specified file does not exist, or is not readable.
Examples:Line starting with # and in blue are commentry; your script should not produce them.
$ ./q3-sort.sh file.txt
# if file.txt exists in current directory will create output files
# alpha-file.txt, and date-file.txt in the current directory
$ ./q3-sort.sh x.txt y.txt
Usage: q3-sort.sh
$ ./q3-sort.sh
Usage: q3-sort.sh
$ ./q3-sort.sh blah.txt
blah.txt does not exist
$ ./q3-sort.sh /long/path/file.txt
# If file.txt exists in /long/path/ will create output files
# alpha-file.txt and date-file.txt in the current directory.
# Dont forget to check the contents of the output files to make sure
# that they are in the correct order.
Will be assessed by running the script on a slightly modified version of the example data file and comparing the output files with known correct versions. Script will also be inspected to check for the method used.
Assessed: 25 marks

Q4 Re-arranging files
Suppose there is directory containing files with names of the form f21.txt, f62.txt etc. For example:f07.txt
f11.txt
f12.txt
f24.txt
f26.txt
f35.txt
f51.txt
f59.txt
f68.txt
f71.txt
f73.txt
f75.txt
f96.txt
f99.txt
Each filename starts with f, followed by a 2 digit number, followed by .txt
Write a script called q4-arrange.sh to re-arrange the files into directories so they end up like this:d0/f7.txt# was f07.txt
d1/f1.txt# was f11.txt
d1/f2.txt# was f12.txt
d2/f4.txt# was f24.txt
d2/f6.txt# etc
d3/f5.txt
d5/f1.txt
d5/f9.txt
d6/f8.txt
d7/f1.txt
d7/f3.txt
d7/f5.txt
d9/f6.txt
d9/f9.txt# was f99.txt
Your script should not take any command-line arguments. It should operate in the current working directory. Directories should be created if they do not already exist, and there is to be at least one file inside. In this example, no directories d4 or d8 need to be created. The original files should be moved/renamed (not just copied). Initially, if there are any files/directories that do not match the fNN.txt pattern, they should be ignored. There are several ways to do this, but the simplest is to use a pair of nested for loops.
NOT ASSESSED.

Q5 Use of parallel; evaluation of the scheduler
Introduction: Using parallel to execute several processes at once
The GNU parallel command is a powerful utility to get multiple instances of a command to run at the same time.
Here is an example of running the simple command echo Output N 10 times, but in parallel as many times as there are processors/cores on the machine. For example, using a 4-core machine, it will run the command 4 times in parallel (with values 1,2,3,4), then run it again 4 times in parallel (with values 5,6,7,8) and finally run it two more times in parallel (with values 9,10).
$ parallel -j4 echo Output {} ::: {1..10}
Explanation:
Red: parallel command that starts and controls multiple processes. The values after ::: at the end control how many processes will be run in total; and the values are substituted into the {} part of the payload command. Using {1..10} produces a sequence of numeric values; but the values are arbitrary; they do not need to be sequential, or even numeric (so adding ::: one two three four five six seven eight nine ten would have had the same effect; try this out). The -j4 option indicates that parallel should execute 4 processes (or jobs) at a time. On a machine with 4 cores/cpus, this would be equivalent to completely leaving out the -j option. See the documentation for other suitable values for the -j option.
Blue: the payload command that is actually executed multiple times. Note the use of {} to indicate the place where the parallel arguments will be placed.
When you first run parallel it will print out a message explaining how to properly attribute your usage in any academic publications you produce by using it. To get rid of this message, run parallel citation and follow the instructions.
Assessed question: Evaluation of the linux scheduler under various conditions
For this exercise you will perform some experiments, collect the results and then write and submit a short analysis of the results.
You task is to consider the overall performance (of the system process scheduler) in relation to these three criteria:
A.overall turnaround time (elapsed/wall clock);
B.overall CPU utilisation;
C.impact on other interactive processes that are running on the machine.
You can use the results of the time program, and by monitoring the processes that are running using top or htop. You may well have to devise other strategies, especially to assess criteria C.
Consider these example of a long running processes (similar to those from lab1):
ssh-keygen -G /tmp/mods-X.candidate -b 768
grep -r $(number -l 1) /usr/share/doc > /tmp/grep1.txt
The experiment is to run multiple copies of these commands, so that in total twice as many processes are run as there are cores/CPUs available. So on a machine with 4 cores, the experiments involve running a total of 8 processes each time.
For each command, you will need to run the experiment three times and assess the performance against the criteria (A, B, C) given above:
1.Run 2N copies of the process strictly sequentially (use the -j1 option for parallel).
2.Run N copies of the process concurrently, and then run another N copies of the process concurrently (this is the default behaviour of parallel with no -j option, or use the option -j4 on a 4 core machine option).
3.Run all 2N copies of the process concurrently (use the option -j0, or -j8 option on a machine with 4 cores).
Here are the two commands you need to run as part 1 of your experiment:
$ /usr/bin/time -o ssh-timing1.txt -v parallel -j1 ssh-keygen -G /tmp/mods-{}.candidate -b 768 ::: {1..8}

$ /usr/bin/time -o grep-timing1.txt -v parallel -j1 grep -r $(number -l {}) /usr/share/doc > /tmp/grep{}.txt ::: {1..8}
Note that you need to adjust the total number of processes run if your machine does not have 4 CPUs/cores.
For parts 2 and 3, you need to change the value of the -j option and make sure you adjust the names of the output files so that the results of each experiment are kept separate.
Write a brief plain-text document (q5-answer.txt). Your document should include:
The exact command lines, involving parallel, used for each of 1, 2, 3;
for each of the criteria A, B, C, give a brief explanation (one or two sentences) of which approach (1, 2 or 3) is best (there may not be a clear winner in every case);
for each criteria, give evidence that supports your answers; for example, you could compare specific lines of output collected by the time program; for C you need to briefly explain how you evaluated it.
Will be assessed by considering whether your answer is clearly stated and is backed up by the evidence supplied. Where suggested commands are given they will be checked.
Assessed: 30 marks

Contents:
Preamble
Q1 Simple conditional scripts and variables
Q2 Files, searching and finding
Q3 Shell scripting
Q4 Re-arranging files
Q5 Use of parallel; evaluation of the scheduler
Introduction: Using parallel to execute several processes at once
Assessed question: Evaluation of the linux scheduler under various conditions
Up: CO2017 home page
Prev: Lab 1
CW 301 Help Desk
Semester Two 2018/19
The help desk schedule is shown below. The help desk will run until 29th March.
Information about the expertise of the GTAs is here.
Monday:
13001400 Pamela (from 28th January)16001700 Hakeem
Tuesday:
13001400 Jakob
Wednesday:
11001200 Hao
Thursday:
11001200 Mirzhan
Friday:
09001000 Rafael15001600 Bello (from 28th January)17001800 Xiang
Previous
Up
University Home
Science & Engineering
Informatics Home
University A-Z
Author: Gilbert Laycock ([email protected]).$Revision: 1380 $ University of Leicester 2019/01/18. Last modified: 29th January 2019, 16:34:21Informatics Web Maintainer. Any opinions expressed on this page are those of the author.