AC Assignment Chef

[Solved] COMP1007 Lab 9-Fundamental Statistics with Python

5.0 1 customer review Digital download

Digital download

$25.00

Availability
In stock
Checkout
One item

Upload assignment

Send a ZIP file

First, we summarize how to sort data in Python. Second, we learn how to use NumPy and SciPy packages for basic statistics. Given a data set, we will:

I. Sorting Data

Case 1. If the data are stored in a list, we can use list.sort() function to sort the data. L = [5, 2, 3, 1, 4] L.sort() print(L) # [1, 2, 3, 4, 5] Case 2. If the data are stored in a dictionary, we can first get the keyvalues pairs through the dict.items() function, and then use the Python builtin function sorted() to sort the keyvalue pairs according to the key.
d = {x:3, a:89, k:5, d:10} L = sorted(d.items())print(L) # [(a, 89), (d, 10), (k, 5), (x, 3)]
Case 3. If the data are stored in a Numpy ndarray, we can use numpy.ndarray.sort() function to sort the ndarray inplace. import numpy as np array = np.array([7, 5, 3, 2, 6, 1, 4]) array.sort() print(array) # [1 2 3 4 5 6 7]

II. Counting Student Scores and Creating a Graph of Score Distributions with Median and InterQuartile Range

  1. Import required library packages.
import numpy as np from scipy import stats import pandas as pdfrom matplotlib import pyplot as plt
  1. Download the data file csv from the course web page and save it in D drive.
  1. Explore the csv file using Excel. We can see that the file has 3 columns the index, student id, and scores. The index and student id columns are not used for the statistics. We focus on the scores column.
  1. Load the file in Python. And, we store the values of the scores column in the NumPys
df = pd.read_csv(quiz1_scores.csv, index_col=index) scores = df[scores].values Remark: df is a DataFrame; df[scores] is a Series, df[scores].values is an ndarray.
  1. Finding the median.
Remark: Median is the value separating the higher half from the lower half of a data set. median = np.median(scores)
  1. Finding the qth percentile.
Remark: A percentile is a measure used in statistics indicating the value below which a given percentage of observations in a group of observations falls. For example, the 20th percentile is the value below which 20% of the observations may be found. Median is in fact the 50th percentile. per75 = np.percentile(scores, 75) per25 = np.percentile(scores, 25)
  1. Getting the Inter Quartile Range (IQR).
Remark: IQR is equal to the difference between 75th and 25th percentiles. iqr = stats.iqr(scores)
  1. Separate student scores into 10 groups evenly and count the number of scores in each group.
  1. Create labels for each group. We will use them as the x axis labels of the chart.
labels = [below 10, 10 to 19, 20 to 29, 30 to 39, 40 to 49, 50 to 59, 60 to 69, 70 to 79, 80 to 89, 90 or above]
  1. There are 10 groups, but we need to create a sequence with total 11 numbers (0, 10, 20, , 90, 100) that define the bin edges.
edges = np.linspace(0, 100, 11) The linspace() function returns total 11 float numbers 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, and 100. You may also use the following statement to get the same values (but as integers). edges = np.arange(0, 101, 10)
  1. Use NumPys histogram() function to tally scores into the appropriate interval and count the number of students in each interval.
Remark: A histogram is an accurate representation of the distribution of numerical data. To construct a histogram, the first step is to bin (or bucket) the range of values, i.e., divide the entire range of values into a series of intervals, and then count how many values fall into each interval. We have two methods to generate the histogram: hist, edges = np.histogram(scores, bins=edges) The histogram() function returns two ndarrays hist stores the resulted data that is the count of each interval; bins stores the list of bin edges. Notice that the size of edges is one more than the size of hist. The following is the counting result:
Interval Count
below 10 3
10 to 19 2
20 to 29 15
30 to 39 26
40 to 49 51
50 to 59 42
60 to 69 39
70 to 79 23
80 to 89 11
90 or above 8
  1. Print the histogram result.
print(Score Distribution) for i in range(len(hist)): print(%s: t %d % (labels[i], hist[i]))
  1. Plot a graph.
    1. To plot a histogram with 10 bins, we need to find the center values for each bin:
bin_centers = 0.5 * (edges[1:] + edges[:1]) The statement above does the following thing:
edges[1:] + edges[:1] temp x 0.5 bin_centers
10 0 10 5
20 10 30 15
30 20 50 25
90 100 190 95
  1. Plot the graph using the bin_centers and histogram result data, and add meaningful labels on X axis. To see the stepbystep visual effect, you need to type the %matplotlib qt command in the IPython console first.
plt.plot(bin_centers, hist) plt.xticks(bin_centers, labels, rotation=vertical) You can also choose the bar style for the histogram: plt.bar(bin_centers, hist, width=10) plt.xticks(bin_centers, labels, rotation=vertical)
  1. Add an indicator of Inter Quartile range Q1, Q2, and Q3
q1 = median iqr/2
q2 = median q3 = median + iqr/2 plt.axvline(x=q1, color=lightgrey, linestyle=) plt.axvline(x=q2, color=red, linestyle=:) plt.axvline(x=q3, color=lightgrey, linestyle=)
The axvline() function draws a vertical line in the graph. We can customize the line with the parameters:
  1. Add graph title and text labels.
plt.title(Score Distribution 1, fontsize = 24) plt.text(q1, 5, Q1=%.2f% q1, fontsize = 16) plt.text(q2, 10, Median=%.2f% q2, fontsize = 16) plt.text(q3, 15, Q3=%.2f% q3, fontsize = 16)

III. Creating a Graph with Mean, Standard Deviation, Reference Standard Normal Distribution Curve

In this section, we are going to create another graph using the same data set. The graph shows the mean, standard deviation, and a reference curve. You may add the example code of this section after the previous one. At the end, you will see two graphs. The second graph is created by the code of this section. Otherwise, you need to repeat the steps from 1 to 8 of the previous section first.
  1. Finding the mean, maximum, minimum, and standard deviation.
s_mean = scores.mean() s_max = scores.max() s_min = scores.min() s_std = scores.std() The functions include mean(), max(), min() and std() are the builtin functions of ndarray.
  1. Compute the density data using the Probability Density Function.
    1. Use the pdf() function to compute the reference data.
pdf = stats.norm.pdf(range(100), s_mean, scale = s_std) We provide a range (0 to 99, 100 elements), the peak (s_mean) and scale (standard deviation).
  1. Scale the resulted data up so as to match the histogram data scale.
Pdf = pdf / pdf.max() * hist.max()
  1. Plot the graph using the bins and histogram result data. Then, change the xaxis labels.
plt.plot(bin_centers, hist) plt.xticks(bin_centers, labels, rotation=vertical)
  1. Plot the density data as a reference curve.
plt.plot(Pdf, )
  1. Add indicators for the mean and standard deviation.
  1. Add line and label for mean:
plt.axvline(x=mean, color=green) plt.text(mean, 10, $mu = %.2f$ % mean) The string $mu = %.2f$ is a formatted string for printing the string with special symbols. The mu in the string will be replaced by the symbol but the string must be enclosed in quotation marks and dollar signs ($ $).
  1. Add lines and labels for standard deviation:
for i in range(2,3): if (i != 0): x = mean + std * i plt.axvline(x=x, color=lightgrey, linestyle=:) plt.text(x, 0, $ %d sigma$ % i)
  1. Draw the legend.
plt.legend([Score Counts,Reference]) There are total 7 lines in the graph including the score counts (result of the histogram), reference (the result of pdf), mean, and the lines about the standard division. But, we dont show all of them in the legend. In the statement above, we pass a list with two strings that mean we want to show the first two lines with the labels stored in the list.

Exercise

Imagine that we have a survey data file retrieved from an online survey system. The columns contained in the survey data includes the responded date, customer IDs, salary range, age, number of children, gender and the choices of question 1 to 6. However, we only focus on the salary range. Now, you need to create a graph to show the salary distribution with the median, IQR, and reference curve. You need to:
Label Description
Below 15K The salary is less than $15,000
15K <= $ < 20K The salary is larger than or equal to $20,000 but less than $15,000.
20K <= $ < 25K The salary is larger than or equal to $25,000 but less than $20,000.
$95K or above The salary is larger than or equal to $95,000
Assume that the minimum amount is $10,000, and the maximum amount is $100,000. The sample output of the graph: