[SOLVED] python Spark graph network Correlation Networks

Correlation Networks

Due: 21219.
Objectives:
Experiencing basic network properties
Preparing a network dataset

Tasks:
Install netwrokx library or use python 3.7 distribution with preinstalled networkx.
Download the following datasets from UCI Machine Learning Repository
https:archive.ics.uci.edumldatasetsgeneexpressioncancerRNASeq
https:archive.ics.uci.edumldatasetsDexter
https:archive.ics.uci.edumldatasetsArcene
https:archive.ics.uci.edumldatasetsAmazonCommercereviewsset
https:archive.ics.uci.edumldatasetsReuter5050
https:archive.ics.uci.edumldatasetsDrivFace
https:archive.ics.uci.edumldatasetsDailyandSportsActivities
https:archive.ics.uci.edumldatasetsp53Mutants
https:archive.ics.uci.edumldatasetsGisette
https:archive.ics.uci.edumldatasetsMicroMass
https:archive.ics.uci.edumldatasetsCNAE9
https:archive.ics.uci.edumldatasetsParkinson27sDiseaseClassification
https:archive.ics.uci.edumldatasetsGastrointestinalLesionsinRegularColonoscopy
https:archive.ics.uci.edumldatasetsMultipleFeatures
https:archive.ics.uci.edumldatasetsISOLET
https:archive.ics.uci.edumldatasetsSECOM
https:archive.ics.uci.edumldatasetsSmartphoneBasedRecognitionofHumanActivitiesandPosturalTransitions
https:archive.ics.uci.edumldatasetsUJIIndoorLoc
https:archive.ics.uci.edumldatasetsFMA3AADatasetForMusicAnalysis
https:archive.ics.uci.edumldatasetsDetectMalaciousExecutable28AntiVirus29
https:archive.ics.uci.edumldatasetsMadelon
https:archive.ics.uci.edumldatasetsDynamicFeaturesofVirusShareExecutables
https:archive.ics.uci.edumldatasetsCharacterFontImages
https:archive.ics.uci.edumldatasetsLSVTVoiceRehabilitation
https:archive.ics.uci.edumldatasetsSemeionHandwrittenDigit
https:archive.ics.uci.edumldatasetsOPPORTUNITYActivityRecognition
https:archive.ics.uci.edumldatasetsNoisyOffice
https:archive.ics.uci.edumldatasetsNorthix
https:archive.ics.uci.edumldatasetsEpilepticSeizureRecognition
https:archive.ics.uci.edumldatasetsUltrasonicflowmeterdiagnostics
https:archive.ics.uci.edumldatasetsIDA2016Challenge
https:archive.ics.uci.edumldatasetsAPSFailureatScaniaTrucks
https:archive.ics.uci.edumldatasetsMusk28Version129
https:archive.ics.uci.edumldatasetsMusk28Version229
https:archive.ics.uci.edumldatasetsWeightLiftingExercisesmonitoredwithInertialMeasurementUnits
https:archive.ics.uci.edumldatasetsGasSensorArrayDriftDatasetatDifferentConcentrations
https:archive.ics.uci.edumldatasetsPhysicalUnclonableFunctions
https:archive.ics.uci.edumldatasetsGasSensorArrayDriftDataset
https:archive.ics.uci.edumldatasetsREALDISPActivityRecognitionDataset
https:archive.ics.uci.edumldatasetsdetectionofIoTbotnetattacksNBaIoT
https:archive.ics.uci.edumldatasetsLowResolutionSpectrometer
Generate 41 correlation networks from the above datasets such that:
Nodes are features
Undirected weighted Edges are derived from Pearson correlation values
The networks should be sparse but having one giant connected component.
Prepare a CSV file listing the all graphs. The CSV should have the following fields and field names
DatasetName same as in the links above
CorrelationPearson
Threshold
n number of vertices
m number of edges after applying the threshold
maxdeg maximal degree
diameter
effectivediameter90 i.e. 90 of node pairs have distance of effectivediameter90 or less
averagelocalclusteringcoefficient
components number of connected components
Save the graphs into files using writeedgelist with comma delimiters and weights. The file names should be the DatasetNameCorrelationThreashold.txt
Pick three graphs of your choice and print their layouts.

Submission instructions:
Solution should contain three files:
A .csv file from task 4 named HW2YourID1YourID2.csv
A .zip file with 41 networks from task 3 named HW2YourID1YourID2.zip
Report named HW2YourID1YourID2.pdf with the table from task 4 and three graphs from task 6.