[SOLVED] 代写 algorithm Spark parallel graph Question 1

Question 1
5 / 5 pts
Download the adjective-noun data sets here.
Load it into spark and use divide-and-conquer to find the first (adj, noun) pair in which the noun is ‘unification’. Print the corresponding adjective.  The skeleton code is provided below.  One solution is to use filter() to find all pairs where the noun is ‘unification’, and then report the first one.  This is inefficient.  The better idea is to find, in parallel, the first such pair in each partition (if one exists), and then find the first partition that returns such a pair.
numPartitions = 10

lines = sc.textFile(path_to_file, numPartitions)
pairs = lines.map(lambda l: tuple(l.split())).filter(lambda p: len(p)==2)
pairs.cache()

# FILL IN YOUR CODE HERE
Your Answer:
numPartitions = 10

lines = sc.textFile(path_to_file, numPartitions)
pairs = lines.map(lambda l: tuple(l.split())).filter(lambda p: len(p)==2)
pairs.cache()

# FILL IN YOUR CODE HERE
def f(iterator):
for a in iterator:
if(a[1]==’unification’): 
yield a

result= pairs.mapPartitions(f)
print result.collect()[0][0]

Question 2
5 / 5 pts
Design a parallel divide-and-conquer algorithm for the following problem: Given two strings of equal length, compare them lexicographically. Output ‘<‘, ‘=’, or ‘>‘, depending on the comparison result. The skeleton code is provided below.  Your code should run on all partitions of the rdd in parallel.
x = ‘abcccbcbcacaccacaabb’
y = ‘abcccbcccacaccacaabb’

numPartitions = 4
rdd = sc.parallelize(zip(x,y), numPartitions)

# FILL IN YOUR CODE HERE

Your Answer:
x = ‘abcccbcbcacaccacaabb’
y = ‘abcccbcccacaccacaabb’

numPartitions = 4
rdd = sc.parallelize(zip(x,y), numPartitions)

# FILL IN YOUR CODE HERE
def f(iterator):
for a in iterator:
if(a[0] a[1]): 
yield “>”


t= rdd.mapPartitions(f)

#if strings are equal printing ‘=’
if t.count() == 0:
print “=”
else:
print t.collect()[0]