Project final: Bring Your Own Project!

Option 1: Implementation

If you are interested in gaining more experiences with Go beyond the Project MapReduce, you probably will like this option. At the high level, this option is about adding more features to the simple MapReduce framework you have implemented previously.

For this option, the recommended steps are following:

1. Read the original MapReduce paper from Google to gain deeper understanding about the design.
2. Pick one feature that interests you the most and sketch out what you need to do to incorporate the said feature into your existing MapReduce framework.
3. Design a few corresponding test cases.
4. Implement the feature based on the test cases.

Option 2: Case Study

If you are interested in seeing how classical distributed systems techniques are being used in modern distributed systems, then you might like this option. At the high level, you will pick an open-source distributed system and do a deep dive of its internal with the key goal of identifying which techniques are used and for what.

Here are a list of modern distributed systems that you might find interesting to explore:

1. Apache Spark is an in-memory data processing framework. Github link: https://github.com/ apache/spark
2. Alluxio which is a virtual distributed storage system. Github link: https://github.com/Alluxio/ alluxio
3. Pytorch Distributed which allows training deep learning models on different workers. Github link:

https://github.com/pytorch/pytorch/tree/master/torch/distributed

Option 3: Measurement

At the high level, you will pick a distributed systems related technology, learn how to use it, and measure the performance. For example, if you want to know more about grpc which is Googles RPC system, the recommended steps are following:

1. spend some time reading the docs for grpc: https://www.grpc.io/docs/
2. make sure you understand how to measure event times with precision with functions clock gettime() and gettimeofday().
3. think about performance metrics that are important for RPC
4. come up with a list of factors such as data types and network conditions that might lead to different RPC performance.
5. vary each factor while keeping the other factors constant to perform the corresponding measurements.
6. (Optional) If you are really interested in RPC and are curious about how grpc compares to other systems such as Apache Thrift, then you can repeat the above steps and do a shoulder-to-shoulder comparison.^[1]

Here are a list of distributed systems technology that you might find interesting to measure:

1. Any cloud-based services, such as Virtual Machines, Containers, and serverless: as a first-time user, you can get one-year free access to most of AWS services.
2. Any open-source distributed systems: including the ones mentioned in the Option 2: Case Study.

Checkpoint Contributions

Checkpoint is not required for this project.

Deliverables and Grading

When submitting the project, please make sure to satisfy the following requirements:

• Relevant deliverables to this project such as code and PDF writeup.
• A document called README.txt explaining the project, any defects, and anything that you feel the teaching staff should know when grading the project. Only plaintext write-ups are accepted; Markdown is allowed.
• A document called self assessment.txt that describes your groups self-assessment of your performance (see below).

Please compress all the files together as a single .zip, named YOUR WPI USERNAME project final.zip, archive for submission. As with all projects, please only use standard zip files for compression; .rar, .7z, and other custom file formats will not be accepted.