[SOLVED] Hw assignment 1 (a1): bag of words cs6120

Integrating clustering techniques with dimension reduction in unsupervised learning presents a fascinating study area. Dimension reduction, a process that streamlines complex, high-dimensional datasets into a more manageable form, is essential for efficient data analysis and visualization. Techniques like Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), and Uniform Manifold Approximation Projection (UMAP) are instrumental in this context. Applying clustering methods such as k-means, c-means, hierarchical clustering, DBSCAN, HDBSCAN, and Expectation-Maximization (EM) to dimensionally reduced datasets offer a comprehensive understanding of how these algorithms can identify patterns and groupings effectively. This approach facilitates a practical application of these algorithms and deepens the knowledge of their collective impact in enhancing data analysis, particularly within unsupervised learning.

In this assignment, you are provided with 40,000 physician notes authored by test-takers of the USMLE. These notes, written for ten standardized patients, offer a unique dataset for analysis. The notes contain a natural ten clusters as the patients are the same for all note writers. The task is a good example of unsupervised learning where the ground truth can be used for post-hoc analysis. Your tasks are as follows:

Please submit a fully executed jupyter notebook identifying question number and steps. Make sure to add comments to your solution.