D-DS-FN-23 Sample Questions & Answers

The summary() function in R is specifically designed to provide a quick statistical summary of a variable. For a numerical vector, it returns the minimum, 1st quartile, median, mean, 3rd quartile, and maximum values. This single command gives a concise overview of central tendency (mean, median), dispersion (quartiles, min/max), and distribution shape.

The AUC represents a model's ability to discriminate between positive and negative classes across all possible thresholds. A higher AUC (closer to 1.0) indicates better performance. An AUC of 0.85 means the logistic regression model is superior to the decision tree (AUC 0.78) in its ability to correctly rank a randomly chosen positive instance higher than a randomly chosen negative one. It does not directly state overall accuracy, which depends on a single chosen threshold.

The Elbow method is a common heuristic for finding the optimal number of clusters in K-means. It involves running the algorithm for a range of K values and plotting the WSS for each. The point where the rate of decrease in WSS sharply slows, forming an 'elbow' in the plot, is considered a good estimate for the optimal K. Gini impurity is for decision trees, and Lift charts are for classification models.

Apache Spark, and particularly its Spark Streaming library, is designed for scalable, high-throughput, fault-tolerant processing of live data streams. It processes data in micro-batches, providing near real-time capabilities. While tools like Pig and Hive are excellent for batch processing, and HBase is a NoSQL database, Spark is the premier choice in the Hadoop ecosystem for real-time stream computation.

The NameNode is the centerpiece of the HDFS architecture. It acts as the master server and maintains the filesystem metadata, including the directory tree and the mapping of file blocks to the DataNodes where they are stored. It does not store the data itself but directs clients to the correct DataNodes. DataNodes store the data blocks, and TaskTrackers (in YARN, the NodeManager/ApplicationMaster) execute tasks.

R-squared, or the coefficient of determination, is a statistical measure that represents the proportion of the variance for a dependent variable that's explained by an independent variable or variables in a regression model. An R-squared of 0.82 means that 82% of the fluctuations in house prices can be accounted for by the model's inputs.

Window functions perform calculations across a set of table rows that are somehow related to the current row. To calculate a rolling 7-day average, you would use AVG(sales) OVER (PARTITION BY product_id ORDER BY sales_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW). This allows you to compute the average over a moving 'window' of rows without collapsing them like a GROUP BY clause would.

The 'Naïve' in Naïve Bayes comes from its core assumption that all features are conditionally independent given the class. If features are highly correlated, this assumption is violated, which can lead to poor probability estimates and reduced model performance. Decision Trees, on the other hand, do not make this assumption and can handle correlated features effectively by selecting the most informative feature at each split.

Time Series Analysis is specifically designed for analyzing and forecasting data points collected over time. The presence of a repeating 12-month pattern indicates seasonality, a key component that time series models like SARIMA (Seasonal ARIMA) or Holt-Winters exponential smoothing are built to handle. Other methods like linear regression or clustering do not account for the temporal dependencies and seasonal patterns inherent in this data.