🎥 Project Presentation Video
Dataset Overview
For this project, I have chosen the "Global Mental Health and Lifestyle Survey Dataset", sourced from Kaggle (contributed by dhrubangtalukdar). This dataset was selected because it provides a high-dimensional, cross-sectional view of modern psychological well-being, offering the complexity required for sophisticated modeling.
Technical Specifications
The dataset strictly adheres to the assignment requirements:
• Rows: 10,000 (Meets the 10K+ requirement).
• Features: 51 (Meets the 15+ requirement).
• Data Types: A diverse mix of Numeric and Categorical variables.
Features Breakdown
The dataset contains a rich mix of Numeric and Categorical features:
• Numeric: Age, Work Hours per Week, Sleep Hours, Screen Time, Caffeine Consumption, Close Friends Count, and psychometric scores (Loneliness, Mood Swings, etc.).
• Categorical: Country, Gender, Education, Occupation, Remote Work Status, Smoking Status, and Diet Quality.
• The Target Variable (Label): Work_Stress_Level – A numeric value on a scale of 1–10.
Research Question & Objective
To what extent can an individual’s Work Stress Level be predicted through a combination of daily lifestyle patterns and socio-cultural backgrounds? Furthermore, how can establishing this predictive baseline enhance clinical assessments and help personalize mental health interventions?
Project Objective: The primary goal is to build an algorithm that predicts an individual's "Standard Stress Baseline". By utilizing socio-cultural factors and lifestyle metrics, the model establishes the expected stress level for a specific profile. This serves as a critical reference point for identifying clinical moments of arousal, allowing for more nuanced and personalized mental health monitoring.
Data Cleaning & Preprocessing
To ensure the integrity of our predictive baseline, I conducted a rigorous cleaning process:
• Duplicate & Null Check: The dataset was found to be exceptionally clean, with 0 duplicate rows and 0 missing values across all 51 features. This completeness allows for high-confidence training without the need for synthetic imputation.
• Categorical Consistency: I processed 13 categorical columns (e.g., Gender, Country, Occupation). All entries were stripped of leading/trailing whitespaces and standardized to uniform casing.
• Structural Integrity: Features like 'Gender' were verified for consistency (Male, Female, Non-binary, Prefer not to say), and 'Country' was mapped to six distinct regions, ensuring no redundancy or typos (e.g., preventing "USA" vs "usa" discrepancies).
Outlier Detection & Handling: The "Clipping" Strategy
During the initial analysis, I identified extreme values in several key numeric features (e.g., Age values beyond 68.5 and Close_Friends_Count beyond 8) using the IQR (Interquartile Range) method. The Decision: Instead of removing these observations, which would reduce our statistical power, I applied Clipping (Winsorization).
Regression models are highly sensitive to extreme outliers, a single data point can "pull" the prediction line and skew results. By capping these values at the statistical upper and lower bounds, I maintained the full 10,000-row dataset while stabilizing the model to perform accurately for the general population.
Population Profile: Descriptive Statistics
A summary of the 10,000 observations reveals a high-stress profile among modern participants:
• Work Stress Baseline: The average stress level is 5.49/10, indicating a moderately high level of occupational pressure.
• Digital Habits: A significant finding is the high average screen time of 7.06 hours per day, which serves as a primary hypothesis for digital fatigue.
• Sleep Patterns: Participants sleep an average of 6.8 hours, which is below the recommended threshold for optimal mental health.
• Social Connectivity: On average, participants report having 4 close friends, establishing a baseline for social support impact.
Key Visualizations & Narrative Insights
To answer the research question, I explored the relationships between lifestyle patterns and stress levels:
- The Drivers of Stress
The analysis identified that Work Hours per Week (0.43) is the strongest positive predictor of stress, followed by Screen Time (0.23). Conversely, Sleep Hours (-0.37) emerged as the most significant protective factor.
- The Sleep-Stress Correlation
The data shows a clear downward trend: as sleep duration increases, work stress levels decrease. This validates the clinical importance of sleep hygiene as a key feature in our proposed algorithm.
- Digital Overload Impact
By analyzing screen time distribution, I observed that individuals reporting high stress (Levels 9-10) consistently spend significantly more hours on digital devices compared to those in the "Low Stress" group.
- Socio-Cultural Distribution & Bias Check
To ensure the model is globally representative and statistically sound, I analyzed both the sample volume and the average stress levels across regions.
Sample Volume (Left): The distribution confirms a well-balanced dataset, with significant representation from the USA, India, UK, Brazil, and Germany. Each region contains enough observations (~1,000+) to provide robust statistical power.
Regional Stress Baseline (Right): Germany showed the highest average work stress (5.60), while the UK showed the lowest (5.40). This subtle variance suggests that while workplace stress is universal, a "Cultural Baseline" is essential for personalizing stress-detection algorithms.
Research Questions & Visual Insights
In this section, I transition from data cleaning to clinical storytelling. I posed five targeted research questions to uncover the patterns that define an individual's "Stress Baseline." Each question is answered through a specific visualization, providing the logic behind our predictive algorithm.
Q1: Does geography dictate a 'Cultural Stress Baseline'?
Visual Approach: Ranked Barplot of Average Stress Level by Country.
The Insight: The data reveals measurable regional variations; participants in Germany (5.60) and the USA (5.55) report higher average stress compared to those in the UK (5.40).
Algorithmic Impact: This confirms that a "one-size-fits-all" threshold is insufficient. Our algorithm must be regionally calibrated, adjusting its sensitivity based on the patient's socio-cultural environment.
Q2: Is the Sleep-Stress relationship consistent enough for predictive modeling?
Visual Approach: Regression Plot (Regplot) of Sleep Hours vs. Work Stress.
The Insight: A clear, consistent negative correlation exists (the red regression line shows a steady decline). As sleep duration increases, stress levels reliably drop.
Algorithmic Impact: This establishes sleep as a Physiological Restoration Metric. If the model detects a sudden spike in stress, it can cross-reference it with the sleep baseline to determine if the arousal is lifestyle-driven or clinical.
Q3: Can digital habits serve as a "Behavioral Red Flag"?
Visual Approach: Boxplot of Screen Time Distribution per Stress Level.
The Insight: There is a notable upward shift in screen time for individuals in the "Extreme Stress" group (Levels 9-10). The median screen time is significantly higher than in the lower-stress groups.
Algorithmic Impact: Digital usage acts as a Leading Indicator. The algorithm can utilize screen time spikes as a "trigger warning" to predict upcoming periods of high arousal.
Q4: Can social connections buffer the impact of workload?
Visual Approach: Correlation Heatmap focusing on Work Hours, Close Friends, and Stress.
The Insight: While work hours are the strongest positive driver of stress (0.43), social support (Close Friends) shows a distinct negative correlation.
Algorithmic Impact: This allows for Resilience Modeling. The system can adjust the "expected stress level" of a patient based on their social safety net; a patient with high social support may have a more "resilient" baseline.
Q5: Do demographic disparities require a more inclusive approach?
Visual Approach: Grouped Barplot of Stress Levels by Country and Gender.
The Insight: Stress distribution is not uniform across demographics. In several regions, certain gender identities report higher variance and higher average stress levels compared to others.
Algorithmic Impact: This ensures Algorithmic Fairness. To prevent bias, the predictive baseline must account for the unique socio-cultural stressors faced by different demographic groups in specific countries, ensuring the algorithm is accurate for everyone.
Define and Train a Baseline Model
Model Strategy
The goal is to establish a predictive baseline for an individual's "Standard Stress Level" (scale 1-10). This baseline acts as the reference point for detecting clinical gaps during therapy.
Feature Selection: For the initial run, I selected primary predictors identified in the EDA: Work Hours, Sleep Hours, Screen Time, Age, Close Friends Count, and socio-cultural factors (Country, Gender).
Train-Test Split:
The dataset was partitioned into an 80/20 split using a fixed Random Seed (42) to ensure results are reproducible and consistent across iterations.
Training:
I utilized a standard Linear Regression model with default parameters via scikit-learn.
Model Evaluation & Metrics
The performance of the initial baseline model was measured using standard regression metrics:
Performance Insights & Interpretation
Visualizing the results reveals the inherent complexity of the task:
Actual vs. Predicted (Scatter Plot): The plot shows a high concentration of predictions around the mean. The lack of a clear diagonal trend indicates that a simple linear combination of raw features is insufficient to capture the subjective nature of stress.
Feature Importance (Coefficients): Work Hours and Screen Time emerged as the most significant positive drivers of predicted stress.
Sleep Hours showed the strongest negative (protective) coefficient, confirming it as a critical baseline metric.
Socio-cultural factors like Country showed low initial coefficients, suggesting they require further engineering to be effective.
This initial failure to capture high variance is a crucial finding. It demonstrates that raw lifestyle data alone contains too much "noise."
Feature Engineering & Unsupervised Learning
To improve the performance of our predictive algorithm and capture non-linear relationships, I expanded the dataset from the initial raw features to a refined set of 42 high-impact variables.
Advanced Feature Creation
I engineered two custom indices designed to capture the "lifestyle balance" that raw metrics often miss:
Lifestyle Index: Calculated as Sleep_Hours_Night - (Screen_Time_Hours_Day / 2). This represents the net "Mental Energy" available to an individual.
Work-Life Ratio: Calculated as Work_Hours_Per_Week / (Hobby_Time_Hours_Week + 1). This measures occupational intensity relative to personal recovery time.
Data Transformation & Scaling
To prepare the dataset for complex modeling, I implemented a robust preprocessing pipeline:
Dynamic One-Hot Encoding: Automatically transformed 13 categorical variables (e.g., Education, Marital Status, Smoking Habits) into binary indicators.
Standard Scaling: Since features like Age and Work Hours have different units, I applied StandardScaler to normalize the data, ensuring that features with larger numerical ranges do not dominate the model's coefficients.
Unsupervised Learning: Patient Segmentation
To move beyond generic predictions, I utilized K-Means Clustering to identify distinct "User Archetypes" based on sleep, work hours, and screen time.
Visualizing the Clusters (PCA)
Since the data has 42 dimensions, I applied PCA (Principal Component Analysis) to reduce the features into two dimensions for visualization. The resulting plot clearly shows four distinct segments of individuals within our dataset.
Cluster Interpretation & Utility
By analyzing the averages of each cluster, we can identify unique "Stress Profiles":
Cluster 2 (The Overworked): Highest work hours (~54 hrs/week) and high stress.
Cluster 1 (The Flexible): Lowest work hours (~24 hrs/week) but surprisingly high average stress, suggesting other lifestyle factors (like loneliness or digital fatigue) are at play.
Utility: These cluster IDs were added back into the dataset as a new feature (User_Cluster). This allows the final models to treat individuals differently based on their lifestyle "archetype," significantly enhancing the precision of our predictive baseline.
Improved Regression Models & Evaluation
In this phase, I retrained the baseline model using the engineered features and compared its performance against two sophisticated machine learning algorithms: Random Forest and Gradient Boosting.
Model Comparison Results
I used an 80/20 train-test split with the same random seed (42) to ensure a fair comparison.
The Winner: Improved Linear Regression
Despite the complexity of Ensemble models like Random Forest, the Linear Regression model with engineered features emerged as the winner.
Why did the simple model win?
Feature Engineering Impact:
Moving from R^2 of -0.002 to +0.003 demonstrates that while the improvement is subtle, the added features like Work_Life_Ratio and User_Cluster successfully captured a linear signal that wasn't there before.
Avoiding Overfitting:
Complex models like Random Forest struggled with the high noise-to-signal ratio of this psychological dataset, leading to overfitting on the training set and poor generalization on the test set (as seen in the negative R^2).
Linearity of Stressors:
The relationship between work hours, sleep, and stress in this dataset appears to be predominantly linear, making a simpler model more robust.
Feature Importance Analysis
Visualizing the coefficients of the winning model confirms the primary drivers of our stress baseline.
Top Drivers:
Financial_Stress, Work_Hours_Per_Week, and Caffeine_Drinks_Day are the strongest positive predictors.
Protective Factors:
Job_Satisfaction and Sleep_Hours_Night show significant negative coefficients, acting as key resilience metrics.
Error Analysis: Where the Model Struggles
To understand the limitations of our regression approach, I analyzed the Residuals (the difference between actual and predicted values).
The "Mean-Reversion" Trap: The residual boxplot shows that the model is accurate for average stress levels (5-6) but fails significantly at the extremes. It tends to over-predict stress for calm individuals (1-2) and under-predict for those in extreme distress (9-10).
Conclusion:
This error pattern proves that stress is a highly subjective experience that cannot be captured by a single continuous number alone. This insight justifies our next move: Transitioning to Classification.
Reframing as a Classification Problem
Strategic Transformation: From Numeric to Categorical
To improve the practical utility of our predictive framework, I reframed the regression task into a three-class classification problem. Instead of predicting a specific score from 1-10, the algorithm now identifies "Stress Zones."
The Transformation Strategy: I utilized a Business Rule Threshold (Custom Binning) to divide the Work_Stress_Level target:
Class 0 (Low Stress): Scores 1–4.
Class 1 (Medium Stress): Scores 5–7.
Class 2 (High Stress): Scores 8–10.
Reasoning: In a clinical setting, knowing the exact decimal of a stress score is less important than identifying the severity level. This 3-tier classification aligns with standard triage protocols, allowing the system to distinguish between a "stable" patient and one requiring immediate "High Stress" intervention.
Data Distribution & Imbalance Analysis
Class Balance Check
Before training the classifiers, I analyzed the distribution of our new categories across both training and test sets to ensure the model wouldn't be biased toward a specific class.
Results (Training Set):
Low Stress (Class 0): 39.6%
Medium Stress (Class 1): 30.1%
High Stress (Class 2): 30.3%
Evaluation Insights
The data shows a slight imbalance, with the "Low Stress" group being the majority (approx. 40%).
Risk: An algorithm might achieve high accuracy simply by over-predicting the "Low Stress" class, while failing to detect the more critical "High Stress" events.
Mitigation Strategy: Because accuracy alone can be misleading in this context, the evaluation will prioritize Recall for Class 2 (High Stress). In our clinical framework, missing a high-stress state (a False Negative) is far more critical than a false alarm. Therefore, I will focus on the F1-Score and Confusion Matrices to ensure balanced performance across all tiers.
Classification Models - Training & Evaluation
In this final phase, I trained three distinct classification models to predict the "Stress Zone" (Low, Medium, or High). This approach allows the system to prioritize clinical severity over precise numeric scores.
Model Comparison & Selection
I evaluated three different algorithms to find the best fit for our high-dimensional dataset:
Detailed Evaluation & Error Analysis
To look beyond simple accuracy, I examined the Confusion Matrix and Classification Report for each model.
Identified Patterns & Mistakes:
Majority Class Bias: The Logistic Regression model exhibits a strong bias towards the "Low Stress" category (Recall: 0.89). While it achieves the highest accuracy, it often misclassifies "Medium" and "High" stress patients as "Low."
The "Grey Area" Challenge: Across all models, there is significant confusion between "Medium" and "High" categories. This confirms that lifestyle data alone has a "ceiling" in its ability to distinguish between moderate and severe distress without physiological real-time input.
KNN Adaptability: Interestingly, KNN achieved the highest recall for the "Medium" class (0.27), suggesting that "proximity-based" logic is better at identifying transitional stress states than linear logic.
The Winning Model: Logistic Regression
Winner: Logistic Regression (Accuracy: 39.8%).
Despite the sophistication of Random Forest, the Logistic Regression model performed best overall. This is because the underlying relationships in this dataset are largely linear and "noisy." Logistic Regression’s simplicity acted as a form of regularization, preventing it from overfitting to the noise in the psychological survey data, unlike the Random Forest which struggled to generalize.
This model proves that while lifestyle data can set a "base prediction," it is the clinical discrepancy (the gap between this prediction and real-time data) that will provide the most value for mental health practitioners.
Final Summary & Research Reflection This project set out to explore to what extent a "Standard Stress Baseline" can be predicted using lifestyle and socio-cultural data. Our findings demonstrate that while these factors provide a significant signal—improving the model's predictive power from a negative baseline to a positive $R^2$ and achieving 40% classification accuracy—the high degree of subjectivity in stress reporting remains a major challenge.
However, the value of this model lies not in "perfect prediction," but in establishing a personalized reference point. By integrating engineered features like the Work_Life_Ratio and unsupervised User_Clusters, we have successfully moved from generic assessments to profile-based baselines. For the algorithm ecosystem, this baseline serves as the essential "starting line": it allows the clinical algorithm to distinguish between a patient's chronic lifestyle stress and acute moments of arousal during therapy. Ultimately, this research proves that while lifestyle data sets the stage, it is the combination of this socio-cultural baseline with real-time physiological data that will enable truly personalized mental health interventions.