Capoo Quantitative Evaluation Report

Workload and Usability Analysis of Difficulty Levels L1 and L2

Abstract

This report evaluates the workload and usability of the platformer puzzle game “Capoo” at two difficulty levels (L1 and L2) using NASA TLX and SUS. Ten classmates participated in the test, and data were analyzed using the Wilcoxon signed-rank test. Results indicate that L2 had a slightly higher workload than L1 (28.75 vs 24.58), but the difference was not significant (W = 36, p > 0.05). Similarly, L2 had a lower SUS score than L1 (43.5 vs 45.5), but the difference was also not significant (W = 24, p > 0.05). This suggests that the difficulty variation in Capoo has a limited impact on user experience.

Introduction

Objective

This report aims to evaluate the user experience of the platformer puzzle game “Capoo” at low difficulty (L1) and high difficulty (L2) using quantitative methods, comparing workload and usability differences.

Background

NASA TLX is a tool for measuring subjective workload across six dimensions (Hart & Staveland, 1988). SUS is a reliable usability assessment tool (Brooke, 1986). This study combines both methods to analyze the impact of Capoo’s difficulty on player experience.

Goals

• Quantify the workload (NASA TLX) and usability (SUS) of Capoo at L1 and L2.
• Use statistical tests to determine the significance of differences.

Methodology

Participants

• Number: 10 volunteers.
• Characteristics: Classmates with no specific gaming experience requirements.
• Selection Method: Random recruitment.

Experimental Design

• Difficulty Levels: Capoo includes L1 (low difficulty) and L2 (high difficulty).
• Testing Order:
  • 5 users played L1 first, then L2.
  • 5 users played L2 first, then L1.
  • This minimizes learning effects.

Data Collection

• Tools:
  • NASA TLX: 6 dimensions (raw scores).
  • SUS: 10-question survey.
• Procedure: Each user played one difficulty level and then filled out the NASA TLX and SUS forms, resulting in four scores per participant.

Scoring Method

• NASA TLX: Dimension score = (Rating - 1) × 25, Total score = (∑ Dimension scores) / 6.
• SUS: Odd-numbered questions = Rating - 1, Even-numbered questions = 5 - Rating, Total score = (∑ Score contributions) × 2.5.

Data Analysis

• Tool: Wilcoxon signed-rank test.
• Online Calculator: Statology Wilcoxon Test Calculator
• Significance Level: α = 0.05.

Results

Data Overview

• Averages:
  • L1 NASA TLX: 24.58, L2 NASA TLX: 28.75.
  • L1 SUS: 45.5, L2 SUS: 43.5.

Statistical Analysis

• NASA TLX:
  • Wilcoxon test result: W = 36 (n=8, excluding zero values).
  • Critical value (n=8, α=0.05): 3.
  • Conclusion: W > 3, no significant difference.
• SUS:
  • Wilcoxon test result: W = 24 (n=8, excluding zero values).
  • Critical value (n=8, α=0.05): 3.
  • Conclusion: W > 3, no significant difference.

Discussion

Interpretation of Results

• Workload: L2 NASA TLX (28.75) is slightly higher than L1 (24.58), mainly due to increased physical demands (e.g., jumping) and mental effort (e.g., puzzle complexity), but the difference is not significant.
• Usability: L1 SUS (45.5) is slightly higher than L2 (43.5), suggesting that the increased difficulty of L2 slightly reduced perceived usability, but not significantly.

Comparison with Expectations

• It was expected that L2 would have a higher workload and lower usability. The observed trend aligns with expectations but does not reach statistical significance, possibly due to insufficient difficulty differences.

Design Insights

• Increase the difficulty of L2 by making jumps and puzzles more challenging to amplify workload differences.
• Optimize L2’s control smoothness (SUS Q1 and Q6 had lower scores) to reduce inconsistencies.

Limitations

• Small sample size (10 participants) limits statistical power.
• The difficulty difference between L1 and L2 may not be significant enough to fully reflect puzzle and platforming challenges.

Conclusion

Capoo’s L2 workload is slightly higher than L1 (28.75 vs 24.58), and its SUS score is lower than L1 (43.5 vs 45.5), but neither difference is statistically significant (NASA TLX W = 36, SUS W = 24, p > 0.05). It is recommended to enhance L2’s difficulty and optimize control experience to improve player immersion.

Appendix

Game Design Updates

1. Enhance L2 Jumping Difficulty: Increase platform height and introduce moving obstacles to heighten physical demand.
2. Optimize Puzzle Consistency: Standardize puzzle hint styles to improve SUS Q4 (consistency) scores.
3. Implement Dynamic Difficulty Adjustment: Adjust jumping and puzzle complexity based on player performance.

Raw Data

L1 NASA TLX

L2 NASA TLX

L1 SUS

L2 SUS