5. Evaluation

5.1 Qualitative Evaluation#

5.2 Quantitative Evaluation: NASA TLX & SUS#

Objective

This report aims to evaluate the user experience of the platformer puzzle game “Capoo” at low difficulty (L1) and high difficulty (L2) using quantitative methods, comparing workload and usability differences.

Background

NASA TLX is a tool for measuring subjective workload across six dimensions (Hart & Staveland, 1988). SUS is a reliable usability assessment tool (Brooke, 1986). This study combines both methods to analyze the impact of Capoo’s difficulty on player experience.

Goals

• Quantify the workload (NASA TLX) and usability (SUS) of Capoo at L1 and L2.
• Use statistical tests to determine the significance of differences.

Methodology

Participants

• Number: 10 volunteers.
• Characteristics: Classmates with no specific gaming experience requirements.
• Selection Method: Random recruitment.

Experimental Design

• Difficulty Levels: Capoo includes L1 (low difficulty) and L2 (high difficulty).
• Testing Order:
  • 5 users played L1 first, then L2.
  • 5 users played L2 first, then L1.
  • This minimizes learning effects.

Data Collection

• Tools:
  • NASA TLX: 6 dimensions (raw scores).
  • SUS: 10-question survey.
• Procedure: Each user played one difficulty level and then filled out the NASA TLX and SUS forms, resulting in four scores per participant.

Scoring Method

• NASA TLX: Dimension score = (Rating - 1) × 25, Total score = (∑ Dimension scores) / 6.
• SUS: Odd-numbered questions = Rating - 1, Even-numbered questions = 5 - Rating, Total score = (∑ Score contributions) × 2.5.

Data Analysis

• Tool: Wilcoxon signed-rank test.
• Significance Level: α = 0.05.

Results

• Averages:
  • L1 NASA TLX: 24.58, L2 NASA TLX: 28.75.
  • L1 SUS: 45.5, L2 SUS: 43.5.

Graphical Representation

Statistical Analysis

• NASA TLX:
  • Wilcoxon test result: W = 36 (n=8, excluding zero values).
  • Critical value (n=8, α=0.05): 3.
  • Conclusion: W > 3, no significant difference.
• SUS:
  • Wilcoxon test result: W = 24 (n=8, excluding zero values).
  • Critical value (n=8, α=0.05): 3.
  • Conclusion: W > 3, no significant difference.

Discussion

Interpretation of Results

• Workload: L2 NASA TLX (28.75) is slightly higher than L1 (24.58), mainly due to increased physical demands (e.g., jumping) and mental effort (e.g., puzzle complexity), but the difference is not significant.
• Usability: L1 SUS (45.5) is slightly higher than L2 (43.5), suggesting that the increased difficulty of L2 slightly reduced perceived usability, but not significantly.

Comparison with Expectations

• It was expected that L2 would have a higher workload and lower usability. The observed trend aligns with expectations but does not reach statistical significance, possibly due to insufficient difficulty differences.

Design Insights

• Increase the difficulty of L2 by making jumps and puzzles more challenging to amplify workload differences.
• Optimize L2’s control smoothness (SUS Q1 and Q6 had lower scores) to reduce inconsistencies.

Limitations

• Small sample size (10 participants) limits statistical power.
• The difficulty difference between L1 and L2 may not be significant enough to fully reflect puzzle and platforming challenges.

Conclusion

Capoo’s L2 workload is slightly higher than L1 (28.75 vs 24.58), and its SUS score is lower than L1 (43.5 vs 45.5), but neither difference is statistically significant (NASA TLX W = 36, SUS W = 24, p > 0.05). It is recommended to enhance L2’s difficulty and optimize control experience to improve player immersion.

Appendix

Game Design Updates

1. Enhance L2 Jumping Difficulty: Increase platform height and introduce moving obstacles to heighten physical demand.
2. Optimize Puzzle Consistency: Standardize puzzle hint styles to improve SUS Q4 (consistency) scores.
3. Implement Dynamic Difficulty Adjustment: Adjust jumping and puzzle complexity based on player performance.

5.3 Code Testing and Debugging#

We used black-box testing with equivalence partitioning to validate the game. Test cases were designed based on input types and game states, without looking at the internal code. We focused on transitions (e.g. main menu to level select), controls (e.g. movement, jumping), and interactions (e.g. keys, traps, water). Each feature was tested using representative inputs from different equivalence classes to ensure correct behavior.

5.4 Summary#

The evaluation process confirmed that Capoo is easy to use and cognitively manageable for casual players, even as difficulty increases. While workload and usability metrics did not show significant shifts, qualitative feedback provided actionable insights for polishing level design and improving onboarding. Heuristic evaluation revealed critical UI flaws that were successfully addressed. Ongoing testing ensured stability, performance, and a better user experience across devices.