README.md

---
language: en
license: mit
tags:
- computer-vision
- object-detection
- yolov8
- gesture-recognition
- gaming
pipeline_tag: object-detection
library_name: ultralytics
---
# Model Description
### Overview
This model detects hand gestures for use as input controls for video games. It uses object detection to recognize specific hand poses from a webcam or standard camera and translate them into game actions.
The goal of the project is to explore whether computer vision–based gesture recognition can provide a low-cost and accessible alternative to traditional game controllers.

### Training Approach
The model was trained using the nano version of YOLOv8 (YOLOv8n) through the Ultralytics training framework.
The model was trained from pretrained YOLOv8n weights and fine-tuned on a custom hand gesture dataset.

### Intended Use Cases
* Gesture-controlled video games with simple control schemes
* Touchless interfaces
* Interactive displays
* Public kiosks
* Smart home media controls
* Desktop navigation
***
# Training Data
### Dataset Sources
**The training dataset was constructed from two sources:**

Rock-Paper-Scissors dataset
* Source: Roboflow Universe
* Creator: Audrey
* Used for the first three gesture classes
* Dataset URL: https://universe.roboflow.com/audrey-x3i6m/rps-knmjj

Custom gesture dataset
* Created by recording a 30-second video of the author performing gestures
* Video parsed into frames at 10 frames per second
* Images manually selected and annotated

### Dataset Size
| Category         | Count     |
| ---------------- | --------- |
| Original Images  | 444       |
| Augmented Images | 1066      |
| Image Resolution | 512 × 512 |

### Class Distribution
| Class    | Gesture     | Annotation Count |
| -------- | ----------- | ---------------- |
| Forward  | Open Palm   | 169              |
| Backward | Closed Fist | 210              |
| Jump     | Peace Sign  | 187              |
| Attack   | Thumbs Up   | 121              |

### Data Collection Methodology
The dataset combines stock gesture images with a custom dataset created from recorded video frames.

**The custom dataset was generated by:**
* Recording a short gesture demonstration video
* Extracting frames at 10 FPS
* Selecting usable frames
* Annotating gesture bounding boxes
* This process produced 236 custom images that were merged with the stock dataset.

### Annotation Process
All annotations were created manually using Roboflow.
Bounding boxes were drawn around the visible hand gesture in each image.
Due to failure to import annotation metadata from the original dataset, all 444 images were annotated manually.
Estimated annotation time: 2–3 hours

### Train / Validation / Test Split
| Dataset Split | Image Count |
| ------------- | ----------- |
| Training      | 933         |
| Validation    | 88          |
| Test          | 45          |

### Data Augmentation
**The following augmentations were applied:**
* Rotation: ±15 degrees
* Saturation adjustment: ±30%

*These augmentations expanded the dataset from 444 to 1066 images.*

### Dataset Availability
Dataset availability: https://universe.roboflow.com/b-data-497-ws/hand-gesture-controls

### Known Dataset Biases and Limitations
* Small dataset size
* Class imbalance (thumbs-up has fewer examples)
* Mixed image quality between stock and custom images
* Limited diversity in backgrounds and lighting conditions
* Limited number of subjects (primarily one person)

*These factors may affect model generalization.*
***
# Training Procedure
### Framework
Training was performed in Google Colab using altered Python code from a YOLOv11 training run. Code was taken and altered for YOLOv8n from [here](https://oceancv.org/book/TrainandDeployObj_YOLO.html).

### Model Architecture
Base model: YOLOv8n (Nano)

**Reasons for selection:**
* Lightweight architecture
* Low inference latency
* Lower hardware requirements
* Faster training times
* Suitable for real-time applications

### Training Configuration
| Parameter               | Value                        |
| ----------------------- | ---------------------------- |
| Epochs                  | 200 (training stopped early) |
| Early stopping patience | 10                           |
| Image size              | 512 × 512                    |
| Batch size              | 64                           |

### Training Hardware
| Component     | Specification    |
| ------------- | ---------------- |
| GPU           | A100 (High Ram)  |
| VRAM          | 80 GB            |
| Training Time | ~40 minutes      |

### Preprocessing Steps
* Images resized to 512×512
* Bounding box annotations normalized
* Augmented images generated before training
***
# Evaluation Results
### Overall Metrics

**Final model performance at epoch 41:**
| Metric    | Score |
| --------- | ----- |
| mAP@50    | 0.97  |
| mAP@50–95 | 0.78  |
| Precision | 0.93  |
| Recall    | 0.91  |
| F1 Score  | 0.94  |

*These results exceed the predefined project success criteria.*

**Per-Class Performance**
<img alt= "Per-Class Performance" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/perclass_perf.png" width="1000" height="180"></img>

**Sample Class Images**
<img alt= "Sample Images" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/sample_images.png" width="1100" height="700"></img>

### Key Visualizations
<img alt= "Confusion Matrix" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/confusion_matrix_normalized.png" width="1100" height="700"></img>
<img alt= "F1 Curve" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/BoxF1_curve.png" width="1100" height="700"></img>
<img alt= "Precision-Recall Curve" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/BoxPR_curve.png" width="1100" height="700"></img>

### Performance Analysis
The model achieved high precision and recall across all gesture classes, indicating strong detection performance on the test set.

Several factors contributed to this performance:
* A small number of distinct gesture classes
* Highly visible and consistent hand poses
* A balanced dataset for most classes
However, the dataset size is relatively small, which may inflate evaluation scores and limit generalization.

Failure cases were observed in several situations:
* Complex or cluttered backgrounds
* Low confidence detections
* Ambiguous or blurred gesture poses
These issues highlight areas where the model could be improved with more diverse training data.
***
# Limitations and Biases
### Known Failure Cases
<img alt= "Failure Cases" src="https://huggingface.co/cvtechniques/VideoGameHandGestures/resolve/main/failure_cases.png" width="1100" height="700"></img>
The model struggled with some of the photos from the RPS dataset as these images contain complex backgrounds, partially occluded hands, or ambiguous gestures. 

### Data Biases
Potential biases include:
* limited subject diversity
* similar backgrounds across many images
* dataset partially composed of stock imagery
* limited environmental variability

### Environmental Limitations
Model performance may degrade when:
* lighting conditions vary significantly
* gestures are performed at unusual angles
* hands are partially occluded
* gestures appear at extreme scales or distances

### Inappropriate Use Cases
This model should not be used for:
* complex gesture recognition (complex 3D control schemes)
* sign language recognition
* high-precision human-computer interaction systems
* any safety-critical applications

### Sample Size Limitations
The dataset is relatively small for object detection training, which may limit generalization to new users or environments.
Future improvements to the model would likely be a larger and more diverse dataset. Best course of action would be to remove stock images dataset and culminate gesture videos using diverse individuals, backgrounds, etc.
***
# Future Work
Potential improvements include:
* collecting a larger and more diverse gesture dataset
* increasing the number of gesture classes
* improving image quality and environmental diversity
* exploring hand keypoint detection models instead of object detection
* Keypoint estimation could allow detection of more complex hand gestures and improve gesture recognition accuracy.