In this study, we present a novel and cost-effective method for detecting individual finger movements using a custom pressure sensor array based on Hall effect sensors and magnets. Twelve sensors were placed radially around the forearm, and machine learning techniques, specifically a random forest regressor, were employed to distinguish between finger curls. With only 5 minutes of training data, our approach achieved a mean absolute error (MAE) of 35, demonstrating its potential as a viable and inexpensive sensor solution for finger movement and force detection. This forearm armlet-based system has applications in fields such as prosthetics, robotics, and human-computer interaction, offering a promising alternative to traditional sensor technologies.

Introduction

Hand amputations can significantly impact an individual’s quality of life and ability to perform daily tasks. Despite losing their hand, many people with amputations retain an intact forearm with functional muscle movement. However, due to the disconnected tendons, detecting and interpreting finger movements becomes challenging. In this study, we sought to develop an innovative and cost-effective sensor solution that could discern individual finger movements by sampling the force exerted by muscle contractions in the forearm. One of our main challenges was finding a low-pressure sensor system with minimal friction. We explored different designs and ultimately used a spring and clevis pin system for the best results, while also considering rubber foam for a more scalable solution in future designs.

We hypothesized that with minimal training data, our approach could provide an inexpensive and accessible solution for a wide range of individuals who could potentially benefit from this technology. To test our hypothesis, we conducted experiments on able-bodied individuals with the intention of validating the efficacy of our approach and warranting further research in the context of hand amputees. Our work was conducted using open-source and open hardware principles to contribute to the scientific community with a non-proprietary alternative to existing forearm sensor technologies. This open approach aims to foster collaboration, innovation, and widespread adoption of our proposed forearm bracelet system.

In this paper, we describe the design and implementation of our custom pressure sensor array, the machine learning techniques employed to recognize finger movements, and the performance of our system in detecting individual finger curls. We also discuss potential applications of our technology in prosthetics, robotics, and human-computer interaction, as well as future directions for research and development.

Materials & Methods

Pressure Sensors:

Custom pressure sensors were designed using Hall Effect sensors mounted on printed circuit boards (PCBs). These PCBs were attached to PLA 3D printed clevis pin holders, which featured a hole for the clevis pin piston positioned 7mm above the Hall Effect sensor. A nail head “foot” was secured to the piston, and a 5mm diameter earth magnet was glued to the foot. The piston was fitted into the clevis pin holder, and a spring and clip were attached to the opposite side. The feet were designed to make contact with the skin, and their surface area could be adjusted to modify the sensor’s overall amplitude.

A second version of the sensor utilized the same PCB but incorporated a rubber foam spring system, a rubber sheet, and a magnet. This alternative design yielded promising results and is recommended for further investigation, as its construction requires less time.

Sensor Array and Data Acquisition: The magnet array was wired together and connected to an ESP32-S2 microcontroller, which utilized MicroPython firmware and its 20 13-bit ADCs to measure the Hall sensor values.

Machine Learning Model

Data Collection:

A Python UDP server was used to collect data from the sensor array band. The data was labeled using a Label Seeking (LASK) system, which was built on a rehabilitation device that consisted of four pistons with springs designed to strengthen the muscles responsible for finger movements. Hall Effect sensors were also incorporated into the LASK system. By providing labels to the machine learning regressor, the LASK system enabled us to identify the specific finger movements we were attempting to recognize.

Model Training:

The Scikit-Learn Python library was employed for machine learning model development. A random forest regressor was selected for training the data due to its versatility and robustness in handling complex relationships between input features and output predictions. The labeled data collected from the LASK system was used to train the random forest regressor, which aimed to identify and distinguish individual finger movements based on the pressure sensor data obtained from the forearm.

Data Collection and Transmission

The 12 sensor data points, along with the corresponding time data, were transmitted over UDP packets to a central computer running a collection server. Simultaneously, the LASK system, responsible for applying labels to the training data, sent its 4 sensor readings, time data, and processor ticks.

Data Pairing and Preprocessing

The data was saved in a JSON format and then passed through a sorting algorithm to match the pairs of data points based on their closest proximity in time. Unmatched data points were discarded. The paired data was then converted into a CSV format to be used as input for the random forest training program in Python.

Feature Selection and Dataset Preparation

The Open Muscle and LASK data pair CSV file was processed to create a feature dataset with 13 attributes, comprising 12 sensor data points and 1 time data point from the Open Muscle system, and a labeled dataset with 5 attributes, including 4 sensor data points and 1 time data point from the LASK system.

Model Training and Evaluation

The random forest regressor was trained using the prepared feature and labeled datasets. An 80/20 split was applied to the original dataset for training and testing purposes. The model’s performance was assessed using mean absolute error (MAE) as a metric. The MAE was found to have an average of 40, with a range between 25-80, depending on the quality of the data.

Testing with Prerecorded and Live Data

To further evaluate the model’s performance, tests were conducted on both prerecorded and live data. The random forest regressor demonstrated its capability to accurately detect finger movements in real-time scenarios, thereby showcasing its potential as a viable and cost-effective sensor solution for finger movement and force detection.

Training dataset with the best results was taken during a recording that was posted on youtube:

We also tested a version that used foam instead of springs.

Initial Test Results

All files needed to test the data found in the demo folder on the GitHub repository.

https://github.com/turfptax/openmuscle/tree/main/Demo

Results for Training on capture_11.csv (~5 minutes of data)

• Mean Squared Error: 7689.885189928801
• Mean Absolute Error: 29.919568537049315

Results for Training on January23.csv (~10 minutes of data)

All of the files are available on github to train the model and run the predictions. https://github.com/turfptax/openmuscle/tree/main/Demo

• Mean Squared Error: 6131.50072204039
• Mean Absolute Error: 23.225226598136324
• Training session video was also recorded here: https://www.youtube.com/watch?v=BRkbyyItcVQ)

The results for the training on January 23rd were also manually split up into sections and fed into the random forest regressor to ensure that there was not an error with the data split. The data yielded the same results.

predictions_vs_actualsDownload
predictions_vs_actuals-Short-SegmentDownload
predictions_vs_actuals-Short-Segment_shorterDownload