Since utilizing the mouse involves monotonous movements of the same small muscle groups for protracted periods of time, overusing the mouse by positioning, traveling, scrolling, and clicking can make these muscle groups tired and overworked. Hence, after repetitive overexposure to monotonous mouse movements, muscle stress (soreness) increases precipitously and can engender[1]:

Also, it can cause soreness and fatigue by putting extra strain on the muscles in the upper back (trapezius muscle) and shoulder (deltoid muscle).

Generally, the aforementioned symptoms are temporary and get better on their own after reducing mouse use. Nevertheless, overexposure to monotonous mouse movements can cause more noxious and enervating disorders such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS).

Repetitive strain injury (RSI) results from forceful, awkward, and/or repetitive use of your limbs, producing damaged muscles, tendons, and nerves. Although RSI is a broad term that encompasses several disorders, general symptoms include tingling or loss of sensation in fingers, inability to grasp objects between thumb and fingers, decrease in the size of hand muscles, and pain in the wrist, elbow, shoulder, or neck. The severity of RSI cases varies widely. Tendonitis is the most common example of RSI, while Carpal tunnel syndrome is a more rare and serious disorder.

Carpal tunnel syndrome (CTS) and Thoracic outlet syndrome (TOS) are two of the most disabling repetitive strain injuries. These conditions are disorders of the tendons, nerves, arteries, or veins, occurring at the wrist and upper arm, respectively. In CTS, repeated bending or use of the wrist and fingers results in the compression of the median nerve (runs along the palm side of the wrist), causing intermittent numbness, tingling, and pain in the side of the hand, including the thumb through the inside of the ring finger[2].

After scrutinizing recent research papers on mouse fatigue, I decided to utilize GSR (galvanic skin response) and EMG (Electromyography) measurements denoting forearm muscle soreness and create a budget-friendly device to forecast muscle soreness levels in the hope of averting permanent mouse-related injuries.

Since muscle soreness levels depending on GSR and EMG measurements fluctuates for different muscle groups, it is not possible to extrapolate and interpret forearm muscle soreness levels by merely employing limited data without applying complex algorithms. Therefore, I decided to build and train an artificial neural network model by utilizing the empirically assigned soreness classes to predict forearm muscle soreness levels based on GSR and EMG measurements.

I decided to employ Wio Terminal in this project since it can easily collect muscle soreness data (GSR and EMG measurements) and run my neural network model after being trained to forecast forearm muscle soreness levels. To obtain the required data to train my model, I connected a GSR sensor (Grove) and an EMG detector (Grove) to Wio Terminal. Also, Wio Terminal can display the collected muscle soreness data on its integrated TFT LCD screen.

Since Wio Terminal supports reading and writing information from/to files on an SD card, I stored the collected data in a CSV file on the SD card to create a data set. In this regard, although Wio Terminal provides Wi-Fi connectivity, I was able to save data packets via Wio Terminal without requiring any additional procedures.

After completing my data set, I built my artificial neural network model (ANN) with TensorFlow to make predictions on forearm muscle soreness levels (classes) based on GSR and EMG measurements. As labels, I employed the empirically assigned muscle soreness classes for each data record while collecting data:

After training and testing my neural network model, I converted it from a TensorFlow Keras H5 model to a C array (.h file) to execute the model on Wio Terminal. Therefore, the device is capable of detecting precise forearm muscle soreness levels (classes) by running the model independently.

Lastly, to make the device as compact and robust as possible while operating, I designed a Pikachu-inspired case (3D printable).

So, this is my project in a nutshell 😃

In the following steps, you can find more detailed information on coding, logging data on the SD card, building an artificial neural network model with TensorFlow, and running it on Wio Terminal.

🎁🎨 Huge thanks to Seeed Studio for sponsoring these products:

⭐ Wio Terminal | Inspect

⭐ Grove - GSR Sensor | Inspect

⭐ Grove - EMG Detector | Inspect

🎁🎨 Also, huge thanks to Creality3D for sponsoring a Creality CR-6 SE 3D Printer.

🎁🎨 If you want to purchase some products from Creality3D, you can use my 10% discount coupon (Aktar10) even for their new and most popular printers: CR-10 Smart, CR-30 3DPrintMill, Ender-3 Pro, and Ender-3 V2.

🎁🎨 You can also use the coupon for Creality filaments, such as Upgraded PLA (200g x 5 Pack), PLA White, and PLA Black.

Step 1: Designing and printing a Pikachu-inspired case

Since I wanted to collect forearm muscle soreness data and run the model while working or studying, I decided to design a Pikachu-inspired case to create a compact and sturdy device operating flawlessly on my desk. I thought adding Pikachu to my case would be funny due to Pikachu's notorious Thunder Shock attack stunning the opponent's muscles.

I designed the main case and its back cover in Autodesk Fusion 360. You can download their STL files below.

For the Pikachu affixed to the main case, I utilized this model from Thingiverse:

Then, I sliced 3D models (STL files) in Ultimaker Cura.

Since I wanted to create a solid structure for the case and apply a stylish theme to the device, I utilized these PLA filaments:

Finally, I printed all parts (models) with my Creality CR-6 SE 3D Printer. Although I am a novice in 3D printing and it is my first FDM 3D printer, I got incredible results effortlessly with the CR-6 SE :)

Step 1.1: Assembling the case and making connections & adjustments

// Connections// Wio Terminal :
//                                Grove - GSR sensor
// A0  --------------------------- Grove Connector
//                                Grove - EMG Detector
// A2  --------------------------- Grove Connector

To collect and display forearm muscle soreness data, I connected the GSR sensor (Grove) and the EMG detector (Grove) to Wio Terminal. Since Wio Terminal has one Grove port supporting analog sensors, I connected the EMG (Electromyography) detector via a Grove male jumper conversion cable, as shown in the schematics below.

GSR (galvanic skin response) sensor needs to be calibrated by adjusting the integrated resistor until the sensor generates 512 as the output signal on the serial monitor before wearing it.

You can get more information regarding coding in Step 3.

After printing all parts (models) and completing sensor connections successfully, I fastened all components to the main case and made connection points rigid by utilizing screws and a hot glue gun.

Finally, I attached the back cover to the main case via screws and affixed Pikachu to the top via the hot glue gun.

Step 2: Setting up Wio Terminal

Since Wio Terminal supports reading and writing information from/to files on an SD card, I decided to utilize a CSV file on the SD card so as to log the collected forearm muscle soreness data without applying any additional procedures. However, before proceeding with the following steps, I needed to set up Wio Terminal on the Arduino IDE and install the required libraries for this project.

#️⃣ To set up the Seeed SAMD Arduino Core, open the Arduino IDE, click Tools ➡ Board ➡ Boards Manager, and search for Wio Terminal in the search box. Then, install Seeed SAMD Boards.

#️⃣ Download the required libraries for Wio Terminal:

Seeed_Arduino_FS | Download

Seeed_Arduino_Linechart | Download

Step 2.1: Loading and displaying images from the SD card

To display images on the TFT LCD screen integrated into Wio Terminal successfully, I needed to convert them to a compatible BMP file format and then load them from the SD card.

#️⃣ First, open Microsoft Paint to rescale images to the required sizes and save them as the 24-bit bitmap (.bmp) files in the bmp folder.

#️⃣ To convert the 24-bit bitmap (.bmp) files to Wio Terminal's required BMP file format, download the bmp_converter.py file and save it to the bmp folder.

#️⃣ Then, modify the folder location in the bmp_converter.py file and run it.

#️⃣ Enter 1 for 8-bit BMP file format conversion or 2 for 16-bit BMP file format conversion.

#️⃣ Finally, the bmp_converter.py file converts all the given 24-bit bitmap (.bmp) files and saves them to rgb332 (8-bit) or rgb565 (16-bit) folders in the bmp folder.

#️⃣ To display the converted BMP files on the TFT LCD screen, move them to the SD card.

#️⃣ Then, copy the RawImage.h file to the sketches on the Arduino IDE.

// To draw the 8-bit color image on screen, starting from point (x, y):
drawImage<uint8_t>("path to sd card image", x, y); 

// To draw the 16-bit color image on screen, starting from point (x, y):
drawImage<uint16_t>("path to sd card image", x, y);

Step 3: Collecting and storing forearm muscle soreness data by GSR and EMG w/ Wio Terminal

Step 3.1: Logging forearm muscle soreness data in a CSV file on the SD card

After uploading and running the code for collecting forearm muscle soreness data and saving information to a given CSV file on the SD card on Wio Terminal:

💪💻 The device displays real-time GSR and EMG measurements as line charts on the TFT screen.

💪💻 If Button A (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Relaxed [0] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:

💪💻 If Button B (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Tense [1] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:

💪💻 If Button C (configurable button) is pressed, the device appends a new row (data record) to the mouse_fatigue_data_set.csv file on the SD card by adding the Exhausted [2] muscle soreness class under the Soreness data field. Then, if the device saves the data record successfully to the given CSV file on the SD card, it shows:

💪💻 If Wio Terminal cannot open the given CSV file successfully, the device displays the error message on the TFT screen.

💪💻 Also, the device prints notifications and sensor measurements on the serial monitor for debugging.

After logging the collected forearm muscle soreness data in a CSV file on the SD card for 15 days, I elicited my data set with eminent validity :)

Step 4: Building an Artificial Neural Network (ANN) with TensorFlow

When I completed collating my forearm muscle soreness data set and assigning labels, I had started to work on my artificial neural network model (ANN) to make predictions on muscle soreness levels (classes) based on GSR and EMG measurements.

I decided to create my neural network model with TensorFlow in Python. Thus, first of all, I followed the steps below to grasp a better understanding of my data set so as to train my model accurately:

As explained in the previous steps, I assigned muscle soreness classes empirically while logging data with the device. Since the assigned classes are stored under the Soreness data field in the mouse_fatigue_data_set.csv file, I preprocessed my data set effortlessly to assign labels for each data record (input):

After scaling (normalizing) and preprocessing inputs, I obtained two input variables and one label for each data record in my data set. Then, I built an artificial neural network model with TensorFlow and trained it with my training data set to acquire the best results and predictions possible.

Layers:

To execute all steps above and convert my model from a TensorFlow Keras H5 model to a C array (.h file) so as to run it successfully on Wio Terminal, I developed an application in Python. As shown below, the application consists of three code files and three folders:

First of all, I created a class named Mouse_Fatigue in the main.py file to execute the following functions precisely.

⭐ Include the required modules.

import tensorflow as tffrom tensorflow import kerasimport matplotlib.pyplot as pltimport numpy as npimport pandas as pdfrom tflite_to_c_array import hex_to_c_arrayfrom test_data import test_inputs, test_labels

⭐ In the __init__ function, define the required variables to build the neural network model and read the forearm muscle soreness data set from the given CSV file.

def __init__(self, csv_path):        self.inputs = []        self.labels = []        self.model_name = "mouse_fatigue_level"        self.scale_val = 1000        # Read the collated soreness data set (GSR and EMG):        self.df = pd.read_csv(csv_path)

I will elucidate each file and function in detail in the following steps.

Note: The bmp folder is thoroughly explained in Step 2.1.

Step 4.1: Visualizing the forearm muscle soreness data set

Since it is essential to understand a given data set to pass appropriately formatted inputs and labels to a neural network model, I decided to visualize my data set and scale (normalize) it in Python after extracting it from the mouse_fatigue_data_set.csv file saved in the data folder.

⭐ In the graphics function, visualize the requested data column (field) from the given data set by utilizing the Matplotlib library.

def graphics(self, column_1, column_2, x_label, y_label):        # Show the requested data column from the data set:        plt.style.use("dark_background")        plt.gcf().canvas.set_window_title('Mouse Fatigue Estimation by GSR and EMG Values')        plt.hist2d(self.df[column_1], self.df[column_2], cmap="coolwarm")        plt.colorbar()        plt.xlabel(x_label)        plt.ylabel(y_label)        plt.title(x_label)        plt.show()

⭐ In the data_visualization function, scrutinize all data columns (fields) before scaling and preprocessing the given data set so as to build a neural network model with appropriately formatted data.

def data_visualization(self):        # Scrutinize data columns to build a model with appropriately formatted data:        self.graphics('GSR', 'EMG', 'GSR', 'EMG')

Step 4.2: Assigning labels and scaling (normalizing) data records to create inputs

After visualizing my data set, I needed to create inputs from data records to train my neural network model. Therefore, I utilized these two data elements to create inputs:

Then, I scaled (normalized) each data element to format them properly and thus extracted these scaled data elements from my data set for each data record:

⭐ In the scale_data_and_define_inputs function, divide every data element into their required values so as to make them smaller than or equal to 1.

⭐ Then, create inputs with the scaled data elements, append them to the inputs array, and convert this array to a NumPy array by using the asarray function.

⭐ Each input includes two parameters [shape=(2, )]:

def scale_data_and_define_inputs(self):        self.df["scaled_GSR"] = self.df["GSR"] / self.scale_val        self.df["scaled_EMG"] = self.df["EMG"] / self.scale_val        # Create the inputs array by utilizing the scaled variables:        for i in range(len(self.df)):            self.inputs.append(np.array([self.df["scaled_GSR"][i], self.df["scaled_EMG"][i]]))        self.inputs = np.asarray(self.inputs)

As explained in the previous steps, I assigned muscle soreness classes for each data record under the Soreness data field to be utilized as labels:

⭐ In the define_and_assign_labels function, get predefined labels [0 - 2] under the Soreness data field for each input and append them to the labels array.

def define_and_assign_labels(self):        self.labels = self.df["Soreness"]

Step 4.3: Training the model (ANN) on muscle soreness levels (classes)

Since my forearm muscle soreness data set is already limited, I decided to utilize all of my data set as the training data set instead of splitting it into training and testing data sets. Thus, I created a separate testing data set in the test_data.py file.

After defining the training data set, I scaled (normalized) the testing data set inputs to format them appropriately.

def split_data(self):        # (training)        self.train_inputs = self.inputs        self.train_labels = self.labels        # (test)        self.test_inputs = test_inputs / self.scale_val        self.test_labels = test_labels

Then, I built my artificial neural network model (ANN) by utilizing Keras and trained it with the training set for 150 epochs.

You can inspect these tutorials to learn about activation functions, loss functions, epochs, etc.

def build_and_train_model(self):        # Build the neural network:        self.model = keras.Sequential([            keras.Input(shape=(2,)),            keras.layers.Dense(64, activation='relu'),            keras.layers.Dense(32, activation='relu'),            keras.layers.Dense(8, activation='relu'),            keras.layers.Dense(3, activation='softmax')        ])        # Compile:        self.model.compile(optimizer='adam', loss="sparse_categorical_crossentropy", metrics=['accuracy'])        # Train:        self.model.fit(self.train_inputs, self.train_labels, epochs=150)			...

After training with the training set (inputs and labels), the accuracy of my neural network model is between 0.83 and 0.88.

Step 4.4: Evaluating the model accuracy and converting the model to a C array

After building and training my artificial neural network model, I tested its accuracy and validity by utilizing the testing data set (inputs and labels).

The evaluated accuracy of the model is 0.9375.

        ...		        # Test the model accuracy:        print("\n\nModel Evaluation:")        test_loss, test_acc = self.model.evaluate(self.test_inputs, self.test_labels)         print("Evaluated Accuracy: ", test_acc)

After evaluating my neural network model, I saved it as a TensorFlow Keras H5 model (mouse_fatigue_level.h5) to the model folder.

def save_model(self):        self.model.save("model/{}.h5".format(self.model_name))

However, running a TensorFlow Keras H5 model on Wio Terminal to make predictions on muscle soreness levels is not eligible and efficient considering size, latency, and power consumption.

Thus, I converted my neural network model from a TensorFlow Keras H5 model (.h5) to a TensorFlow Lite model (.tflite). Then, I modified the TensorFlow Lite model to create a C array (.h file) to run the model on Wio Terminal successfully.

To revise the TensorFlow Lite model as a C array, I applied the hex_to_c_array function copied directly from this tutorial to the tflite_to_c_array.py file.

⭐ In the convert_TF_model function, convert the recently trained and evaluated model to a TensorFlow Lite model by applying the TensorFlow Lite converter (tf.lite.TFLiteConverter.from_keras_model).

⭐ Then, save the generated TensorFlow Lite model to the model folder (mouse_fatigue_level.tflite).

⭐ Modify the saved TensorFlow Lite model to a C array (.h file) by executing the hex_to_c_array function.

⭐ Finally, save the generated C array to the model folder (mouse_fatigue_level.h).

def convert_TF_model(self, path):        #model = tf.keras.models.load_model(path + ".h5")        converter = tf.lite.TFLiteConverter.from_keras_model(self.model)        #converter.optimizations = [tf.lite.Optimize.DEFAULT]        #converter.target_spec.supported_types = [tf.float16]        tflite_model = converter.convert()        # Save the recently converted TensorFlow Lite model.        with open(path + '.tflite', 'wb') as f:            f.write(tflite_model)        print("\r\nTensorFlow Keras H5 model converted to a TensorFlow Lite model!\r\n")        # Convert the recently created TensorFlow Lite model to hex bytes (C array) to generate a .h file string.        with open("model/{}.h".format(self.model_name), 'w') as file:            file.write(hex_to_c_array(tflite_model, self.model_name))        print("\r\nTensorFlow Lite model converted to a C header (.h) file!\r\n")

Step 5: Setting up the model on Wio Terminal

Step 6: Running the model on Wio Terminal to make predictions on muscle soreness levels

My neural network model predicts possibilities of labels (muscle soreness classes) for each given input as an array of 3 numbers. They represent the model's "confidence" that the given input array corresponds to each of the three different muscle soreness classes based on GSR and EMG measurements [0 - 2], as shown in Step 4:

After importing and setting up my neural network model as a C array on Wio Terminal successfully, I utilized the model to run inferences to forecast muscle soreness levels.

After executing the code for running inferences on Wio Terminal:

💪💻 The device displays real-time GSR and EMG measurements as line charts on the TFT screen.

💪💻 If the 5-way switch is pressed, the device runs inference with the model by employing the most recently generated GSR and EMG measurements as the input.

💪💻 Then, the device displays the output, which represents the most accurate label (muscle soreness class) predicted by the model.

💪💻 Each muscle soreness level (class) has a unique image (16-bit BMP file) and color code to be shown on the TFT screen when being detected as the output:

As far as my experiments go, the device operates impeccably while predicting forearm muscle soreness levels (classes) :)

Videos and Conclusion

After completing all steps above and experimenting, I have employed the device to predict and detect forearm muscle soreness levels so as to get prescient warnings to prevent permanent or temporary disorder risks related to mouse overuse such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS).

Further Discussions

By applying neural network models trained on GSR and EMG measurements in detecting muscle soreness levels while using the mouse, we can achieve to:

💪💻 alleviate the exacerbating effects of mouse overuse,

💪💻 palliate temporary symptoms of overexposure to monotonous mouse movements,

💪💻 prevent permanent or temporary disorder risks related to mouse overuse such as Repetitive strain injury (RSI) and Carpal tunnel syndrome (CTS),

💪💻 avert permanent mouse-related injuries.

References

[1] Computer Mouse - Common Problems from Use, Canadian Centre for Occupational Health & Safety, June 2017, https://www.ccohs.ca/oshanswers/ergonomics/office/mouse/mouse_problems.html.

[2] Ergonomics & Computer Use, The Trustees of Princeton University, https://uhs.princeton.edu/health-resources/ergonomics-computer-use.