viEwMotion

Attempting to build a sensor platform that is easier to integrate, more compact, etc. Taking inspiration from Curt White's "Hack $35 Activity Trackers for Mental Health & More"

Using the research that Curt has already done and made available I hope to make a simpler and less expensive wrist band sensor.

A small, wearable, wristband housing sensors, microcontroller and BLE transmitter.

I successfully etched a new board (after thickening traces, etc as much as possible).  I cleaned and etched the board, re-flowed all the SMD parts, drilled holes for jumpers and header pins (and hand soldered all of that stuff) it was time to check everything under the microscope (everything looked good, including continuity across ground and power nets).

Now to plug the board into the Adafruit Feather board... again nothing seemed to work!  The board connected to my PC without a problem, loading and running simple programs like Blink also worked fine.  Time to look deeper!  Fairly quickly I found that I was getting no voltage out at the 3.3v pin even though continuity from the board to the pin seems good.

Just in case I re-built the circuit on a solder-less breadboard and hooked it up to an Arduino Uno (actually I tested the pulse sensor functions first, then the GSR and finally both together) thankfully my circuits and data capture software all still functions correctly.

In the morning I'll retest the Feather Wing board by hooking it up to the Arduino Uno to see if it actually works, if not I'll need to see what's wrong and then figure out why my Adafruit Feather board is not supplying voltage through the 3.3v pin.

Outside of facial expression, what are the actual physiological measures that correlate with emotional response. The first thing most of us think of involve a Polygraph. While a Polygraph can be useful, it's main focus is on detection of stress or arousal (like in a Fight or Flight response, only more subtle). So, a Polygraph can determine areas that elicit a stress or arousal response in a subject but they are not really designed to determine the actual emotional response of a subject. In addition

Current theories on emotional response and categorization focus on 2 dimensions: Arousal and Valance.

Distinguish between 4 emotional states:

picture

Sympathetic – Fight or Flight

Parasympathetic – Rest and Digest

Sympathetic and Parasympathetic responses to emotion (present as a chart)

Sympathetic Responses

• Heart rate increases
• Respiration increases
• Blood Pressure increases
• Skin conductance decreases
• Decreased circulation to extremities

Parasympathetic Responses

• Decreased Heart Rate
• Decreased Blood Pressure
• Decreased Respiration
• Decreased skin conductance
• Increased circulation to extremities

This is not an exhaustive list but I will continue to add to it.

Emotion and Physiology

• Neuro-Tools: Emotion Detection January 16, 2017 https://acuityets.wordpress.com/2017/01/16/neuro-tools-emotion-detection/ accessed 3/28/17
• Broek E, Schut MH, Westerink J, Tuinenbrejier K. Unobtrusive Sensing of Emotion (USE). Journal Of Ambient Intelligence and Smart Environments (2009) 287-299.
• Kreiberg S, Autonomic nervous system in emotion: A Review. Biological Psychology 84(2010) 394-421
• Levenson RW, The Autonomic Nervous System and Emotion. Emotion Review, vol , no. (April 2014) 100-112.

Heart Rate Detection (ECG and Photoplysmography)

• Texas Instruments Application Report SLAA655-March 2015
• How to Design Peripheral Oxygen Saturation (SpO2) and Optical Heart Rate Monitoring (OHRM) Using the AFE4403
• Arduino Measures Heart Beat Rate From Fingertip http://embedded-lab.com/blog/arduino-heart-rate-meter-seven-segment-led-display/
• http://makezine.com/projects/ir-pulse-sensor/

GSR/EDA

• A Wearable Sensor for Unobtrusive, Long-Term Assessment of Electrodermal Activity. IEEE Transactions on Biomedical Engineering. Vol. 57 No.5 May 2010.
• https://hackaday.io/projects/6701/logs LMV324 for GSR/EDA
• https://www.etc.cmu.edu/projects/biofeedback/research/gsr.doc
• ttp://www.trans4mind.com/psychotechnics/gsr.html
• https://www.media.mit.edu/galvacticator/faq.html

Heart Rate Variability

• Goss CF & Miller FB. Dynamic metrics of heart rate variability. August 29, 2013, 4 pages. ArXiv:1308.6018.
• Lee C, Yoo SK, Park Y, Kim N, Jeong L, Lee B. Using Neural Network to Recognize Human Emotions from Heart Rate Variability and Skin Resistance. Proceedings of the 2005 IEEE, Engineering in Medicine and Biology 27^th Annual Conference, Shanghai China, Sept. 1-4,2005. p5523-5525
• Bailon R, Laouini G, Groa C, Orini M, Laguna P, Meste O. The Integral Pulse Frequency Modulation Model with Time Varying Threshold: Application to Heart Rate Variability Analysis During Exercise Stress Testing. IEEE Transactions on Biomedical Engineering, Vol 58, No 3 March 2011. p642-652
• Linares L, Medez AJ, Lado MJ, Oliviera DN, Vila XA, Conde I. An open source tool for heart rate variability spectral analysis. Computer Methods and Programs in Biomedicine 103 (2011) 39-50
• Zhao M, Adib F, Katabi D. Emotion Recognition using Wireless Signals. MobiCom'16, Oct 3-7, 2016 http://dx.doi/10.1145/29373750.2973762
• Valderas MT, Bolea J, Laguna P, Vallverdu M, Bailon R. Human emotion recognition using heart rate variability analysis with spectral bands based on respiration.
• Lane RD, McRae K, Reiman EM, Chen K, Athern GL, Thayer JF. Neural correlates of heart rate variability during emotion. NeroImage 44 (2009) 213-222
• Orini M, Bailon R, Enk R, Koelsch S, Mainardi L, Laguna P. A method for continuously assessing the autonomic response to music-induced emotions through HRV analysis. Med Biol Eng Comput (2010) 48-423-433
• Williams DP, Cash C, Rankin C, Bernardi A, Koenig J, Thayer JF. Resting heart rate variability predicts self-reported difficulties in emotion regulation: a focus on different facets of emotion regulation. Frontiers in Psychology, March 2015, vol 6, article 261
• Valenza G, Citi L, Lanata A, Scilingo EP, Barbieri R. Revealing Real-Time Emotional Responses: a Personalized Assessment based on Heartbeat Dynamics. Scientific Reports, 4:4998, DOI: 10.1038/srep04998

It's been quite a long time coming but I've finally got some updates on this project.  I've been repeatedly delayed due to a vacation, massive computer failures, waiting for a new computer (then loading all the software etc.) and finally an illness.  None of this has completely stopped me from working on this project but it all has definitely slowed me down.

After experimenting with the PulseSensorAmped (a great project and the documentation is really great) I decided to go back to using the MAXREFDES117.  The main reasons for this decision are:

• more pre-processing in the breakout
• easier control of the light intensity settings, etc.

In addition I've decided to switch to a different circuit for GSR/EDC that makes some aspects of GSR measurement and interpretation simpler:

• self adjusting to background skin resistance (tonic?)
• easier to use

Circuit was pulled from research paper (link), with the addition of the precision voltage divider.  In the same paper, the authors, looked at other testing sites for GSR/EDA that were comparable to the palm and fingers but allowed the subject greater freedom of movement.  There testing revealed that the palmar wrist area proximal (near) to the hand worked as well as the fingers and palm.  They even had success with a subject using a wrist strap for a week of continuous readings while performing normal daily activities.

In initial tests I used the above circuit along with the MAXREFDES117 breakout board, both connected to the Adafruit Feather Bluefruit, to develop a simple program to make readings of GSR and HRV and send them back to the computer for interpretation.  (The ultimate goal, obviously, is to send the information via BLE back to a processor).

I put this together after working out a number of bugs while building the circuits on a solder-less breadboard and testing different configurations. 

One major issue, turned out to be the lack of insulation on the face of the MAXREFDES117 board that contacts the skin (used by itself it would not be an issue, but when coupled with GSR measurements it added current and potential the the GSR circuit).  Once I insulated all the possible skin contacts in the MAXREFDES117 all the GSR data returned to normal.

I fabricated one initial (singled sided) board  that mounts directly onto the Adafruit Feather Bluedfruit so that I could start more comprehensive testing of individual GSR and HRV changes in relation to emotional stimulation.  After designing the board, etching it, populating it, re-flowing it, drilling out holes, soldering jumper wires, soldering pins to attach to the Adafruit Feather... something went wrong!

It turned out that either during etching, re-flow, drilling or hand soldering -  a number of thinner tracks were either cut or pulled up.  So, I had to re-etch a new board, populate it and drill out the holes for the connecting pins to the Feather board and the smaller holes for the jumper wires.  This is where I'm at right now.  I'll have more to post in the afternoon (once I have the jumpers and header pins done)!!!

Initially i visualized a watch like design to house the sensors etc. that could be worn but I've now moved to using a wristband instead. The reasons for this change are: comfort (a soft, mildly stretchy wrist band - think sweat band - is more comfortable than a watch band) and better sensor placement for the heart rate sensor and for the GSR sensor.

In addition to the above changes, I've also decided to switch to the Maxim breakout board (MAXREFDES#117) due to its better stability (and my better understanding about how to use it) as well as switching to a different circuit design for the GSR sensor module.

The above circuit is taken from: A Wearable Sensor for Unobtrusive, Long-Term Assessment of Electrodermal Activity. IEEE Transactions on Biomedical

Engineering. Vol. 57 No.5 May 2010 (with some minor changes - primarily the addition of the MAX6520 precision voltage divider). This circuit seems to need less adjustment between individuals. I've had to play around with the code for this setup (uses 2 analog readings to then calculate GSR) to ensure that I get enough resolution in the resultant signal. I may add an additional amplifier stage to each of the analog signals.

In the same article this circuit design came from, the researchers tested the palmar side of the wrist for GSR measurements compared to fingers and both were equally as accurate. In addition, there tests involved longer periods (one subject wore the sensor for an entire week, comfortably) and their wrist band with Ag/AgCl electrodes showed little in the way of motion artifacts, etc., basically it was able to provide a stable and consistent signal even with a subject engaging in daily activities of living.

As for the pulse sensor, further research led me to conclude the the back of the wrist is a poor site for recording a photoplethysmography signal (due to a lower concentration of surface capillaries). Thankfully, through experimentation, I found that the radial side of the palmar portion of the wrist seems to provide consistent readings when compared to a Polar Chest strap.

The Adafruit Feather Bluefruit seems to be a perfect choice so far. I've been able to get the BLE communication working though I am still figuring out how to send the sensor data (I can manually send data from the serial monitor and receive it on an iPhone using Adafruit's Bluefruit Application iOS App / Android App  

Presently I have everything on a breadboard with leads to the sensors held in place on my wrist with a wristband.

Just ordered some new parts to work on a wearable prototype to read and collect bio-metric data. These parts don't quite fit the form factor that I'm aiming for but they make it much easier to work towards a more optimal design.

• Microcontroller - Adafruit Feather M0 Bluefruit BLE
• Pulse Sensor - Pulse Sensor Amped

I choose the Adafruit Feather M0 Bluefruit BLE as my microcontroller because it has a number of elements included that simplify prototyping:

1. Arduino compatible microcontroller (lots of community support)
2. Integrated USB communication (easier debugging)
3. On board connection for LiPo power (including voltage regulator and charging)
4. On board Bluetooth BLE

I decided to change my pulse (photoplethysmography) sensor from the MAXREFDES177 Pulse/Ox sensor to the Pulse Sensor Amped - for 2 reasons:

1. Only interested in pulse data (to determine heart rate) - don't need the oxygen saturation data.
2. The Pulse Sensor Amped can directly output IBI (inter beat interval) data directly from a microcontroller (using open source software).

It is likely possible to get this data from the Maxim breakout but the learning curve is to steep for me now. Another plus is that the Pulse Sensor Amped uses green LED's which are likely to make data collection from the wrist easier.

Just a quick update and now I'm getting back to working on this.

This project is an outgrowth of my Voight-Kampff project. My goal is to use what I've learned (as well as what I'm continuing to learn) about physiological markers for emotion and focus this on better methods of facilitating communication for people who have difficulty expressing emotions though facial expressions, vocal intonation or body movements.

Currently there are a number of applications for detecting emotions through analysis of facial expressions (Microsoft's Emotion API, Empatica's facial analysis, etc). These do a truly amazing job at detecting emotional expression and their abilities are growing daily but they only work with those who have no impediments in expressing emotions.

The above technologies offer great promise in helping people learn to recognize the non-verbal signals of emotion (facilitate recognition of emotional states in others). This is an application that is being pursued, especially to help people who fall in the autism spectrum.

For those who do have difficulty (due to various issues) in outwardly expressing emotion, there does not seem to be similar efforts (at least that I've found so far - please let me know if I'm incorrect!). The primary goal of viEwMotion is to address that very issue.

There are numerous other potential applications for viEwMotion ranging from research into emotional response (especially by Citizen Scientists), game design and play (increased immersion due to game response to player emotions) and even market research (analyzing emotional response to advertisements, etc.).