1. General descriptionThemain goal of the project is to create a system which will be able to
detect various diseases at their early stages. Thanks this feature, the people should have possibility to apply a proper treatment earlier – what should dramatically increase the efectiveness of such treatment and lower its cost. Sometimes it may even lead to avoid illnesses – for example a person may avoid a cold if a proper pill is taken before getting a high temperature. But to do so – such person must be aware of coming disease.
The algorithm of detection used by the device is based on detection of patterns in data gathered by various sensors. Most of data is gathered by contactless sensors and without need of any actions taken by people whose health is supervised. The system is also able to collect data from any kind of intelligent wirstbands, smart watches or even intelligent scales. The device works 24 hours a day/7 days a week – so it has enough data to perform a reliable detection.
Based on all collected data and using a library of patterns the system is able to
detect possible diseases and other health problems. Thanks the wide
range of sensors it can be used also for supervising of young
childrens – especially during their sleep. The library of
patterns is created based on feedback from the supervised persons.
Thanks the connection with other similar devices, the results can
be compared and based on results of such comparisons the system can
create even more reliable patterns for broader range of diseases.
It is also possible to create a central repository or repositories
(using popular nowadays Big Data techniques) maintained by
a trustworthy organisation (govermental agency). Such repository will be responsible for
preparation and propagation of libraries of patterns created based
on a data from medical researches and data gathered anonymously
from devices from around the world.
The main applications are:
a) early detection of different kind of diseases – from cold to cancer
b) supervising young children and seniors
Fig.1. System's diagram
The vital part of the device is the sensor suite. It contains two main sensors:
- optical – consists of two cameras – one standard and one IR sensitive in a 8-12 um wavelength area. Both cameras have the same optical axis - it a crucial assumption. The purpose of the first camera is to recognize a person using a face recognition algorithm. The purpose of the second one is to measure temperature of objects in a field of a view of the first camera. Image from the tv camera may be used to detect also clothes and other things that can have impact on a precision of a temperature measurement. Thanks such construction the device can measure instantly a temeparature of persons whose health is supervised. It can also detect other events – like for example a sudden fall of a person or even changes in the way how these persons move. As an IR camera an AMG8853 sensor from Panasonic may be used (optionally Melexis MLX 90620/90621). Both sensors are quite expensive - so if we want to lower a cost af this device, we can use a cheap contactless thermometer with two servos (pan and tilt) and measure a temperature of persons located by tv camera. This solution is technically more complicated but is cheaper and may provide more precise measurements because in this case the device measures temperature of one person at the time. As a TV camera the system uses any popular 2-5 Mpx visible light camera.
- audio – consists of three identical microphones. Microphones create a triangle – thanks this form the system is able to find the place where sound come from. Connecting this information with information from the camera about the identity of the person located in this direction – the system is able to connect person with alarming sounds – like cough, whistling breath etc. The system not only detects such alarming sounds but also performs their analysis. It is able to detect for example a type of cough – and use this information in algorithms responsible for detection and recognition of diseaes. Except microphones this sensor uses also amplifiers with programmable gain.
The heart of the system is a computer responsible for the whole processing and communication. The computer is based on a multicore processor – its type hasn't been chosen yet. The most apropriate choice seems to be an SoC containing ARM cores (all Pi family for example - Raspberry Pi, Orange PI etc.) – but Intel products might be also taken into consideration. Of course the computer is equipped with a suitable amount of memory (based on requirements of an operating system) and all needed peripherals (USB for configuration purposes, digital interfaces used to connect cameras, analog converters to gather data from microphones etc.).
The device communicates with environement using two ways – WiFi and Bluetooth LE. The first is used in a communication between similar devices (when in one home/building there is more than one device) and between the device and internet. BLE serves as a connector between the device and smatrwatches/intelligent wirstbands.
In a prototype a RaspberryPi 3 will be used as a main computer.
Except components mentioned above – the system uses of course also other typical components – like DC/DC converter, leds and buttons used as UI etc.
Fig.2. Proposed enclosure with location of sensors.
The device will use an operating system – it can be Linux or Windows (there is an ARM version of the Windows 10). The software will be probably writtenin C/C++ and will use available open source libraries. Data from camera will be processed using OpenCV library, audio data – probably with a use of Essentia library. The rest of the software – detection algorithms, handling the data from sensors, storing data etc – will be dedicated for the device. This software should also be open sourced. Thanks this there shouldn't be problems with making copies of the device by private persons and companies interested in production of this product. This allows also to improve the hardware and software of the device by a community and gives a chance to the others to audit the code in search for malicious code.
Because the device will have access to sensitive data (it gathers images and sounds from within the house) the crucial thing is security. It will be provided at many different levels:
- the device will use a trustworthy and hacker-resistant OS
- the device won't store any images or sounds. They will be used only for detection/recognition purposes.After that they will be removed from memory.
- The communication between devices will be very limited – they won't communicate all the time.
- additional security software – like firewalls – will be installed
- software used on the device will be an opensource – everyone can check what is inside of it
5. How to use a device
The device should be located in a room, where people spend most of their time. If ther is more then one such place – in each of them one device should be located. In this case they all will be connected via WiFi and one of them is a master and other are slaves (slave may have different construction – for example they don't have to have a BLE interaface – so they can be even cheaper). The place, where the device is located should have a good field of view – it should be located in the way, which allow it to see people faces (not for the all time – after it recognizes a person it can track it without visibility of the face). Microphones shouldn't be covered – they must hear sounds from the room. The device located in this way will track people, measure their temperature, analyse their behaviour (movements) analyse their voices (cough etc.). Based on gathered data it will be able to predict incoming diseases and detect all other non-standard behaviours. Information about results of its work can be send to owners by email, text. The device itself is also able to alert people – it can blink leds mounted on its housing. If the device is used in hospital or to supervise older people – results can be send to persons responsible for taking care of supervised persons.
Users may have access to data gathered by the device – in the simplest variant - by using a control panel via a web browser (or dedicated application on a PC or on a tablet/phone). If there is such need – data will be stored in the cloud – and handled by pplications installed there.
The device will be able to learn based on gathered data and feedback from supervised persons. Each such person should (via panel mentioned before) inform the device about ilnesses. Thanks this the device will be able to identify patterns connected with a certain disease.
Devices will be able to communicate between each other – in a decentralised manner. They will disseminate patterns detected by themselves. Thanks this other devices should learn based not only on their own experiences but also based on data from other devices.
As i've mentioned before – patterns may also be prepared by trustworthy organisations based on information received from devices. Devices should send there only detected patterns and information about diseases connected with these patterns, this data will be verified by specialists and used to prepare libraries of patterns. Such libraries will be send back to devices.