Close
0%
0%

FaceAssure: cloud based pan-tilt security camera

A security camera product that presents that tracks a person's face using openCV in the cloud.

Public Chat
Similar projects worth following
We developed a security camera product called that presents the owner a comprehensive view of the area under surveillance. The system provides the ability to detect and track people as they move across the area automatically, and to notify the owner if a suspected breach of property has occurred.

Our product goals were implemented through the design of a web-enabled camera system. The core camera system uses a custom designed PCB camera module, comprising primarily of a microcontroller unit, wifi module and 2 megapixel camera sensor. The camera module is installed on a pan-tilt servo mount capable of 2-dof rotation. Since an embedded system would not have enough compute power for real-time image processing, our computer vision goals were achieved by leveraging cloud computing and deploying a Tornado web server with computer vision processing. A web client is provided for potential customers to view a video stream of their cameras online.

Board Design.

We use Eagle to create our embedded board design, including version 1.0 and version 2.0 . Here is the version one, our first prototype for testing of images transmission. 

As we can see, this version includes WiFi module, MCU and camera module. Two buttons are used for reset and setup of WiFi chip. Moreover, two mountain holes are implemented to fix our board on the enclosure. However, our design still needs improvement, since PWM control are not included yet and size of the board is also not good for the final purpose. Then we developed our second version.

Version 2.0 is almost the same as the previous from components angle, however we still make a lot of progress in our design. First, two sets of PWM control headers are implemented, including 5V0, GND and PWM control signal. Also, in order to fit our board in to the pan-tilt base, we change size of the board. We make it slimmer, with 30mm width which is exactly the width of the pan-tilt base. Still we have some tiny parts to improve. For example, two PWM headers should not be so close to J-link headers, since it is not easy to insert two parts at the same time, which means we cannot download program and test PWM control at the same time. Well, it might not be perfect but still could be accepted, because we are 99% sure that our PWM control works well, we don't need to test them together. But from the application point of view, maybe in the next version we should fix this problem.

Image transmission using Camera Module.

The camera’s Zentri AMW004 wifi module, is able to receive more 3-4 image frames per second from the Microcontroller Unit (MCU). SPI is used for communication between the MCU and Camera module. Once the image is transmitted to the MCU buffer, the MCU is able to initiate a HTTP POST request through the AMW wifi chip via UART. We maximized the UART transmission rate by increasing the UART baud rate 8x faster at 921600 Hz rather than 115200 Hz, and reducing the size of the camera buffer to about 30KB (supports most 640x480 JPEG images).

Google Cloud server solution. 

We used a micro EC2 compute to host a python Tornado web server, which is supported by OpenCV. The server scripts can be found in the GitHub repository.

Tornado web server.

The cloud server receives all images via HTTP POST and performs person recognition and tracking in real time for a single camera. Once the coordinates and size of the person are determined, a PID controller determines the angles that pan-tilt base should now adopt. These values are transmitted via a HTTP RESPONSE to the camera using the same TCP connection. 

Web Client

The Python server script offers a simple web interface from which to view the image stream. This was achieved using a javascript function that refreshed the JPEG image every 100ms and a browser cache bypass.

OpenCV detection and tracking. 

We used OpenCV Haar-Cascade feature identification and MOSSE tracking to detect and track people as they move across the room. Haar-Cascade detection is used to identify faces (frontal and profile), half body and full body to a reasonable degree of accuracy. Once a region of interest is identified, a MOSSE tracker follows the person, identifies the appropriate bounding box and overlays the bounding box graphic on the JPEG image, while updating the PID controller accordingly.

PWM servo actuators. 

This part is relatively easy, because we could use example file in the Atmel Studio. We need to define pins of MCU to work as PWM mode, and follow the configuration from the example code, things are gonna be fine. The PWM servos now respond accordingly to the angle values provided over the HTTP RESPONSE from the server. The MCU...

Read more »

cloud.zip

Server script using Python 3.x, Tornado web server, and OpenCV. Contains a 'webcam' folder that servers a simple HTML/js web application that show displays the video stream

Zip Archive - 496.67 kB - 06/11/2019 at 19:59

Download

atmel firmware.zip

Atmel studio codebase for the ATSAM4S8BA.

Zip Archive - 6.39 MB - 06/11/2019 at 19:58

Download

eagle.zip

Contains schematics for our custom PCB embedded board

Zip Archive - 55.86 kB - 06/11/2019 at 19:58

Download

  • 1 × Adafruit #1967 Min Pan-Tilt Kit
  • 1 × OV2640 Omnivision Camera Module
  • 1 × AMW004 Silicon Lab Wifi Module
  • 1 × ATSAM4S8BA Microprocessors, Microcontrollers, DSPs / ARM, RISC-Based Microcontrollers
  • 1 × Custom PCB See EAGLE files

  • Status Report 3

    Nathaniel Wong06/11/2019 at 20:05 0 comments

    Design Updates

    Our team made process in redesigning the camera security system in these key areas:

    Image transmission. The camera’s Zentri AMW004 wifi module, is able to receive more image frames per second from the Microcontroller Unit (MCU), achieving up to 3-4 frames per second. This was done by increasing the UART baud rate 8x faster at 921600 Hz rather than 115200 Hz, and reducing the size of the data buffer.

    Google Cloud server solution. The cloud server now receives all images and performs person recognition and tracking in real time for a single camera. Once the coordinates and size of the person are determined, a PID controller determines a new angle the pan-tilt base should now adopt and transmits the data via HTTP.

    OpenCV facial recognition. The OpenCV library has been exploited to identify faces (frontal and profile), half body and full body to a reasonable degree of accuracy. Once a region of interest is identified, a CSRT tracker follows the person and updates the PID controller accordingly.

    PWM servo actuators. The PWM servos now respond accordingly to the angles provided by the HTTP connection; this is achieved by updating the duty cycles for each servo.

    Progress made on the project has been acceptable - we are on target to meet the project’s minimum viable product goals, including extending the system to incorporate multiple cameras. however we will not be able to explore our stretch goals of including an LCD screen.

    Planning and Organization

    Figure: Gantt Chart for the project

    Item Desc.

    Mfg. Part #

    Unit Price

    1000 Unit Price

    Quantity

    In Final Design?

    Total Unit Price

    Total Bulk Price

    Mini Pan-Tilt Kit

    1967

    $18.95

    $15.15

    1

    Yes

    $62.34

    $53.29

    LCD screen

    181

    $9.95

    $7.96

    1

    No

    Omnivision camera module

    2640

    $9.44

    $7.55

    1

    Yes

    Silicon Labs Wifi Module

    AMW004

    $27.69

    $25.63

    1

    Yes

    Microchip Technology MCU

    ATSAM4S8BA-AU

    $5.26

    $4.45

    1

    Yes

    Custom PCB

    $1.00

    $0.50

    1

    Yes

    Figure: Bill of Materials for the project

    Results of Market Research

    Our non-expert interviewees found a product that could recognize specific faces to be very useful in monitoring their private spaces, for example, alerting against front door package thieves or trespassers. On the topic of facial recognition, one interviewee noted that constant face detection alerts could get stale because of false alarms, and suggested using criminal face databases to reduce false alarms.

    We interviewed Prof. Jack Tumblin for his expert input on the computer-vision aspects of the security camera that we are making. Most of his suggested surrounded our efforts to reduce latency on the camera, including using grayscale images rather and RGB for transmission to achieve compression. He also suggested using a more direct form of data transmission rather than UART, or bypassing the MCU directly and using a shared bus between the sensor and wifi chip to transmit images as quickly as possible.

    In terms of image quality, Prof. Tumblin suggested reducing exposure time to reduce motion blur. To compensate, we would have to increase aperture size (if possible) and ISO settings.

    Next Steps

    By the end of the quarter, we will fine tune camera system to ensure that the face detection, tracking, and actuation work seamlessly. Amongst some measures, we will increase the UART baud rate to increase throughput, alter the server and CV configurations to improve latency, and tweak the PID controller to improve responsiveness to the detected person.

    References

    https://www.cnet.com/pictures/security-cameras-with-facial-recognition-tech-inside/

    https://docs.opencv.org/3.3.0/d7/d8b/tutorial_py_face_detection.html

    https://www.adafruit.com/product/1967

    https://www.adafruit.com/product/181

    https://usa.banggood.com/3pcs-Mini-OV2640-Camera-Module-CMOS-Image-Sensor-Module-for-Arduino-p-1452968.html

    https://www.digikey.com/product-detail/en/silicon-labs/AMW004/1586-1012-ND/5269916

    https://www.digikey.com/product-detail/en/microchip-technology/ATSAM4S8BA-AU/ATSAM4S8BA-AU-ND/3593645

  • Status Report 2

    Nathaniel Wong06/11/2019 at 20:04 0 comments

    Design Updates

    Our team made significant process in reconfiguring the camera module into a web client in 4 key areas:

    Image transmission. The camera’s Zentri AMW004 wifi module, as controlled by the Microcontroller Unit (MCU), is now capable of transmitting images to the host server on Google Cloud. This is done by sending HTTP POST requests with the appropriate headers for a image/jpeg Content-Type, and the Content-Length matching the size of the image file. At time of report, the Camera system is able to output about one frame of image output every second to the server in this manner.

    Google Cloud server solution. The Google Cloud compute instance is the workhorse of the security camera system. Currently, a python Tornado web server accepts HTTP requests from any properly configured device on the internet that sends the appropriate headers. The contents of the request are stored as a jpeg device on the disk and a HTTP response indicates that the transaction was successful. A web interface allows a user to watch the image feed ‘live’ conditions, at a 200ms local refresh rate (the camera continues to be the bottleneck at one frame per second). Where appropriate, the response text will include data for x and y coordinate for the location of a detected face (see section on OpenCV).

    PWM servo actuators.  The servos were successfully tested using the breakout boards. We were able to control the servo positions on the pan-tilt base using PWM and altering the value of the PWM duty cycle.

    OpenCV facial recognition. The OpenCV library for python has been integrated into the web server. Currently, we are using a Haar Feature-based cascade classifier for face and eye detection. The classifier works reasonably well on the low pixel count of the images, but only when the face is directed squarely at the camera. It is not capable of distinguishing between faces of different individuals either.

    The overall state of the project is healthy. So far, we are on target to meet the project’s minimum viable product goals. More time will be needed than expected to integrate the PWM servos into the current solution as this will require the wifi module to interface with the microcontroller seamlessly.

    At the current moment, the stretch goals of incorporating and LCD screen and per-user facial recognition may not be feasible given the amount of development time and testing required to complete the features in the current iteration.

    Planning and Organization

    Figure: Gantt Chart for the project

    Item Desc.

    Mfg. Part #

    Unit Price

    1000 Unit Price

    Quantity

    In Final Design?

    Total Unit Price

    Total Bulk Price

    Mini Pan-Tilt Kit

    1967

    $18.95

    $15.15

    1

    Yes

    $62.34

    $53.29

    LCD screen

    181

    $9.95

    $7.96

    1

    No

    Omnivision camera module

    2640

    $9.44

    $7.55

    1

    Yes

    Silicon Labs Wifi Module

    AMW004

    $27.69

    $25.63

    1

    Yes

    Microchip Technology MCU

    ATSAM4S8BA-AU

    $5.26

    $4.45

    1

    Yes

    Custom PCB

    $1.00

    $0.50

    1

    Yes

    Figure: Bill of Materials for the project

    Results of Market Research

    Our non-expert interviewees found a product that could recognize specific faces to be very useful in monitoring their private spaces, for example, alerting against front door package thieves or trespassers. On the topic of facial recognition, one interviewee noted that constant face detection alerts could get stale because of false alarms, and suggested using criminal face databases to reduce false alarms.

    We will be interviewing Prof. Jack Tumblin for his expert input on the computer-vision enabled security camera product that we are making.

    Next Steps

    By the next presentation, we will have the servos autonomously operating as faces are detected by the camera system and the cloud server. This will require steps to be taken to integrate PWM functionality into the codebase as well as the ability to parse HTTP responses. Should this fail, we will operate the servos either by manually commands from a web interface or on an automated timed loop, panning across the entire room.

    We will aim to improve the OpenCV...

    Read more »

  • Status report 1

    Nathaniel Wong06/11/2019 at 20:04 0 comments

    Design Requirements

    • Camera is configured as a web client
    • Deploy and cloud server to maintain connection with camera modules perform image processing
    • Use computer vision techniques to detect and classify faces seen in camera feeds
    • Cameras track face movements using mounted servo system
    • Alert is sent if the person’s face is not recognized

    Design Description

    Figure: Block diagram of proposed multi-camera system with central cloud server

    Using a hub and spoke model will result in a high latency between sensors and actuators, because a TCP round-trip will be required to process each video frame.

    The benefit to this is that the central server is capable of higher image processing loads. We will use a combination of Tornado, a lightweight python web server, and OpenCV, a powerful real-time computer vision library built on python, to our advantage on the cloud compute instance. This will allow the viewing and autonomous control of multiple video feeds..

    Planning and Organization

    Action by

    Start Date

    End Date

    Status

    Define project goals

    Nat

    Apr 2, 2019

    Apr 9, 2019

    Complete

    Research components

    QK

    Apr 2, 2019

    Apr 9, 2019

    Complete

    Prepare order

    Nat

    Apr 2, 2019

    Apr 9, 2019

    Complete

    Create new PCB layout

    QK

    Apr 9, 2019

    Apr 11, 2019

    Active

    Setup Amazon AWS services

    Nat

    Apr 9, 2019

    Apr 16, 2019

    Complete

    Camera client sends images to AWS

    Nat

    Apr 16, 2019

    Apr 23, 2019

    Active

    Use breakout board to test LCD&Pan-tilt

    QK

    Apr 17, 2019

    Apr 24, 2019

    Upcoming

    Put together breakout prototype

    QK

    Apr 18, 2019

    Apr 25, 2019

    Upcoming

    AWS server processes images with CV

    Nat

    Apr 23, 2019

    May 9, 2019

    Upcoming

    Make revisions and make embedded PCB

    QK

    Apr 25, 2019

    May 2, 2019

    Upcoming

    Put together embedded prototype

    QK

    May 9, 2019

    May 16, 2019

    Upcoming

    Make revisions and design revised PCB

    QK

    May 16, 2019

    May 23, 2019

    Upcoming

    Pan-tilt base reacts to location of human face

    Nat

    May 9, 2019

    May 30, 2019

    Upcoming

    Put together final prototype

    All

    May 30, 2019

    Jun 12, 2019

    Upcoming

    Prepare final presentation

    All

    Jun 4, 2019

    Jun 12, 2019

    Upcoming

    Figure: Gantt Chart for the project

    Item Desc.

    Mfg. Part #

    Unit Price

    1000 Unit Price

    Quantity

    Mini Pan-Tilt Kit

    1967

    18.95

    15.15

    1

    LCD screen

    181

    9.95

    7.96

    1

    Figure: Bill of Materials for the project

    Splitting of work

    Qiankai will largely focus on the hardware aspects of the design and fabrication process, including PCB design, servo motor control and final integration of different modules.

    Nathaniel will design the central server functionality, modify the camera to accept wifi connections to the server and stream images, and implement computer vision techniques.

    Results of Market Research

    Based on home ownership data, about 40 million middle to upper income urban households own their properties globally, and many of these households face the real threat of petty household theft. To that end, they may find value in an autonomous visual security system that deters thieves while providing homeowners of critical alerts and valuable visual evidence of trespassers.

    List of competitors

    One major brand is SimpliSafe, which provides an array of household security camera products, entry sensors and motion sensors. At a price of US$500 for a complete solution, this may be prohibitive for even that most security-conscious of users. The products also do not include face detection and recognition

    NestCam is another serious competitor that features built-in facial recognition for outdoor cameras, and is able to recognize ‘familiar persons’ with in-app alerts. Cloud storage backs up the video feeds as part of the product package. The camera retails for $349, a hefty price tag for a singular camera, or $600 for a pair.

    Other brands: Arlo, Allianca, Blink, Nest Cam, Zmodo, Dlink, Netgear, Logitech

    Results of Interviews

    We conducted interviews with two potential users, but did not manage to interview an expert. Our interviewees found a product that could recognize specific faces to be very useful in...

    Read more »

View all 3 project logs

Enjoy this project?

Share

Discussions

Deltaprintr wrote 10/04/2019 at 00:06 point

Is it possible to modify the PCB so that it's direct to USB transmission?

  Are you sure? yes | no

Nathaniel Wong wrote 10/04/2019 at 12:34 point

yes, it’s possible to communicate with the Atmel SAM4S micro controller through USB. You’ll need Atmel Studio and the Atmel SAM-ICE JTAG programmer.

  Are you sure? yes | no

Deltaprintr wrote 10/04/2019 at 14:24 point

Would I need to modify any of the code? (not a programmer) 

  Are you sure? yes | no

rachel wrote 06/17/2019 at 02:49 point

A clever and creative solution to achieve real-time performance!

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates