Serial Sensor

Free app for using your phones camera as a sensor and build some fun stuff. Additionally build in sensors are supported.

Public Chat
Similar projects worth following
So I wanted to get into java, later some image processing. I started to do so with my android phone and things quickly got out of hand. After billions of hours spent, there is a free app which does the following:
* Stream camera based sensors, currently a line- and a position sensor
* Build in, simple camera calibration
* Send data via USB, Bluetooth (HC-Modules) or to CSV file
* Stream phone internal sensors (accelerometer and so on)

What's different to already available stuff? Uitilizes the camera and it looks like a real app.

I'll be sharing some information the upcoming days how stuff is implemented.

Check the logs for more details!

Camera based sensor are currently in beta state. The line sensor is already working robustly for different surfaces & light conditions. For the position sensor (incl. velocity & heading) I'll show what you currently can expect.

  • Position sensor - current performance

    SerialSensor03/06/2021 at 10:48 0 comments

    Here's a video on the position sensor output (csv output from the phone is visualized and manually synced with the video). Output of the position sensor is the position (x, z in meters), orientation (rad) and velocity in m/s.

    For the position estimation, features in the camera images are tracked. With these tracked features you can estimate the displacement between two images (also kown as visual odometry). The position estimation gets fused with the phones accelerometer readings, which allows an output rate which is as low as your accelerometer rate (e.g. 2ms).

    Whats the problem?

    Only one camera is used, therefore depth information is not available. You can anyways estimate a displacement which is correct in direction and relative magnitude, but it differs from the realworld displacement by a unkown scale. And there is no way around this, as long as you do not get some distance information from elsewhere (like from speed sensors over time or a Lidar or whatever). Even super cool AI mono camera visual odometry cannot solve this.

    A simple solution to this is, assume you are moving on a even surface, and you know your height above that surface. So as long as this assumption is valid, the output is correct.

    The used phone in the video is a Huawei P10, which works really well. Compution time is ~10ms for the corner tracking (320x240 done in RenderScript) and another ~5ms for the displacement estimation and fusion (done in Java).

    Tested with a Huawei P10, Samsung S6, LG V30 and my favorite, the Samsung S3 which works not so well, you barely can see anything in the images, lol.

  • How does the line detection work

    SerialSensor02/22/2021 at 18:14 0 comments

    So how does the line detection work? If you already have some experience with image processing, the follwoing might not be to interesting for you.

    So internally the line sensor generates an output that looks like that:

    The red crosses beeing potential line points while the green circled ones are assumed to be from the line which we are interested in. As you can see, there is some nice shadow which also creates potential line detections.

    But how do you get there?

    Using the sobel operator you can calculate the gradient image, meaning that there is a high response when there is a color change in the image. For the scene above, the gradient image looks like:

    So you can see the line, but also the shadow is creating a high response. Potential line points are calculated by assuming that our line is between these high respones, which explains why the "shadow points" are between the line edge and the actual shadow edge.

    Finally for extracting the line which we walk along the valley. So we start at some point, beginning from the bottom and check if we do find a nice detection within the next row that is withing our valley. The below image illustrates that, the left and right point cannot belong to the current evaluted point, since there is a "hill" (or maxima) in between.

    So what's the sensor output now?

    You'll get the first and the last detected line point's lateral position, as well as the distance in between these two points in meters (check the sensor details inside the app for more inforamation). The provided distance between these two points can then be used to go faster / slower depending on how much line the camera can currently detect.

    How you go from the image to meters?

    The line points are transformed into a 3D frame of refernce (again, see sensor details inside the app). This is done using the intrinsic camera parameters and the height of your phone over ground. The intrinisc camera parameters are known,  since you are doing a camera calibration before you can use the line sensor.

    For this to work, we assume that the line and your phone is on an even surface.

    Whats not working so far?

    With this approach you cannot detect horizontal lines.

    Computation times?

    Galaxy S3 (with lineage OS...yeah) average of ~30ms

    LG V30 average of ~9ms

  • Day 1: A boring line follower

    SerialSensor02/21/2021 at 15:22 0 comments

    My first application done with Serial Sensor is a boring line follower. Watch the video, you'll be suprised.

View all 3 project logs

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates