A Smart Fridge With Brains

All prior smart fridge attempts have basically been tablets glued to fridges. My idea is a blueprint for a fridge that is actually "smart".

Similar projects worth following
Benefit society / the environment:
- Reduce wasted food through spoilage - prevent overproduction, overbuying, and general waste production
- Save people money - money that can be better spent elsewhere instead of on food that ends up in the trash
- If people get used to wasting less, they will need less fridge space - could lead to smaller more energy efficient fridges
- For the elderly, could enable relatives to keep an eye on nutrition, make sure they are getting enough to eat, and enable automatic ordering of food to be delivered

Most basic overview of idea: A combination of cameras mounted on the outside of the fridge and weight sensors inside the fridge would track which items enter and exit the unit, WITHOUT requiring input from the user. The fridge would "learn", and could be trained to recognize non-standard goods. Object recognition software would be implemented alongside weight tracking to determine what food objects are entering, exiting, and sitting in the fridge.

Here is my project pitch (4/4/2015):

To me, the real defining failure of all smart fridge attempts to date has been the requirement of user input. Fridges that claim to be "the future" still require users to manually input (through keypresses or barcode scanning) what items they have purchased, and even where they are placing them within the fridge, which defeats the entire purpose of a "smart" fridge. I see no difference from these prior attempts and someone simply hot-gluing a tablet with inventory control software on it to the door of their fridge. When I have relatives that have trouble figuring out how to log back into websites like Facebook if their session expires, I find it hard to believe that consumers would ever be accepting of a smart fridge that requires manual entry of items. In fact, that option is already available to most consumers through mobile applications - anyone can create a list of what is in their fridge with the most basic of smartphones and text editing application. My project hopes to be a blueprint for a smartfridge that goes beyond manual inventory control, and becomes a seamless part of a consumer's kitchen and lifestyle. In addition it will use EXISTING food packaging technologies, instead of requiring manufacturers to implement costly innovations like RFID, which they will avoid and never implement.

Basic idea for project:
A combination of cameras mounted on the outside of the fridge and weight sensors inside the fridge would track which items enter and exit the unit, WITHOUT requiring input from the user. The fridge would "learn", and could be trained to recognize non-standard goods. Object recognition software would be implemented alongside weight tracking to determine what food objects are entering, exiting, and sitting in the fridge. A database would be maintained of standard expiration lengths for various food groups, so the fridge could track which items are likely closest to expiring, and provide that information to the user if requested. Ideally, my project would exploit existing synergies with other existing software that is popular among users, such as automatically adding items to a grocery list application (e.g. OurGroceries) as they expire.

I have to note that my educational background is not in electronics or computer science; it is in business (cue audible groans, boos, and hissing). This is worth mentioning, because I want my project to partially be a blueprint for commercial applications of a "smarter" smart fridge. For example, I could encourage companies to adapt QR codes on packaging to provide my fridge with additional information, such as lot numbers, in exchange for providing valuable user data. Example: manufacturer prints QR codes on milk cartons that includes embedded nutritional information, lot number, and expiration date. In exchange, they get access to valuable data on how many consumers are stocking their milk cartons, how long they keep them for, how fast they go through the carton... the possibilities are endless. To sweeten the deal for consumers, in exchange for providing this anonymous data, money collected from sales of aggregated data could be used to subsidize the manufacturing costs of the smart fridges, meaning a lower purchase price for consumers, which would speed up adoption. Furthermore, the more manufacturers that contribute to the database, the more access consumers would have to information that would benefit them, such as if any products in their fridge have been recalled.

I should note that there are extremely recent technological developments that bring to mind additional wonderful applications for this project - in particular, the announcement of the Amazon Dash Replenishment Service (DRS). If developments continue at this pace, it would be likely my fridge could tie into an API for a service like AmazonFresh, and predictively order food to be delivered to your front door exactly when you run out of your existing stock, meaning less gas wasted on...

Read more »

  • Dr. Strain Gauge (Or: How I Learned to Love Pre-made Scales)

    Joshua T04/18/2015 at 21:29 0 comments

    My speed of progress on this project has been a little slower than anticipated, mostly due to school (BLERGH), but I have made a little bit of progress. To me, the most important part of this concept for a smart fridge is the way (or should I say weigh) I am envisioning it using weight sensors to make intelligent assumptions without user input. Example: you take a carton of milk out of the fridge, pour yourself a glass, and then put the carton back on the fridge shelf. Because the fridge has weight sensors, it knows the weight of the carton before and after you consumed some, so combined with publicly available nutrition information, it can tell you exactly how many calories of milk you consumed, without requiring a single button to be pressed.

    Here is the thing though, weight sensing seems to a lot more complex than you might think. There are so many sensors out there that can be easily integrated into MCU projects (simply hook sensor output up to ADC), but I had no idea where to even start with weight. I started researching and it seems like the most commonly used sensor is a strain gauge. This is essentially a flexible film that has an embedded resistor. As the gauge flexes (or is strained), the resistance changes. However, this change in resistance is miniscule, so you need to use the strain gauge in a Wheatstone Bridge configuration in order to actually measure the change in resistance. After that, you need a special kind of amplifier, an instrumentation amplifier, to amplify the signal before it can be fed into a microcontroller, if you want a wide range of values to be read. This video is an excellent overview of the process:

    All of this is certainly feasible to do, but the costs start adding up really quickly. For some reason it is really hard to source strain gauges (also often sold in the form of load cells) for cheap. When you add in the cost of an instrumentation amplifier (hard to find under $3 per IC), things really start getting pricey. I think this comment thread (on this HAD article) best sums up the dilemma:

    At this point, I think the best option for this project might actually be to just head over to Goodwill (or some other thriftstore) and purchase some already made kitchen scales, then disassemble them and see if there is any easy way to tap into the output. Even on Amazon, I can get a digital kitchen scale for around $7, coming from inside the US, which is LESS than a single order of strain gauges.

    In the meantime, while I was thinking about all of this, I was trying to come up with alternative ways to measure weight, other than the go-to method of strain gauges. I happened to have a few Hall Effect sensors laying around, so I came up with a concept for using them to measure the change in distance between a platform (holding the object to be weighed) and a base platform, separated by some sort of spring. Based on price, this might be a more cost-effective method, considering I probably only paid about a penny for each sensor, but much more labor-intensive to implement. I'm not actually going to use this method in my project, but it was something fun to play around with. Here is short video of me goofing around with this concept:

    I should also note that I am far from the first person to experiment with using hall effect sensors to measure weight. Hall effect sensors are commonly used in automotive applications for measuring force/displacement:

    The best example I could find of a hall-effect sensor scale was actually a project featured on Hackaday previously:

  • Research Stage: Part 1

    Joshua T04/06/2015 at 03:12 0 comments

    I am just getting started with this project, which means all I have at the moment is a concept and a list of desired features. However, I have started researching what I would need to bring something like this to life, and have discovered some things worth noting:

    The first thing to note is that I will likely not be using OpenCV. When I first thought of attempting a smart fridge with object recognition, my very first thought was that I would have to use OpenCV, which I believed was the industry standard. However, after actually looking at the website and documentation, I think OpenCV, although powerful, is too complex and non-user-friendly for my needs.

    Instead of OpenCV, I think I will try using Caffee, which appears to be very powerful, easy to use, and is also covered under a more generous license (BSD). The demo on their site is absolutely amazing and makes it seem as though the software would be a perfect match for my needs.

    The other thing I realized is that my original plan, to use a raspberry pi, is probably not a good idea. The computational requirements for near real-time object recognition exceed what the raspberry pi was designed for, and although there have been developments in optimizing Caffee for the device, the fastest it appears to run is at 3 seconds of processing per frame, which is too slow for my needs. I think what I will use instead is an actual laptop or desktop computer. I have an old laptop that is not in use that would be great for running a lightweight linux distro and Caffee, so I will probably try that before spending money on any new hardware.

View all 2 project logs

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates