AI-driven Web-based Lab Assistant | UNO Q & Gemini

Details

After hearing about the launch of the brand-new Arduino UNO Q, designed as the first SBC (single-board computer) with Arduino's philosophy of bridging the gap between employing professional development tools and implementing them as novices when creating introductory projects or as experts while prototyping complex mechanisms rapidly yet stably, I thought it would be a great opportunity to redesign my previous AI-driven lab assistant project and enable more developers, beginner or expert, to replicate, experiment, or improve this new AI-based ancillary lab assistant thanks to the built-in Arduino UNO Q features and its beginner-friendly development platform — Arduino App Lab.

As you may know, if you have read one of my previous project tutorials, I prefer building my AIoT projects on the target development boards and environments from scratch and enjoy developing unique methods, applications, and mechanisms to collect custom training data and achieve intended device features, strictly following my methodology of developing proof-of-concept research projects. Nonetheless, in this project, I heavily focused on developing all lab assistant features based on the provided UNO Q and Arduino App Lab characteristics, such as the built-in Bricks, native microprocessor-microcontroller communication procedure, and Linux-oriented SBC board architecture, to ensure that anyone with a UNO Q can effortlessly replicate and examine this lab assistant without needing to have a deep understanding of all aspects of this project; coding, web design, neural network training, LLM-implementation, 3D modeling, etc. In this regard, I hope this project serves as an entry point for developing research projects, encouraging readers to reverse-engineer the features of this AI-driven lab assistant to gain a deeper understanding of AIoT development on the edge.

As I was taking inspiration from my previous lab assistant project, I heavily modified the device structure and added a lot of new features specific to this iteration, for instance, designing a unique PCB (UNO Q shield) for utilizing various lab sensors to conduct LLM-assisted basic lab experiments. After months of hard work, I managed to complete the reimagined AI-driven ancillary lab assistant structure and develop all the features I envisioned on UNO Q by solely employing the Arduino App Lab development environment, providing foundational building blocks (Bricks).

🤖 To build the ancillary lab assistant structure:

✍🏻 I designed a unique PCB as a UNO Q shield (hat) to connect the selected lab sensors and create the analog lab assistant interface, including the capacitive fingerprint sensor.

✍🏻 Then, I modeled 3D parts to design the ancillary lab assistant base, containing the USB camera and the analog interface.

✍🏻 Finally, I designed a modular lab sensor ladder, organizing all sensors and secondary experiment tools, to create a compact but easy-to-use instrument.

🤖 To accomplish all of the ancillary lab assistant features I contemplated, performed by an Arduino App Lab application:

🛠️ I trained an Edge Impulse object detection model to identify various lab equipment.

🛠️ I programmed the MCU (STM32) to collect real-time sensor information and manage the analog lab assistant interface.

🛠️ I developed a feature-rich web dashboard as the primary user interface and control panel of the lab assistant, hosted directly by the Arduino App Lab.

🛠️ I incorporated Google Gemini to enable the lab assistant to generate LLM-based lessons about the detected lab equipment.

🛠️ Thanks to the built-in background Linux MPU-MCU communication service (Arduino Router), I built the interconnected interface background in Python, handling the data transfer between the web dashboard, the analog interface (MCU), and the Qualcomm QRB (MPU) running the essential App Lab Bricks (Docker...

Components

1 × Arduino UNO Q 4GB

1 × UGREEN 5-in-1 100W USB-C Hub with 4K@60Hz HDMI and 3 * USB-A 3.0

1 × A4 Tech PK-910H USB Webcam (1080p)

1 × Raspberry Pi 15W 5.1V / 3.0A USB-C Power Supply

1 × Custom PCB

Build Instructions

Collapse

1

Development process, thinking in terms of creating an Arduino App Lab application, and final results

As mentioned in the introduction, the development process of this AI-driven lab assistant differs quite a bit from my previous AIoT projects since I built a single application within the confines of the Arduino App Lab development environment, specifically constructed to capitalize on the dual-brain (MPU-MCU) nature of UNO Q, even though I developed a feature-rich web dashboard and analog lab assistant interface individually. Arduino App Lab provides built-in Bricks (Docker containers) for adding various fundamental attributes to an App Lab application, such as web UI hosting, inference running for custom models, etc., and manages all of the operations of the included Bricks while executing the completed application. Thus, although I still utilized specific programming languages to develop the different aspects of the lab assistant App Lab application, Arduino for programming the STM32 microcontroller (MCU), Python for the application backend (Qualcomm MPU), and HTML, CSS, JavaScript for the web dashboard, as a whole, I built a single application that the App Lab runs and manages.

I think the most prominent feature of UNO Q, with the support of the App Lab, is the built-in RPC (Remote Procedure Call) managed by the Arduino Router background Linux service, which enables developers to borrow and run functions between Qualcomm MPU and STM32 MCU interchangeably. App Lab also provides a built-in web socket to establish data transfer between the web dashboard and the Python backend. In this regard, it makes the communication between the STM32 MCU and the web dashboard effortless through the same Python backend. In light of these built-in features, I decided to build the second iteration of my AI-driven lab assistant with UNO Q.

Generally, I thoroughly explain the setup process of my interconnected software and hardware applications according to the employed development boards, modules, environments, third-party APIs, etc. However, since I only utilized the built-in Arduino App Lab attributes to enable anyone with a UNO Q to replicate and examine this project effortlessly, I highly recommend inspecting the official Arduino UNO Q specifications and tutorials.

To be able to use UNO Q as a single-board computer and connect a USB camera, I needed to use a USB-C hub (dongle) with reliable HDMI, USB-A, and USB-C external power ports. Nonetheless, since UNO Q does not have a dedicated GPU, the processing power was too slow to run the App Lab and develop the lab assistant application solely in the SBC mode, especially the web dashboard. Thus, I utilized the SBC mode and the network mode simultaneously, supported by the App Lab, to access UNO Q remotely on any machine connected to the local network. In this regard, I was able to build the lab assistant App Lab application by accessing the full capacity of the Arduino App Lab.

#️⃣ Since I needed to capture screenshots for this tutorial while utilizing the SBC mode, I installed a simple program to enable taking screenshots on Debian-based Linux distributions via the terminal.

sudo apt install xfce4-screenshooter

I documented the overall development process for the finalized ancillary lab assistant in the following written tutorial. Even though I exhibited all of the lab assistant features in the tutorial, I highly recommend checking the project demonstration videos that thoroughly showcase the device structure and real-time user experience of the analog assistant interface and the web dashboard.
2
Integration and use cases of Google Gemini
Since I wanted users to generate AI lessons based on questions about specific lab equipment and inspect the LLM-generated lessons via the web dashboard, I decided to fine-tune the large language model responses appropriately to obtain lessons directly in the HTML format. According to my previous experiments with different large language models that I conducted while developing LLM-oriented projects, Google Gemini produced reliable, informative, and concise HTML pages about simple inquiries. Thus, I decided to utilize Google Gemini to enable the ancillary lab assistant to produce AI lessons. Furthermore, Google Gemini has a very low barrier to entry for utilizing its primary chat application and API services.

#️⃣ First, to be able to integrate Google Gemini into my Arduino App Lab application, I opened Google AI Studio and created a new API key specific to this project.

#️⃣ Since the App Lab already provides a Brick to integrate and use cloud LLMs in Python, I only needed to register the produced API key into my custom application. I will explain how to utilize Bricks in detail in the following steps.
#️⃣ Although I enabled users to produce AI lessons freely on different lab equipment based on predefined or specific questions, I decided to make the web dashboard present LLM-generated but curated guides with simple experiment tips about the selected lab sensors. To ensure the consistency between the user-generated AI lessons and the static (default) lab sensor guides with experiment tips, I employed the official Google Gemini chat application to produce dedicated HTML pages for each lab sensor.

❓ Such as: "Create me an HTML page explaining Gravity: Factory Calibrated Electrochemical Alcohol Sensor and the importance, dangers, and usage of alcohol in labs."

#️⃣ Since I am not a talented graphic or logo designer, I also decided to employ Gemini to produce custom logos and CSS animations for the web dashboard. I specifically made Gemini to contain each CSS animation in a separate HTML page, which helped me to manage my primary web dashboard layout and Gemini-generated elements. For each Gemini-generated static lab sensor information page, animation page, and logos (images), I added the gemini moniker to their file names.
- gemini_alcohol_concentration.html
- gemini_fingerprint_waiting.html
- gemini_text_to_speech_stop_logo.png
3

Continue reading the full project tutorial

Continue reading the full project tutorial