Although 3D printing is relatively new to the dental industry, 3D-printed dental casts exhort countless benefits and opportunities when creating dental products. However, after perusing recent articles on 3D printing applications in the dental industry, I noticed that there are not many promising tools or methods to inspect the 3D-printed cast accuracy and efficiency for dental technicians. Therefore, I decided to build a user-friendly and accessible device employing an object detection model to classify 3D-printed casts in the hope of assisting dental technicians in detecting cast accuracy and malfunctions.

To extrapolate and interpret dental cast categories (classes) accurately, I needed to collect data from actual 3D-printed dental casts in order to train my object detection model with notable validity. Therefore, I purchased numerous dental cast STL files identified with different tags (labels) and printed them with my SLA (HALOT-ONE) and FDM (CR-200B) 3D printers:

Since Sony Spresense is a high-performance development board intended for edge computing in sensor analysis, machine learning, image processing, and data filtering, I decided to utilize Sony Spresense in this project. To capture images and store them on an SD card to train my object detection model, I connected the Spresense extension board and the Spresense camera board to the Spresense main board (CXD5602). Then, I utilized a TFT LCD touch screen (ILI9341) to display the video stream and captured images. Also, I added a tiny thermal printer to the device so as to print the detection result after running my object detection model on Sony Spresense.

After completing my data set by taking pictures of 3D-printed dental casts, I built my object detection model with Edge Impulse to make predictions on dental cast accuracy categories (classes). I utilized Edge Impulse FOMO (Faster Objects, More Objects) algorithm to train my model, which is a novel machine learning algorithm that brings object detection to highly constrained devices. Since Edge Impulse is nearly compatible with all microcontrollers and development boards, I had not encountered any issues while uploading and running my model on Sony Spresense. As labels, I appended three main categories (classes) to the file names while capturing and storing pictures:

After training and testing my object detection (FOMO) model, I deployed and uploaded the model on Sony Spresense. Therefore, the device is capable of detecting dental cast accuracy categories (classes) by running the model independently without any additional procedures.

Lastly, to make the device as robust, sturdy, and compact as possible while experimenting with 3D-printed dental casts, I designed a dental-themed case with a sliding side cover (3D printable).

So, this is my project in a nutshell 😃

In the following steps, you can find more detailed information on coding, capturing dental cast pictures, storing pictures on an SD card, building an object detection (FOMO) model with Edge Impulse, and running it on Sony Spresense.

🎁🎨 Huge thanks to DFRobot for sponsoring a Tiny (Embedded) Thermal Printer.

🎁🎨 If you want to purchase products from DFRobot, you can use my $5 discount coupon.

🎁🎨 Also, huge thanks to Creality for sending me a Creality CR-200B 3D Printer.

Step 1: Designing and printing a dental-themed case

Since I focused on building a user-friendly and accessible device for dental technicians in this project, I decided to design a robust and compact case allowing the user to capture dental cast images flawlessly. To avoid overexposure to dust and prevent inadvertent loose wire connections, I added a sliding side cover to the case. Also, I decided to fasten a tooth replica to the device and emboss a dental logo on the sliding side cover so as to emphasize the dental theme gloriously.

I designed the main case and its sliding side cover in Autodesk Fusion 360. You can download their STL files below.

For the tooth replica affixed to the top of the tiny thermal printer, I utilized this model from Thingiverse:

Then, I sliced all 3D models (STL files) in Ultimaker Cura.

Since I wanted to create a solid structure for this device with a sliding side cover and complement the dental theme gloriously, I utilized these PLA filaments:

Finally, I printed all parts (models) with my Creality CR-200B 3D Printer. It is my first fully-enclosed FDM 3D printer, and I must say that I got excellent prints effortlessly with the CR-200B :)

If you are a maker planning to print your 3D models to create more complex projects, I highly recommend the CR-200B. Since the CR-200B is fully-enclosed, you can print high-resolution 3D models with PLA and ABS filaments. Also, it has a smart filament runout sensor and the resume printing option for power failures.

According to my experience, there are only two downsides of the CR-200B: relatively small build size (200 x 200 x 200 mm) and manual leveling. Conversely, thanks to the large leveling nuts and assisted leveling, I was able to level the bed and start printing my first model in less than 30 minutes.

#️⃣ Before the first use, remove unnecessary cable ties and apply grease to the rails.

#️⃣ Test the nozzle and hot bed temperatures.

#️⃣ Go to Settings ➡ Leveling and adjust four predefined points by utilizing the leveling nuts.

#️⃣ Finally, attach the spool holder and feed the extruder with the filament.

#️⃣ Since the CR-200B is not officially supported by Cura, select the Ender-3 profile and change the build size to 200 x 200 x 200 mm. Also, to compensate for the nozzle placement, set the Nozzle offset X and Y values to -10 mm on the Extruder 1 tab.

Step 1.1: Assembling the case and making connections & adjustments

// Connections
// Sony Spresense (w/ Extension Board) :  
//                                2.8'' 240x320 TFT LCD Touch Screen (ILI9341)
// D7   --------------------------- CS 
// D8   --------------------------- RESET 
// D9   --------------------------- D/C
// MOSI --------------------------- SDI (MOSI)
// SCK  --------------------------- SCK 
// 3.3V --------------------------- LED 
// MISO --------------------------- SDO(MISO) 
//                                Tiny (Embedded) Thermal Printer
// TX   --------------------------- RX
// RX   --------------------------- TX
// GND  --------------------------- GND
//                                Control Button (A)
// D2   --------------------------- +
//                                Control Button (B)
// D4   --------------------------- +
//                                Control Button (C)
// D14  --------------------------- +
//                                Keyes 10mm RGB LED Module (140C05)
// D3   --------------------------- R
// D5   --------------------------- G
// D6   --------------------------- B

First of all, I connected the Spresense extension board and the Spresense camera board to the Spresense main board to capture images and connect components easily. Since the Spresense main board operates at 1.8V, it is not suitable to connect components directly to the main board without the expansion board, which supports 5V or 3.3V logic level voltage.

Then, I connected a TFT LCD touch screen (ILI9341) to the expansion board so as to display the video stream, captured images, and the predicted dental cast accuracy category (class) after running the object detection (FOMO) model. Also, I utilized a tiny (embedded) thermal printer to print the predicted class and its details to deduce dental cast specifications and accuracy effortlessly.

Since the Spresense extension board cannot power the tiny (embedded) thermal printer due to its operating voltage and current, I connected a USB buck-boost converter board to my Xiaomi power bank to elicit stable 9V to supply the thermal printer. The higher input voltage (voltage range is 5~9V) means faster printing speed and more clear printed records.

To append labels to the file names while capturing dental cast images and storing them on the SD card, I added three control buttons (6x6), as shown in the schematic below. Also, I added a 10mm common anode RGB LED module (Keyes) to indicate the outcomes of operating functions.

After completing breadboard connections and adjustments successfully, I made the breadboard connection points rigid by utilizing a hot glue gun.

After printing all parts (models), I fastened all components to their corresponding slots on the main case via the hot glue gun.

Then, I placed the sliding side cover via the dents on the main case.

Finally, I affixed the tooth replica to the top of the tiny thermal printer via the hot glue gun.

Step 2: Printing numerous dental casts (models)

Since I needed to collect images from numerous dental casts, I decided to print actual dental casts (impressions) with my resin (SLA) 3D printer. Therefore, I purchased plenty of dental casts with different tags (labels) from Cults:

Then, I sliced all dental 3D models (STL files) in the latest release of HALOT BOX.

To print my dental casts similar to the casts molded by dental technicians, I utilized a special resin:

Finally, I printed all dental casts (models) with my Creality HALOT-ONE Resin 3D Printer. Although I am a novice in resin 3D printing, and it is my first SLA (resin) 3D printer, I got incredible results effortlessly with the HALOT-ONE :)

Also, I sliced some dental 3D models again in Ultimaker Cura to print them with my FDM 3D printer (CR-200B) in order to improve my object detection model's accuracy on 3D-printed dental casts molded with different materials or methods.

After printing dental casts, I captured their images via the Spresense camera board, as explained in the following steps.

Step 3: Setting up Sony Spresense on the Arduino IDE

Before proceeding with the following steps, I needed to set up Sony Spresense on the Arduino IDE and install the required libraries for this project.

#️⃣ First of all, mount the provided shading seal on IC4 in the Spresense main board to prevent malfunctions under direct or in the proximity of an intense light source, e.g., sunlight or a bright lamp.

#️⃣ To add the Sony Spresense board package to the Arduino IDE, navigate to File ➡ Preferences and paste the URL below under Additional Boards Manager URLs.

#️⃣ This package also includes the required libraries for the extension board modules (such as the SD card module) and the camera board.

https://github.com/sonydevworld/spresense-arduino-compatible/releases/download/generic/package_spresense_index.json

#️⃣ Then, to install the required core, navigate to Tools ➡ Board ➡ Boards Manager and search for Spresense.

#️⃣ After installing the core, navigate to Tools → Board → Spresense Boards and select Spresense.

#️⃣ To update the Spresense firmware so as to use the latest version of the Spresense Arduino libraries, go to Tools → Programmer → Spresense Firmware Updater. Then, select Burn Bootloader.

#️⃣ Finally, download the required libraries for the tiny (embedded) thermal printer and the TFT LCD touch screen (ILI9341).

Adafruit-Thermal-Printer-Library | Download

Adafruit_ILI9341 | Download

Adafruit-GFX-Library | Download

Step 3.1: Displaying images on the ILI9341 TFT LCD touch screen

To display images (RGB) on the ILI9341 TFT LCD touch screen successfully, I needed to convert PNG or JPG files into the.c (source) file format.

#️⃣ First of all, download GIMP.

#️⃣ Then, upload an image (RGB) and go to Image ➡ Scale Image... to resize the uploaded image.

#️⃣ Finally, export the image as a.c (source) file.

#️⃣ To generate the data array in the right format, select only the Save as RGB565 (16-bit) option.

#️⃣ After exporting the image, import the generated data array to the code and print it on the screen.

tft.drawRGBBitmap(10, 10, (uint16_t*)(data_collect.pixel_data), (int16_t)data_collect.width, (int16_t)data_collect.height);

Step 3.2: Printing bitmaps with the tiny thermal printer

#️⃣ First of all, utilize an image editing program to save an image (black and white) as a 1-bit BMP, such as the built-in Paint program on Windows.

#️⃣ Then, install and run the LCD Assistant.

#️⃣ Upload the monochromatic bitmap and select Horizontal in the Byte orientation section under Settings.

#️⃣ If necessary, change the image width because the LCD Assistant and the thermal printer handle images in horizontal groups of eight pixels. Therefore, if the image width is not a multiple of eight, it is truncated (cropped) to the nearest smaller 8-pixel boundary.

#️⃣ Convert the image (bitmap) and save the output (data array).

#️⃣ Finally, add the data array to the code and print it via the thermal printer.

printer.printBitmap(80, 80, dental_logo);

Step 4: Capturing and storing images w/ Sony Spresense

Inspect.

Step 4.1: Saving the captured pictures to the SD card as samples

After uploading and running the code for capturing pictures and saving them to the SD card on Sony Spresense:

🦷🖼️ The device turns the RGB LED to blue if the SD card module and the camera board connections with the Spresense main board are successful.

🦷🖼️ Then, the device turns the RGB LED to magenta as the default color and displays the video stream on the ILI9341 TFT LCD touch screen.

🦷🖼️ If the control button (A) is pressed, the device pauses the video stream and captures a picture. If the device captures the picture successfully, it turns the RGB LED to yellow, appends the Cast [0] dental cast category (class) number with the current date & time to the file name, and stores the recently captured image on the SD card.

0_D_2022.06.29__18.12.05.JPG

🦷🖼️ Then, the device displays the captured image information on the ILI9341 TFT LCD touch screen.

🦷🖼️ Finally, the device resumes the video stream and turns the RGB LED to green.

🦷🖼️ If the control button (B) is pressed, the device pauses the video stream and captures a picture. If the device captures the picture successfully, it turns the RGB LED to yellow, appends the Failed [1] dental cast category (class) number with the current date & time to the file name, and stores the recently captured image on the SD card.

1_D_2022.06.29__18.08.02.JPG

🦷🖼️ Then, the device displays the captured image information on the ILI9341 TFT LCD touch screen.

🦷🖼️ Finally, the device resumes the video stream and turns the RGB LED to green.

🦷🖼️ If the control button (C) is pressed, the device pauses the video stream and captures a picture. If the device captures the picture successfully, it turns the RGB LED to yellow, appends the Implant [2] dental cast category (class) number with the current date & time to the file name, and stores the recently captured image on the SD card.

2_D_2022.06.29__18.45.36.JPG

🦷🖼️ Then, the device displays the captured image information on the ILI9341 TFT LCD touch screen.

🦷🖼️ Finally, the device resumes the video stream and turns the RGB LED to green.

🦷🖼️ If Sony Spresense throws an error while operating, the device turns the RGB LED to red and prints the error details on the serial monitor.

🦷🖼️ Also, the device prints notifications and the captured image data on the serial monitor for debugging.

As far as my experiments go, the device operates impeccably while capturing dental cast images and saving them to the SD card :)

After capturing images of numerous dental casts with different tags and storing them on the SD card, I elicited my data set, including training and testing samples for my object detection (FOMO) model.

📌 Dental cast tags:

Even though my 3D-printed dental casts have different tags pertinent to a dental problem, I categorized them under the dental cast accuracy classes (Cast, Failed, and Implant) since it is not possible to detect every cast individually with a limited data set and an object detection model running on an edge device.

Step 5: Building an object detection (FOMO) model with Edge Impulse

When I completed capturing dental cast images and storing them on the SD card, I had started to work on my object detection (FOMO) model to make predictions on the dental cast accuracy categories (classes).

Since Edge Impulse supports almost every microcontroller and development board due to its model deployment options, I decided to utilize Edge Impulse to build my object detection model. Also, Edge Impulse provides an elaborate machine learning algorithm (FOMO) for running more accessible and faster object detection models on edge devices such as Sony Spresense.

Edge Impulse FOMO (Faster Objects, More Objects) is a novel machine learning algorithm that brings object detection to highly constrained devices. FOMO models can count objects, find the location of the detected objects in an image, and track multiple objects in real-time, requiring up to 30x less processing power and memory than MobileNet SSD or YOLOv5.

Even though Edge Impulse supports JPG or PNG files to upload as samples directly, each training or testing sample needs to be labeled manually. Therefore, I needed to follow the steps below to format my data set so as to train my object detection model accurately:

Since I appended the assigned classes to the file names while capturing and storing dental cast images, I preprocessed my data set effortlessly to label my image samples on Edge Impulse:

Plausibly, Edge Impulse allows building predictive models optimized in size and accuracy automatically and deploying the trained model as an Arduino library. Therefore, after scaling (resizing) and preprocessing my data set to label samples, I was able to build an accurate object detection model to predict dental cast accuracy categories (classes) and run the model on Sony Spresense effortlessly.

You can inspect my object detection (FOMO) model on Edge Impulse as a public project.

Step 5.1: Uploading images (samples) to Edge Impulse and labeling samples

Inspect.

Step 5.2: Training the FOMO model on dental cast categories

Inspect.

Step 5.3: Evaluating the model accuracy and deploying the model

Inspect.

Step 6: Setting up the Edge Impulse FOMO model on Sony Spresense

Inspect.

Step 7: Running the FOMO model on Sony Spresense to classify dental casts

My Edge Impulse object detection (FOMO) model scans a captured image and predicts possibilities of labels (dental cast accuracy categories) to recognize an object on the given captured image. The prediction result (score) represents the model's "confidence" that the detected object corresponds to each of the three different dental cast accuracy categories (classes) [0 - 2], as shown in Step 5:

After executing the dental_model_classifier_run_model.ino file on Sony Spresense:

🦷🖼️ The device turns the RGB LED to blue if the camera board connection with the Spresense main board is successful.

🦷🖼️ Then, the device turns the RGB LED to magenta as the default color and displays the video stream on the ILI9341 TFT LCD touch screen.

🦷🖼️ If the control button (B) is pressed, the device pauses the video stream and captures a picture. If the device captures the picture successfully, it turns the RGB LED to yellow, resizes the currently captured image, and converts its data format to GRAYSCALE.

🦷🖼️ Then, the device runs an inference with the Edge Impulse object detection (FOMO) model.

🦷🖼️ Finally, the device displays the detection result (dental cast accuracy category) and the generated bounding boxes on the ILI9341 TFT LCD touch screen.

🦷🖼️ Each dental cast accuracy category (class) has a unique color code to be shown on the ILI9341 TFT screen when being predicted (detected) by the FOMO model:

🦷🖼️ Also, the device prints the predicted label (dental cast accuracy class) information and details via the tiny (embedded) thermal printer.

🦷🖼️ After printing information successfully, the device resumes the video stream and turns the RGB LED to green.

🦷🖼️ Also, the device prints notifications and the captured image data on the serial monitor for debugging.

As far as my experiments go, the device recognizes objects with different dental cast accuracy categories (classes) precisely and shows accurate bounding boxes around the detected objects :)

Videos and Conclusion

After completing all steps above and experimenting, I have employed the device to predict and detect dental cast accuracy categories (classes) of various 3D-printed dental casts so as to check potential mistakes and inaccuracies.

Further Discussions

By applying object detection models trained on captured 3D-printed dental cast images in detecting dental cast accuracy categories, we can achieve to:

🦷🖼️ obtain more stable, flexible, and precise 3D-printed dental casts,

🦷🖼️ preclude potential mistakes and inaccuracies on the 3D-printed dental casts before diagnosis,

🦷🖼️ predict and detect dental problems faster and more straightforward,

🦷🖼️ assist dental technicians who are novices in 3D printing.

References

[1] Erik Borgwardt, Great benefits of using 3D-printing and digital models in the dental industry, March 15, 2017, https://elosmedtech.com/great-benefits-of-using-3d-printing-and-digital-models-in-the-dental-industry/