Close

Does this project spark your interest?

Become a member to follow this project and don't miss any updates

NoteOn Smartpen

Slim, wireless, self-contained. No special paper, no base station.

6.2k 11 92 67

This project was created on 08/20/2014 and last updated a month ago.

Description
Current smartpens are not very convenient to use. Most are quite bulky, and all require either special paper or a base station that clips on to the page. NoteOn aims to be a better smartpen; one that is slim, wireless, and self-contained. The pen uses inertial measurement to track its movement, allowing it to be used with any notebook, post-it note, or napkin. Notes and sketches can be sent to a mobile device over Bluetooth, and from there to a cloud service like Evernote or Google Drive.

Currently, two prototype PCBs and one case have been manufactured. Work on the firmware is ongoing. See the Design Document and build logs for more details.
Details

The goal of the NoteOn project is to create a smart pen that is better than the ones currently on the market. It should be small, wireless, and self-contained.

  • Small: Most smart pens are 12mm or more in diameter. Normal writing implements generally range from 8-10mm. The NoteOn prototype is 10.0mm in diameter using an off-the-shelf battery and 2-layer board. The battery and PCB both limit the minimum size of the pen. With a custom battery and four layer PCB the diameter could probably be shrunk to 8mm.
  • Wireless: NoteOn includes a Bluetooth 4.0 LE transceiver to transfer data wirelessly to mobile devices and computers. From there it can be uploaded to an Internet document service like Evernote or Google Drive.
  • Self-Contained:  This is the most important aspect of NoteOn. All currently  available smartpens require either special paper or a base station that attaches to the page. NoteOn uses inertial measurement to track its motions without any external aids. It can be used with any notebook, sticky note, or napkin.

Electronics

The heart of NoteOn is the ST LSM9DS0TR, a 9-axis MEMS inertial measurement unit. It contains a 3-axis accelerometer, gyroscope, and magnetometer. The data from the IMU is supplemented by the ST LIS3DSHTR, a 3-axis accelerometer. This auxiliary accelerometer is placed on the opposite end of the PCB from the IMU, and will be used to help locate the precise center of rotation movements (each accelerometer experiences a different rotation based on its distance from the pivot point).

The IMU and auxiliary accelerometer are connected to the microcontroller by the I2C bus. The microcontroller is an ST STM32F302K8U6. It contains an ARM Cortex-M4 core with a floating-point coprocessor, 64kB flash, and 16kB RAM. I chose this microcontroller because it included a USB interface in a small QFP-32 package. Most chips with USB interfaces are physically too wide for the NoteOn PCB.

For Bluetooth 4.0 connectivity, NoteOn uses the Nordic nRF8001. It connects to the microcontroller over SPI. Unlike many Bluetooth interfaces, it doesn't have an internal microcontroller that needs to be programmed, making it a good fit for a system that already has a powerful uC. The Bluetooth antenna bears mentioning: I don't have the skills or equipment to design an antenna matching network, so the output of the balun is fed directly into a piece of wire. The RF performance is undoubtedly horrible, but I think it will work well enough for the prototype.

Also on the SPI bus is the Micron N25Q512A13GF840E, a 512Mbit (64Mbyte) NOR flash memory. I would have liked to for NoteOn to have more memory, but most flash chips are too wide.

Power is provided by a GoldPeak GP0836L17 170mAh lithium battery. (This battery is not available from GoldPeak, but clones of it are marketed as replacements for the Sony MH-100 headset. See BatteryBob #201747 and various sellers on Ebay.) This was the smallest diameter lithium-ion battery I could find. Although NiMh batteries are more commonly available in these small sizes, their low voltage makes the power supply design difficult.

The battery is charged from USB power by a Skyworks AAT3693IDH-AA-T1. Power is provided to the system through a single 3.0V rail generated by a Fairchild FAN53610AUC30X, a 3Mhz buck regulator in an exceptionally small 6-CSP package. A dual diode powers the regulator from the USB when it is available. (My first iteration of the design used a fully integrated diode, charger, and regulator IC. Unexpectedly, discrete parts took up less space.) 

The battery voltage is monitored by an ST STC3115IJT Li-ion gas gauge. It connects to the microcontroller over the I2C bus.

No provision is made to switch off the power rail when the pen is in sleep mode, so sleep mode is achieved by putting all the ICs into their low-power states. This is something I'd like to address in the next design revision.

A single momentary switch and blue LED are the pen's user interface. The switch turns the pen on and off,...

Read more »

Components
  • 1 × ST STM32F302K8U6 ARM Cortex-M4F microcontroller with 64kB ROM/16kB RAM
  • 1 × ST LSM9DS0TR 9-axis (accelerometer, gyroscope, and magnetometer) IMU
  • 1 × ST LIS3DSHTR 3-axis accelerometer
  • 1 × Nordic nRF8001-R2Q32-T Bluetooth 4.0 LE transceiver
  • 1 × Micron N25Q512A13GF840E 512Mbit SPI NOR Flash Memory
  • 1 × Skyworks AAT3693IDH-AA-T1 Li-Ion Battery Charger

Project logs
  • On Pause

    a month ago • 2 comments

    I'm putting the NoteOn project on pause for a bit. Developing NoteOn's firmware has been more tedious than I expected, and I've lost interest in the project as a result. I'll probably start working on NoteOn again 6-12 months from now. In the meantime, feel free to use or adapt it however you like. The design files are licensed CCv4-BY-SA and are available here. The firmware is licensed GNU GPLv3 and is available on GitHub.

    I won't be proceeding further in the HaD Prize. Thanks to Hackaday and Supply Frame for putting it on, and good luck to everyone still in the running.

  • Current Status

    a month ago • 1 comment

    (Apologies for the lack of updates; some family stuff has prevented me from working on NoteOn for the last few weeks.)

    As of 2014-09-07, the IMU driver (which fetches data from the IMU and aux. accelerometer) has been completed, and prototype 2's defective aux. accelerometer has been replaced.

    Auxiliary Accelerometer Replacement Process

    I started out by removing the passive components surrounding the defective aux. accelerometer using a soldering iron, in order to give myself more room to work. (402-sized components can be easily removed by wiping them away with a large blob of solder.) I applied Kapton tape to the adjacent microcontroller to shield it from the hot air, and bundled the SPI debugging wires underneath the board to keep them from melting.

    I used an Aoyue 850++ hot-air rework station to remove the defective device, then used an iron and solder wick to remove excess solder from the pads.

    I decided to pre-tin the pads on the new device with solder balls, rather than using solder paste. The only solder paste I have is lead-free, which requires higher temperatures. I stuck the device upside-down on a piece of Kapton tape, applied copious liquid flux, and ran a molten solder blob over the pads.

    Before attaching the new device, I pre-heated the PCB for about ten seconds with a medium air flow. I then applied flux, aligned the device with tweezers, and heated it for 25 seconds with a low air flow. (A higher air flow is much more effective at heating, but it can also blow the chip right off the board.)

    After replacing the passive components I removed earlier, the new aux. accelerometer responded over I2C and produced valid data.

  • Current Status

    2 months ago • 0 comments

    As of 2014-08-20, the electrical and mechanical design of the NoteOn prototype is complete. One case and two PCBs have been manufactured. 

    The first PCB, prototype 1, was damaged by a static shock which disabled its crystal oscillator, so I've been using prototype 2 for development. Unfortuneatly, prototype 2's auxilliary accelerometer doesn't work. (It responds over I2C, but the acceleration readings are always pegged to their maximum value. Checking against prototype 1's working aux. accel shows that the software is not at fault.) I've purchased a replacement accelerometer, and will swap it in once I can aquire a hot air rework tool. All other parts of prototype 2 appear to work fine.

    Development on the device firmware is ongoing. The board has been brought up, and I've written drivers for all nessecary internal peripherals, excluding USB. Drivers for the battery monitor IC and external flash memory have been written and tested. I'm currently working on a streaming driver for the IMU. (The driver will stream data continuously from the IMU, with the IMU caching its data in its internal FIFO between reads.) After that's done, I plan on getting Bluetooth communication working.

View all 6 project logs

Discussions

doubleman2014 wrote 22 days ago null point

I want to know how the data sent to the PC or other device by the bluetooth.

Are you sure? [yes] / [no]

webwicht wrote a month ago null point

Hi Nick, you did a great job with this project and I would really like to have one pen and work on the firmware. Although I am not sure if the resolution is sufficient for the intended purpose the device has certainly potential for other things. I was thinking of a 3d air mouse or a level for a drill . To me it seems that one needs another reference system to compensate drift of the accelerometers. At which point you got stuck? What was the difficulty that made you to stop this wonderful project?

Are you sure? [yes] / [no]

Nick Ames wrote a month ago null point

I put NoteOn on hold because I'm tired of writing and debugging the firmware. I ran into a lot of small issues that slowed down the speed of development. Programming is my least favorite aspect of embedded development, and I can only do so much before I become burned out on a project. Unfortunately, that happened before I could make NoteOn fully functional.

Are you sure? [yes] / [no]

dorianpicard1 wrote a month ago null point

You must create a sort of tutorial so people can build their own pen. I would love to have the same!

Are you sure? [yes] / [no]

Michael Oneppo wrote a month ago null point

Hi Nick,

While I'm really impressed with your hardware skills, I'm very dubious that the pen will be able to interpret/replay notes with any accuracy. I prototyped multiple similar approaches without any success in extracting legible information. Have you tested the output of the pen? Can we see what that looks like, graphically?

I'm not trying to be a downer, I guess I'm hoping you found a solution to the problems I was encountering.

Are you sure? [yes] / [no]

Suckiden wrote a month ago null point

Very interesting project. Do you think your calculations will be so accurate that is possible to lay down the pen for a minute and return later to you paper still knowing where the pen tip is?

Are you sure? [yes] / [no]

Edi Eco wrote a month ago null point

You have a wonderful project! This smart pen is my dream, I really would buy one for my personal use. I'd like to ask if you have started the mathematical modeling to process the IMU and accelerometer data, noise filtering, etc. Thank you and congratulations!

Are you sure? [yes] / [no]

atomkemp wrote a month ago null point

Unbelievable. I would love to swap my Livrescribe for this. Awesome work!

Are you sure? [yes] / [no]

Pixel Pirate wrote 2 months ago null point

This is really cool! I was thinking about something like this the other day but couldn't figure out how to accurately track the movement of the pen, but you're idea is rather brilliant!
Tell me, how accurate is it with translating movement into words?

Are you sure? [yes] / [no]

PointyOintment wrote 2 months ago null point

This looks really cool! I never thought of using and IMU for tracking writings with a smartpen. What's the reason for the redundant accelerometer?

Are you sure? [yes] / [no]

Nick Ames wrote 2 months ago null point

Thanks! It's to help locate the center of rotation during movement. The IMU and the auxiliary accelerometer are located on opposite sides of the pen, and will experience very different accelerations depending on the center of rotation.

Are you sure? [yes] / [no]

Similar projects