Close
0%
0%

NoteOn Smartpen

Slim, wireless, self-contained. No special paper, no base station.

Similar projects worth following
Current smartpens are not very convenient to use. Most are quite bulky, and all require either special paper or a base station that clips on to the page. NoteOn aims to be a better smartpen; one that is slim, wireless, and self-contained. The pen uses inertial measurement to track its movement, allowing it to be used with any notebook, post-it note, or napkin. Notes and sketches can be sent to a mobile device over Bluetooth, and from there to a cloud service like Evernote or Google Drive.

NoteOn development is currently on hold. So far, two prototype PCBs and one case have been manufactured, and firmware drivers for all the peripherals except the Bluetooth transceiver have been completed.

The goal of the NoteOn project is to create a smart pen that is better than the ones currently on the market. It should be small, wireless, and self-contained.

  • Small: Most smart pens are 12mm or more in diameter. Normal writing implements generally range from 8-10mm. The NoteOn prototype is 10.0mm in diameter using an off-the-shelf battery and 2-layer board. The battery and PCB both limit the minimum size of the pen. With a custom battery and four layer PCB the diameter could probably be shrunk to 8mm.
  • Wireless: NoteOn includes a Bluetooth 4.0 LE transceiver to transfer data wirelessly to mobile devices and computers. From there it can be uploaded to an Internet document service like Evernote or Google Drive.
  • Self-Contained: This is the most important aspect of NoteOn. All currently available smartpens require either special paper or a base station that attaches to the page. NoteOn uses inertial measurement to track its motions without any external aids. It can be used with any notebook, sticky note, or napkin.

Electronics

The heart of NoteOn is the ST LSM9DS0TR, a 9-axis MEMS inertial measurement unit. It contains a 3-axis accelerometer, gyroscope, and magnetometer. The data from the IMU is supplemented by the ST LIS3DSHTR, a 3-axis accelerometer. This auxiliary accelerometer is placed on the opposite end of the PCB from the IMU, and will be used to help locate the precise center of rotation movements (each accelerometer experiences a different rotation based on its distance from the pivot point).

The IMU and auxiliary accelerometer are connected to the microcontroller by the I2C bus. The microcontroller is an ST STM32F302K8U6. It contains an ARM Cortex-M4 core with a floating-point coprocessor, 64kB flash, and 16kB RAM. I chose this microcontroller because it included a USB interface in a small QFP-32 package. Most chips with USB interfaces are physically too wide for the NoteOn PCB.

For Bluetooth 4.0 connectivity, NoteOn uses the Nordic nRF8001. It connects to the microcontroller over SPI. Unlike many Bluetooth interfaces, it doesn't have an internal microcontroller that needs to be programmed, making it a good fit for a system that already has a powerful uC. The Bluetooth antenna bears mentioning: I don't have the skills or equipment to design an antenna matching network, so the output of the balun is fed directly into a piece of wire. The RF performance is undoubtedly horrible, but I think it will work well enough for the prototype.

Also on the SPI bus is the Micron N25Q512A13GF840E, a 512Mbit (64Mbyte) NOR flash memory. I would have liked to for NoteOn to have more memory, but most flash chips are too wide.

Power is provided by a GoldPeak GP0836L17 170mAh lithium battery. (This battery is not available from GoldPeak, but clones of it are marketed as replacements for the Sony MH-100 headset. See BatteryBob #201747 and various sellers on Ebay.) This was the smallest diameter lithium-ion battery I could find. Although NiMh batteries are more commonly available in these small sizes, their low voltage makes the power supply design difficult.

The battery is charged from USB power by a Skyworks AAT3693IDH-AA-T1. Power is provided to the system through a single 3.0V rail generated by a Fairchild FAN53610AUC30X, a 3Mhz buck regulator in an exceptionally small 6-CSP package. A dual diode powers the regulator from the USB when it is available. (My first iteration of the design used a fully integrated diode, charger, and regulator IC. Unexpectedly, discrete parts took up less space.)

The battery voltage is monitored by an ST STC3115IJT Li-ion gas gauge. It connects to the microcontroller over the I2C bus.

No provision is made to switch off the power rail when the pen is in sleep mode, so sleep mode is achieved by putting all the ICs into their low-power states. This is something I'd like to address in the next design revision.

A single momentary switch and blue LED are the pen's user interface. The switch turns the pen on and off, and the LED lights...

Read more »

  • 1 × ST LSM9DS0TR 9-axis (accelerometer, gyroscope, and magnetometer) IMU
  • 1 × ST LIS3DSHTR 3-axis accelerometer
  • 1 × Nordic nRF8001-R2Q32-T Bluetooth 4.0 LE transceiver
  • 1 × Micron N25Q512A13GF840E 512Mbit SPI NOR Flash Memory
  • 1 × Skyworks AAT3693IDH-AA-T1 Li-Ion Battery Charger

View all 6 components

  • On Pause

    Nick Ames09/22/2014 at 22:16 2 comments

    I'm putting the NoteOn project on pause for a bit. Developing NoteOn's firmware has been more tedious than I expected, and I've lost interest in the project as a result. I'll probably start working on NoteOn again 6-12 months from now. In the meantime, feel free to use or adapt it however you like. The design files are licensed CCv4-BY-SA and are available here. The firmware is licensed GNU GPLv3 and is available on GitHub.

    I won't be proceeding further in the HaD Prize. Thanks to Hackaday and Supply Frame for putting it on, and good luck to everyone still in the running.

  • Current Status

    Nick Ames09/07/2014 at 22:15 2 comments

    (Apologies for the lack of updates; some family stuff has prevented me from working on NoteOn for the last few weeks.)

    As of 2014-09-07, the IMU driver (which fetches data from the IMU and aux. accelerometer) has been completed, and prototype 2's defective aux. accelerometer has been replaced.

    Auxiliary Accelerometer Replacement Process

    I started out by removing the passive components surrounding the defective aux. accelerometer using a soldering iron, in order to give myself more room to work. (402-sized components can be easily removed by wiping them away with a large blob of solder.) I applied Kapton tape to the adjacent microcontroller to shield it from the hot air, and bundled the SPI debugging wires underneath the board to keep them from melting.

    I used an Aoyue 850++ hot-air rework station to remove the defective device, then used an iron and solder wick to remove excess solder from the pads.

    I decided to pre-tin the pads on the new device with solder balls, rather than using solder paste. The only solder paste I have is lead-free, which requires higher temperatures. I stuck the device upside-down on a piece of Kapton tape, applied copious liquid flux, and ran a molten solder blob over the pads.

    Before attaching the new device, I pre-heated the PCB for about ten seconds with a medium air flow. I then applied flux, aligned the device with tweezers, and heated it for 25 seconds with a low air flow. (A higher air flow is much more effective at heating, but it can also blow the chip right off the board.)

    After replacing the passive components I removed earlier, the new aux. accelerometer responded over I2C and produced valid data.

  • Current Status

    Nick Ames08/21/2014 at 04:50 0 comments

    As of 2014-08-20, the electrical and mechanical design of the NoteOn prototype is complete. One case and two PCBs have been manufactured. 

    The first PCB, prototype 1, was damaged by a static shock which disabled its crystal oscillator, so I've been using prototype 2 for development. Unfortuneatly, prototype 2's auxilliary accelerometer doesn't work. (It responds over I2C, but the acceleration readings are always pegged to their maximum value. Checking against prototype 1's working aux. accel shows that the software is not at fault.) I've purchased a replacement accelerometer, and will swap it in once I can aquire a hot air rework tool. All other parts of prototype 2 appear to work fine.

    Development on the device firmware is ongoing. The board has been brought up, and I've written drivers for all nessecary internal peripherals, excluding USB. Drivers for the battery monitor IC and external flash memory have been written and tested. I'm currently working on a streaming driver for the IMU. (The driver will stream data continuously from the IMU, with the IMU caching its data in its internal FIFO between reads.) After that's done, I plan on getting Bluetooth communication working.

  • Thoughts on ARM vs. AVR

    Nick Ames08/21/2014 at 04:42 0 comments

    Until now, I've only used AVR microcontrollers. Although not very powerful, they have good documentation and a high-quality toolchain. (Avr-libc is fantastic.) As NoteOn will require some heavy number crunching (for a microcontroller), I equipped it with an STM32F302, which contains an ARM Cortex-M4 with a floating-point coprocessor. 

    Setting up a toolchain was difficult. After trial and error, I settled on gcc-arm-embedded for the compiler and utilities with libopencm3 for chip support. (Using ST's library wasn't an option.  It comes with a restrictive license which would have prevented NoteOn from meeting the Hackaday Prize openness requirement.) I patched stm32flash to support the F302 in order to download code via the serial bootloader. (Dfu-util may work for downloading code over USB, but I haven't tried it.)

    After getting the toolchain working, I found development much harder than with AVRs. The chip is more complicated, but the real issue is libopencm3's lack of documentation. (ST's terse datasheets don't help.) For the most part, the only documentation is the source code.

    However, the sweet nectar of DMA, 32-bit word size, and the NVIC (interrupt controller) soothe the pain of development. While AVRs will still be my first choice for small projects, I won't hesitate to use an ARM if the chip is doing any heavy lifting.

  • Machining the NoteOn Case

    Nick Ames08/21/2014 at 04:08 0 comments

    I use my Makerbot Thing-o-matic to create most of my project enclosures. It's simple, easy, and inexpensive. For the NoteOn case, this wasn't an option. Most of the parts have thin walls and require a precise fit. Instead, I machined the parts on my Taig Lathe.

    The parts were made of black polycarbonate, delrin (for the ink cartridge sleeve) and clear acrylic (for the button/status indicator).

    Most of the parts are tube-shaped. They just need to be turned to size on the outer diameter and drilled. A tip on drilling large holes in polycarbonate: don't use a pilot drill. I don't know why, but a pilot hole causes the larger drill to bind horribly. Also, don't be too aggressive, or the plastic will melt.

    For the PCB shelf, USB slot, and tip switch wire holes, I used my milling attachment with a simple fixture. The fixture is just a 10mm hole with a slot in it that clamps on the OD of the parts.

    To get a matte finish on the outside of the pen, I used 380 and 500 grit sandpaper with cutting oil (Cool Tool II). Some hot water and ivory soap afterwards leaves the parts nice and clean.

  • Leadless Packages: Less Difficult Than They Seem

    Nick Ames08/21/2014 at 03:50 0 comments

    Space is at a premium on the NoteOn PCB. The PCB is only 8.1mm wide, just barely larger than a SOIC package. To make the design fit, I used leadless packages for all the ICs. Most of the chips are QFP or LGA packages. The voltage regulator and battery monitor are chip-scale BGAs. 

    This was the first time I've used leadless packages on a project, and I was worried that there would be problems with short circuits and dry joints underneath the chips. To my surprise, everything went fine. In this build log, I'll go over the process I used.

    My PCBs were purchased from OSHPark. They produce high-quality boards, and their $5/sq. in. price is a fantastic value if your designs are small. .003" Kapton solder paste stencils were ordered from OSHStencils. I purchased a set of their board guides to rest the stencil on.

    The boards were stenciled with ChipQuik SMD291SNL lead-free solder paste. I had some issues with paste bridging between pads, so I used a needle to scrape the paste out of the way.

    The boards were reflowed in my temperature-controlled convection toaster oven. It's not fancy: a thermocouple and on-off control of the oven are all that's needed. Ideally, the thermocouple would be attached to the board being reflowed, but trying to attach a thermocouple to a board as a dense as the NoteOn PCB would be a recipe for disaster. Instead, I taped the thermocouple to a one of the spare NoteOn PCBs and placed the actual board near it.

View all 6 project logs

Enjoy this project?

Share

Discussions

Radu Motisan wrote 07/07/2017 at 20:34 point

This is an exciting project. I guess it's time for you to get up and get the job done.

  Are you sure? yes | no

saurabh wrote 07/31/2015 at 10:17 point

This is very interesting and though provoking.You must be very proud to design this.You have created thousand possibility of extending this work.Thanks for sharing your efforts.

Far above from being just commendable.

  Are you sure? yes | no

doubleman2014 wrote 09/29/2014 at 09:36 point
I want to know how the data sent to the PC or other device by the bluetooth.

  Are you sure? yes | no

webwicht wrote 09/23/2014 at 17:52 point
Hi Nick, you did a great job with this project and I would really like to have one pen and work on the firmware. Although I am not sure if the resolution is sufficient for the intended purpose the device has certainly potential for other things. I was thinking of a 3d air mouse or a level for a drill . To me it seems that one needs another reference system to compensate drift of the accelerometers. At which point you got stuck? What was the difficulty that made you to stop this wonderful project?

  Are you sure? yes | no

Nick Ames wrote 09/24/2014 at 07:36 point
I put NoteOn on hold because I'm tired of writing and debugging the firmware. I ran into a lot of small issues that slowed down the speed of development. Programming is my least favorite aspect of embedded development, and I can only do so much before I become burned out on a project. Unfortunately, that happened before I could make NoteOn fully functional.

  Are you sure? yes | no

dorianpicard1 wrote 09/23/2014 at 05:15 point
You must create a sort of tutorial so people can build their own pen. I would love to have the same!

  Are you sure? yes | no

Michael Oneppo wrote 09/22/2014 at 19:27 point
Hi Nick,

While I'm really impressed with your hardware skills, I'm very dubious that the pen will be able to interpret/replay notes with any accuracy. I prototyped multiple similar approaches without any success in extracting legible information. Have you tested the output of the pen? Can we see what that looks like, graphically?

I'm not trying to be a downer, I guess I'm hoping you found a solution to the problems I was encountering.

  Are you sure? yes | no

Suckiden wrote 09/22/2014 at 13:21 point
Very interesting project. Do you think your calculations will be so accurate that is possible to lay down the pen for a minute and return later to you paper still knowing where the pen tip is?

  Are you sure? yes | no

Edi Eco wrote 09/22/2014 at 13:08 point
You have a wonderful project! This smart pen is my dream, I really would buy one for my personal use. I'd like to ask if you have started the mathematical modeling to process the IMU and accelerometer data, noise filtering, etc. Thank you and congratulations!

  Are you sure? yes | no

atomkemp wrote 09/22/2014 at 13:02 point
Unbelievable. I would love to swap my Livrescribe for this. Awesome work!

  Are you sure? yes | no

Pixel Pirate wrote 09/01/2014 at 21:49 point
This is really cool! I was thinking about something like this the other day but couldn't figure out how to accurately track the movement of the pen, but you're idea is rather brilliant!
Tell me, how accurate is it with translating movement into words?

  Are you sure? yes | no

PointyOintment wrote 08/20/2014 at 19:23 point
This looks really cool! I never thought of using and IMU for tracking writings with a smartpen. What's the reason for the redundant accelerometer?

  Are you sure? yes | no

Nick Ames wrote 08/20/2014 at 21:11 point
Thanks! It's to help locate the center of rotation during movement. The IMU and the auxiliary accelerometer are located on opposite sides of the pen, and will experience very different accelerations depending on the center of rotation.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates