Hack Chat Transcript, Part 1

A event log for Open-Source AR Hack Chat

Making the world a better place to see

dan-maloneyDan Maloney 05/31/2023 at 21:250 Comments

Dan Maloney
11:57 AM
Hi Raj, welcome aboard! We'll get started in a minute or two

Dan Maloney12:00 PM
Hello and welcome back to the Hack Chat. I'm Dan, and I'll be moderating today along with Dusan as we welcome Raj Nakaraja for a chat about Open-Source AR.

Hi Raj, welcome. Care to start us off with a little about what you do?

Raj Nakarja12:00 PM
Hey Dan! Excited to get started :)

Raj Nakarja12:01 PM
Thanks for the warm welcome! I'm Raj, head of Engineering at Brilliant Labs. We're a startup making open-source AR/heads up display glasses, and my job is leading the hardware and firmware design

Pinembour joined  the room.12:02 PM

Dan Maloney12:03 PM
So, as you can probably tell from the way I promoted this, I tend to think of AR as being able to see other parts of the EM spectrum as an overlay on the regular view of the world. I suspect that's not all there is to it, though. How far off the mark am I?

Dan Maloney12:04 PM
Nto gonna lie -- kinda want a Geordi LaForge visor ;-)

Raj Nakarja12:06 PM
Haha, that's a pretty cool idea honestly. I imagine myself being bombarded with colors as soon as a microwave turns on, or the phone rings. Well, there are tons of use cases for AR glasses. One favorite of mine was a startup a while ago that was doing a circuit overlay on top of PCBs. The idea was that you could see the current and voltages in realtime while you're probing your board

Raj Nakarja12:07 PM
I think they got bought up by one of the big PCB software companies and I never heard of them again. We want to avoid that happening by being open source and building a community around our devices

Dan Maloney12:09 PM
Yeah, that company was started by a guy we had on the chat a couple of times. It was a really good idea that just sort of disappeared. Forgot all about it, actually.

Raj Nakarja12:10 PM
If anyone wants to recreate it, they can try it out on our Monocle!

Dan Maloney12:13 PM
Can you go into Monocle a bit? I'm particularly interested in the optics -- having a big goofy thing stuck to your face always seemed like a barrier to general AR, but Monocle is much more streamlined.

Dan Maloney12:13 PM


Brilliant Labs

Pocket-sized AR for imaginative hackers. At Brilliant Labs, we're building an open-source ecosystem to support developers and creatives reimagining the future.

Read this on Brilliant Labs

Raj Nakarja12:16 PM
For sure! Well firstly, here's the full datasheet: There's a brief description down the page on the optics, but in short it's a tiny micro OLED which projects down to a beam-splitter. The image then appears as a screen floating a couple of meters in front of you

Raj Nakarja12:16 PM

Dan Maloney12:18 PM
The reason I ask is for those of us with "old eyes". Seems like we'd have a hard time focusing up close, but maybe that's not how it works, since the apparent image is so far off

Raj Nakarja12:19 PM
Exactly. It looks like a tablet screen at arm's length basically

Raj Nakarja12:20 PM
The display is driven by an FPGA which allows for some nice graphics acceleration, and Bluetooth networking is done via an nRF52. The nRF runs MicroPython, so actually, Monocle is totally python driven

Raj Nakarja12:21 PM
Additionally, there's a 5MP camera, microphone, leds, and a couple of touch sensors

Dan Maloney12:25 PM
Would you want to use this on your dominant eye? Or would you put it on the weak eye and leave the dominant eye free to see the real world?

Raj Nakarja12:26 PM

Raj Nakarja12:26 PM
This is how it looks in real life. You can compare the text size here to my laptop screen

kjansky112:27 PM
What is the experience like I mean I've always had a hard time seeing through a microscope where the instinct is to close the other eye but you are really supposed to mentally turn off the eye not looking through the microscope?

Raj Nakarja12:29 PM
I personally put it on my weaker eye. It's not really something you have to focus on, in fact, the focal distance is technically at infinity, so it's always in focus. We've put the display below the horizon so it's like looking down at a book or your phone

xBeau12:30 PM
I'm curious what you have found most useful to use your Monicle for day to day, are you focused on any particular use cases?

Raj Nakarja12:31 PM
Text requires a bit of focusing I find, but for shapes and icons, it's not so much of a problem. There it can become more of a background thing

kjansky112:32 PM
Is there some kind of focus for nearsightedness as the focus at infinity is a blur w/o corrective lenses?

Raj Nakarja12:35 PM
There are quite a few things popping up. One example which went sort of viral was a Stanford student who connected it to ChatGPT. He'd use voice to ask it a question, and then it would print the response on the screen. I can find the tweet if you're interested. Another guy made a gym training gadget with timers and things. A lot of dev tools popping up were folks wanted info about their PC or network or something like that. All just running from python scripts over BLE from their PC

Dan Maloney12:35 PM
As for use cases, I keep coming back to "Wouldn't it be cool for surgery?" On the one hand, I don't want a surgeon who needs help finding his or her way around the surgical field. OTOH, I want the best possible outcomes from surgery. Tough to say where AR would fit-- maybe for training?

Dan Maloney12:36 PM
Or maybe it overlays the patient's vital signs?

Raj Nakarja12:39 PM
For sure! It's still quite early for this type of stuff, and I think we're days from things going crazy when Apple makes an announcement next week. There will be a lot of amazing immersive AR stuff coming, but we're starting off pretty simple. Monocle doesn't take up your whole vision, isn't 3D, and doens't track the environment in any way. However, as this industry evolves, we're going to see better displays and better controllers where it's possible to really overlay things onto the real world smoothly and without bulky hardware

kjansky112:40 PM
A worry with AI is when it hallucinates, what if in the middle of heart surgery it decides you are Vulcan "The insides of a Vulcan are completely different from a human's insides. They must be, if a Vulcan's heart is right around where a human's liver would be. This is great if a Vulcan were to get wounded in the chest region. But this is not so good if a Vulcan were to get wounded on the right side of their abdomen"!

Raj Nakarja12:40 PM
On that journey, our ideas of AR experiences will evolve. Who would have imagined all the smartphone apps we have today, before the smartphone arrived. For sure there'll be a lot of cool ideas

Raj Nakarja12:44 PM
Yeah, the same probably goes for AI. It's still so early and we probably have to be a bit careful in what we connect the AIs to. As they evolve though, maybe they will become mature tools that can genuinely offload the gruntwork we do every day, and let us focus on getting more done quicker

Raj Nakarja12:45 PM
I definitely think AI and AR will become intertwined though. That's partly why we added the camera. We hope someone figures out how to implement interesting AI models on the FPGA that make use of the camera

Dan Maloney12:48 PM
What's the battery life like? Seems like this might chew through batteries pretty fast

Raj Nakarja12:49 PM
This is the GPT app on Monocle It was with quite early firmware and he threw it together in a weekend. Most the processing is happening on his laptop, but we're working on a real version of this app that goes directly via a phone up to openAI

kjansky112:49 PM
Will general acceptance in public be a problem as the Google Glass found out or is a matter of desensitization takes place it will become less weird just like people talking to themselves in public has become commonplace with the advent of the smartphone.

Also, people just point their phone's camera at everything, everywhere and anytime. Why do people have different feeling about a smartphone and smart glasses when they are capable of the same?

Raj Nakarja12:50 PM
The normal runtime without the camera running is 45 mins, and with the camera it's half that. Monocle comes with a charging case however that has a battery inside. You can get 6 charges out of the case

Dan Maloney12:51 PM
Not that bad actually, all things considered.

Raj Nakarja12:53 PM
Yeah, I think cameras have become so common now that most folks aren't all that bothered. Google Glass was so early in that regard, it almost feels like a totally different era back then

Raj Nakarja12:55 PM
However, I do think people talking to themselves and pointing at things in the air probably won't be the way forward. That's a really tough UX problem. In our next product (Frame) we'll have a control ring that will work as a scroll wheel. You'll be able to look around and then click/scroll from the ring to interact much more subtly

Dan Maloney12:55 PM
Back to the optics for a second: that seems like it would be a tough thing to build. What's the manufacturing process like?

Joe joined  the room.12:57 PM

Raj Nakarja12:59 PM
Yeah, manufacturing is not easy, but out of all the different display technologies, it's still giving us great yields. We worked with an amazing optics partner who did tons of science on this for us, and we even got a patent on it. Mostly to prevent patent trolls. There's a lot of science in the patent which you can find online. The optics are two different types of plastic that are bonded with optical glue. The display is also bonded. Each plastic piece is simply injection molded and assembled in a jig

Dan Maloney1:02 PM
Sounds like a cool process, I'll have to read the patent.

Raj Nakarja1:02 PM
I remember looking at pages of documents they sent over with fringing diagrams, distortion, ghosting, and tons of stuff like that. Each was some trade-off for another. Then they were like. "Pick the one you prefer"

Dan Maloney1:03 PM
I recall doing an article on injection molded lenses, and seem to recall it was pretty challenging for the home-gamer. But not impossible

Raj Nakarja1:04 PM
The science is way over my head, but our CEO Bobak has become a real expert in this stuff. If you want to know more, you can jump into our discord and ask him. We also have the optics partner in there who pitch in sometimes

kjansky11:04 PM
Yeah the battery life for small wearables that are environmentally interactive is still a drawback, a possible solution would incorporate some type of wireless charging or even powering from something like a "docking" baseball cap with a larger capacity reserve.

Raj Nakarja1:05 PM
Yeah for sure. We had to go to a pretty special manufacturer to get the precision needed for these

Raj Nakarja1:06 PM
There's one guy in our discord who's made a couple of 3D-printed charging cradles. You couple probably extended it into a wearable battery-type thing. The charging pads on Monocle simply take 5V as an input

Raj Nakarja1:07 PM
*you could

Raj Nakarja1:08 PM There's one down the bottom

Dan Maloney1:08 PM
OK then, just noticed it's after 1:00, so we have to let Raj get back to work. Me too, actually ;-)

Raj, thanks for stopping by today, this was very interesting and I appreciate your time. And thanks to everyone for coming along for the ride!

Raj Nakarja1:11 PM
Thanks for the opportunity, and great talking to you all :) If you're interested to learn more about Monocle (and our future stuff), be sure to check out the docs:, our website, and the Discord where we hang out. Catch ya round!

Dan Maloney1:11 PM
Thanks Raj!