The scope works using a comparator to sample the input waveform. This is an old idea, which seems to date back to S. P. McCabe III's Masters Thesis in 1975, "A sampling voltage tracker for analyzing high speed waveforms." I haven't read it yet, but next time I'm in Los Angeles, maybe I'll swing by UCLA and have a look.
This approach was finely honed by a team at NIST, who wanted to use such a sampler for high-resolution measurements of RF waveforms, initially to replace thermal sensors in true-RMS-reading RF voltmeters. Since the NIST work is US-Govt. funded, the papers they published are not protected by copyright. Here's a good introduction, highlighting some of their latter work on the idea.
The basic idea is to capture a single comparison result at each trigger. So, at the first trigger, you ask if the waveform at the trigger time is above or below a specific voltage. By trying a number of voltages, and recording the greater/less-than answers from the comparator, you can eventually deduce the voltage at that time. If you've ever explored successive-approximation analog-to-digital converters, you'll recognize a similar process. Unfortunately, the binary search used in most SAR ADCs is sensitive to noise, and is not always suitable for a sampling oscilloscope. There are a number of alternative approaches, and a small body of literature on research in this area. The simplest (and perhaps slowest) method is just to try all possible voltages. It's dumb, but it works, and is resilient in the presence of noise.
I'm just going to link the PDF schematic here. The ADCMP582 comparator is the front-end. An MCP1501 2.048 V reference, MCP4921 12-bit DAC, and ADA4000 op-amp form the reference voltage generator that the input sample is compared to. The reference voltage has a range of -2 to +3 V, covering the entire input range of the comparator. In future versions, this may be reduced to +/- 2V. This DAC uses SPI and is relatively slow. I'm currently evaluating faster, parallel DACs.
I swapped op-amps at the last minute in the design, and accidentally connected the inputs backwards (doh!), so these are incorrect in the schematic and board layout, and the board requires a little rework to swap these pins before it will work correctly.
An SY89296U programmable delay line handles timing of the samples. This allows 1024 different sample points, spaced nominally 10 ps apart, for an equivalent time sample rate of 100 GSa/s. On the first sampler prototype, this timebase is uncalibrated. I have a prototype timebase calibration board which has proven a way to calibrate the timebase, and it will be intergrated into the next version.
Since the comparator and delay line are ECL parts, there's a little ECL interface logic to connect to the CMOS parts of the design.
The brains of the PCB is a PIC16F15376 8-bit microcontroller. I chose this because of familiarity with the 8-bit PIC world and previous experience with the built-in peripherals. The inclusion of a 32-bit uC here wouldn't accelerate the design much and simply add complexity to this first prototype. The next spin currently has an FPGA to handle anything that should go faster.
Another key component is the Si53360 clock distribution buffer. This CMOS part handles the clock and trigger timing while adding very little jitter (120 fs RMS). The two inputs to the clock fanout part allow the sampler to be triggered from an external source, or to generate its own sampling clock. This functionality directly parallels the Tektronix 11801 sampling oscilloscope. It works pretty well.
Again, a link to a pdf of the layout. This first prototype was put together quickly to get some data before a next spin, so it's a little rough. At these speeds, though, there's no substitute for measurements with the actual device, so it makes sense to build and test early and often. This is just the first prototype which contained enough stuff to make actual measurements.
The board is a 4-layer design fabricated at OSH Park. One of the three PCBs I received seems to have a short between two internal layers, but I haven't fully tracked it down yet, so I'm not pointing any fingers. Besides, to make a good footprint for end-launch SMA connectors, you are forced to violate the design rules, especially about copper pullback at the board edges.
A key piece, and probably one I don't have quite right yet, is keeping the power supplies quiet. I have two separate regulators for the 3.3 V supplies for the ECL and CMOS parts of the design, but this in itself is probably not sufficient. One of the tasks for the next spin is to take a very close look at the power distribution network with an eye to reducing jitter.