Close
0%
0%

Moving the DVD archive to an 8TB hard drive

Transferring 2TB of optical storage created from 1997-2010 to the hard drive.

Similar projects worth following

The original vision was a big UNIX box sitting in the middle of a living room, like an Onyx 2.  Some of the early UNIX boxes might have been designed as pieces of furniture.  It would contain all the media the lion kingdom ever had, on some kind of 200GB internal disk.  Optical storage became cheaper & more reliable than hard drives in the late 1990's, so the plan shifted to a giant robot which would make thousands of disks behave like a slow hard drive.

The robot was originally supposed to load the disks into a desktop drive.  Commercially produced robots of the time did it for a lot more money & much less capacity.  The lion kingdom thought it could save money by relying more on machine vision & wood.  As the problem grew in complexity, the robot was pared down to just an organizer.  

The robot was built twice.  The 1st one was made of balsa.  

 The 1st tower had 1 disk per row & took an eternity to fabricate.  3 more  towers were built with 2 disks per row & a low density loading tower.  It was fully functional, by this point.  It would transfer disks from high density towers to a loading tower closer to the computer, where a lion could swap them in the drive.  Various attempts to move the disks into the drive by suction proved too bulky & slow.

They still had no caddies & risked getting scratched.  After discovering the weight of the disks was greatly underestimated & ambient light could erase them, it was rebuilt out of MDF with caddies.  3 high density towers & a low density loading tower were built.  The 1st tower was discarded, after finding problems.

The amount of sawdust from fabricating those was quite disastrous.  This worked as designed for many years, until the capacity was used up.

After encountering the need to build another tower, a very high error rate for optical storage, very high tweeking required to keep the robot running, exploding rent prices, & magnetic storage improving, the last optical disk was burned in 2011.  By then, the robot wasn't running anymore because there wasn't enough room for the crane to move around.  Most of it was dismantled for spare parts.  The towers may eventually be sold.

Detailed video of Heroine2200 when it was new, showing some of the problems it had to overcome. Based on later videos, it needed a lot of manetenance. Lions don't remember most of how it worked. It clearly could have been much better with modern methods, but there would be no point.


The error rate grew as these disks aged, manely in the outer tracks.  They were damaged by ambient light & oxygen getting into the laminations.  The oxygen leakage became famous, but was never fully solved.  Considering how sloppy the tracking of a removable disk like this is, how the disk flops up & down while its diameter oscillates in & out, how it gets scratched, it's amazing optical disks stored as much as they did.  

A very useful utility in those days was 

https://cdn.hackaday.io/files/1592676814079328/badfile.c

Since most of the data was multimedia & most of it can now be gotten from the goo tubes, errors are tolerable, but there was no way to preserve the block order in a file copy.  Without preserving the block order, the quicktime/mp4 files couldn't be played.  The mighty dd command

dd if=/cdrom/janet1/track01.wav of=janet1/track01.wav bs=4096 conv=noerror

would shift the blocks when it encountered an error.  badfile.c would pad the bad sectors with 0.  The lion kingdom went through many DVD drives.  The best one was a 2013 Macbook Pro.  They all eventually died.

THE DESTINATION DRIVE

The dream of a single UNIX box storing an entire life's media finally became attainable, though still requiring...

Read more »

badfile.c

Copies damaged files while preserving the block order.

x-csrc - 2.73 kB - 06/25/2018 at 04:16

Download

  • Forgotten codecs

    lion mclionhead07/06/2018 at 00:43 0 comments

    Not surprisingly, a lot of files can't be decoded anymore because of lost codec support.  



    A bit of debugging gets it most of the way, for something which can be bought from Bezos Inc in higher quality.

    MPlayer still decodes everything properly, but can't do anything else.  

     The Xena episodes at 15fps were manely IVTC, so there was no way to increase the framerate by bobbing.  The bobbing would work when IVTC occasionally missed, but it usually just showed the fields out of order, making playback studder.  It was a reasonable sacrifice to get higher spatial resolution.

    After 2 weeks with 3 drives working, all 2TB of data that was known to be unique was transferred.  It was easier to transfer everything than figure out what was worth it.  Most definitely wasn't worth it.  Since many were backups, it was about eyeballing directories to discover dupes.

    Tokyo in 2002, a small world to live out an entire life, with little to seek after besides just surviving.  It might be why people in even poorer countries than US & millenials travel a lot more.

  • Forgotten files

    lion mclionhead07/04/2018 at 21:57 0 comments

    It's surprising how many videos the lion kingdom has no memory of recording.  In most cases, lions can remember making a video on an invention, but nothing about how the invention was made.  It's a case of lion intelligence being augmented by machines.

    In this case, it was a forgotten 2 way serial port.  It sent data over 72Mhz & received data over the audio channel of a 2.4Ghz cam.  The cheapest UART radios of the time were $40 & there was no such thing as editing config files on a phone & sending them over bluetooth.  72Mhz & 2.4Ghz couldn't operate simultaneously.  They needed a 1/2 second delay between sending over 72Mhz & receiving over 2.4Ghz.  It took a piece of heroic PIC assembly language to make it transparent to the user.  It was never used.

  • The Xena transfer

    lion mclionhead07/03/2018 at 03:27 0 comments

    After 1 week, all the Xena episodes were transferred. They were all on 1 or 2 CD-R's, manely 15fps, Oxygen split screens which could be somewhat increased in framerate by converting frames to fields. Some were rerecorded after the dot coms crashed & Oxygen converted to a sane full screen. Some were recorded from WB.


    They used 2 or more encoding passes, to get the file size as close to a CD-R as possible. Only near the very end was inverse telecine good enough to encode a complete episode at 24fps on a CD-R. They were all MPEG-2 video + MPEG-2 audio in program streams, with no B frames. There was no source code for compressing audio + video in Quicktime.


    Some had the black levels lowered, but theoretically the lion only lowered to the darkest part of the video. No such thing as histogramming during playback, in 2001. The very last episodes were non standard Quicktime mashups of MPEG-4 + OGG Vorbis, at 24fps, on a single CD-R, in 2004. The quality was lightyears ahead of its time.


    Remembered the router had a very slow DVD drive so 3 disks could be copied, simultaneously. Virtualbox constantly crashes when reading the Mac DVD drive. The most reliable drive is a Matsushita in the Macbook. The fastest drive is a Lucky Goldstar.


    It was a highly debated move.  There weren't any significant errors.   For only the price of an Amazon prime subscription, you can view all the Xena episodes in full framerate, full digital, low definition.  The only compromise is a lousy player with no dynamic range compression, no bookmarking, & no access once you stop feeding Bezos.

    The past obsession with codecs was quite different than working with video for a living.  Doing it for a living means creating set top boxes.  You're implementing standards on the cheapest hardware available, with almost no regard for quality.  There's some concern with making the network streaming reliable, making the GUI as responsive as possible.  The amount of information in the number of bits is all defined in standards from long ago, far below the theoretical limits & the GUI maxes out at awful by the price of the hardware.  Nowadays, there's little need to max out a codec anyway, since you can throw endless bits into an 8GB hard drive & the artifacts are invisible in 4k.

View all 3 project logs

Enjoy this project?

Share

Discussions

Clayton G. Hobbs wrote 07/04/2018 at 23:28 point

There's a typo in the title.  I doubt 2 TB of DVDs could fit into a mere 8 GB, no matter how good your compression is. 😉

  Are you sure? yes | no

Owen Trueblood wrote 07/04/2018 at 15:00 point

Amazing! Would love to live in a space with one of these doing its thing in the background. Any chance of getting a video of it in operation?


  Are you sure? yes | no

Mike Szczys wrote 06/26/2018 at 18:28 point

Oh my, those towers of robotic DVD racks are awesome! You've got to give us more details on that build :-D

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates