Share Button

With a growing list of devices in our day to day lives, personal data storage needs have expanded beyond a single machine’s storage.

Functionally, this is similar to paper and non-digital copy days where one would end up dealing with the following –

  • Printed photographs
    • Stored in albums or just envelopes stacked somewhere
  • Original, copies and hand-written documents of various types
    • Photocopies of documents
    • Printed documents
    • Stored in filing cabinets, very hard to organize
  • Audio and Video
    • Vinyl records (gaining traction again)
    • Music audio tapes
    • Home video tapes
    • Entertainment / movies (VHS, CD, DVD and older)
    • Phone voicemail tapes
    • Home security camera footage
    • Stored physically next to the entertainment center in a shelf or storage totes

 

Here is my list of personal data storage needs –

  • Photographs and Videos
    • 10,000 photos and growing
    • 800GB+ of personal videos and growing
    • From cell phones
      • Uploaded to social media
      • Copied to local HDD in a date organized folder structure
      • Backed up with phone backups on the desktop
    • From DSLR
      • Copied to local HDD in a date organized folder structure
    • From GoPro and other cameras (after motorcycle or flying trip)
      • Copied to local HDD in a date organized folder structure
    • From other people’s devices
      • Received on email
      • Received through social media, manually saved somewhere
    • From security cameras
      • Stored in a DVR or shared storage device
    • Processed content generated with video / photo editing software
  • Documents
    • Financial – Bank statements, shopping receipts, etc.
    • DMV registration and payment receipts
    • Medical – Bills, insurance and payment receipts
    • Home mortgage, insurance and warranty documents
    • Personal documents – passport, travel history, old mark sheets and certificates
  • Books
    • Kindle / iBooks – cloud sources
    • PDF and other formats
  • Audio / Video / Entertainment
    • iTunes, Amazon MP3 and other sources
    • MP3 collection encoded from audio CDs
    • Cached copies of iTunes and other online media

 

ALL of the above are now available for download / local storage or accessible on demand from the cloud. One could argue that everything can be uploaded to one cloud service or another but that quickly gets very expensive and those cloud services often don’t survive through mergers and acquisitions with competition.

 

Of course enterprise storage needs extend beyond the basics but if you consider the “functional requirements” for an average personal data storage solution, they are not very different from basic enterprise storage needs –

  • Must maintain data consistency
    • Should not corrupt data without external influence
    • Must handle scenarios including corruption initiated between the keyboard and the chair
  • Must be redundant
    • Failure of a single (or even double) storage component should not cause complete loss of data
  • Accessibility
    • Based on personal preference, data must be available on multiple devices
  • Availability
    • Must be up and running when needed, in some cases for days (weekends?)
  • Must support Disaster Recovery
    • Home fire scenario
    • One copy stored off-site
  • Speed
    • Must be fast enough to handle regular data backups without hours of transfer time
    • Must be fast enough to handle photo and video processing workloads without choking the editing workflow
  • Upgradable / Expandable
    • Must support swapping out disks to add more space in the future
  • Cost
    • Must not cost a fortune
  • Support
    • For commercial products, some level of support would be good

 

Here are some of the storage options that were traditionally considered only in enterprise settings –

Larger Internal Storage

  • Data Consistency – Good
  • Redundancy – Possible but most consumer desk/laptops do not include RAID
  • Accessibility – Limited to local machine, except network sharing which is not very convenient
  • Availability – Good
  • Disaster Recovery – Needs extra work
  • Speed
    • Good – Typically 5400 RPM / 3Gbps SATA (slow)
    • Better – 7200 RPM / 6Gbps SATA. 12Gbps options not yet that common
    • Best – Larger SSDs – Still very expensive
  • Expandability – Possible but painful to reinstall / copy everything
  • Cost – Varies with needs
  • Support – Bundled with unit

DAS – External Direct Attached Storage

  • Data Consistency – Can vary with unit
  • Redundancy – Possible with Hardware or Software RAID
  • Hardware RAID is more expensive
  • Software RAID is less reliable
  • Accessibility – Limited to local machine, except network sharing which is not very convenient
  • Availability – Good
  • Disaster Recovery – Needs extra work
  • Speed – Depends on interface and disk speed
  • Expandability – Possible but most RAID units force you to re-create / re-copy everything
  • Cost – Varies
  • Support – Bundled with unit
  • Examples – DROBO, OWC ThunderBay, etc.

NAS – Network Attached Storage

  • Data Consistency – Can vary with unit
  • Redundancy – Possible with Hardware or Software RAID
  • Hardware RAID is more expensive
  • Software RAID is less reliable
  • Accessibility – Very good
  • Availability – Good
  • Disaster Recovery – Needs extra work
  • Speed – Slow – Network access adds to latency and transfer speeds
  • Expandability – Possible but most RAID units force you to re-create / re-copy everything
  • Cost – Expensive
  • Support – Bundled with unit
  • Examples – Lot of them – DROBO, Synology, QNAP, Netgear ReadyNAS, Buffalo Linkstation, etc.

Cloud Storage

  • Data Consistency – Good
  • Redundancy – Good
  • Accessibility – Good
  • Availability – Good
  • Disaster Recovery – Good
  • Speed – Very Slow – Network access adds to latency and transfer speeds
  • Expandability – Very easy, not a factor
  • Cost
    • Expensive for online mount-able access like Dropbox or Box.com
    • Cheap if only used for off-site data dumps – Backblaze.com
  • Support – Varies by provider
  • Examples – Backblaze, Dropbox, Box.com, Various cloud drives – iCloud, Google Drive, etc.

 

Reading through the options above, there is no single solution to the list of “functional needs”. To cover everything, a combination of the following is typically what works –

  • Medium sized local SSD storage
    • Provides speed and availability
  • (Optional) Large direct attached storage
    • Provides speed, expandability, availability
  • Large Network attached storage
    • Provides expandability, availability and accessibility
  • Data replication to the Cloud for off-site storage
    • Provides Disaster recovery, availability, accessibility

Data transfer speeds for local, DAS and NAS vary wildly depending on the components used, specially the choice of hard drives, SSDs and controllers.

But that is a topic for another article.