The TCB's server infrastructure is primarily dedicated to providing a centralized data storage for all of the Resource's needs, protecting the valuable data from loss by hardware failure and, to a lesser extent, human error. Since it was first developed in 1999, the system has proved both reliable and readily expandable, offering remarkably few compromises as our data needs have grown exponentially. We expect that we will be able to continue this growth, and hope that our experiences may help other groups that are struggling with the same needs.

Cluster GPU

Cluster GPU

Cluster GPU

Cluster GPU

SunFire X4270 x2 with
Dell PowerVault MD 3260
(240TB Storage)
Intel Xeon E5-2637 -
24 TB SSD Internal Storage
SunFire X4200 with
Sun StorageTek 2500
SunFire X4540-AMD Opteron(tm) -
86 TB Internal storage

System Specifications

  • Disk Arrays
    • Sun StorageTek 2500 x3 (8.4TB)
    • Dell PowerVault MD 3260 x4 (240TB)
  • File Servers
    • SunFire X4200 x2
    • SunFire X4270 x2
    • SunFire X4500 - 42TB x2
    • SunFire X4540 - 42TB
    • SunFire X4540 - 86TB x3
    • Intel Xeon X5650 x2
    • Intel Xeon E5-2637 - 24 TB SSD x2

File Servers

We use redundant file servers running Oracle Solaris, Linux, FreeBSD to share our files across our network. These systems offer three major benefits:

  1. An enterprise-class - ie stable and powerful - NFS server package.
  2. Support for the ufs and zfs file system and filesystem snapshots.
  3. Security and reliability.

The disks are divided up into the following categories:

/home and /Projects

These disks are backed up regularly. /home partitions are made for every user of the system; each disk is shared across many (40+) users with quotas.

/Projects partitions are allocated to users that show need for them. In general, they are allocated on a per-person basis; exceptions to this rule are permitted only after discussion with the sysadmin team. To this end, partitions are generally either 150GB or 300GB.

/Scr

These disks are not backed up, but are otherwise treated like /Projects spaces. They are also on ZFS pools with quota.

Disk Arrays

Unlike many groups, we use ZFS RAID and the ZFS file system, a feature with Oracle Solaris, Linux and FreeBSD Operating System. ZFS allows for greatly simplified storage management and snapshot based backups.

Backups

Backups for home and project spaces are sent nightly via ZFS snapshot to multiple backup servers in a different machine room. Three nightly snapshots are retained on the live server to allow users to recover accidentally deleted files without administrator assistance.