Teraflow Testbed: Sloan Digital Sky Survey
Website
http://www.ncdm.uic.edu/http://www.teraflowtestbed.net/
Collaborators
Australia: University of MelbourneChina: Chinese Academy of Sciences (CAS); Computer Network Information Center; National Astronomical Observatory
Germany: Garching Computing Centre, Max-Planck-Institut für Plasmaphysik
Japan: Institute for Cosmic Ray Research, University of Tokyo; JGN2
The Netherlands: SARA, University of Amsterdam
Russia: Russian Academy of Sciences; Space Research Institute (IKI)
South Korea: Korea Astronomy and Space Science Institute; Korea Institute of Science and Technology Information; KREONet2
Switzerland: CERN
United States: National Center for Data Mining, University of Illinois at Chicago; Johns Hopkins University; University of California San Diego; NASA Goddard Space Flight Center; StarLight; TransPAC2
Description
The Teraflow Testbed was extended to Russia in 2007. GLORIAD involved its Russian partners in the Teraflow project now that there are several gigabits of bandwidth between Chicago and Moscow (3 x 1Gb VLANs on TransLight/StarLight from Chicago to Amsterdam, and 10 Gb from Amsterdam to Moscow funded by the Russians).
On 14 May 2007, the first experiment was to download and create a mirror site for Sloan Digital Sky Survey (SDSS) data on a data server at the Russian Space Research Institute (IKI). SDSS data was transferred from University of Illinois at Chicago (UIC) National Center for Data Mining (NCDM) servers to an IKI mirror site over the TransLight/StarLight link. NCDM moved 1.4 TeraBytes (TB) of data in about 4.5 hours over a 1 Gbps VLAN between NCDM in Chicago and Moscow. Using NCDM’s transport protocol UDT (UDP-based Data Transfer), the SDSS 2.5 TB catalog was compressed to 1.4 TB, split into 60 files, distributed, and then decompressed in Moscow to its original size.
In December 2007, IKI installed NCDM’s UDT relay server in order to disseminate large scientific databases and visualization content. IKI has climate change and space weather databases ready for delivery to institutions that are part of the World Data Centers System under ICSU. These databases will be a major addition to the SDSS database. IKI plans to add open source port relaying services to the UDT protocol stack and will maintain the code at SourceForge. IKI also plans to use UDT to share satellite data with the National Geophysical Data Center at NOAA in Boulder.