Airbus Defence And Space Launches a Petascale Archive for Satellite Data With StorNext Scale-out Storage

Archiving Petabytes of Satellite Data

European Space Agency (ESA) is launching two satellites for each Sentinel mission—within a few years of the program’s start, ESA will have four Sentinel satellites in orbit. Each satellite streams more than 1PB per year, for up to 15 years. Retaining all this data required a new archive. “We archive not only the raw, level 0, instrument data but also data processed by ESA’s Payload Data Ground Segment, which generates level 1 and level 2 products,” says Mark Curtis, technical design authority for Airbus Defence and Space. “To provide a long-term archive for all that data, we needed a storage solution that could scale to tens of petabytes.”

Managing Millions of Files, Leveraging the Economics of Tape

“When there are four satellites in orbit, we’ll need to archive 40,000 files per day,” says Curtis. “We needed a robust file system to support tens of millions of files.” And it needed to support a wide variety of sizes and types.

“Controlling costs is critical since we need to store tens of petabytes of data for the Sentinel missions,” says Curtis. “Tape is very economical: Each tape cartridge can store a huge amount of data inexpensively, and tape libraries require less power and cooling than higher speed storage systems.” And tape is well suited for applications where large volumes of data are streamed.

Building a Scalable Multi-tier Archive with StorNext

The Airbus team selected a multi-tier StorNext solution, including StorNext AEL6000 tape archives, metadata appliances, and disk cache—all powered by StorNext. StorNext was chosen in part because of previous experiences with Quantum at Infoterra, a supplier of images for Google Earth that is now part of Airbus Defence and Space.

“Quantum has always listened to our requests in improving their solutions,” says Curtis. “In the case of StorNext, Quantum incorporated input from us and other organizations, which helped produce a stronger storage solution.”

The archive initially supports 18PB of data, but can scale up to 37PB. “StorNext allows us to easily scale our archive," Curtis explains, "so we can support new satellite missions and collect additional data well into the future.”

Enabling Continuous High-speed Data Ingest and Access

“Our current configuration supports 1.1GB/s performance with the StorNext file system,” says Curtis. “As a result, we can rapidly ingest all the raw and processed data we receive, and quickly retrieve it when necessary.” In most cases, the Airbus team can retrieve individual files from the archive in under a minute, and Fibre Channel connections mean that large transfers are also delivered rapidly. The system supports a full range of operating systems, including both Linux and Windows.

Safeguarding Data While Keeping Staff Levels Low

The system’s extended data life management (EDLM) features help safeguard data. “The policy-based EDLM features of the StorNext tape archive help proactively maintain the integrity of data on the tapes over the long term,” explains Curtis. “If a problem develops with a tape, StorNext automatically migrates data to better tape, without our administrators having to lift a finger. EDLM provides a huge advantage.”

“Administration costs with StorNext are quite low,” says Curtis. “With the Quantum solution, two of us can easily manage this growing petascale archive of satellite data.”

Gaining Flexibility for Unexpected Data Growth

StorNext also helps meet changing requirements. A newly planned relay satellite system, for example, is expected to increase storage needs beyond original estimates. “Fortunately, StorNext enables us to easily scale our environment. We can add new archive systems—and move to LTO-7—to meet these new data collection requirements without a major overhaul,” says Curtis.

“As ESA launches the next series of Sentinel satellites, storage requirements could change again,” says Curtis. “With the StorNext solution, we’re prepared.”