Parallel Log-structured File System (PLFS)

Parallel Log-structured File System (PLFS)

Together, Los Alamos National Laboratory (Los Alamos) and EMC, are enhancing, designing, building, testing and deploying new cutting-edge technologies in an effort to meet some of the nation’s most difficult information technology challenges. Thus far, the LANL and EMC collaboration has been engaged in high-performance computing and data storage research, as well as large-scale analytics.

One of EMC’s innovative products is a flash appliance, called the Active Burst Buffer Appliance, or aBBa, which helps extreme scale high-performance computing set-ups run faster and smoother. It acts like a very fast book mark: when one of the millions of parts that make up a supercomputer fails; the tightly coupled application can quickly get back to where it was.

A key software development in the operation of and other computing platforms is Parallel Logstructured File System (PLFS). PLFS, designed and developed through a Cooperative Research and Development Agreement with Los Alamos and EMC, is an open-source, extremely scalable data-management middleware library that can be used with everything from small clusters of computers to the largest supercomputers in the world. This technology is file-system agnostic and could improve computing efficiency significantly.

PLFS is a parallel IO abstraction layer that rearranges unstructured, concurrent writes by many clients into sequential writes to unique files (N-1 into N-N) to improve the efficiency of the underlying parallel file system. EMC is now demonstrating aBBa at High Performance Computing (HPC) trade shows. PLFS also features as a core technology in the DOE’s FastForward program, targeted at eliminating barriers on the path to exascale computing.

PLFS work at Los Alamos is built on a foundation of years of storage and I/O leadership in HPC, directly impacts execution of the Laboratory’s mission via a variety of computing platform as well as highlight the Laboratory’s broader contribution as this key HPC technology is developed and deployed for the first time at extreme scale locally at Los Alamos.

Parallel storage infrastructure is a growing market segment in an overall multi-billion dollar industry. Los Alamos has been instrumental, first in developing parallel storage infrastructure for HPC, and now in partnering with companies who are using these concepts, including PLFS, to extend parallel storage infrastructure to a customer base spanning every possible market segment; see, for instance,