Automating Real Time Management, Manipulation, and Storage of Big Data…NSA Releases NiagaraFiles to Open Source Software

Automating Real Time Management, Manipulation, and Storage of Big Data…NSA Releases NiagaraFiles to Open Source Software


The value of discrete pieces of intelligence to national security varies for many reasons, but among them, its perishability may be the most important. To ensure that the most important information was identified and prioritized across its communications infrastructure, NSA developed NiagaraFiles (NiFi)—technology that has helped manage the flow of data for NSA and across the entire Intelligence Community (IC).

NiFi automates the management, manipulation, and storage of large streams of data in real time. The technology can interpret and transform information in a variety of formats, facilitating transfer across different systems and agencies. The technology also provides interactive command and control of data flows, allowing engineers to quickly make changes in how data is assessed and transmitted. By automatically embedding context into discrete data flows as they move across and between systems, NiFi creates a fine-grained chain of custody for information, which led to organic interest across the IC.

The technology was released in 2014 to the open source software (OSS) community as Apache NiFi, where its source code is freely available to view, modify, and adapt. Open sourcing engages a much larger development community to continue the technology’s evolution and ensure it remains secured from emerging vulnerabilities. The improved Apache Nifi has been distributed across the Agency and has become the foundation of many corporate dataflow services. More than 60 non-federal contributors have developed features for Apache NiFi that are important for both government and industry. For example, within a month of the OSS release, a contributor from Germany identified a way to reduce the time it takes to compile NiFi’s codebase by 75%, allowing even greater agility and flexibility as the technology continues to evolve.

Several hundred companies worldwide are using Nifi, including Fortune 500 companies ExxonMobil, AT&T, CapitalOne, and the United Kingdom’s British Gas. Hortonworks, a Silicon Valley company that specializes in the commercial distribution of open source big data software, has developed two NiFi products as part of its suite of open source big data technology offerings.

As a side benefit, releasing technology to OSS is of interest to IT professionals because it promotes a free, collaborative exchange of ideas that spurs rapid creative, scientific, and technical advances. Tony Kurc, an NSA technical director and recruiter, says that “highlighting Apache NiFi as an OSS technology that originated at NSA is a great tool in attracting the brightest new hires.” Examples of how companies use NiFi are available at

The NSA patent portfolio features all of the agency's technologies that are available to license. View the NSA patent portfolio here: