The value of discrete pieces of intelligence to national security varies for many reasons, but among them, its perishability may be the most important. To ensure that the most important information was identified and prioritized across its communications infrastructure, NSA developed NiagaraFiles (NiFi)—technology that has helped manage the flow of data for the National Security Agency (NSA) and across the Intelligence Community (IC).
NiFi automates the management, manipulation, and storage of large streams of data in real time. The technology can interpret and transform information in a variety of formats, facilitating transfer across different systems and agencies. By automatically embedding context into discrete data flows, NiFi creates a fine-grained chain of custody for information, which led to organic interest across the IC.
Released to the open source software (OSS) community as Apache NiFi (nifi.apache.org/powered-bynifi.html), the technology’s source code is freely available to view, modify, and adapt. More than 60 contributors have developed features for Apache NiFi that are important for both government and industry. For example, within a month of the OSS release, a German contributor reduced the time it takes to compile NiFi’s codebase by 75%, allowing even greater agility and flexibility as NiFi continues to evolve.
Several hundred companies worldwide are using NiFi, including ExxonMobil, AT&T, and British Gas. Silicon Valley company Hortonworks developed two NiFi products as part of its open source big data technology offerings. In 2018, Hortonworks merged with Cloudera, a leading developer of cloud-based software solutions, which will bring together Hortonworks' NiFi and Cloudera's data warehousing and machine learning technology to create next-generation, artificial intelligence-powered big data solutions.
As a side benefit, releasing technology to OSS is of interest to IT professionals because it promotes a free, collaborative exchange of ideas that spurs rapid creative and scientific advances. An NSA technical director and recruiter says that “highlighting Apache NiFi as an OSS technology that originated at NSA is a great tool in attracting the brightest new hires.”
About the NSA TTP
The NSA TTP establishes partnerships between NSA and industry, academia, and other government agencies to help accelerate mission, advance science, foster innovation, and promote the growth and commercialization of technology originally created for Agency mission.
Questions? Contact NSA’s Technology Transfer Program
Office of Research & Technology Applications, NSA Research
www.nsa.gov/techtransfer | tech_transfer@nsa.gov