BROOKLYN, New York, Wednesday, July 21, 2021 – This week at the 38th International Machine Learning Conference (ICML 21), researchers from the NYU Center for Cyber Security at the NYU Tandon School of Engineering reveal new features that drive the ability of neural networks to make inferences about encrypted data.
In the article “DeepReDuce: ReLU Reduction for Fast Private Inference”, the team focuses on linear and nonlinear operators, key characteristics of neural network infrastructures which, depending on the operation, introduce a heavy toll in time and time. computing resources. When neural networks calculate on encrypted data, many of these costs are incurred by the Rectified Linear Activation Function (ReLU), a nonlinear operation.
Brandon Reagen, professor of computer science and engineering and electrical and computer engineering and a team of collaborators including Nandan Kumar Jha, a Ph.D. and Zahra Ghodsi, a former doctoral student under the supervision of Siddharth Garg, developed a framework called DeepReDuce. It offers a solution through the rearrangement and reduction of ReLUs in neural networks.
Reagen explained that this change requires a fundamental reassessment of where and how many components are distributed in neural network systems.
“What we’re trying to do is rethink how neural networks are designed in the first place,” he explained. “You can skip a lot of those computationally and time consuming ReLU operations while still getting high performing networks with 2-4 times faster runtime. “
The team found that, compared to the state of the art for private inference, DeepReDuce improved accuracy and reduced the number of ReLUs by up to 3.5% and 3.5 times, respectively.
The investigation is not only academic. As the use of AI grows alongside concerns about the security of personal, corporate, and government data, neural networks are increasingly performing calculations on encrypted data. In such scenarios involving neural networks generating private inferences (PIs) of hidden data without disclosing the inputs, it is the nonlinear functions that exert the greatest “cost” in time and power. Since these costs increase the difficulty and time it takes for learning machines to perform PI, researchers have struggled to alleviate the load that ReLUs place on such calculations.
The team’s work is based on an innovative technology called CryptoNAS. Described in an earlier article whose authors include Ghodsi and a third doctorate. student, Akshaj Veldanda, CryptoNAS optimizes the use of ReLUs because one could rearrange the way rocks are arranged in a stream to optimize water flow: it rebalances the distribution of ReLUs in the network and removes redundant ReLUs.
DeepReDuce expands CryptoNAS by further streamlining the process. It includes a set of optimizations for the judicious removal of ReLUs after CryptoNAS reorganization functions. Researchers tested DeepReDuce using it to remove ReLUs from traditional networks, finding that they were able to significantly reduce inference latency while maintaining high accuracy.
Reagan, along with Mihalis Maniatakos, an assistant research professor in electrical and computer engineering, is also part of a collaboration with data security firm Duality to design a new chip designed to handle computation on fully encrypted data.
Research on ReLUS was supported by ADA and the Data Protection in Virtual Environments (DPRIVE) program of the US Defense Advanced Research Projects Agency (DARPA) and the Center for Applications Driving Architectures.
About New York University Tandon School of Engineering
The NYU Tandon School of Engineering dates from 1854, when the New York University School of Civil Engineering and Architecture and the Brooklyn Collegiate and Polytechnic Institute were founded. A January 2014 merger created a comprehensive school of engineering and applied science education and research as part of a global university, with strong ties to the engineering programs of NYU Abu Dhabi and NYU Shanghai. NYU Tandon is rooted in a vibrant tradition of entrepreneurship, intellectual curiosity, and innovative solutions to humanity’s most pressing global challenges. Research at Tandon focuses on the vital intersections between communication / computing, cybersecurity and data science / AI / robotics systems and tools and the critical areas of society they influence, including emerging media, health, sustainability and urban life. We believe that diversity is integral to excellence and we create a dynamic, inclusive and equitable environment for all of our students, faculty and staff. For more information, visit engineering.nyu.edu.
Warning: AAAS and EurekAlert! are not responsible for the accuracy of any press releases posted on EurekAlert! by contributing institutions or for the use of any information via the EurekAlert system.