Revolutionizing AI: How Lightwave-Connected Chips are Expanding Model Size and Accelerating Training Speed

AI expansion is limited by data transfer speeds between processing chips, yet utilizing light for data transfer could eliminate the limitations

A new method for connecting chips could assist in overcoming the “memory wall,” which restricts computing speed and the expansion of AI models currently, by transferring data along adjustable pathways of light instead of electrical wires.

This technology will be configured by a project led by U-M, funded by a $2 million grant from the National Science Foundation’s Future of Semiconductors initiative. The project also comprises researchers from the University of Washington, University of Pennsylvania, Lawrence Berkeley National Laboratory, along with contributions and direction from four industrial collaborators: Google, Hewlett Packard Enterprise, Microsoft, and Nvidia.

Despite the fact that data processing speed nowadays is 60,000 times greater than it was two decades ago, the velocity of data transfer between computer memory and processors has only increased by 30-fold. This delay has established the rate of data transfer as a bottleneck for the expansion of AI models, which have been increasing 400-fold every two years since 1998. Enhanced communication is crucial for overcoming these limitations on AI capability.

“Our recommended technology could allow high-performance computing to match the enormous data volumes that are being supplied to the swiftly expanding AI models,” stated Di Liang, U-M professor of electrical and computer engineering and principal investigator for the project. “By implementing optical connections between chips, we believe we can transmit tens of terabits per second, which exceeds state-of-the-art electric connections by over 100 times.”

Currently, data travels between various memory and processor chips via metallic connections soldered onto a single physical unit referred to as an interposer, which resembles a motherboard. Data can be transmitted within one interposer or across interposers on linked servers known as computing nodes.

The metallic connections are permanently wired into the interposer, which restricts data transfer bandwidth and signal fidelity since faster electrical signals lose energy as heat and can electromagnetically disrupt adjacent connections. Consequently, permanently wiring connections to all the various processors and memory chips is not feasible. A contemporary supercomputer chip can comprise over 900,000 cores, or separate processing units, and that figure will continue to rise with the growth of AI model sizes.

“Each of those processors will require communication with a substantial quantity of memory,” remarked Mo Li, professor of electrical and computer engineering at the University of Washington, and co-principal investigator for the project. “Managing the communication throughout the entire package is crucial. In my perspective, optical connections will be the only feasible solution going forward.”

Light can travel further than electrons and convey a significantly larger volume of data with considerably less energy loss, and the researchers intend to capitalize on these properties in their new interposer design. Light pulses will move between chips through refractive paths in their interposer known as optical waveguides. A receiver on each chip will convert the data back into an electrical signal for the computer to process.

The waveguide network is also reconfigurable—both during production and within a computer—thanks to a unique phase-changing material embedded in the interposer. When subjected to a laser or electric voltage, the material’s refractive index alters, causing the light to bend in various directions as it traverses the waveguide.

“It’s somewhat analogous to opening and closing roads,” commented Liang Feng, professor of materials science and electrical and systems engineering at the University of Pennsylvania and co-principal investigator. “If a company markets a chip utilizing this technology, they will be able to modify the connections on different batches of chips and servers without altering the layout of the other components.”

The researchers will develop a traffic-management software that observes which sections of the interposer require communication at any particular moment and adjust the necessary voltage to establish optimal connections in real-time.

“Altering the connections enables us to restructure the network based on the AI models we wish to execute, or whether we are training or running a model,” noted Reetuparna Das, associate professor of computer science and engineering and co-investigator of the project.

In addition to technological advancement, the initiative will also link U-M students with industry partners and provide invaluable hands-on experience.

“These collaborations allow students to confront real-world challenges in designing swiftly evolving technology,” Liang expressed. “Textbooks fail to adequately address these contemporary issues as the pace of advancement makes it unfeasible for textbooks to keep pace. The optimal way to acquire relevant skills is by collaborating with industry on the challenges they prioritize.”