Business Wire

Red Hat Launches the llm-d Community, Powering Distributed Gen AI Inference at Scale

Forged in collaboration with founding contributors CoreWeave, Google Cloud, IBM Research and NVIDIA and joined by industry leaders AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI and university ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
Forbes

Cerebras Now The Fastest LLM Inference Processor; Its Not Even Close

The company tackled inferencing the Llama-3.1 405B foundation model and just crushed it. And for the crowds at SC24 this week in Atlanta, the company also announced it is 700 times faster than ...
InfoQ

GPULlama3.java Brings GPU-Accelerated LLM Inference to Pure Java

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...
Semiconductor Engineering

Four Architectural Opportunities for LLM Inference Hardware (Google)

“Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI ...
Business Wire

Enfabrica Unveils Industry’s First Ethernet-Based AI Memory Fabric System for Efficient Superscaling of LLM Inference

MOUNTAIN VIEW, Calif.--(BUSINESS WIRE)--Enfabrica Corporation, an industry leader in high-performance networking silicon for artificial intelligence (AI) and accelerated computing, today announced the ...
Semiconductor Engineering

Ultra-low-bit LLM Inference Allows AI-PC CPUs And Discrete Client GPUs To Approach High-end GPU-Level (Intel)

A new technical paper titled “Pushing the Envelope of LLM Inference on AI-PC and Intel GPUs” was published by researcher at Intel. “The advent of ultra-low-bit LLM models (1/1.58/2-bit), which match ...
NextBigFuture

OpenAI Strawberry LLM Reasoning Needs More Compute and Energy for Inference

Jim Fan is one of Nvidia’s senior AI researchers. The shift could be about many orders of magnitude more compute and energy needed for inference that can handle the improved reasoning in the OpenAI ...
InfoQ

Cactus v1: Cross-Platform LLM Inference on Mobile with Zero Latency and Full Privacy

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...