Large language models are powerful, but factory floors don’t have A100s. In my research at the NCU Wireless Ad-Hoc & Sensor Networks Lab, I worked on a question that matters for real industrial deployment: how small can an LLM get before it stops being useful?

The Goal: LLMs on Factory Edge Devices

The target scenario is industrial edge deployment — running language models close to the machines they serve, where network connectivity is unreliable and latency budgets are tight. One motivating application in our lab: robotic-arm G-code generation, where a model translates task descriptions into machine instructions.

A 3B-parameter model is already “small” by LLM standards, but still too heavy for constrained edge hardware. So the goal became: compress Llama-3.2-3B into something with a ~1B footprint, without losing the capabilities that make it worth deploying.

Approach: Distillation + Parameter-Efficient Fine-Tuning

The core technique is knowledge distillation: a compact student model learns to match the behavior of the larger teacher, rather than learning from raw data alone. Combined with LoRA/QLoRA for parameter-efficient fine-tuning, this keeps training affordable while preserving the teacher’s task performance.

The interesting part of this work isn’t any single trick — it’s navigating the throughput-latency-memory tradeoff. Every compression decision buys you speed or memory at some cost in quality, and the only way to make those decisions honestly is to measure.

Benchmarking

I benchmarked the compressed models on a 4x A100 GPU setup, evaluating quality retention against the teacher alongside inference speed and memory footprint.

Results:

• 67% parameter reduction — Llama-3.2-3B compressed to 1B
• 2× inference speedup
• 95%+ of teacher performance retained

What I Learned

1. Distillation is an engineering problem as much as an ML problem. The training recipe matters, but so do evaluation discipline and honest baselines.
2. Tradeoff curves beat single numbers. A “2× faster” claim is meaningless without stating what you paid for it — which is why quality retention was tracked for every configuration.
3. Edge constraints sharpen your thinking. When you can’t just scale up, you’re forced to understand what the model actually needs to do.

This work connects directly to my broader interest in ML systems: making models not just accurate, but deployable — under real latency, memory, and reliability constraints.