💻 Featured Work

Rethinking Weight Tying: Pseudo-Inverse Tying for LM Stable Training and Updates
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

Pseudo-Inverse Tying is a novel weight-tying approach that improves the semantic coherence of language models by keeping input and output token geometries synchronized during training, enhancing stability, interpretability, and lightweight adaptation.

Semantic-based Optimization for Repairing LLMs: Case Study on Code Generation
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

STAR is a novel semantic-based optimization approach for LM repair that efficiently locates and patches buggy neurons using statistical insights and analytical formulas, outperforming prior methods in effectiveness, efficiency, and minimizing side effects.

Semantic-Aware Layer-Freezing for Computation-Efficient Fine-Tuning of LMs
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

Our semantic-based layer freezing approach improves the efficiency of language model finetuning by determining where to finetune, outperforming existing methods through a detailed semantic analysis of the model’s inference process.

Neuron Patching: Semantic-based Neuron-level LM Repair for Code Generation
Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang

MINT is an efficient and reliable technique for repairing large language models in software engineering. It can successfully solve model failures by patching merely 1 or 2 neurons, outperforming state-of-the-art methods in coding tasks.

Towards Top-Down Automated Development in Limited Scopes: A Neuro-Symbolic Framework from Expressibles to Executables
Jian Gu, Harald C. Gall

Deep code generation integrates neural models into software engineering for generating code but requires enhancements for project-level tasks, suggesting a taxonomy on code data and introducing a semantic pyramid framework to improve software development processes.