Scott, J. (CSE) – Mechanistic Specialization Does Not Guarantee Performance: Evidence from Dual AttentionTransformers
Dual Attention Transformers (DATs) extend decoder-only Transformers with a dedicated relational-attention stream, making them a natural architecture for abstract identity rules such asABA and ABB. Surprisingly, we find that comparably sized GPT-2 models outperform DATs on these tasks. We investigate this gap with two complementary mechanistic analyses. First, causal mediation analysis shows that DATs exhibit […]