Long-Context Models

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
We introduce PowerAttention, a novel sparse attention design that facilitates effective and complete context extension through the theoretical analysis.
Lida Chen, Dong Xu, Chenxin An, Xintao Wang, Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang, Yanghua Xiao, Wei Wang
PDF Cite Code
PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention