Remember Everything You Want and Manage the Rest