Keep on to blur preview images; turn off to show them clearly
Smooth DiLoCo: https://t.co/ub6hGVmFhu ( @aaron_defazio et al) Non-distributed alternative, w/ less memory overhead (only one extra buffer) and a continuous update instead of the periodic (rather violent) outer update of classical DiLoCo Curious to see expanded to distributed!
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne
RT @MustafaShukor1: VL-JEPA is out! A non-generative vision-language model, based on JEPA. Different from typical data-space autoregressive…
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne
RT @vivek_2332: Notes on "DiLoCo" (Distributed Low-Communication) paper: 1. The Problem with Large-Scale Training Today -> modern large sc…
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne
Tested Gemini’s hieroglyphic translation. 𓇋𓈖𓆓𓁷 𓇾 𓇋𓅱𓀀 𓁷 𓂓𓏏𓀋 𓂋 𓁹𓏏 𓊃𓇋𓄿 𓅓 𓆷𓂝𓈇 "Hello world, I am working to make intelligence from sand."
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne
RT @GabrielTeston: Heading to @NeurIPSConf in San Diego. I’ve got some DiLoCo stickers to give away! 👾 ❤️ Come check out our poster. 🗓️…
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne
RT @tkhoury: Exciting to see further improvements to DiLoCo from the Google DeepMind team
Distributed Learning @ deepmind | DiLoCo, DiPaCo. Continual Learning PhD @ Sorbonne