We introduce a learning-based framework to optimize tensor programs for deep learning workloads. Efficient implementations of tensor operators, such as matrix multiplication and high dimensional convolution, are key enablers of effective deep learning systems. However, current systems rely on manually optimized libraries, e.g., cuDNN, that support only a narrow range of server class GPUs. Such reliance limits the applicability of high-level graph optimizations and incurs significant engineering costs when deploying to new hardware targets. We use learning to remove this engineering burden. We learn domain-specific statistical cost models to guide the search of tensor operator implementations over billions of possible program variants. We further accelerate the search using effective model transfer across workloads. Experimental results show that our framework delivers performance that is competitive with state-of-the-art hand-tuned libraries for low-power CPUs, mobile GPUs, and server-class GPUs. • We formalize the new problem of learning to optimize tensor programs and summarize its key characteristics. • We propose a machine learning framework to solve this problem. • We further accelerate the optimization by 2× to 10× using transfer learning. 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c m G 4 K 2 + v E 7 a V 1 X P r X r N 6 0 r 9 N o + j C G d w D p f g Q Q 3 q c A 8 N a A E D h G d 4 h T f n 0 X l x 3 p 2 P Z W v B y W d O 4 Q + c z x / J b Y z p < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " h P + 6 L r U f 2 d 3 t Z a l d q a Q Q v E K M X y w = " > A A A B 2 X i c b Z D N S g M x F I X v 1 L 8 6 V q 1 r N 8 E i u C o z b n Q p u H F Z w b Z C O 5 R M 5 k 4 b m s k M y R 2 h D H 0 B F 2 5 E f C 9 3 v o 3 p z 0 J b D w Q + z k n I v S c u l L Q U B N 9 e b W d 3 b / + g f u g f N f z j k 9 N m o 2 f z 0 g j s i l z l 5 j n m F p X U 2 C V J C p 8 L g z y L F f b j 6 f 0 i 7 7 + g s T L X T z Q r M M r 4 W M t U C k 7 O 6 o y a r a A d L M W 2 I V x D C 9 Y a N b + G S S 7 K D D U J x a 0 d h E F B U c U N S a F w 7 g 9 L i w U X U z 7 G g U P N M 7 R R t R x z z i 6 d k 7 A 0 N + 5 o Y k v 3 9 4 u K Z 9 b O s t j d z D h N 7 G a 2 M P / L B i W l t 1 E l d V E S a r H 6 K C 0 V o 5 w t d m a J N C h I z R x w Y a S b l Y k J N 1 y Q a 8 Z 3 H Y S b G 2 9 D 7 7 o d B u 3 w M Y A 6 n M M F X E E I N 3 A H D 9 C B L g h I 4 B X e v Y n 3 5 n 2 s u q p 5 6 9 L O 4 I + 8 z x 8 4 x I o 4 < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D p r g t I z i 2 4 E q 9 y 5 / 8 T y q d C x Q O 5 8 = " > A A A B 3 X i c b Z B L S w M x F I X v + K y 1 a n X r J l g E V 2 X G j S 4 F N y 5 b s A 9 o h 5 J J 7 7 S x m c y Q 3 B H K 0 F / g x o U i / i 1 3 / h v T x 0 J b D w Q + z k n I v S f K l L T k + 9 / e 1 v b O 7 t 5 + 6 a B 8 W D k 6 P q m e V t o 2 z Y 3 A l k h V a r o R t 6 i k x h Z J U t j N D P I k U t i J J v f z v P O M x s p U P 9 I 0 w z D h I y 1 j K T g 5 q 4 m D a s 2 v + w u x T Q h W U I O V G o P q V 3 + Y i j x B T U J x a 3 u B n 1 F Y c E N S K J y V + 7 n F j I s J H 2 H P o e Y J 2 r B Y D D p j l 8 4 Z s j g 1 7 m h i C / f 3 i 4 I n 1 k 6 T y N 1 M O I 3 t e j Y 3 / 8 t 6 O c W 3 Y S F 1 l h N q s f w o z h W j l M 2 3 Z k N p U J C a O u D C S D c r E 2 N u u C D X T d m V E K y v v A n t 6 3 r g 1 4 O m D y U 4 h w u 4 g g B u 4 A 4 e o A E t E I D w A m / w 7 j 1 5 r 9 7 H s q 4 t b 9 X b G f y R 9 / k D t Z i L l g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D p r g t I z i 2 4 E q 9 y 5 / 8 T y q d C x Q O 5 8 = " > A A A B 3 X i c b Z B L S w M x F I X v + K y 1 a n X r J l g E V 2 X G j S 4 F N y 5 b s A 9 o h 5 J J 7 7 S x m c y Q 3 B H K 0 F / g x o U i / i 1 3 / h v T x 0 J b D w Q + z k n I v S f K l L T k + 9 / e 1 v b O 7 t 5 + 6 a B 8 W D k 6 P q m e V t o 2 z Y 3 A l k h V a r o R t 6 i k x h Z J U t j N D P I k U t i J J v f z v P O M x s p U P 9 I 0 w z D h I y 1 j K T g 5 q 4 m D a s 2 v + w u x T Q h W U I O V G o P q V 3 + Y i j x B T U J x a 3 u B n 1 F Y c E N S K J y V + 7 n F j I s J H 2 H P o e Y J 2 r B Y D D p j l 8 4 Z s j g 1 7 m h i C / f 3 i 4 I n 1 k 6 T y N 1 M O I 3 t e j Y 3 / 8 t 6 O c W 3 Y S F 1 l h N q s f w o z h W j l M 2 3 Z k N p U J C a O u D C S D c r E 2 N u u C D X T d m V E K y v v A n t 6 3 r g 1 4 O m D y U 4 h w u 4 g g B u 4 A 4 e o A E t E I D w A m / w 7 j 1 5 r 9 7 H s q 4 t b 9 O 2 N q x n r V m 4 v / e b 3 M R D d B z m W a G Z R s u S j K B D E J m X 9 N h l w h M 2 J q C W W K 2 1 s J G 1 N F m b H Z V G w I 3 u r L 6 6 R 9 V f f c u t d y a 4 3 b I o 4 y n M E 5 X I I H 1 9 C A e 2 i C D w w Q n u E V 3 p x H 5 8 V 5 d z 6 W r S W n m D m F P 3 A + f w D I L Y z l < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " N P r u Y L n 6 6 / p u O z A M t M M 3 t S F g c 5 w = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E q M e i F 4 8 t 2 A 9 o Q 9 l s J + 3 a z S b s b o Q S + g u 8 e F D E q z / J m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I B F c G 9 f 9 d g o b m 1 v b O 8 X d 0 t 7 + w e 1 w m q U H J l o v C V B A T k / n X Z M g V M i O m l l C m u L 2 V s D F V l B m b T c m G 4 K 2 + v E 7 a V 1 X P r X r N 6 0 r 9 N o + j C G d w D p f g Q Q 3 q c A 8 N a A E D h G d 4 h T f n 0 X l x 3 p 2 P Z W v B y W d O 4 Q + c z x / J b Y z p < / l a t e x i t > for yo in range(1024 / ty): for xo in range(1024 / tx): C[yo*ty:yo*ty+ty][xo*tx:xo*tx+tx] = 0 for k in range(1024): for yi in range(ty): for xi in range(tx): C[yo*ty+yi][xo*tx+xi] += A[k][yo*ty+yi] * B[k][xo*tx+xi] for yo in range (128): for xo in range (128)
AutoTVM REF is targeted at accelerating deep learning workloads and uses machine learning to guide auto-tuning of deep learning primitives.
Learning to Optimize Tensor Programs
ArXiv
ArXiv
Mathematics Computer Science