  In this thesis, we develop various techniques for working with sets in
machine learning. Each input or output is not an image or a sequence, but a
set: an unordered collection of multiple objects, each object described by a
feature vector. Their unordered nature makes them suitable for modeling a wide
variety of data, ranging from objects in images to point clouds to graphs. Deep
learning has recently shown great success on other types of structured data, so
we aim to build the necessary structures for sets into deep neural networks.
  The first focus of this thesis is the learning of better set representations
(sets as input). Existing approaches have bottlenecks that prevent them from
properly modeling relations between objects within the set. To address this
issue, we develop a variety of techniques for different scenarios and show that
alleviating the bottleneck leads to consistent improvements across many
experiments.
  The second focus of this thesis is the prediction of sets (sets as output).
Current approaches do not take the unordered nature of sets into account
properly. We determine that this results in a problem that causes discontinuity
issues with many set prediction tasks and prevents them from learning some
extremely simple datasets. To avoid this problem, we develop two models that
properly take the structure of sets into account. Various experiments show that
our set prediction techniques can significantly benefit over existing
approaches.
