  Quantum properties, such as entanglement and coherence, are indispensable
resources in various quantum information processing tasks. However, there still
lacks an efficient and scalable way to detecting these useful features
especially for high-dimensional quantum systems. In this work, we exploit the
convexity of normal samples without quantum features and design an unsupervised
machine learning method to detect the presence of quantum features as
anomalies. Particularly, given the task of entanglement detection, we propose a
complex-valued neural network composed of pseudo-siamese network and generative
adversarial net, and then train it with only separable states to construct
non-linear witnesses for entanglement. It is shown via numerical examples,
ranging from 2-qubit to 10-qubit systems, that our network is able to achieve
high detection accuracy with above 97.5% on average. Moreover, it is capable of
revealing rich structures of entanglement, such as partial entanglement among
subsystems. Our results are readily applicable to the detection of other
quantum resources such as Bell nonlocality and steerability, indicating that
our work could provide a powerful tool to extract quantum features hidden in
high-dimensional quantum data.
