  Pattern lock is a general technique used to realize identity authentication
and access authorization on mobile terminal devices such as Android platform
devices, but it is vulnerable to the attack proposed by recent researches that
exploit information leaked by users while drawing patterns. However, the
existing attacks on pattern lock are environmentally sensitive, and rely
heavily on manual work, which constrains the practicability of these attack
approaches. To attain a more practical attack, this paper designs the
PatternMonitor, a whole pipeline with a much higher level of automation system
againsts pattern lock, which extracts the guessed candidate patterns from a
video containing pattern drawing: instead of manually cutting the target video
and setting thresholds, it first employs recognition models to locate the
target phone and keypoints of pattern drawing hand, which enables the gesture
can be recognized even when the fingertips are shaded. Then, we extract the
frames from the video where the drawing starts and ends. These pre-processed
frames are inputs of target tracking model to generate trajectories, and
further transformed into possible candidate patterns by performing our designed
algorithm. To the best of our knowledge, our work is the first attack system to
generate candidate patterns by only relying on hand movement instead of
accurate fingertips capture. The experimental results demonstrates that our
work is as accurate as previous work, which gives more than 90\% success rate
within 20 attempts.
