Rule Training for VMI Sketch in Developmental Testing based on a Deep Neural Network
Abstract
In this paper, we present a framework that explains the scores of sketches by learning rules used in developmental tests.
To achieve this, we propose a deep neural network model that considers a target and the corresponding sketch images as inputs.
The proposed method is divided into plain and residual models according to the presence of residual connections to compare their performance.
In addition, each model includes the subtraction and concatenation approaches to fuse two feature maps.
To verify the performance of the proposed method, we conduct experiments over all settings that combine the proposed models with the fusion method.
The results show that the proposed framework can be used for visual-motor integration analysis by determining scores and providing explanations.
Introduction
Developmental tests are used to evaluate a child’s abilities and compare them with those of others of a similar age [1].
Given that early intervention yields a better prognosis in many developmental disorders, it is important to check children’s developmental levels early through developmental tests.
The types of developmental disabilities that can be diagnosed through developmental tests include intellectual disabilities, autism spectrum disorder (ASD), and attention-deficit/hyperactivity disorder (ADHD).
In general, infants between 12 and 24 months of age who do not know how to write naturally have difficulty in performing developmental tests using writing.
Instead, developmental tests can be performed using sketches.
Some examples of such tests include the Rey–Osterrieth complex figure (ROCF) [2] and visual-motor integration (VMI) [3].
Moreover, VMI can be an important variable in handwriting, and Sanghavi and Kelkar [4] found it to be the most important factor in handwriting performance.
VMI refers to the ability to coordinate one’s eyes and hands together smoothly and efficiently [4].
Measures of VMI skills are widely used to evaluate the mental and psychological state of children [5, 6].
The most widespread tool for VMI assessment is the Beery-Buktenica developmental test of visual-motor integration, known as the Beery VMI [6].
It uses sketches that consist of diverse figures.
The figures used in the Beery VMI range from simple forms such as a line and a circle to complex designs such as a cube and a three-dimensional star.
The test is performed using test sheets that comprise of six boundaries, and the figures are displayed in the upper row.
The children see the figures and draw them on the corresponding spaces below.
Subsequently, experts inspect the drawn sketches only in the boundaries and score each sketch with 0 or 1 points according to the criteria (rules) provided in the Beery VMI scoring manual [6].
Studies have been conducted with the Beery VMI to assess specific groups in terms of ethnicity, disorder, age, etc.
Specifically, Coallier et al.
[7] used the test for kindergarten children.
Green et al. [8] compared the VMI abilities of children with ASD with typical development groups.
Geldof et al. [9] verified the deficit of VMI ability in children born prematurely.
On the other hand, some studies have used automated methods to recognize the forms in a sketch or retrieve images using them.
Yu et al.
[10] and Zhang et al.
[11] used deep neural networks (DNNs) to perform sketch recognition.
Wang et al.
[12] and Sain et al.
[13] studied a sketch-based image retrieval method that used sketches as input to find similar images.
Although some previous works have considered the Beery VMI and sketches, few studies have used DNNs to describe the sketches collected in the Beery VMI.
In this paper, we propose a framework called a sketch rule-training network (SRTN) to explain scoring results in the VMI analysis.
From a clinical perspective, the explanation of the scoring result is more important than the result itself.
The reason for a given score can be inferred based on the rules because the Beery VMI has figure-specific explicit rules.
However, the score of any given figure may differ depending on subjective judgements of experts.
To solve the issue, a rule inference algorithm that reflects such empirical cases is required.
To incorporate diverse rules into single network, we use target figures as input data in addition to the sketches.
Proposed Method
The proposed rule-training method for sketch explanation requires two images, including a target figure and a sketch image.
The VMI data used includes obvious rules, which the proposed neural network learns to enable the model to infer explanations of the input sketch.
The overall architecture of the proposed framework is demonstrated in Figure 1.
For rule training, the target figure images are also required along with the sketch images.
As a network outputs the rule codes, they must be decoded to obtain the text explanation.
We used a convolutional neural network (CNN) to learn them with a single integrated model because the types of figures and the scoring rules used in this study were diverse.
The overall architecture of the proposed model is shown in Table 1.
The used neural network is largely divided into a plain model using a general convolutional layer and a residual model using a residual block such as VGG and ResNet, respectively.
The output nodes of the proposed network are set to 6, which is the maximum number of criteria for all figure items.
In addition, 1 and -1 outputs represent 1 andpoints respectively, and when there were fewer than 6 criteria, the output of the remaining nodes was set to 0.
The explanation results were decoded by the inference and corresponding rules.
As shown in Table 1, one input is a sketch, while the other is a target figure representing the items shown in the sketch.
The figure images are input together because the neural network can extract features depending on which figure form the sketch exhibits.
Consequently, information can be learned from various figures with a single integrated neural network.
The input sketch and figure image pass through two independent networks to output two feature maps.
The output feature maps [Table] are combined into a single map to predict whether the scoring criterion is satisfied.
Two approaches are used to combine the feature maps.
The first is a subtraction that calculates the difference between the two feature maps, while the other is concatenation that attaches all information without loss.
After the trained DNN inference with the input images, the explanation is generated by decoding the result.
A rule set was determined from predefined sets of VMI figures according to the target figure as a neural network input, and the rules are combined with the inference codes in a way that depends on whether the corresponding rules are satisfied.
Since the number and order of rules in each figure are predefined, the decoder maps the encoded output directly to the learned rules.
When it is determined whether the rule is satisfied, the explanation is completed by combining decoded rules with appropriate phrases.
Moreover, the score that indicates whether all rules in the figure are satisfied is also decoded.
According to the VMI manual, the score for a figure is 1 when all the criteria are satisfied, and the score is 0 otherwise.
The examples of the rules on two test figures are displayed in Table 2.
[Table]
Experimental Results
The sketch data collected included twelve types from the Beery VMI test, but only six types of sketches were used in the experiment.
A total of 133 test sheets were used, and the number of individual figures was 399 because a single sheet consists of three figures.
To reserve additional training data, the target figures and sketch examples presented in the VMI manual were additionally collected.
Furthermore, various data augmentation methods such as image re-scaling were used to increase the training data.
To train and perform inferences only on a fixed area of the test sheet, the target figure and sketch areas were detected and segmented.
The individual figure and sketch images were saved with a resolution of 1,000 × 1,000 pixels and then resized to 224 × 224 pixels before being used in the DNN models.
Table 3 shows the number of data samples used in the experiment.
[Table]
Table 4 shows the performance results of all DNN models.
In the plain model, the concatenation approach exhibited a n accuracy of 79.03%, which was 3.22% higher than the accuracy of the subtraction approach at 75.81%.
Also, the accuracy of the subtraction was 80.65%, and the concatenation approach showed an accuracy of 82.26% in the residual model, which was 1.61% higher.
Additionally, the performance over 12 types of figures was measured together for comparison with 6 types.
As expected, training all collected data adversely affected the performance due to the lack of sufficient data. [Table]
In the plain model, the accuracy of the subtraction was 71.21% and the accuracy of the concatenation was 74.24%, i.e., the accuracy of the concatenation approach was about 3.03% higher than the subtraction approach.
In the residual model, the accuracy of the subtraction and concatenation approaches was 71.21% and 69.70%, respectively.
In this model, the subtraction approach performed 1.51% better than the concatenation approach.
[Figure]
As a result, Figure 2 demonstrates examples of explanations generated by the decoding inference output through the mapping to scoring rules according to the individual target figures.
The prediction and the label match in Figure 4(a), whereas they do not match in Figure 4(b).
The decoding results indicate that the DNN, which was trained on the rules in the proposed method, was able to explain the VMI sketches.
Since the experimental data were collected from actual developmental tests with less than 48 months infants, most of the sketches correspond to the first six types of figures. If more data are collected and extended sketches to other developmental tests, it is expected that the proposed method can be contributed to the early screening of developmental disorders.
Conclusions
We have described an automated scoring method and an associated explanation-generating method designed to learn rules using a deep neural network to measure visual-motor coordination ability using the Beery VMI sketch data.
The DNN was designed as a plain model and a residual model, and the subtraction and concatenation approaches were used as a fusion of feature maps in each model.
The results of an experiment verified that the architecture of the proposed model can be used to explain the scoring results.
It showed a good performance of up to 82.26% accuracy by decoding the rules, although there were differences in performance depending on the method used.
In future study, we plan to explore contrastive learning to improve the performance of explanations with better sketch representations.
We expect that this will result in a performance improvement.
Also, to generalize the rule training, utilization of a finite state machine on the rule set could also be considered. Acknowledgments
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT)
(2019-0-00330, Development of AI
Technology for Early Screening of Infant/Child Autism Spectrum Disorders based on Cognition of the Psychological Behavior and Response)