Paper: Unsupervised Feature Selection using a Basis of Feature Space and Self-Representation Learning (ICLR-2024)

Reviewer comment (Novelty):
In the end of Introduction section, the second and third contribution points is not sufficient, as these constraints of regularization are not proposed in this article.

Author response:
We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that we have revised the contribution points of our proposed method. The main contributions of the paper are summarized below: 

1- To the best of our knowledge, GRSSLFS is the first feature selection method that applies a basis of linearly independent features into a unified framework of subspace learning and self-representation.

2- GRSSLFS characterizes the challenges of subspace learning and self-representation by utilizing a basis matrix derived from the feature space. The objective is to eliminate unnecessary features and identify informative ones within high-dimensional data.

3- GRSSLFS constructs a basis for the feature space by incorporating the variance information of features into the Basis Extension method. The goal is to create a feature basis with the highest variance score. This ensures that the basis for the feature space comprises elements with the most dispersion in space, thereby minimizing the accumulation of features.

Reviewer comment (Theory):
In the methodology section, some formula calculations are confusing and not very convincing. Such as the multiplication in formula (6) and the optimization target in the optimization goal (formula 7). These issues will be described in detail in subsequent questions 1 and 2.

Author response:
We greatly appreciate the reviewer for providing valuable comments that helped clarify certain aspects. To address the concerns raised, we thoroughly reviewed all relevant matrices involved in the formulations (6) and (7) and included their sizes which were omitted in the initial submission. Specifically, we appended the sizes of matrices $\mathbf{U}$ and $\mathbf{V}$ after equation (6) and the size of matrix $\mathbf{B}$ before equation (7).

Reviewer comment (Reproducibility):
In the methodology section, the description of the algorithm is not complete enough. The specific process of selecting features according to the matrix U in Algorithm 1 has not been described in detail.

Author response:
We thank the reviewer for raising this issue. Regarding this comment, we have added an explanation about choosing the features according to the matrix $\mathbf{U}$, in the revised manuscript, just after the main problem in equation $(9)$ on page 6. Please note that, as we explain in Section 3.2 on subspace learning and based on the term  $\|\mathbf{X}-\mathbf{XUV}\|$, which is reflected in  $\|\mathbf{B}-\mathbf{BGUV}\|$ in our basis-based model, we can interpret the matrix $\mathbf{X}\_I=\mathbf{XU}$ as the approximation of selected sub-matrix. We have
$$\mathbf{X}\_{I}=\mathbf{X} \mathbf{U}=[\mathbf{f}\_{1},\ldots,\mathbf{f}\_{n}]\mathbf{U}=\mathbf{f}\_{1}\mathbf{u}^{(1)} + \cdots + \mathbf{f}\_{n}\mathbf{u}^{(n)}, $$
where $\mathbf{u}^{(i)}, i=1,\ldots,n$ are the rows of the matrix $\mathbf{U}$. The above representation indicates that each row of the matrix $\mathbf{U}$
 can be interpreted as a weight in displaying the sub-matrix of selected features. So, we use the norms of the rows of the matrix $\mathbf{U}$ as the importance measure of each feature (column) of the input data $\mathbf{X}$.

Reviewer comment (Theory):
In the section of methodology, the equation (6) is confusing and not so clear. It seems impossible to subtract the matrix XUV of shape m*m from the matrix X with the shape m*n?

Author response:
With respect to the second comment of the previous part, we have adjusted the way we introduce the matrices $U$ and $V$ after formula (6) by incorporating their respective sizes. This subtraction is feasible because the matrix $X$ is $m \times n$, matrix $U$ is $n \times k$, and matrix $V$ is of size $k \times n$. Therefore, the confusion has been resolved by including the dimensional information of the matrices.

Reviewer comment (Theory):
As the feature matrix B is fixed by the VBE method proposed in section 3.3, it is unclear why the basis coefficient matrix G in equation (7) is a parameter to be optimized. Why the matrix G can not be determined by equation (4) directly and reduce the number of parameters.

Author response:
This is an insightful question, and initially, we believed it could aid in reducing computations by fixing the basis matrix $\mathbf{B}$ and computing $\mathbf{G}$ from Equation (4). However, it's essential to note that Equation (4) constitutes the first term of our objective function in Equation (8). To be more precise, in addition to relation (4), the matrix $\mathbf{G}$ simultaneously plays a crucial role in preserving the geometrical information of the data in the final equation of subsection 3.1 and is actively involved in the subspace learning process in the concluding equation of Subsection 3.2. Therefore, to ensure that all parts of the objective function in (8) decrease during the updating process, we must consider $\mathbf{G}$ as a part of the whole and update it within the general framework, as we did. If we only compute $\mathbf{G}$ based on equation (4), we can only guarantee the decrease of the first term of the objective function.

Reviewer comment (Presentation):
In section 3.1, subspace learning that introduces graph regularization seems to be existing methods. Should this part of the content be moved to related work?

Author response:
The reviewer highlighted the graph regularization term within the framework of manifold learning, which is an established method that has been widely used in dimensionality reduction techniques. However, the novelty of our regularization term lies in establishing a connection between the concept of the basis matrix and the geometrical structure of the data. Specifically, we introduce a new term, $\mathrm{Tr}\left(\mathbf{B}\mathbf{G}\mathbf{L}\mathbf{G}^T\mathbf{B}^T\right)$, where the basis matrix $\mathbf{B}$ plays a crucial role, facilitated by the coefficient matrix $\mathbf{G}$. Therefore, as a consequence of our self-representation framework in Section 3.1, we used the basis matrix $\mathbf{B}$ for preserving the geometry of data.

Reviewer comment (Theory):
The subspace learning module is the key component, but its derivation from selected bases highly relies on the assumption that XU=B. This might not hold if B is selected according to the proposed variance basis extension method. Moreover, the selection based on the subspace learning module lacks convincing explanation since G does not exactly represent X based on B as a fixed set of feature vectors.

Author response:
We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that the assumption $\mathbf{X}\mathbf{U}=\mathbf{B}$ holds and there always exists a matrix like $\mathbf{U}$. The reason is that in our proposed Variance Basis Extension (VBE) method which is explained in Subsection 3.4 (Subsection 3.3 in the old version), we generate the basis matrix as $\mathbf{B}=[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] \in \mathbb{R}^{m \times m} $ in which the basis vectors $\mathbf{f}\_{i\_j}, j=1,..,m$ are selected from the input features set $\\{\mathbf{f}\_{1}, \mathbf{f}\_{2}, \cdots, \mathbf{f}\_{n} \\}$. 
These are the columns of the data matrix $\mathbf{X}$. Therefore, we have $\\{\mathbf{f}\_{i\_1}, \mathbf{f}\_{i\_2}, ,\cdots, \mathbf{f}\_{i\_m} \\} \subseteq \\{\mathbf{f}\_{1}, \mathbf{f}\_{2}, ,\cdots, \mathbf{f}\_{n} \\}$.
Now, as in the proof of Theorem 3.1,  we can define the matrix $\mathbf{U} \in \mathbb{R}^{n \times m}$ as
$\mathbf{U}=[\mathbf{e}\_{i\_1},\ldots,\mathbf{e}\_{i\_m}]$, where $\mathbf{e}\_{i\_j} \in \mathbb{R}^{n}$ (for $j=1,\ldots,m$) is a vector whose $i\_j$-element is 1 and other elements are 0. 
Then it can be seen that 
$$\mathbf{X}\mathbf{U}=[\mathbf{f}\_{1},\mathbf{f}\_{2},\cdots,\mathbf{f}\_{n}] \begin{bmatrix}
0 & 0 &  &  & 0 \newline
\vdots & \vdots &  &  &  \newline
0 & 0 &  &  &  \newline
1 & 0 &  &  &  \newline
0 & \vdots & \cdots & \cdots & \vdots \newline
\vdots & 1 &  &  &  \newline
 & 0 &  &  & 1 \newline
 & \vdots &  &  & \vdots \newline
0 & 0 &  &  & 0 
\end{bmatrix} =[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] =\mathbf{B}.$$
Thus, the $0,1$-matrix $\mathbf{U}$ always exists in our method of generating the basis matrix, and the equality $\mathbf{X}\mathbf{U}=\mathbf{B}$ is satisfied.

On the other hand, considering that the matrix $\mathbf{B}$ serves as a basis for the feature space, it follows from a fundamental theorem in linear algebra (namely, each vector in a vector space can be expressed as a linear combination of basis members) that a matrix $\mathbf{G}\in\mathbb{R}^{m\times n}$ exists such that $\mathbf{X}$ can be represented in its basis form, denoted as $\mathbf{X} = \mathbf{BG}$. According to the $0,1$-matrix $\mathbf{U}$ mentioned above, it can be seen that 
$\mathbf{X}=\mathbf{B}\mathbf{G}=\mathbf{X}\mathbf{U}\mathbf{G}$. Hence, $\mathbf{G}$ as the coefficient matrix can be used to recover the data matrix $\mathbf{X}$ from the basis matrix $\mathbf{B}$.
This explanation is provided to show how the matrices $\mathbf{G}$ and $\mathbf{B}$ are used in the framework of subspace learning.

Reviewer comment (Theory):
It is confusing to explain (4) as the self-representation problem if B is an arbitrary basis matrix since they may not come from the input data matrix X. Taking PCA for example, the columns of B are orthogonal, but they are not from the input feature space. Moreover, B defined as a square matrix of size m is inconsistent with the selected r bases in section 3.3.

Author response:
We thank the reviewer for raising this point. In response to the reviewer's comment, we would like to mention that in general, a basis for a vector space, generated by a set of vectors such as $\\{\mathbf{f}\_1,\ldots,\mathbf{f}\_n\\}$, can be formed in two ways:

**First case:** the basis members are chosen directly from the vectors $\\{\mathbf{f}\_1,\ldots,\mathbf{f}\_n\\}$. A well-known method for this selection is the Basis Extension (BE) method, detailed in the manuscript. The BE method is capable of constructing a set of basis vectors for any vector space, including the feature space in our problem. In this scenario, as addressed in the initial question from the reviewer, there exists a $0,1$-matrix $\mathbf{U}$ such that $$\mathbf{X}\mathbf{U}=[\mathbf{f}\_{1},\mathbf{f}\_{2},\cdots,\mathbf{f}\_{n}] =[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}] =\mathbf{B}.$$
This implies that the columns of $\mathbf{B}$ are derived from the input feature space.

**Second case:** the basis members are extracted using the vectors $\\{\mathbf{f}_1,\ldots,\mathbf{f}_n\\}$. For instance, as noted by the reviewer, the basis vectors of PCA can be utilized to ensure that the constructed basis vectors are mutually perpendicular. However, this is not the case for our proposed feature selection framework.

In response to the other reviewer's comment, we would like to mention that as stated in Subsection 3.1 (just after Equation (2)),  our manuscript assumes $\mathrm{rank}(\mathbf{X}) = m$ and $m \leq n$ throughout. Consequently, the matrix $\mathbf{B}$ is consistently a square matrix of size $m$. 

It is worth emphasizing that, even in scenarios where $\mathrm{rank}(\mathbf{X}) < m$ or $m > n$, the creation of a basis matrix for the feature space is still viable and does not pose any challenges in the development of our feature selection framework. To elaborate further, for any matrix $\mathbf{X}$ with $\mathrm{rank}(\mathbf{X})=r$ (where it is proved that $r$ is always smaller than $m$), the BE method can be applied to generate the matrix basis $\mathbf{B}\in\mathbb{R}^{m\times r}$. This basis matrix can then be utilized in our proposed feature selection method. In such cases, it is sufficient to assume that the matrix $\mathbf{G}\in\mathbb{R}^{r\times n}$, and the matrix product $\mathbf{BG}$ will be in $\mathbb{R}^{m\times n}$ and will have meaningful implications. Moreover, it can be easily seen that the expression $\mathbf{X}-\mathbf{BG}\in\mathbb{R}^{m\times n}$ will be well-defined. To address the reviewer's comment, we have incorporated a brief explanation of this issue in a remark on Page 4 of the revised manuscript.

Reviewer comment (Theory):
The derivation in section 3.2 depends on the assumption that XU=B. As B is a set of feature vectors selected from input data, it is unclear whether the assumption still holds or not. Similarly for Theorem 3.1, it is trivial to have if the assumption holds.

Author response:
We thank the reviewer for raising this point. In response, we should mention that considering the explanations provided to respond the Weaknesses section, in our algorithm for generating the basis (in our proposed Variance Basis Extension (VBE) method in subsection 3.4), we select the basis vectors $\mathbf{f}\_{i\_1}, \cdots, \mathbf{f}\_{i\_m}$ from the columns of the input data matrix $\mathbf{X}$, denoted as ${\mathbf{f}\_{1}, \mathbf{f}\_{2}, \cdots, \mathbf{f}\_{n}}$. Therefore, in our subspace learning formulation in subsection 3.2, the $0,1$ matrix $\mathbf{U}$ exists such that $\mathbf{XU}=\mathbf{B}$.

Because of  this issue, in the proof of Theorem 3.1, we explained how to construct this $0,1$ matrix $\mathbf{U},$ and then, as the reviewer mentioned, the proof is straightforward.

Reviewer comment (Theory):
The variance basis extension is to simply change the selection order of feature vectors in terms of variance score of feature vectors. It is possible that for each individual feature, the variance is high, but is it similar to say the largest amount of data dispersion?

Author response:
We thank the reviewer for raising this point. In order to address the reviewer’s question, we would like to confirm that the variance of a feature measures its dispersion or range. When a feature exhibits high variance, it suggests that its values are widely spread over a broader range. In the context of the variance basis extension, altering the selection order of feature vectors based on their variance scores implies a focus on features with significant variability or dispersion.

However, it is important to note that while high variance indicates a wide spread of values for an individual feature, it does not necessarily mean that this feature contributes the most to the overall dispersion or variability in the dataset. The idea of the "largest amount of data dispersion" is better captured by considering the overall variance or variability of the entire dataset, which takes into account the combined effects of all features. In summary, high variance for individual features reflects their own level of dispersion, but identifying the features contributing the most to the overall data dispersion requires considering the collective impact of all features. For more information, we would like to mention that the theoretical investigations into this issue can be found in some related studies, such as ``Unsupervised feature selection based on variance-covariance subspace distance, Neural Networks, 2023."

Reviewer comment (Theory):
For completeness, authors should describe the derivation process on how equations (9)-(11) are obtained. Since all three equations are fractional, is it possible that any of the denominators can be zero? How is it handled?

Author response:
In response to the reviewer's comment, we would like to mention that as mentioned on Page 6 of the paper, the extraction of the update rules introduced for $\mathbf{G}$, $\mathbf{U}$, and $\mathbf{V}$ involves three steps:

(1) Calculating the gradient of the objective function in regards to one of the variables with the assumption of keeping the other variables constant;

(2) Setting the obtained gradient equal to zero;

(3) Applying the Karush-Kuhn-Tucker conditions (for example, in our problem, we have $\mathbf{M}\_{ij}\mathbf{G}\_{ij}=\mathbf{N}\_{ij}\mathbf{U}\_{ij}=\mathbf{W}\_{ij}\mathbf{V}\_{ij}=0$).

To ensure that the denominator of the fraction does not become zero, two common workarounds are employed in the problems based on matrix factorization:

**First option:** Utilizing the assumption of non-negativity for the input matrices of the problem. Notably, one of the most famous dimension reduction methods, Non-negative Matrix Factorization (NMF), assumes the non-negativity condition for the data matrix $\mathbf{X}$ and the unknown matrices used in the optimization problem, such as the matrices $\mathbf{G}\geq 0$, $\mathbf{U}\geq 0$, and $\mathbf{V}\geq 0$ in our proposed framework. This guarantees that the updated matrices $\mathbf{G}$, $\mathbf{U}$, and $\mathbf{V}$ remain non-negative, avoiding the zero denominator problem. Theoretical investigations into this issue can be found in some studies related to NMF, such as Lemma 1 on Page 3 of the following reference:

$\bullet$ ``A convergent algorithm for orthogonal nonnegative matrix factorization. Journal of Computational and Applied Mathematics, 260, 149-166."

**Second option:** This method, which is applicable even when assuming non-negativity, has a more practical aspect. To prevent the denominator from reaching zero and causing numerical calculation errors, a small non-negative number, such as $\epsilon = 10^{-10}$, is used. Specifically, we set

$ denominator \leftarrow \max(denominator,\epsilon).$

Employing this strategy ensures that fraction calculations are performed without encountering zero-related problems.

In the framework designed for our problem, the first option (assuming non-negativity of inputs) is used. Additionally, when implementing the method, the second option (comparing the denominator with a very small non-negative number) is also considered.

It is worth mentioning that even if the assumption of non-negativity on the input variables is not considered, there is a crucial principle in matrix analysis stating that any matrix $\mathbf{X}$ with real elements can be expressed as the difference of the two non-negative matrices.
This is demonstrated as follows:

$$\mathbf{X} = \mathbf{X}^+ - \mathbf{X}^{-},$$

where $\mathbf{X}\_{ij}^+ = \frac{|\mathbf{X}\_{ij}| + \mathbf{X}\_{ij}}{2}$, and $\mathbf{X}\_{ij}^- = \frac{|\mathbf{X}\_{ij}| - \mathbf{X}\_{ij}}{2}$. By utilizing this representation and the two methods mentioned above, the problem of encountering zero in dimension reduction problems based on matrix factorization is effectively prevented.

Reviewer comment (Theory):
In Algorithm 1, the selected features are derived from U. However, U is not directly related to the input X instead to B and G, unless BG=X. However, B is selected feature vectors from input space. It is unclear why the assumption can hold. So why is the selection rule proper?

Author response:
The reviewer's great attention to this issue is mostly appreciated. In order to address the reviewer's question, we would like to highlight that the primary objective of this paper is to further investigate the impact of a basis for the feature space on tasks related to feature learning. As discussed in the paper, we first defined the self-representation problem based on the basis matrix as follows:

$\min\_{\mathbf{G}\in\mathbb{R}\_+^{m\times n}}\Vert\mathbf{X}-\mathbf{BG}\Vert\_F^2,$

where $\mathbf{B}=[\mathbf{f}\_{i\_1},\ldots,\mathbf{f}\_{i\_m}]\in\mathbb{R}^{m\times m}$ is a basis matrix. Moreover, in order to learn a low-dimensional representation of high-dimensional data, we established the subspace learning problem based on the basis matrix as:

$\min\_{\mathbf{G}, \mathbf{U}, \mathbf{V}}\Arrowvert \mathbf{B}-\mathbf{BGUV}\Arrowvert\_F^2,
$
where $\mathbf{G}\in\mathbb{R}\_+^{m\times n}, \mathbf{U}\in\mathbb{R}\_+^{n\times k}$, and $\mathbf{V}\in\mathbb{R}\_+^{k\times m}$, such that $k\leq n$ is the dimension of the selected feature space. And, we finally defined our proposed feature selection method based on the above-mentioned problems designed via the basis for the feature space. In summary, our proposed feature selection method employed the following structure:

\begin{equation*}
\small
\textbf{Dataset}\,\,\mathbf{X}\rightarrow\textbf{Construct the Basis}\,\,\mathbf{B}\,\,\textbf{for the Feature Space}\rightarrow\textbf{Define the Feature Selection Problem Using}\,\,\mathbf{B}.
\end{equation*}

To provide further clarification, in contrast to classical feature selection methods that select a subset of features by searching the original data feature set, our proposed feature selection method conducts the selection process by searching through a basis of features rather than all features. As discussed in Subsection 3.3 of the manuscript, the selected feature matrix $\mathbf{X}\_{\text{selected}}$ can be written as $\mathbf{X}\_{\text{selected}}=\mathbf{BGU}.$

On the other hand, considering that the matrix $\mathbf{BG}$ serves as a basis representation for the data matrix $\mathbf{X}$, it follows from a fundamental theorem in linear algebra (namely, each vector in a vector space can be expressed as a linear combination of basis members) that $\mathbf{X}$ can be represented in its basis form, denoted as $\mathbf{X} = \mathbf{BG}$. Consequently, it can be inferred that 
$\mathbf{X}\_{\text{selected}} = \mathbf{XU}$. 

 In order to address the reviewer's question, it is worthwhile to note that 

**1-** The representation $\mathbf{BGU}$ for the selected feature matrix  $\mathbf{X}\_{\text{selected}}$ is based on the aforementioned theorem in linear algebra, stating that each vector in a vector space can be written as a linear combination of basis members (for instance, in our case, $\mathbf{X} = \mathbf{BG}$). 

**2-** The selected feature matrix $\mathbf{X}{\text{selected}}$ and the matrix $\mathbf{U}$ are explicitly connected through the relationship $\mathbf{X}\_{\text{selected}} = \mathbf{BGU}$. In essence, our proposed feature selection method conducts the selection process by exploring a basis of features rather than considering all features.

**3-** The selected feature matrix $\mathbf{X}\_{\text{selected}}$ and the matrix $\mathbf{U}$ can be implicitly linked through the relationship $\mathbf{X}\_{\text{selected}} = \mathbf{XU}$.

In both scenarios, the matrix $\mathbf{U}$ functions as the feature weight matrix, and the feature selection process is carried out based on $\mathbf{U}$.

Reviewer comment (Evaluation):
The application to the PneumoniaMNIST dataset is quite interesting. However, the way of presenting the outcomes can be improved significantly. For example, what is the interested region? How many selected features are in the interested region? How do other compared methods perform? The validation is not quantified. How many radiologists are involved in the evaluation? What is the performance measured? These plots shown in Fig. 2 deliver less useful information except that more red points are accumulated in the center when the number of selected features increases.

Author response:
Our responses are as follows:

**1- What is the interested region?**

For patients diagnosed with pneumonia, the chest X-rays reveal various diagnostic features, particularly in the cardiac section. For more information, please refer to ``*Chest Radiology: Patterns and Differential Diagnoses, Elsevier Health Sciences, 2017.*" Notably among these characteristics are an augmented cardiac shadow, alterations in cardiac positioning, the presence of effusion, pericardial effusion, and instances of cardiomegaly. In this study, the application of our proposed feature selection method to the PneumoniaMNIST dataset is assessed with the goal of precisely identifying and analyzing the cardiac silhouette in chest X-ray images, which is located in the central part of the chest. 

**2- How many selected features are in the interested region?**

For our proposed method when the number of selected features is 50 and 100 (as depicted in *Figure 1* on Page 9), all the selected features are consolidated within our area of interest. 

**3- The validation is not quantified. How many radiologists are involved in the evaluation? What is the performance measured?**

The validation process incorporated the expertise of two board-certified radiologists. Both professionals thoroughly reviewed the results of chest X-ray images, offering valuable insights to ensure the accuracy and reliability of our proposed method. Their assessments played a pivotal role in validating the model's performance within a clinical context. Additionally, given the study's exclusive focus on detecting the cardiac silhouette without making any diagnoses, both radiologists concurred that assessing specific performance criteria is unnecessary. For this reason, our analysis is focused solely on identifying and examining the cardiac silhouette in chest X-ray images.

**4- How do other compared methods perform?**

In agreement with the reviewer, we have included results related to the selection of 100 features using comparative methods in *Section J* on Page 23 (refer *Figure 8* in the Appendix). Examining this figure, it is evident that some methods, including our proposed method, RMFFS, and VCSDFS, exhibit a tendency to identify the central area of chest X-ray images. Conversely, some other methods, while successfully identifying portions of the central areas, may overlook additional regions that could provide valuable information. These methods fail to capture such information in radiological analysis.

Finally, in response to the reviewer's questions, we have incorporated additional explanations regarding ``the region of interest" and clarified our purpose for conducting the test on the PneumoniaMNIST dataset in the revised manuscript. Furthermore, the results concerning the selection of 100 features by other comparative methods are presented in *Figure 8* of the revised manuscript.

Reviewer comment (Theory):
The motivation of the proposed method is not clear. In Eq.(8), the first three terms have been well explained, but the final regularization term has not been explained.

Author response:
We appreciate the reviewer's thorough attention to these issues. To respond to the reviewer's comment regarding the motivation, we would like to mention that the main goal of the feature selection process is to opt for a subset of features with the purpose of diminishing noise and improving the efficiency and effectiveness of a learning task, such as classification and clustering. In essence, this process of selection aims to identify features that are important for the learning task while ensuring that the chosen set of features is non-redundant, thereby strengthening robustness. Within this framework, a crucial factor in feature selection is redundancy, which involves examining the similarity among features and assessing how effective it is to introduce a new feature into a designated feature set for data analysis.

In the field of linear algebra, a direct connection exists between redundancy and the concept of linear independence. Specifically, a set of vectors in a vector space is deemed linearly dependent if non-zero scalars can be identified in a way that their linear combination equals zero.
This suggests that it is possible to express one of these vectors as a combination of the others. In contrast, a collection of vectors is deemed linearly independent if there is no linear dependence among them. Consequently, we can deduce from this principle that a set of vectors exhibiting linear dependence includes redundancy, enabling the removal of one vector from the initial set without altering the span of the remaining vectors. A significant benefit of eliminating linearly dependent features from the initial feature set is that it aids in fulfilling a crucial aim in feature selection, which is the reduction of redundancy. In the pursuit of this goal, the concept of a basis, a fundamental element in the theory of linear vector spaces, becomes crucial.

Each basis for the feature space exhibits two essential traits. Firstly, it covers the entire feature space, and secondly, all features within a basis are linearly independent. These attributes positively and constructively influence the discriminative feature selection process. The initial property guarantees that a basis can represent the complete feature space through its elements, implying that the fundamental characteristics of the original features are present in the basis with a significantly reduced number of elements compared to the entire feature space. The second characteristic of a basis, which involves its elements being linearly independent, leads to a significant reduction in data redundancy. Exploiting the advantages of a basis in a vector space, the primary objective of this paper is to further investigate the impact of a basis for the feature space on tasks related to feature learning.