CumulantBased DOA Estimation of Noncircular Signals against Unknown Mutual Coupling To effectively find the direction of noncircular signals received by a uniform linear array (ULA) in the presence of nonnegligible perturbations between array elements, i.e., mutual coupling, in colored noise, a direction of arrival (DOA) estimation approach in the context of high order statistics is proposed in this correspondence. Exploiting the noncircularity hidden behind a certain class of wireless communication signals to build up an augmented cumulant matrix, and carrying out a reformulation of the distorted steering vector to extract the angular information from the unknown mutual coupling, by exploiting the characteristic of mutual coupling, i.e., a limited operating range and an inverse relation of coupling effects to interspace, we develop a MUSIClike estimator based on the rankreduction (RARE) technique to directly determine directions of incident signals without mutual coupling compensation. Besides, we provide a solution to the problem of coherency between signals and mutual coupling between sensors coexisting, by selecting a middle subarray to mitigate the undesirable effects and exploiting the rotationinvariant property to blindly separate the coherent signals into different groups to enhance the degrees of freedom. Compared with the existing robust DOA methods to the unknown mutual coupling under the framework of fourthorder cumulants (FOC), our work takes advantage of the larger virtual array and is able to resolve more signals due to greater degrees of freedom. Additionally, as the effective aperture is virtually extended, the developed estimator can achieve better performance under scenarios with high degree of mutual coupling between two sensors. Simulation results demonstrate the validity and efficiency of the proposed method.

DOA Estimation Using FourthOrder Cumulants in Nested Arrays with Structured Imperfections Recently developed super nested array families have drawn much attention owing to their merits on keeping the benefits of the standard nested arrays while further mitigating coupling in dense subarray portions. In this communication, a new mutual coupling model for nested arrays is constructed. Analyzing the structure of the newly formed mutual coupling matrix, a transformation of the distorted steering vector to separate angular information from the mutual coupling coefficients is revealed. By this property, direction of arrival (DOA) estimates can be determined via a grid search for the minimum of a determinant function of DOA, which is induced by the rank reduction property. We also extend the robust DOA estimation method to accommodate the unknown mutual coupling and gainphase mismatches in the nested array. Compared with the schemes of super nested array families on reducing the mutual coupling effects, the solutions presented in this paper has two advantages: (a) It is applicable to the standard nested arrays without rearranging the configuration to increase the interelement spacing, alleviating the cross talk in dense uniform linear arrays (ULAs) as well as gainphase errors in sparse ULA parts; (b) Perturbations in nested arrays are estimated in colored noise, which is significant but rarely discussed before. Simulations results corroborate the superiority of the proposed methods using fourthorder cumulants.