One has to really stretch the modern definition of "colonization" and its moral implications for it to be applicable to the 7 centuries of Muslim Spain, but not make it applicable to the immediately preceding Visigothic and Suevi kingdoms of Hispania, some of the Germanic tribes who invaded and took over during the fall of the Roman Empire.

Even if we were to accept invasion of Iberia by the Umayyad Caliphate of Arabia in 711 as an exercise in colonisation, the Caliphate collapsed by 750, and Muslim rule in Iberia was succeeded by a complex succession of mostly local rule by the united Emirate or Caliphate of Cordova, or broken into smaller Taifas, and fluctuating levels of control by Berber dynasties which also controlled parts of North Africa—but notably would sometimes seat their court in Seville or Cordova.

Was that colonisation? If it was, it's hard to argue that the similar political and territorial changes taking place in the Christian kingdoms, counties, and principalities of Northern Spain, often by descendants of the Visigothic invaders, were not also acts of colonisation. Another example would be the Marca Hispanica, established by Charlemagne, and direct predecessor of the County of Barcelona and hence the Principality of Catalonia, a realm of the Crown of Aragon which is a constituent part of modern Spain.

Spanish mediæval history is a constant shuffle of rulers and territorial changes, which can neither be ascribed to the oversimplified idea of the Christian Reconquista (=Reconquest) nor can it be regarded as an example of colonisation in the modern sense. Regarding this complexity, consider that the Coat of Arms of Spain [0], in its many historical variations, explicitly references the Crown of Castile (Kingdoms of Leon and Castile), the Crown of Aragon, the Kingdom of Navarre, and the Kingdom of Granada. The first two entities were united by dynastic union, and the latter two were taken by conquest. To what extent was the conquest of Muslim Granada a decolonisation event, and not the conquest of Christian Navarre?

Finally, regarding Andalusia being the poorest part of the country [2] and its history with serfdom, this is more due to the repopulation carried out by the Kingdom of Castile after the (re?)conquest of territories from Muslim rulers. Much land was granted to Christian noble families and feudal lordships were established as reward to them, to a greater extent than had happened in the North of Spain in the earlier period. In fact, the effect of repopulation and land awards as recompense can be seen to a certain extent in the map of municipalities of Spain [1]. Notice that the Northern Meseta, mostly modern Castilla y León (Northwest square of inland) is much more granular than the bottom half, including the Southern Meseta, mostly Castilla-La-Mancha and Extremadura (inland area below north-south midline of Spain just north of Andalusia), Andalusia and Murcia. That is the case even though the Northern and Southern Mesetas are not significantly different in terms of climate and geography. Their only differences is how the Southern one was more systematically repopulated by the Crown of Castile, as was the rest of Southern Spain.

[0] https://en.wikipedia.org/wiki/Coat_of_arms_of_Spain

[1] https://es.wikipedia.org/wiki/Municipio_(Espa%C3%B1a)#/media...

[2] There were of course many later contributing factors, including the effect of the Spanish Empire, the effect of 19th century protectionist economic policies, as well as the whole 20th century. Here I'm only focusing on the how the late mediæval period had its effect.