I assume you mean AI in the sense of powerful machine learning systems - one would think if we consider a general AI, you might consider rephrasing the question along the lines of 'What if an advanced AI behaves unethically'? (If you get my point there...)

First, I think you need a way to define 'ethics' and 'morals', with a 'ontology' and a 'epistemology' to derive a metaphysic for the system (and for my $.02, aesthetics arises from here). Until we can have a bit of rigor surrounding this, it's a challenge to discuss ethics in the context of an AI, and AI in the context of the metaphysical stance it takes towards the world.

To quickly illustrate - comments below discuss encoding race or gender bias (for example). I think this is a metaphysical problem (application of Wisdom to determine correct action) that we mistake for an epistemic issue (it came from 'bad' encoding).

Next, after we get some rigor surrounding the ethical stances you consider 'good' vs. 'bad' in the context of a metaphysic - you have to consider 'who' is using the system unethically.

Unless you have clarity here, it becomes extremely easy to befuddle, confuse, or mislead (innocently or not) questions regarding 'who'.

- Who can answer for the 'Scope' or strategic context (CEO, Board of Directors, General Staff, Politburo, etc.)

- Who in 'Organizational Concepts' or 'Planning' (Division Management, Program Management, Field commanders, etc)

- Who in 'Architecture' or 'Schematic' (Project Management, Solution Architecture, Company commanders, etc)

- Who in 'Engineering' or 'Blueprints' (Team Leaders, Chief Engineers, NCO's, etc.)

- Who in 'Tools' or 'Config' (Individual contributors, Programmers, Soldiers, etc.)

that constructed the AI.

Then you need to ask which person, group, or combination (none dare call it conspiracy!) of these actors used the system in an unethical manner? Might 'enabled for use' be culpable as well - and is that a programmer, or an executive, or both?

What I'm getting at here, is that there is both a lack of rigor in such questions (in general in this entire area - it's not like it is your fault), a challenge in defining ethical stances in context (which I argue requires a metaphysic), and a lack of clarity in understanding how such systems come to creation ('who' is only one interrogative that needs to be answered, after all).

I would say that until and unless we have some sort of structure and standard to answer these questions, it might be beside the point to even ask...

And not being able to ask leads us to some uncomfortable near-term consequences. If someone does use such a system unethically - how can our system of retributive justice determine the particulars of who to hold accountable, what went wrong, how it was accomplished (in a manner hopefully understandable by lawmakers, government/corporate/organizational leadership, other implementers, and one would think - the victims), understand why it could be used this way, and if it could happen again - just for starters.

The sum total of ignorance surrounding such a question points to a serious problem in how society overall - and then down to the individuals creating and using such tech - is dealing (or rather, not dealing) with this vital issue. We need to start talking along these lines in order to stake out the playing field for everyone NOW, so we actually might have time to address these things, before the alien pops right up and runs across the kitchen table.