Standard_NV6 6 1 x K80 GPU (1/2 Physical Card) 56 380 Azure Standard_NV12 12 2 x K80 GPU (1 Physical Cards) 112 680 Azure Standard_NV24 24 4 x K80 GPU (2 Physical Card) 224 1440 Azure Microsoft Azure Enterprise Operations To operate your application, service, or infrastructure within Microsoft Azure, it is important to understand the roles, access methods, and components that make up a given organizations Azure environment. This section covers each of these areas at a high level. Enterprise Roles and Portals Roles Within a given enterprise enrollment, Microsoft Azure has several roles that individuals play. These roles range from creating subscriptions (covered later in this document) to provisioning resources. The following top roles exist within Azure: Role Quantity/Description Functions/Permissions Enterprise Administrator There may be multiple Enterprise Administrators per Enterprise Enrollment Manage accounts and Account Owners Manage Enterprise Administrators View usage across all accounts View unbilled charges across all accounts Account Owner Each account requires a unique Microsoft Account or Organizational Account Create and manage subscriptions – only the account owner is able to perform these functions Manage Service Administrators and View usage for subscriptions
RA-GRS), the blob type, storage service, storage transactions, and the use of premium storage will affect the overall cost of the Azure Storage solution. accessed, you should also consider the costs involved. Your total cost depends on how much you store, the volume of storage transactions and outbound data transfers, and which data redundancy option you choose. data that is critical to a business may drive the decision to have GRS, whereas data that is less critical may suffice with LRS. Data that must be quickly accessed with the highest possible IOPS may drive the usage of premium storage, where data without that requirement may accept the use of standard storage. These requirements will necessitate and support the higher costs associated with the storage services. Storage containers can be used to further organize data in storage accounts. Decide how you want the data in Azure Storage to be organized. Deciding how to design and build containers is similar to how you would design and build a folder structure on a file server. It is simply how you want to organize the data. By default, all VHDs will be put into a “vhds” folder, but you can change or specify whatever container structure you want to use. Concurrency settings can be modified for Azure Storage accounts. Modern applications usually have multiple users viewing and updating data simultaneously. This requires developers to think carefully about how to provide a predictable experience to their end users, particularly for scenarios where multiple users can update the same data. You can opt to use optimistic or pessimistic concurrency models to manage access to blobs and containers in the Blob service. If you do not explicitly specify a strategy, last writer wins is the default. For IaaS, concurrency settings do not need to be modified.
on encryption, and then you upload blobs or archived VHDs to that storage account and then turn on SSE, those blobs will not be encrypted unless they are rewritten or copied. Table, Queues, and Files data will not be encrypted Subscription-Level Protection Types Data-At-Rest Data-In-Transit Data Access IaaS Performed by the customer by encrypting the virtual hard disk (VHD) files. Microsoft and third-party mechanisms are used. Workloads (such as SQL Server) also support Transparent Data Encryption (TDE). Technologies that assist with this are: Key Vault SQL Server Transparent Data Encryption Azure Disk Encryption Azure Storage Service Encryption Performed by the customer by using transport encryption of traffic traversing exposed virtual machine network endpoints. Microsoft and third-party mechanisms are used. Actions performed by Microsoft include disk encryption using BitLocker Drive Encryption for bulk import/export operations and encrypting traffic between Azure datacenters. Technologies that assist with this are: HTTPS/REST API Azure endpoints Azure Import/Export service Performed by the customer by using native protections within the installed operating system to authenticate and authorize access to the virtual hard disk (VHD) data that is exposed through the operating system and published endpoints (for example, operating system file shares). PaaS Performed by the customer by encrypting data located in Queue, Table, and Blob storage. Uses Microsoft encryption mechanisms. Technologies that assist with this are: Performed by the customer by using transport encryption of traffic traversing storage account network endpoints. Microsoft and third-party mechanisms are used. Performed by the customer by using shared access keys and shared access signatures to provide access to data stored in Queue, Table, and Blob storage.
Key Vault Client-Side Encryption Azure SQL Database Transparent Data Encryption Azure Table Encryption Actions performed by Azure include encryption of traffic between Azure datacenters. Technologies that assist with this are: HTTPS/REST API Storage account endpoints Technologies that assist with this are: Shared access signatures Storage account access keys Storage account endpoints StorSimple Performed by the appliance using AES-256 encryption with Cipher Block Chaining (CBC) prior to saving to the mapped Azure storage account. Performed by the customer by using transport encryption of traffic traversing exposed physical or virtual machine network endpoints. Performed by the appliance using SSL encryption. Performed by the customer by using native protections within the installed operating system to authenticate and authorize access to attached StorSimple volumes. Performed by the appliance by using authentication protocols (such as CHAP), ACLs, network access control, and Role-Based Access Control (RBAC). Feature References BitLocker Drive Encryption Key Vault Encrypt and decrypt blobs in Microsoft Azure
Recommended: Azure Storage supports HTTP and HTTPS; however, using HTTPS is highly recommended. Optional: During shipping, physical media may need to cross international borders. You are responsible for ensuring that your physical media and data are imported and exported in accordance with the applicable laws. Before shipping the physical media, check with your advisors to verify that your media and data legally can be shipped to the identified datacenter. This helps ensure that it reaches Microsoft in a timely manner. Design Guidance When you design storage management, consider the following: Capability Considerations Capability Decision Points Capability Models in Use There is a need to determine whether it is appropriate to store the data in Azure Storage. When moving data into Azure Storage (aside from IaaS VHD files), it is important to examine and determine the suitability of placing that data into Azure. There are three primary factors to examine to determine if data should move to Azure Storage: Access frequency – How often is the data is accessed? Data that is accessed on a frequent or regular basis may not be a good candidate to store in the cloud. Data that is accessed less frequently, such as archive data, is better suited for the cloud. Active data vs. passive data – How much data latency is tolerable? When data that is stored in the cloud needs to be accessed, additional latency should be expected when reading or writing that data. Applications or services that cannot handle latency should not store its data in the cloud. For example, do not install a SQL Server on-premises and store its databases in cloud storage. Privacy requirements –What kind of data is permitted in cloud services for your organization? Based on what country, industry, and customer you Suitability is something that is unique to every customer. It is important to first understand the data privacy laws that may apply to the customer’s country, industry, and any regulatory bodies that may govern it. From there, begin to understand the sensitivity of the data that may be stored. Then look at the technical aspects of access frequency and active vs. passive data. It doesn’t make sense to look at the technical aspects of the data
The following summary of the capabilities of each virtual machine series is provided in the following table: Size Available CPU Cores Available Memory Available Disk Sizes Maximum Data Disks Maximum IOPS Basic_A0 – Basic_A4 1 – 8 768 MB – 14 GB Operating system = 1023 GB Temporary = 20-240 GB 1 – 16 300 IOPS per disk Standard_A0 – Standard_A11 (Includes compute intensive A8-11) 1 - 16 768 MB - 112 GB Operating system = 1023 GB Temporary = 20-382 GB 1 – 16 500 IOPS per disk Standard_A1_v2 – Standard_A8_v2; Compute-intensive Standard_A2m_v2 - Standard_A8m_v2 1 - 8 2 GB – 64 GB Operating system = 1023 GB Temporary SSD disk = 10-80 GB 1 - 16 500 IOPS per disk Standard_D1-D4 Standard_D11-D14 (High memory) 1 - 16 3.5 GB – 112 GB Operating system = 1023 GB Temporary (SSD) =50 – 800 GB 2 – 32 500 IOPS per disk
The load balancer distributes traffic by a simple hashing algorithm across all endpoints that are considered Ready/Healthy. The load balancer support two alternative methods for deciding on the status of the instances that belong to a cloud service role. The two methods are: Using the Guest Agent Responds with HTTP 200 OK only when instance is Ready. It does not monitor the w3wp.exe process that the websites run in. NOTE: When using the guest agent probe, the role itself (WaIISHost.exe) may be running just fine and reporting Ready (thus receiving incoming LB traffic) while the website (w3wp.exe) may be failing and returning HTTP 50x errors to end users. This exposes one of the disconnects between the role and w3wp in that the role and guest agent don’t actually know anything about w3wp status or health, even though w3wp health is what matters for the role instance availability. Load Balancer Probe Overrides the Guest Agent. Checks for HTTP 200 OK every 15s *, and times out if no response is received in 31s * (*=default settings). NOTE: With the custom LB probe, the role itself may be unhealthy or in the process of shutting down while the website is still returning an HTTP 200, thus receiving new incoming connections even when the role is in a potentially bad state or about to be shut down. This exposes the other disconnect between the role and w3wp in that the role doesn’t directly coordinate lifecycle with w3wp. Neither method by itself will provide a full view of the health of the role. A better approach is to implement a combined approach: The Guest Agent is used to monitor overall role health and handle life cycle events like stops. Important: No load balancer probe is implemented directly. Implement a custom RoleEnvironment.StatusCheck event handler in the WaIISHost.exe. This can probe a health probe for the application and signal its health status back to the Guest Agent. This gives the benefit of both worlds of the Azure controlled lifecycle plus a custom HTTP probe. 5.2.1.1 Load Balancer Probing
that needs to be load balanced should be referencing the load balancer. Load Balancers can be internal or external. Virtual IP Address Cloud Services will get a default VIP (Virtual IP Address) when a virtual machine is added to a cloud service. The Virtual IP Address is the address associated with the implicit load balancer. Public IP address is a resource exposed by the Microsoft.Network provider. Public IP Address can be Static (Reserved) or Dynamic. Dynamic Public IPs can be assigned to a Load Balancer. Public IPs can be secured using Security Groups. Reserved IP Address You can reserve an IP Address in Azure and associate it with a Cloud Service to ensure that the IP Address is sticky. Public IP Address can be created in “Static” mode and it offers the same capability as a “Reserved IP Address”. Static Public IPs can only be assigned to a Load balancer right now. Public IP Address (PIP) per VM Public IP Addresses can also be associated to a virtual machine directly. Public IP address is a resource exposed by the Microsoft.Network provider. Public IP Address can be Static (Reserved) or Dynamic. However, only dynamic Public IPs can be assigned to a Network Interface to get a Public IP per virtual machine right now. Endpoints Input Endpoints needed to be configured on a Virtual Machine to be open up connectivity for certain ports. One of the common modes of connecting to virtual machines done by setting up input endpoints. Inbound NAT Rules can be configured on Load Balancers to achieve the same capability of enabling endpoints on specific ports for connecting to the virtual machines. DNS Name A cloud service would get an implicit globally unique DNS Name. For example: mycoffeeshop.cloudapp.net. DNS Names are optional parameters that can be specified on a Public IP Address resource. The FQDN will be in the following format
Control ingress and egress traffic for a single virtual machine Forced tunneling Default route for a gateway that send all non-local traffic to the customer's edge router for processing. Can be combined with Azure Public peering to route traffic from inside a VNet to Azure Public services via an router or firewall. Block outbound Internet traffic in a virtual network Block inbound traffic in a virtual network ExpressRoute is implemented at the BGP routing level. Provides defense-in-depth by role separation. Firewall appliances – single network adapter Software-based firewall that can be placed between virtual machines and the Internet. Requires all traffic to be routed through the firewall, typically by using agents and IPsec. Need additional firewall protection from the Internet Want traffic flow control of all virtual machine traffic Want to log and monitor traffic Firewall appliances – dual network adapters Software-based firewall that can be placed between subnets or between a subnet and the Internet Need additional firewall protection from the Internet and also need high throughput. Want packet capture and inspection Want detailed logging IPsec Traffic authentication and encryption at the server level. Requires machines to be domain joined, Want to use policy-based traffic encryption Want to control which servers can communicate with each other Hardware firewall appliances at the network edge Placing a hardware firewall appliance at the customer’s network edge Want to control ingress and egress traffic between Azure and on-premises
Network Access Approaches When customers extend their datacenters to Azure or deploy an application within the Azure infrastructure, they must select an approach for access control and security, based on an access scenario. Common access scenarios include: Internal users Accessing the solution from on premises Accessing the solution from non-corporate locations Accessing the solution via VPN External users Accessing the solution via the Internet Dependent applications Accessing the solution from on premises Accessing the solution from Azure Accessing the solution from Internet locations Solution Accessing on-premises resources Accessing Azure public services Accessing other Internet-facing services Note that a security-access approach might have multiple options to provide access. For example, accessing an application in Azure via the Internet can be accomplished with different security and traffic routing approaches. Application Access Approach Description When to Use Direct to Azure Internet access is accomplished by exposing the UI tier directly on the Internet. The application needs to be accessed from the Internet, and minimal security is required. The application has no connection to corporate resources.
Virtual Appliance Scenarios The following series of scenarios have been developed and are currently being turned into reference implementations. The diagrams show the scenario network connectivity. NSGs are represented at the subnet level with a shield . UDR are represented at the subnet level using a symbol. Scenario Name Type Appliances Description Protect On premises resources Ingress Firewalls Leverage NVAs to firewall off ingress access to Azure and restrict routing of traffic to on premises destinations. Typically used when the customer has applications that exist in one or more public clouds and want to use Azure as the guardian to on premises. Monolithic Application Ingress Firewalls Leverage NVAs to control ingress traffic via Azure. Traffic is allowed to flow to the UI tier of the monolithic application, but any traffic flow between the layers of the app leverage NSGs to control flow Monolithic Application with NVA flow Ingress Firewalls Leverage NVAs to control ingress traffic via Azure. The NVAs are also connected to every subnet tier of the application and all traffic flow must go through the NVAs. This can be implemented with a single backend NIC or multiple, the preferred approach is multiple NICs to ensure performance. Multiple NVA Stack Ingress Firewalls IPS/IDS Expanded version of the Protect On Premises Resources scenario where additional NVA are added in series to provide additional protection. Public Peering Egress Layer 7 Web Application Firewall Leverage L7 WAF devices to control what Azure based traffic can connect to on premises or via a public Peering connection. This is focused on data loss prevention and requires any system in Azure to flow through the L7 appliances to get to public peering. This allows the customer to control what traffic can flow and to what Azure public services via FQDN limitations. Shared Security Stack Ingress Firewalls IPD/IDS Load Balancers L7 Web Application Firewalls Leverages a multi-tier security stack in Azure to provide a replacement to an on premises security stack. This allows customers to monitor, log, and control all ingress traffic. This is a highly complex implementation and requires multiple vendor solutions typically.
Default Outbound Network Security Group Rules Description Priority Source Address Source Port Destination Address Destination Port Protocol Action Allow virtual network outbound 65000 VIRTUAL_NETWORK * VIRTUAL_NETWORK * * Allow Allow Internet outbound 65001 * * INTERNET * * Allow Deny all outbound 65500 * * * * * Deny Subscription Limits for Network Security Groups Object Resource Management Subscription Limits Network Security Groups 100 per region/per subscription 1 Network Security Group per subnet 1 Network Security Group per network adapter 1 Network Security Group can be linked to multiple subnets or network adapters Network Security Group rules 200 rules per Network Security Group* *Can be increased by Microsoft support personnel to a maximum of 400 rules per Network Security Group. Default tags are system-provided identifiers to address a category of IP addresses. Default tags can be specified in customer-defined rules. Note that the format of the tags is different for ASM versus ARM NSG rules. The default tags are as follows: Tag Description VirtualNetwork (ARM) This default tag denotes all of your network address space. It includes the virtual network address space (IP CIDR in Azure) and all connected
VIP Virtual IP address. Assigned to a virtual machine, cloud service load balancer, or an internal load balancer. Address is private for an internal load balancer and is public for a cloud service load balancer or a virtual machine. Address is shared across all virtual machines within the same cloud service. Always assigned PIP Public IP address. Public instance-level IP address that can be assigned to a virtual machine. A PIP allows direct communication to a virtual machine without going through the cloud service load balancer. Use only when you need to directly communicate with an instance in cloud service Reserved This is a static public-facing VIP address for a cloud service that must be specially requested. There are a limited number of these addresses per subscription. This is an ASM construct. Use only when you need a public facing static IP address Public Static This is a static public-facing VIP address for a load balancer. This is the equivalent of a ASM Reserved IP Address. This is an ARM construct. Use only when you need a public facing static IP address Internal static A static address allocated from the subnet address pool. Internal facing only. The number is only limited by the number of addresses assigned to the subnet address pool. This is implemented as a DHCP reservation. Use only when you need an internal facing static IP address For more information, see . The relationship is illustrated in the following diagram for ARM environments:
Networking Considerations When considering extending Active Directory to Azure, there are two primary areas of networking that need some thought: Connecting on-premises domain controllers to Azure virtual machines Networking the domain controllers in Azure with the virtual networks in IaaS Connectivity to Azure Whether the domain controllers are on-premises or deployed in the Azure, there needs to be connectivity between the Azure virtual network and your on-premises network. If you want to keep domain controllers on-premises, you need an ExpressRoute connection or a Site-to-Site VPN connection to Azure. Every time a virtual machine in Azure needs to access a domain controller, it will traverse this connection over the WAN. Depending on the stability and performance of the connection, this may cause issues. You should ask the following questions: What kind of connection is available between the on-premises network and Azure? If it’s an ExpressRoute connection, you are more likely to keep the domain controllers And it’s even better if it’s an IXP connection instead of NSP. For more information about ExpressRoute, see the ExpressRoute Overview section earlier in this document. What is the cost of network traffic across the connection? If there’s a high cost associated with the connection, and you expect the virtual machines to communicate heavily with Active Directory, it would probably be more cost-effective to include the domain controllers in IaaS. How stable is the network connection with Azure? Accessing domain controllers over the WAN will not work well if the connection is unstable. Inaccessibility to Active Directory can cause many applications and services to be unusable. If you have applications with dependencies on Active Directory, you’ll want to put in a more stable connection (consider ExpressRoute) or put the domain controllers locally in the IaaS tenant. IP Addressing Virtual machines in Azure get IP addresses assigned dynamically from the virtual network that they reside in. To the operating system, this assignment occurs via DHCP. If you try to change the IP address to a static IP address, Azure will isolate the virtual machine, and it won’t be able to communicate on the network.
This should include guidance about all settings that have a security impact including documentation of: Threats – What the setting is designed to mitigate Countermeasures – How to configure the setting correctly to mitigate the threat Potential Impact –The operational impact of the setting configurations What tools are available to check the integrity of the system after deployment? What security considerations need to be taken into account by a developer calling the system’s APIs, if applicable? Is the product is designed to be exposed to Internet traffic without requiring a firewall? Identify what security logs are produced by the product, how they are protected against tampering, and how to access them. Require vendors to minimize and document attack surface changes resulting from installing software. This can be performed by using the Microsoft Attack Surface Analyzer tool. For more information, see: Require vendors to document which threats the product is and is not designed to protect itself and its data against. Typical items that cannot be protected against are identified in: Software digital signatures Require vendors to digitally sign all software binaries for all software products. For more information, see . Software development practices: Gather information about the vendor’s development practices Secure development education Do you educate all engineers about security development? How frequently? How much time per year for each engineer? What methodology is taught? Who teaches the classes? Development security and privacy processes What security development methods do you practice? The Microsoft Security Development Lifecycle? ISO 27034? Something else?
Network Security Group Rule Name Leverage targeted zone in the name Alphanumeric, underscore, period, and the hyphen character are permitted Must start with a letter or number Must end with a letter, number, or underscore Can have up to 80 characters Must be unique within a region Case IBA – InBound Allow IBD – InBound Deny OBA – OutBound Allow OBD – OutBound Deny Priority 100-4096 CZ-NP-IBA-Port-Protocol-Priority CZ-NP-IBA-443-TCP-400 Route Table Name Must begin with a letter or number Must end with a letter, number or underscore May contain only alphanumeric, underscore, period, or hyphen CZUEPDADC001 Virtual Network Gateway Name Must begin with a letter or number Must end with a letter, number or underscore May contain only alphanumeric, underscore, period, or hyphen CZUEPDADC001 Resource Group Name Must be unique within a subscription Length 1-80 characters It can contain only alphanumeric characters and -_(). The name cannot end with a period Application CZE-ANB-PDA-RGP001 Infrastructure CZE-INF-PDA-RGP002 Azure Automation Account Name Must be from 6 to 50 characters Can contain only letters, numbers, and hyphens. The name must start with a letter, and it must end with a letter or a number Must be unique within a subscription Since there is no application specific association CZ-PD-Automation CZ-NP-Automation Azure SQL Logon Name Must be a SQL identifier Cannot be a system name, for example: admin, administrator, sa, root, dbmanager, loginmanager, etc Built-in database user or role like dbo, guest, public, etc. Cannot contain: White space like spaces, tabs, or returns Unicode characters Nonalphabetic characters (“<>|:*?\/#&;,%=) Cannot begin with: Digits (0 thru 9) @, $, + CZ-ANB-PD-SDB-ADM01