Deploy Azure Data Services with Terraform

Terraform-based deployment of almost all Azure Data Services (default deployment settings are in the parenthesis):

  • Azure Service Bus (Standard, namespace,topic,subscription, auth. rules)
  • Azure Data Lake Storage (ZRS, Hot, Secured, StandardV2)
  • Azure Data Factory (w/Git or without)
  • Azure Data Factory linked with Data Lake Storage
  • Azure Data Factory Pipeline
  • Azure DataBricks WorkSpace (Standard)
  • Azure EventHub (Standard, namespace)
  • Azure Functions (Dynamic, LRS storage, Python, w/App.Insights or without)
  • Azure Data Explorer (Kusto, Standard_D11_v2, 2 nodes)
  • Azure Analysis Server (backup-enabled, S0, LRS, Standard)
  • Azure Event Grid (domain, EventGridSchema)

Properties and content

  • 831 strings in total
  • Written about 1 year ago, updated a day ago to fix deprecated expressions
  • Tested with the latest Terraform 0.13.2 and Azure provider 2.27.0 (in fact, works fine with >=0.12 and Azure provider >= 1.35)
  • auth.tf – provider authentication and version settings
  • main.tf – a desired Azure infrastructure
  • terraform.tfvars – controls deployment settings
  • variables.tf – variables list
  • outputs.tf – outputs useful information

Deployment settings (excerpt)

#--------------------------------------------------------------
# What should be deployed?
#--------------------------------------------------------------
servicebus       = true  # Azure Service Bus
datafactory      = true  # Azure Data Factory
datafactory_git  = false # Enable GIT for Data Factory? (don't forget to set Git settings in the Data Factory section)
databricks       = true  # Azure DataBricks
eventhub         = true  # Azure EventHub
functions        = true  # Azure Functions 
functions_appins = true  # Integrate App.Insights with Azure Functions?
eventgrid        = true  # Azure EventGrid
kusto            = true  # Azure Data Explorer (kusto)
analysis         = true  # Azure Analysis Server

Resource block (excerpt)

resource "azurerm_function_app" "rlmvp-svc-function-appins" {
  count                      = var.functions == "true" && var.functions_appins == "true" ? 1 : 0
  name                       = "${var.prefix}function${random_string.rndstr.result}"
  location                   = var.az_region
  resource_group_name        = azurerm_resource_group.az_rg.name
  app_service_plan_id        = azurerm_app_service_plan.rlmvp-svc-appplan[count.index].id
  storage_account_name       = azurerm_storage_account.rlmvp-svc-storacc[count.index].name
  storage_account_access_key = azurerm_storage_account.rlmvp-svc-storacc[count.index].primary_access_key
  # storage_connection_string = azurerm_storage_account.rlmvp-svc-storacc[count.index].primary_connection_string (deprecated; works though)
  app_settings = {
    "FUNCTIONS_WORKER_RUNTIME"       = var.az_funcapp_runtime
    "APPINSIGHTS_INSTRUMENTATIONKEY" = azurerm_application_insights.rlmvp-svc-appins[count.index].instrumentation_key
  }
  tags = var.az_tags
}

Usage guide

  • Open the terraform.tfvars file
  • Indicate the “What Should Be Deployed?” section
  • Use true/false to set your desired configuration
  • Check or change Azure services settings in the appropriate sections (naming convention (prefix/suffix), location, SKUs and etc.)
  • Run terraform init to get required Terraform providers
  • Run terraform plan to initiate pre-deployment check
  • Run terraform apply to start a deployment
  • (optional) terraform destroy to delete Azure resources

Requirements

  • The script uses Service Principal authentication, so define the subscription ID, client ID, tenand ID and principal secret in the auth.tf (or use another authentication type – Managed Identity, if your CI is running on Azure VMs, for instance)
  • If you are going to deploy Analysis Server (enabled, by default), provide valid Azure AD user(s) UPN(s) to set them as administrators of Analysis Server (az_ansrv_users variable, file – terraform.tfvars)

Result

Deployed Azure Resources (all in one resource group)
Terraform output

P.S. feel free to share/commit/fork/slam/sell/copy and do anything that your conscience allows you 🙂

Backup Best Practices in Action – The Backup Bible Part 2

The Backup Bible is an ambitious project produced by Altaro and written by Microsoft MVP Eric Siron that aims to be the definite guide to backup and disaster recovery for companies of all sizes.

The first eBook in the series was published earlier this year and focused on how to create a robust backup strategy including the key stakeholders you’ll need to deal with and the questions you’ll need to ask yourself to prepare yourself for a potential data loss event.

Part 2 – Backup Best Practices in Action – follows on from this starting point explaining how to implement this strategy and showing the reader what secure backup looks like on a day-to-day basis.

The eBook is focused on providing practical implementation using actionable steps in each section providing the reader with the knowledge to bring the theory to life. It covers:

  • Choosing the Right Backup and Recovery Software
  • Setting and Achieving Backup Storage Targets
  • Securing and Protecting Backup Data
  • Defining Backup Schedules
  • Monitoring, Testing, and Maintaining Systems
  • And More!

The Backup Bible is an essential resource for anyone who manages data on a daily basis. For any business, data is your lifeline. A significant data loss can cause irreparable damage. Every company must ask itself – is our data properly protected?

You can download both part 1 and part 2 for free right now! The final part of the series, on disaster recovery, will be released later this year. By accessing the available parts, you’ll automatically receive the final eBook direct to your inbox when it is released later this year!

What are you waiting for? Get your eBook now!