Notes from BackstageCon 2022
Published: 2022-11-03 15:58:18

---
title: Notes from BackstageCon 2022
published: true
description: Interesting thoughts from BackstageCon at Kubecon 2022
tags: 
# cover_image: https://direct_url_to_image.jpg
# Use a ratio of 100:42 for best results.
# published_at: 2022-11-03 15:15 +0000
---
All talks available on this [Youtube Playlist](https://www.youtube.com/playlist?list=PLj6h78yzYM2OKySsTuiip3BqmdYZQRnSf).

- Hosts - Suzanne Daniels - Spotify, DevRel, Martina Iglesias Fernandez - Roadie, CTO

## Opening Remarks - Suzanne Daniels

- Happy Developers Make Happy Code
- Devs are spending less than 30% of their time writing code
- Backstage Stats
  - 1,500+ Contributors
  - 3,200+ Project Forks
  - 400+ Adopters
  - 7,300+ Discord Members
  - 13,000+ contributions
- Development started at Spotify in 2014, open source since 2020
  - Recently moved to CNCF Incubator status
  - 0 released
  - Passed security audit
    - [https://backstage.io/blog/2022/08/23/backstage-security-audit](https://backstage.io/blog/2022/08/23/backstage-security-audit)

## 10x the Effectiveness of Dev Teams with Backstage - David Tuite, CEO, Roadie

- Last 15 yrs brought great strides in eng effectiveness
  - Microservices
  - CI/CD
  - IDEs
- Cost is fragmentation
- Next 15 yrs about defragmentation
- Great code only matters if people knows it exists
- Backstage allows engineers to know about the great code, and use it instead of building new code from scratch repeatedly
- Templates allow teams to get started with new things quickly
- Need to organize with other teams like Sec and Infra, Backstage enables this

## Building for Developer Happiness - Dave Zolotusky, Principal Engineer, Spotify

- Devs are essentials
- Every business is now a software business
- Rising complexity to develop and scale
  - DevSecFinOps
- Devs at Spotify tell us that they're happy in their jobs, and they stay longer than the industry average
- Effectiveness = f(Productivity, Happiness)
  - Unhappy Devs don't do their best work, and deliver bad code
  - ![Slide of white text on black background stating Effectiveness = f(Productivity, Happiness)](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7lo1optz4zzrapoop429.png)
- Developers should be able to choose the right tool for the right job, and Backstage can help nudge to use the right tools, and reduce friction
- Goal to make it a critical part of every devs daily workflow
- Spotify wants to continue to share how they
- **Spotify Plugins for Backstage**
  - Available for all adopters
  - Open Sourcing homegrown and newly built plugins
  - Freemium model
    - This will be a paid solution
  - Highlights list
    - Soundcheck
      - Ensure quality, reliability and alignment of software architecture throughout your org.
    - RBAC
      - Use roles to control access to actions and data in Backstage
    - Skill Exchange
      - Build a marketplace for short-term learning and growth opportunities in R&D
    - Pulse
      - Track metrics that matter and identify gaps in your engineering practices and tools.
    - Insights
      - Measure usage of Backstage in your organization
- ![Slide detailing the info on Soundcheck, RBAC, Backstage, Pulse and Insights](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5jcm2gognsjwh0o5y6xz.png)

## Providing a Curated Experience for Developers with Backstage - Valentina Alaria, Sr. Director, VMware Tanzu

- Why do we care about Developer Experience?
  - Business depend on DevEx, they depend of developer productivity
- Orgs that have higher dev productivity are fundamentally more effective
  - Able to support bigger and greater innovation
  - Able to react to situations, more effectively
- Enterprise Golden Path
  - [https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem/](https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem/)
- Curated yet Customizable is key to offering order, best practices, and predefined solutions to help devs be consistent, and focus on new functionality vs frequent rebuilding and hunting for tools
- Paraphrase - "All about being able to enable security end to end, while reducing burden on developers"
  - It means that you can support various LOB (Lines of Business) needs without having to rebuild every workflow for each team
- How can we enable coordination between teams, especially dev and ops
- VMWare found themselves at a fork in the road, between an in house solution or an integrated version of Backstage
- Initial Backstage releases didn't fit their needs, but development on the product rapidly matured, and so they joined the community
- Backstage now powers VMWare Tanzu Application Platform

## The Evolution of Backstage Backends - Patrik Oldsberg & Johan Haals, Spotify

- Patrick
- Johan - sick, replacement Charles
- Focus on how Backstage Backends, is changing
- During open sourcing, switch to all Node.JS
- No initial backend plugins
- Goal to enable elaborate plugins, but provide consistent patterns of development
  - Simpler Installation
  - Fewer Breaking Changes
  - Less Code
  - Recognizable
  - Easy to evolve and extend
- Current: lots of copy and paste
- New system enables lightweight customizations, while having baked in templating and access for common usage patterns
- ![Diagram with Title Building Blocks. Diagram holds Backend at the top, Services at the middle, and then an interconnected bottom layer of Plugins, Extension Point and Modules.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k8avmyotq3yakrw6arsr.png)
- New model allows sharing of "factories" that can be customized but by default allow it to work between different systems without modification
- This new model will require a migration
- New plugin platform is available now [https://backstage.io/link?new-backend](https://backstage.io/link?new-backend)
- No specific timeline but hoping to launch early next year

## Building an Internal Developer Platform with Backstage Plugins - Taras Mankovski, The Frontside Software, Inc

- Era of DevX - Proactive Investment in DX
  - Even 5 years ago, there was little enthusiasm for improving DevX
- DX Leads to value
- How do we continue to develop the community around backstage
- Leads DX Consulting company - Frontside
  - Backstage Professional Services Partner
- New Plugin Demo
- Internal Developer Platform Tools
  - Allows dev to download a tool that allows them to use a Backstage Instance via CLI
  - Customized for specific instance to enable seamless login and usage
  - ![Slide showing Internal Developer Platforms (IDP) as the title. Mid text "IDP is all tools developers have to use to ship software." The rest of the slide then lists things like Cloud: AWS, Orchestration: Kubernetes, and other common items.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1oei4e5c4690wzgabr3d.png)
- Live demo
- Backstage App is not an IDE
- Provides limited use cases
  - Discovery
  - Templating
  - Doesn't support getting runtime information from other sources
    - Outside of the existing Kubernetes integration
  - Cannot manage secrets
    - Need to manually control them via different systems and tools
  - Cannot manage releases
- Dream tool, allows integrated flows between Backstage and IDEs
  - Give developers the option to choose the right tool, to help eliminate noise, and reduce friction
- "Internal Developer Platforms are a combination of all tools that developers use to run their software."
- Future Ideal State
  - Environments exist in Backstage
    - Different cards following existing UI, showing which deployments exist for a specific user/team
    - Easily see how it connects to Git and various levels (prod/dev/staging)
  - ![Slide title: IDP DX Features: Environments. DX stands for Developer Experience. Mid line: "As a developer, I want to be able to see in what environments this component is deployed." Then a mock up of an ideal Backstage focused on the developer component and deployments is shown.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rihe2e3kvi6hj2jahr6h.png)
  - Secrets Management
  - Releases
  - Logs
- Goal: How to make a Heroku-like DX on IDPs
- ![Slide title: Last missing piece of Heroku-like DX on IDPs. Mid line: "Cloud Native ecosystem has most technical pieces," Then lists: Self-serve databases, push to build workflow, automatic deployment, automatic rollback, auto-scaling, clean web ui, full featured cli.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kxvq0brmot4lfmuuv4vx.png)
- Code for IDP will be available tomorrow
  - [https://frontside.com/backstage](https://frontside.com/backstage)
- ![Slide: Resolvers, IDP specific implementation of each feature, then shows a diagram of the Platform Backend Plugin, and how it connects to Platform API, CLI binaries, resolvers and various plugins and adapters.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n3eu3iftlsxm6gjgwg9y.png)

## Enabling Developer Experience Journey in U.S. Bank - Ravikumar Tadikonda & Poonam Garg, U.S. Bank

- Lead DevX and DevOps Platform at US Bank
- 5th largest Bank in the US, operate globally
- Mission: make it easier to create and manage modern best-of-class software products
- What should we tackle first
  - Our developer and engineer experience, as they constantly faced challenges in building and maintaining solutions
- Common Engineering Challenges
  - ![Slide title: Common Engineering Challenges, and then unchained diagram, showing Day to Day multiple systems, Process, Onboarding New Team Members, Friction, Time to Onboard and Compliance.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lgxodfxkpv9grr79v5og.png)
- Attempted to build a custom inhouse solution
- After that didn't succeed, they retro'd what they could improve and how that could be possible
  - Key: provide a single pane of glass experience
- Comprehensive Approach:
  - How do we enable devs to create and manage
  - Discover and Reuse
  - Insight and Control
- Concerns that the environment is highly regulated industry
- Created user personas
  - Engineers
  - Auditors
  - Managers
  - Compliance and risk control owners
  - Building a platform that can be extended to support future roles
- How is Backstage part of our journey?
  - Focus on Create and Manage at this stage
  - Blueprints/templates
- High level architecture
  - ![Slide title: High-level architecture. Showing Developer Experience Platform at the top along with core capabilities, and below the expanded multipath experience of a developer attempting to accomplish the high level goal of deploying new code.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5lglfi36hyvhfkg7x57y.png)
- Q&A
  - Rec: Karl Haworth - American Airlines Backstage Implementation - [https://www.youtube.com/watch?v=0hdWil6cXw0](https://www.youtube.com/watch?v=0hdWil6cXw0)
  - Manages 1000s of internal applications and even more sub services
    - Have to use approved Open Source libraries and versions
    - Cataloging is key to successful management at US Bank
  - How do you manage catalogue accuracy while aggregating multiple catalogs?
    - System is not 100% comprehensive
    - Use entity providers to sync asynchronous
      - Backstage first then down to custom tools
      - Vice versa for other tools
    - Longterm goals to just be Backstage
  - How did you get started with Backstage?
    - Took one demo java app and it took about a month to create the first template from that app

## Leading Architectural Change with Backstage - Andy Hoffman, Caribou

- DevOps manager at Caribou, FinTech
- Caribou's Story
  - Main concern 2 years ago was Cashflow
  - Originally named MotoReFi
  - Ruby Monolith
  - 10X Growth during the Pandemic thru July 22
  - Need to step back and decompose complexity growth
  - Moved to K8s + ArgoCD
  - Where to go from here?
  - Regulated industry
  - Culture is metrics driven, experimental, and psych safety focused
- Understanding Obstacles
  - "It's impossible to guarantee any engineer you hire will understand or have experience with the full tech stack at your company."
  - Custom Shop to Assembly Line
- How Dev Expectations have changed
  - This shift has happened in the last 3 years very quickly
  - Must have friction remove
  - Need to develop to a polished level internal applications
- K8s is getting a bad rep
  - Complexity and overhead isn't worth it from devs
  - K8s is making the apps less stable
  - New Services take too long to stand up
  - Copypasta everywhere and too much YAML
  - ![Slide title: Fear of: K8s is Hard - Because of it's Unknown Learning Curve, Naming - Unintuitive Architectures, Takes Too Long - Unclear Scope, Slows Us Down - Unfamiliar Patterns.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/eqodhc8grhivv8v0bq4q.png)
- Naming is Hard
  - As apps become distributed, how do you name your functions and apps
- Infra and Sec teams are slowing us down
  - K8s minimal sec footprint can hinder wide open dev practices
- Anti Pattern of Distributed Monoliths is arising
  - All components are needed for any single component to work
- Cloud Native Platforms around Backstage
  - Backstage can reduce complexity
  - DevOps used to enable knowledge sharing
  - Backstage eliminates the need for DevOps/Infra/Automation team in a lot of cases
- ANTLERS - Automatic Normalized Toolchain for Launching Easily Releasable Systems
- Didn't have room to internally scale
  - Hired Roadie to build out their Backstage and ANTLERS
  - Call the system the Barn
- Architecture
  - Hard to know where you'll end up, but it's easy to know where you will start
- Simple & Intuitive Architecture == Simple & Intuitive Scaffolder Implementation
- Define Scope and Boundaries early
  - Be intentional about onboarding as it will lead to positive experiences
  - Rocky onboarding will leave a bad taste
- Dove into Argo
  - Provisions all of the infra as part of the ArgoCD implementation
- World of Pain is when a DevOps engineer doesn't think through about conventions when templating YAML
- Registering Resources at the Domain Level, sign off at levels driven by eng org
- Git driven governance
- Leverage Helm + Backstage Scaffolder
- Maturity Model -\> Focus on Lifecycle Management including building in time for known upgrades as systems become routine
- "How do we ask other teams to accept change, without acknowledging their suffering"

## What is This Tech Insights Anyways? - Jussi Hallila & Irma Solakovic, Roadie

- Tech Insights Plugin
- 2 Core Engineers who developed it
- Searching for an automated way to keep things organized and understand the relative value and usage (software quality)
- Introduced Scorecards to measure the quality of software
  - ![Slide Title: Measure the Quality of Software with Scorecards, and then shows an image of a demo app scoring 75%](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fi7xfuvdmyj0161pm886.png)
- Focused on visualizing what matters most
  - EX Snyk critical vulns over time
- Identify teams that need the most support
  - Low quality ratings across related apps visualizes a previously hidde struggle
- Automated nudges via Slack/ChatOps
- [https://roadie.io/backstage/plugins/tech-insights/](https://roadie.io/backstage/plugins/tech-insights/)
- Backstage directly supports SQLite and Postgres
- Facts
  - Data retrieved from external sources
  - Data already in Backstage
  - Can be customized to query against any database but designed around the SQLite or Postgres
- Calculations
  - Checks - Boolean/True or False
  - Aggegations, Accumulations and Indicators
    - Visualize data from databases and retrieved facts
- Visualizations
  - Scorecards
  - Graph Cards
  - Aggregate visualizations
  - Others
    - Initiatives, migrations and other tech insights
      - Ex: Tracking how teams are doing on migrations like Helm2 to Helm3
      - They used Python2 to Python3
      - Link: [How to get an overview of your tech health (Tech Insights plugin) in Backstage (Demo)](https://www.youtube.com/watch?v=K3xz6VAbgH8)
- How to Use Tech Insights now
  - Create fact retrievers
  - Create checks
  - Configure the app to display scorecards based on checks
- Open to contributions?
- Available on Hosted Backstage at Roadie.com

## Measuring Developer Happiness: Analytics for Backstage - Eric Peterson, Spotify

- @iamEAP
- Worked at Tableau for 8 years, been doing Backstage for 2 years, lives in Rural Sweden
- DevX Story
  - The Review, Build, Feedback Loop
    - Time spent waiting for either peer or CI/CD feedback between builds
- Translating the DX into Data in Backstage
- Show the grammar of Backstage so you can visualize your own devs stories
- Backstage Analytics Architecture
  - ![Slide title: Backstage Analytics Architecture. Bullet Points: Event-based system, Plugins fire events using core API, Apps provide an implemntation, Which translates and forwards to an analytics platform or service. Half of slide devotes to matching diagram.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/80kv9nwf644vgewpkb8w.png)
  - Subject -\> Noun
  - Action -\> Verb
  - Attributes -\> Adjectives
  - Context -\> Adverbs
    - Extension-Derived Context
- Works out of the box, if you use the core createPlugin function in your plugins

## How We Scaled Catalog Ingestion to Hundreds of Thousands of Entities - Damon Kaswell, HP Inc.

- 200K objects needed to be ingested from a single provider
- How do you make data objects you have no control over, searchable?
- HP has just about everything new and everything old in terms of tech stack
- Backstage kept getting slower as their usage increased
- V1 - mostly out of the box Backstage:
  - Catalog Processors emitting locations, read by other Catalog Processors
    - Recursion!
  - No control or schedule around ingestion
  - Locations polluting the catalog
  - Entities would eventually show up, but took a long time, and wasn't reliable
- V2 - Backstage with Entity Providers
  - Still lots of overlap and recursion
  - Didn't resolve slowness, and may have made it worse
  - Did have benefits in ability to clear orphaned resources automatically
    - Esp users who are no longer active
- ![Slide Title: Catalog Processors vs Entity Providers. Neither meets our scaling needs out of the box. Catalog Processor, Pros list: Simple to implement, meature feature, one-stop shop for entity ingestion. Cons list: Leaves behind orphaned entities, creates long delays in entity ingestion, better for entity "side-effects". Entity Provider pros list: Dedicated strictly to ingestion, more flexibility in scheduling, able to clean up orphans. Cons list: Cannot emit relations at all, Only has options for ingestion, full mutation or delta mutation, Delta mutation dependent on external integrations supporting that functionality, full mutation incapable of handling very large data sets, triggering errors such as javacript heap out of memory.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tfnz10fmbkxsyt325snh.png)
- Both design patterns fail to meet scaling needs out of the box
- Started running into DB Concurrency issues with V2
- Solution
  - Incremental Entity Providers
    - Built in incremental backoff
    - Data Sources must support pagination
  - Optimizing the processing loops
    - Commiting to front loading all async operations into the Incremental Entity Provider, and banned them from Catalog Processors
- Success in decreasing the the running average from 4+ hours in mid July to about a half hour for an entity to be found and scheduled, to then be ingested
  - Longest they ever saw was over 1 day
- Newest iteration in dev is ingesting new entities in about 15 minutes
  - With a full challenge of ingesting all resources from zero!
- Looking to open sourcing it in November of this year
- How can the Backstage community focus not just on making tools that get new teams from 0 to 1 quickest, but solutions that will work in 2+ years
  - Enterprise functional deployments
- ![Summary, How we made Backstage catalog ingestion scale. Incremental entity Providers: more flexible scheduling, more resilient, more manageable, able to handle large data sources. Reconfigured processors: no asynchronous operations, processing an entity now takes milliseconds. What we learned: The advice new adopters receive needs to be optimized for scaling out of the box, The design solution you use on day 1 should still work on day 700, lag in processing is cumulative, if there's lag with 1,000 entities, there will be much more lag with 100,000, Backstage can absolutely scale to handle these loads, it's a powerful platform.](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wx9kzpstcxhal1pl7lv7.png)

## A Picture is Worth a Thousand Words: Bring Data Visualization in Backstage - Olivier Liechti, Avalia Systems

- Deep dive demo into visualizing data with Vega.JS and D3.JS and how this can be used inside of Backstage
- Similar to Tech Insights, but need deep understanding of D3.JS and Vega.JS, if skill is already there should be seamless

## What We Learned During Our Journey in Using Backstage at DAZN - Marco Crivellaro, DAZN

- DAZN - Italian Media Company
- Dev Success provides support to 700+ devs in their org
- ![Slide Title: So many questions. Bullet points: which team owns a given service? What does such service do? Where can we find the code? Where is it deployed to? What are its dependencies?](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xt5nhwzgq18u10dfhej9.png)
- The dazn-manifest
- Started to integrate with Backstage in early 2021
- Start small and build piece by piece
  - Can get way too deep in the plugin ecosystem
  - Get to the core of your needs, and get 1 thing working, and then the next
  - Don't try and launch every plugin, as devs will dive deeper than you
- They focused on Software Catalog -\> Tech Docs and are now working on Scaffolder
- Don't be afraid to get your hands dirty
- Could be painful though
  - Custom integrations can be a rabbit hole
- Don't let Backstage become your single source of truth
  - Hinderance to disaster recovery if you lose backstage

## Lightning Talks

### Talking in Backstage: How to Navigate the Business and Engineering Vocabulary - Jorge Lainfiesta, Roadie

- Why? Language is hard
- Developed Course: Introduction to Backstage - https://www.classcentral.com/course/introduction-to-backstage-developer-portals-made--110320
- Adopting Backstage is an org-encompassing initiative
  - Creates a lot of possibility, but will need to work through a lot of org wide friction
- Terms
  - Doers - Get the work of backstage running
  - Sponsors - Communicate and vouch for the need during the building phase
  - Users - Interact with backstage and provide a reason for it to exist
  - Backstage is a platform for making developer portals
  - Developer Portal - A self-service one-stop shop
  - Discoverability - How easy it is to find something
  - Software Catalog - a structured collection of all software assets
    - Not just a list!
    - Can have 100s of 1000s of entities that are searchable and consumable in Backstage
  - Entities - a unit of information stored in a catalog
  - Search - a way of discovering across sources
  - Plugins - your Backstage features
    - New features are added by building internal or integrating external plugins
  - Self service - allows you to access or build something without needing to start a communication process
  - Open Source - Not Free
  - Community - Power

### Building and Contributing Our First Plug-In at TELUS - Isaiah Thiessen, TELUS

- Over 80 open source plugins in Backstage since launch ~ 2 years ago
- How he developer his first plugin
- Creator of the Backstage [Dynatrace plugin](https://github.com/backstage/backstage/pull/11754)
- Fell in love with how good the Backstage plugin docs were
- Backstage Storybook - [https://backstage.io/storybook/?path=/story/plugins-examples--plugin-with-data](https://backstage.io/storybook/?path=/story/plugins-examples--plugin-with-data)
- Learned how to use React as a backend dev
- Combined Frontend Components -\> Backstage Backend Proxy -\> API
- Worked with Tellus' Open Source office to get it shipped