Post
750
š Meta teams use a fine-tuned Llama model to fix production issues in seconds
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
š¤ 42%, is that not too low?
ā”ļø Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
š” So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
š Two-step approach:
ā£ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
ā£ Fine-tuned Llama 2 7B ranks the most likely culprits
š Training pipeline:
ā£ Continued pre-training on Meta's internal docs and wikis
ā£ Supervised fine-tuning on past incident investigations
ā£ Training data mimicked real-world constraints (2-20 potential changes per incident)
š® Now future developments await:
ā£ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
ā£ Improvements in model reasoning should boost accuracy further
Read it in full š https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response
One of Meta's engineering teams shared how they use a fine-tuned small Llama (Llama-2-7B, so not even a very recent model) to identify the root cause of production issues with 42% accuracy.
š¤ 42%, is that not too low?
ā”ļø Usually, whenever there's an issue in production, engineers dive into recent code changes to find the offending commit. At Meta's scale (thousands of daily changes), this is like finding a needle in a haystack.
š” So when the LLM-based suggestion is right, it cuts incident resolution time from hours to seconds!
How did they do it?
š Two-step approach:
ā£ Heuristics (code ownership, directory structure, runtime graphs) reduce thousands of potential changes to a manageable set
ā£ Fine-tuned Llama 2 7B ranks the most likely culprits
š Training pipeline:
ā£ Continued pre-training on Meta's internal docs and wikis
ā£ Supervised fine-tuning on past incident investigations
ā£ Training data mimicked real-world constraints (2-20 potential changes per incident)
š® Now future developments await:
ā£ Language models could handle more of the incident response workflow (runbooks, mitigation, post-mortems)
ā£ Improvements in model reasoning should boost accuracy further
Read it in full š https://www.tryparity.com/blog/how-meta-uses-llms-to-improve-incident-response