Build a Log Aggregation and Analysis Pipeline
by @melbetincome
Design a robust Log Aggregation and Analysis pipeline with architecture and code
Prompt
You are a data engineer. Design and implement a Log Aggregation and Analysis pipeline.
Provide:
1. Architecture diagram description
2. Technology choices with justification
3. Data flow from source to destination
4. Transformation logic and business rules
5. Error handling and retry strategy
6. Monitoring and alerting setup
7. Code implementation (Python/SQL)
8. Testing and validation approach
9. Performance optimization tips
10. Deployment and scheduling
Scale requirements: {scale}
Data sources: {sources}
Use prompt
Fill variables & copy
Customize the placeholders, then copy the ready-to-paste prompt into ChatGPT, Claude, or Cursor.
Ready prompt
Related prompts
Analyze Website Traffic Patterns Data
Perform a comprehensive analysis of Website Traffic Patterns data with insights
Build a Batch Processing Jobs Pipeline
Design a robust Batch Processing Jobs pipeline with architecture and code
Analyze Campaign Performance Comparison Data
Perform a comprehensive analysis of Campaign Performance Comparison data with insights
Analyze Customer Satisfaction Scores Data
Perform a comprehensive analysis of Customer Satisfaction Scores data with insights