Big Data Platform Engineer

Experience Requirements ● 8+ years of overall IT experience ● 5+ years of experience in Cloud Infrastructure Engineering ● 4+ years of experience working with Kubernetes platforms ● 3+ years of experience supporting enterprise-scale Data Platforms ● 3+ years of experience in DevOps/SRE practices ● 2+ years of experience working with Azure Data Services Mandatory Skills Candidates must possess strong hands-on experience in the following areas: ● Apache NiFi ● Azure Event Hub ● Azure Functions ● Enterprise-scale Data Platform Administration These are core and non-negotiable requirements. Profiles without hands-on experience in Apache NiFi and Azure Event Hub will not be considered. Requirements 1. Azure Kubernetes Service – AKS Strong Expertise ● AKS cluster deployment and administration ● Cluster upgrades and patch management ● Node pool management ● Ingress Controllers, including NGINX and Application Gateway ● RBAC implementation ● AKS security hardening ● Autoscaling using HPA, VPA, and Cluster Autoscaler ● Persistent storage management ● Troubleshooting cluster issues ● Network policies Experience ● Production support of AKS environments ● Multi-node and multi-environment AKS setups ● High availability and disaster recovery implementation 2. Kubernetes and Azure Networking Kubernetes Networking ● Kubernetes networking concepts ● Services, Ingress, and Load Balancers ● DNS resolution ● Pod-to-Pod communication ● Pod-to-Service communication ● Network Policies ● Service Mesh concepts, preferably Istio ● CNI Architecture Azure Networking ● Virtual Networks ● Subnets ● Private Endpoints ● Peering ● Network Security Groups ● User Defined Routes ● Azure Firewall ● VPN connectivity 3. Docker and Containerization ● Docker image creation and optimization ● Container lifecycle management ● Registry management ● Security scanning ● Multi-stage builds ● Container troubleshooting ● Azure Container Registry ● Image versioning and governance 4. Azure DevOps and CI/CD CI/CD Expertise ● Azure DevOps Pipelines ● YAML pipelines ● Build and Release pipelines ● Git repositories ● Branching strategies Infrastructure Deployment ● Infrastructure as Code ● ARM Templates ● Bicep ● Terraform Automation ● Deployment automation ● Environment promotion ● Release management 5. Monitoring and Observability Azure Monitor ● Monitoring strategy implementation ● Alert configuration ● Dashboard creation Log Analytics ● KQL query development ● Performance analysis ● Troubleshooting Application Monitoring ● Azure Application Insights ● Distributed tracing ● Metrics collection AKS Monitoring ● Cluster health monitoring ● Pod performance monitoring ● Resource optimization 6. Azure Event Hub Candidates must have strong hands-on experience in: ● Event Hub architecture ● Namespace management ● Consumer groups ● Scaling and throughput units ● Monitoring and troubleshooting ● Security and access control 7. Azure Functions Candidates must have hands-on experience in: ● Deployment and operations support ● Function App management ● Scaling configuration ● Managed Identity integration ● Monitoring and troubleshooting ● CI/CD deployment 8. Apache NiFi Candidates must have strong hands-on experience in: ● NiFi cluster setup and maintenance ● Flow deployment ● Security configuration ● Performance tuning ● Monitoring ● Troubleshooting Preferred ● NiFi Registry ● Cluster scaling ● High availability implementation 9. Infrastructure as Code Must-Have Skills ● Terraform ● Bicep ● ARM Templates Required Capabilities ● Platform provisioning ● Environment consistency ● Version-controlled infrastructure Good-to-Have Skills Platform Engineering ● GitOps using ArgoCD or Flux ● Platform automation ● Self-service platform concepts ● Internal Developer Platform Observability ● Grafana ● Prometheus ● OpenTelemetry Security ● Microsoft Defender for Cloud ● Kubernetes security best practices ● Container security ● Secrets management ● Vulnerability management Data Platform ● Azure Data Factory ● Synapse Analytics Cloud ● Multi-region architecture ● Disaster recovery planning ● Landing Zone implementation Scripting ● PowerShell ● Bash ● Python Key Responsibilities ● Manage and support AKS-based Big Data platform infrastructure. ● Administer Apache NiFi, Azure Event Hub, and Azure Functions. ● Implement and maintain CI/CD pipelines using Azure DevOps. ● Monitor platform health using Azure Monitor and Log Analytics. ● Troubleshoot platform, networking, and infrastructure issues. ● Implement infrastructure automation through Terraform and Bicep. ● Ensure platform security, governance, and compliance. ● Collaborate with Data Engineering teams for platform enablement. ● Perform upgrades, patching, capacity planning, and optimization. ● Support production incidents and platform reliability initiatives.