A production engineer is a specialized role responsible for ensuring the smooth operation, performance, and stability of production applications. They possess deep knowledge of the production environment, monitoring tools, and operational best practices.

What do production engineers do?

A production engineer’s primary responsibility is to troubleshooting and resolve long running issues in production— these issues take the form of data corruption, resource leaks (connection, memory), and performance degradation.

To that affect, they will analyze charts and graphs, add extra logging, profile the application, and setup experiments in production to reproduce the issue. Once the issue is reproduced, they will work to implement the appropriate remediating solutions.

Prevention is better than the cure, and so, production engineers work diligently to improve the monitoring (and alerting) of production applications. Production Engineers will set up dashboards, error tracking and diagnosing tools; alerts and notifications to proactively track and review the health and performance of production applications.

Lastly, production engineers will use their diagnostic tools and internal knowledge of the systems to identify and prioritize performance bottlenecks in applications and their corresponding components. This involves analyzing database queries, implementing caching strategies and implementing queuing systems to to move towards asynchronous workloads.

What production engineers don’t do.

Re-architect and re-implement existing applications. Re-architecting or re-implementing existing applications is typically a complex undertaking that involves various stakeholders, including software architects, developers, and product owner. While production engineers may have a good understanding of application architecture, their expertise lies more in ensuring the smooth operation of the existing systems rather than fundamentally altering them.

Production engineers are not responsible for building and maintaining the deployment (CI/CD) or the underlying infrastructure that runs the applications, e.g., Kubernetes. Production engineers may collaborate with SRE/DevOps and provide guidance on infrastructure requirements, performance considerations, and operational aspects.

Similarly, production engineers are not responsible for Security and Risk Management — they do not implement access controls, vulnerability management solutions, or ensure compliance. Security is a broad and complex field which requires specialised knowledge. Security professionals, e.g., DevSecOps folks, are better equipped to understand and address compliance needs, conduct security audits, and manage the associated documentation. Incidentally, their specialised access to the production environment makes them a security risk and poor candidates to enforce security.

Conclusion

Production engineers play a crucial role in maintaining and optimizing the production environment. Their expertise lies in troubleshooting and resolving issues, improving monitoring and alerting, and identifying performance bottlenecks.

👋🏽 Is postgresql making you sad? Reach out, say hi 👋🏽! I’m available for hire.