How to interview DevOps candidates?

A little bit of sour-sweet memories

I’ve been in many job interviews. Few as a candidate but many more as a Technical Leader or Hiring Manager. There are many questions I’ve been asked and many tests I passed. I remember one test, where I had to install and configure Apache Tomcat on a Virtual Machine, under Windows (which I don’t use for years), without access to internet. There were more traps there, like some files in the locations where Tomcat is normally installed were blocked for write with chattr immutable attribute. I was able to configure the application to run, but then I asked what’s the point of this exercise? Main explanation was to “understand my knowledge”. I asked more questions:

• How often do you work without access to Internet?
• Do they look for people with encyclopedic knowledge or those that can do the work?
• What about topics where you have to discover how to do something, that no one else did before?
• Do you often work on a systems where others prepare “traps” for you?

Just to cut it here. I’ve been irritated, by the test. They’ve been irritated by my questions, although they admit they touch some points.

They propose me a job, but I rejected. I didn’t want to work there. So how to do it better, what could be a better approach?

Review as an interview

That’s something we practice more often recently. Idea is simple, let see how candidate works in similar topics and environment we do. We prepare a repo, which is just not in the best shape (to say it gently). Actually it’s completely fine if you take one of the “skeletons in the closet”, remove sensitive stuff and throw more rubbish at the top of it like a cherry 😄

You share this repo with candidate day or two before the interview. It should be enough time to get familiar with it, but we don’t want to make it too expensive and spend a whole week on it. We don’t expect any changes, any work in front.

The test question is: Check this repository and we would like to discuss what kind of improvements you would suggest?.

In our repo we have:

• basic Terraform infra with unencrypted resources and few spooky policies,
• some Python code,
• some secrets committed,
• some comments of TODO/TBD/FIXME type,
• some rubbish accidentally committed,
• etc.

Project should generally represent a landscape of technologies YOU use. There’s something for everyone and we already collected twice as much good advices, than we expected to hear 😄

It’s more natural for people to work with such “test”. There will be people that will have a lot of Cloud improvement suggestion, but won’t touch the code, or vice versa. Some people would advice to replace Jenkinsfile with Github Actions, others would prefer to deploy with Ansible. You can guide this person through the repo, ask more targeted questions. Those are usually a good hook points to ask about specific technology experience or project’s size. You can see if they think big or small, if scale is similar to your expectations.

I’ve read a blog¹ which I read and it’s quite aligned with what we do.

DevOps interview questions

The list below is with me for many years right now, but I still use some of the questions from time to time.

This is the more old school way of doing interviews. I collected a bunch of good questions, that I often use as a starters, to worm up a candidate or just go one by one if candidate is too stressed to discuss freely.

General

1. Please tell us about one of your success stories – something you’re proud of, something you did and you really enjoyed (best if this is technical story).
2. Please tell us about one of your failure stories – something you did but you’re not proud of it. (best if this is technical story)?

3. If you could choose anything, what your dream job would look like? What you would like to do from the morning till the evening?

4. Do you know what the term Twelve-Factor App  mean? Could you tell something about it?

5. What’s the difference between Continuous Integration/Continuous Delivery/Continuous Deployment?
6. What Configuration Management tools do you know, which will you choose and why?

Attitude

1. How lazy are you? I.e. which tasks would you not automate?

2. How much access should Developers have to a production system? Set out reasons why.

3. Have you taken part in Agile/Scrum teams?
4. Are you comfortable suggesting improvements to an architecture? Can you provide occasions you have?
5. How would you handle a technical disagreement with a colleague?
6. Do you consider yourself a developer, sysadmin, or other? Which is more appealing to you?
7. Security, Performance, Features, Reliability. list them in importance.
8. Who should be responsible for deployments?
9. The team are planning on deploying an application change you believe is fundamentally unsound. How do you deal with this?

Development

1. Describe Language experience. MUST have at least good knowledge of on of Python, Ruby, Java, GoLang. Just Bash doesn’t count.
2. Do you use GIT (or other SCM) in your daily duties?
3. Can you name and explain any GIT workflows?
4. Talk about a reasonable size development project you’ve created. What did you use? Which Frameworks?
5. Describe the difference between a unit tests, integration test and performance tests. Talk about the tools you would use for each. At which stage of the development would you use them?
6. Have you ever been working with SSO software? SAML/OAuth/Open ID Connect?
7. Have you been working with Spring and dynamic properties? SpringCloud/Consul?
8. How do you test infrastructure deployments? How do you version infrastructure changes?
9. Describe the concept of idempotent calls and how it applies to CM tools.
10. Please provide Github repo. If no Github repo, why not?
11. What kind of communication patterns/protocols in micro-services ecosystem do you know?
12. What are the advantages of using micro-services?
13. How can you introduce an API / contract breaking change?

DBs

1. What DB experiences do you have?
2. What DB engines do you used?
3. We have shared between few apps DB schema. It’s starting to be a problem. What would you suggest?
4. Types of SQL joins?

Jenkins

1. Can you elaborate on best practices on Jenkinsfile writing?
2. Do you know how to create Jenkins pipeline libraries?
3. Do you know what are best practices for pipeline libraries creation?

Docker

1. What are good practices for writing Dockerfiles?
2. What problems arrive in services that run on Docker?
3. We have 4 different environments: DEV, QC, UAT, PROD – there are small differences between them (ex. different login/password/IP on DB server). We use Docker to prepare images with our applications. What would you suggest to do, to reflect those small differences between environments but without need to build different Docker images for them?

Monitoring/Metrics

1. Have you ever been working with ELK stack?
2. Do you know Grafana?
3. What about Sensu/Nagios/Zabbix?

Linux

1. How will you check who lately logged to server via SSH or by accessing machine?
2. How will you check open connections on server?
3. How will you check what service is running on port 2345?
4. How many connections are opened now to port 2345?
5. You have no space left on device error but df -h shows a lot of free space – what could go wrong?
6. How to change number of available inodes on existing volume?
7. Apache server is running under huge load (ex. 300 or 3000) but you could login to it and system is responsive. What could be wrong?
8. You have a lot of Apache processes in state D – how will you kill them?
9. How to create Zombie process? How to kill it? Is there any mechanism that should kill them automatically?

CDN/WAF/LB

This one is very specific, but I was looking for people with this specialization.

1. Have you ever been working with Apache?
   • mod_rewrite
   • mod_proxy?
   • mod_security?
   • other modules?
2. Are you familiar with Nginx?
   • rewrites?
   • proxy?
   • NAXSI?
3. What’s difference between rewrite and redirect?
4. Have you ever been working on reverse proxy configuration?
5. What is CDN, when to use it?
6. What headers are used in HTTP for resource caching? What’s mine difference between them, which one is better?
   • Expires vs Cache-Control
   • https://devcenter.heroku.com/articles/increasing-application-performance-with-http-cache-headers#http-cache-headers 
   • https://developer.mozilla.org/pl/docs/Web/HTTP/Headers/Cache-Control 
   • https://developer.mozilla.org/pl/docs/Web/HTTP/Headers/Expires 
7. Do you have any experiences with load-balancers (eg. HAProxy)?
8. Can you list few load balancing algorithms? (random, round-robin, weighted variants, sticky)
9. What “sticky session” mean?
10. Do you know what Server Side Includes or Edge Server Includes?
    • How would you build whole HTML page from multiple micro services (one page with sections from different services)?
11. Can you explain how HTTP/HTTPS request/response look like (as deep as it’s possible)?
    • DNS
    • TCP handshake (or UDP)
    • SSL - RTT-0
    • Nice explanation can be found here: https://blog.cloudflare.com/introducing-0-rtt/ 
12. What configuration would you suggest to shorten time of connection in HTTP/2? (TLS 1.3, RTT-0, session-tickets, keep-alive)