Software Engineering Due Diligence for Non-Engineers

I wrote up this list after I got a referral from a PE firm that bought a lemon. They had hired an auditor to evaluate the company prior to purchase, but the auditor evaluated the company as though it had strictly third-party technology. Is MFA required to log into email, are there database backups, etc. All sufficient questions for evaluating IT health if you're buying a regional beverage distributor or boutique hotel, where the first-party software edge is minimal, but if the business runs on first-party software, you need to ask a whole other set of questions, which MBAs and company-buyers usually don't know.

The order isn't important; I wrote these down off the top of my head in response to an email.

How to evaluate responses to these questions

Ideally, you're asking these questions face-to-face with the senior technology leadership. If they push back, don't understand, or insist the answers don't matter ("We've never had an outage!"), you're going to need to replace them.

The questions seeking specific numbers are probably not going to be in their heads. If they can answer exactly on the spot, you need to hold on to them.

For the other questions, they should be able to deliver broad answers ("Oh, we usually refresh the test environment from a production backup, but I'd have to check on the date…") on the spot. For any follow-ups, the amount of time it takes to get an answer is one measure of the quality of the organization. Your favorite LLM can help evaluate the answers, but beware answers that look like they were fabricated out of whole cloth by a robot.

Systems

For each system maintained and/or written by the company, please answer the following:

What is the average time elapsed between merge and deploy per change?
What is the average frequency of deployment per week?
What is the mean time to recovery for an incident?
What is the percentage of changes which fail in production?
Is all business logic (including stored procedures) under version control? Which version control system is in use?
How are schema changes managed? When was the last change, when was the last one which was rolled back, and why?
How is it deployed? Please distinguish between automated and manual steps.
Can an employee deploy changes to production without review by another employee? How? Are there special permissions involved, and if so, how many people at the company have them?
How often are disaster recovery drills performed?
What percentage of the code is covered by automated tests? How is it measured?
Describe the process for manual QA of changes, if any.
Are all of the steps necessary for a new employee to work with the system written and error-free?
If the system handles PII or secrets (e.g. passwords, keys, SSNs, credit card numbers), describe how that information is handled in transit and at rest, as well as the audit trail for access to that information.
Detail single points of failure in the system.
Describe the data backup process.
When was the last time a backup restoration was tested?
Describe the system's architecture.
How is the system's infrastructure managed?
How long ago was the last change to the system's architecture, and how do you know?
Detail all distinct environments for the system.
Describe monitoring systems in place, including logging infrastructure, log searchability, retention time, data scrubbing, and alerting.
Is there a written disaster recovery policy, and when was it last updated?
When was the last test of a simulated system outage conducted?
What was the longest outage for the system, what was the cause, and how was it resolved?
How is internal user access managed?
Do changes to user permissions have an auditable log?
How many current full-time employees are considered particularly knowledgeable about this system?
How is the performance of the system monitored, and what SLAs does it offer?
Who are the top three contributors to the system in the past year, and are they still active employees?

Engineering and Product

Describe the organization of the engineering and product organizations.
How many systems had distinct accounts which had to be deactivated for the last director- or higher-level member of the engineering team to depart the company?
Has the number of engineers increased or decreased over the last year?
Please provide copies of written processes (e.g. hiring, promotion, product development, software engineering, etc).