Measuring success for Internal Tools

Being a PM for an Internal Tool differs a lot from being a PM for consumer or enterprise tools. Here your co-workers are your users.

Benefits of being an Internal Tool PM -

• Conducting User Interviews is ridiculously easily (Since you know where your users sit in the office 😉)
• You usually understand and know the problem you are trying to solve beforehand because you yourself are also a user.
• You don’t have to spend hours and days to find the product market fit. You know what is the demand, you just need to find the best solution.

For the common consumer or enterprise based tools, we have the Pirate Metrics -

Measuring the success of internal tools as suspected, is very different than tracking the usual KPIs. But you still have to be careful about deciding the actionable metrics and don’t spend too much time on Vanity Metrics.

Here are few things that I measure when I have to define success of my internal tools.

Quantitative KPIs -

1. User acquisition - Acquisition metric is a little tricky to calculate for internal tools. There might be cases where there is no direct acquisition cost with respect to marketing. However, sometimes I measure cost in terms of time and effort. For example, hours spent on training, internal marketing through one-on-one channels, meetings, team gatherings, mails etc.

Customer Acquisition Cost/Effort (CAC/CAE) - Divide your total acquisition costs by the number of new customers

If you have organised say 10 sessions for marketing, training and education of 1 hour each. If the prep time is 1 additional hour for each session. Let’s assume you gained 10 new users per session. Then you can have rough estimation of Customer Acquisition Effort.

CAE = 20 hours/100 users = 0.2 Hours or 12 mins for every new user

You can also include any logistics cost/effort as well.

1. User Adoption -

• Number of people registered on your tool.
• Number of Business Teams registered on your tool.

• Once your tool/feature matures, here is a better metric since simply tracking the number of users coming to your platform might not give you anything actionable -

  • Adoption Rate - The percentage of new users
  • Adoption Rate = number of new users / total number of users.
• Time to Start - On an average how much time it takes for user to start using the tool (You might have to define what “Start” means for your tool, I measure the time difference between acquiring required access and the first time user logged into the tool).
• Activation Time - On an average how much time it takes a new user to complete their first task on your tool.

1. User Engagement -

• Number of users / Business Teams logging daily in your tool
• Number of hours users / Business Teams spending on your tool

In some type of tools - Number of tasks completed per hour / day / weekly Number of people using your tool daily/weekly/monthly/yearly (Define user segments based on usage, you might have highly active, active, passive, occasional users)

1. User Satisfaction -

• Net Promoter Score (NPS) - Subtract the percentage of Detractors from the percentage of Promoters. The score is a whole number that ranges from -100 to 100, and indicates customer happiness.

• Customer Satisfaction Score (CSAT) - Respondents can usually rate their satisfaction with answers in the “Very Satisfied – Very Unsatisfied” range. Alternatively, you can also let respondents rate their answers in the 1-5 or 1-10 range.

• Responsiveness/Service -
• Number of tickets raised through Platforms such as ServiceNow/Jira
• Number of User Support Queries
  • Categories of User Support Queries
  • Average resolution time
  • Average reply time
  • First contact resolution rate
  • Replies per resolution
  • Support Cost
• Performance -
  • Service Availability - Percentage of actual uptime (in hours) of service relative to the total numbers of planned uptime (in hours).
  • % of planned Outages - Percentage of outage (unavailability) due to implementation of planned changes, [Outage due to planned changes] / [Total Outage Hours]
  • % of unplanned Outages - [Outage due to unplanned changes] / [Total Outage Hours]
  • % of availability Service Level Agreements (SLAs) met
  • Number of open vulnerabilities - Infrastructure / Servers
  • % of (critical) components with automated availability monitoring
  • % of critical components not covered by a defined availability plan

1. Retention -

• Customer Churn Rate (CCR) - The percentage of customers that have been lost over a specific period of time.
• Customer Retention Rate (CRR) - The percentage of customers staying with you over time.
• Loyal Customer Rate (LCR) - Your loyal customer rate (LCR) allows you to define which customers are truly loyal to you. They will always be the first one to try out new features. They will always give you constructive feedback.
• Time Between usage surge of users
• Customer Lifetime Value (CLV) - Usually it refers to how much revenue is generated by a single customer. But based on your tool goal you can tweak its meaning. For example, there was one tool whose primary purpose was to build a model. I could say the number of models built and deployed by the user using my tool is my tool’s CLV.

1. Revenue -

For internal tools, most of the time you might not be focussing on growth in revenue, profit or cash flow. But you should definitely track you ROI(Return on Investments). In case of internal tools ROI benefits might not be very evident in monetary terms but you can measure the benefits in terms of the metrics that can be measured in your favour. Make a compelling case for your tools and set goals.

Sometimes we also take pride in calculating the below but it varies from time to time -

• Number of features delivered over a quarter/year.
• Number of user pain points and direct/indirect users feedback got incorporated
• Number of features promised vs actually delivered

Some Qualitative KPIs -

Quantitative metrics answers the what and how much type of questions but with Qualitative metrics you will know the why. Remember to talk to your users, potential or new, understand their sentiments and never skew their answers.

User Feedback -

You can ask users direct feedback through mails/direct interactions. If you have any forums such as Confluence/Internal Discovery platforms. You can look at the ratings/reviews provided by the users. Qualitative data can be captured by -

• User Surveys
• Focus Groups
• Dropping by your users desk and taking them for a coffee

Final Thoughts

The challenges faced by internal tools PM are very different than consumer facing PMs. The KPIs are different in both cases but some common patterns can always be there. You should also consider the possibility that your internal products may one day be commercialised and become customer facing. Have a clear roadmap and a vision for your internal products, find the one metric that matters, stay competitive, scale effectively, drive value for your business and bring digital transformation.