Wouldn’t it be great if everything we wanted to purchase and integrate into our technology stacks would just work the way we wanted without needing to worry about it or even test things out? Well, anyone who has been around technology knows that is not the case. There is not always a ‘one size fits all’ technology that is going to meet the need for every organization and use case. We need methods to be able to evaluate new technology products and systems before we take the plunge of purchases to ensure that the given tech is going to be a good fit for the organization. Luckily, we have multiple means available for accomplishing this ‘try before you buy’ concept when it comes to evaluating new products and services. The options we have are proofs of concept, pilot programs, and proof of value studies.
Before we dig into each of these product evaluation options, let’s start where I often like to, which is with requirements. If we do not begin with a clear picture of what it is that we are trying to accomplish, it is going to be very difficult for us to definitively determine whether the evaluation is a success or a failure. These requirements or outcomes that we are trying to achieve in these evaluation programs are called success criteria. Having defined success criteria gives us a guide to perform and analyze a proper evaluation. Now, let’s touch on the different evaluation options listed above.
Proof of Concept
In my opinion, the proof of concept (POC) is going to be the lowest lift, and minimally intrusive evaluation option for a new technology. The purpose of a POC study or implementation is to just verify that a new product or service functions as advertised for the specific use case in question. Depending on the technology being evaluated, combined with the use case in question, a proof of concept could be a relatively small effort. Most likely in a POC, we are not going to be delving too deeply into the details of the product or performing a lot of stress testing. We are really just wanting to answer the question of “Does this product, at a high level, do what I need it to do?”. For more in depth evaluations, we look to implementing a pilot program.
Pilot Programs
As introduced in the section above, pilot programs really take the POC a step further. A pilot program is going to get more people involved in evaluating a potential new solution. A subset of users would be selected to test out the new product over a period to time to perform that in depth analysis and stress testing that was mentioned to not necessarily take place in a proof of concept study. Some key points and tasks that should be part of a pilot program include:
- Defining objectives.
- Carefully select the participants. These should be individuals that would be key users, or power users of the system if it is purchased and integrated into production.
- Have a clear test plan defined to make sure the success criteria of the pilot program are met.
- Have known methods defined for communicating and delivering feedback throughout the pilot program.
- If the product ends up getting purchased, make sure to use the lessons learned from the pilot, when deploying the system into production.
Proof of Value
At face value, I think a proof of value (POV) study can be somewhat nebulus. The goal of a POV study is to come to a decision about whether or not implementing a new technology solution will add value to an individual process or the organization overall. We need a way of quantifying that value into terms that make sense such as cost or time savings. In my option, this definitely means getting the proper business units involved to make sure this value calculation is as close to reality as possible.
While proofs of concept, pilot programs, and proof of value studies take time and effort to complete, we would really be doing a disservice to the organizations we represent if we did not go through the due diligence of proper evaluations before purchasing new technology solutions. The evaluation concepts described in this article give us those different options to try before we buy.
Netication 