Helping you to choose the right technologies for your requirements.
This content was archived in July 2019
About this guide
With technology now underpinning every aspect of learning, teaching, research and administration, the range of technology options can be bewildering especially to the non-specialist.
This guide replaces the system selection infoKit (now archived) which was created when the options for administrative systems really boiled down to selecting and integrating a range of “best of breed” products, or choosing a monolithic Enterprise Resource Planning (ERP) type system with less specialised functionality and the range of systems supporting learning and teaching was limited.
The technology context has now changed significantly and the range of options is much broader. The underlying model on which the system selection guidance was based on remains valid. The model is about defining requirements and planning a means of evaluating different options. It is quite a generic model aimed at helping you take better decisions.
We have therefore refreshed the model to take account of a much wider set of circumstances in which it might be used. You can find out more about the range of technology options currently available in the suite of resources on improving organisational efficiency.
The core audience for this guide is those who have been charged with replacing legacy systems or infrastructure within their institution. The guidance does however apply equally well where you are selecting a tool to do something new. The model is applicable to any type of application/tool and any scale of implementation. We identify components which are key to the approach and others which are optional and generally suitable only in very large scale or costly projects.
The model was adapted by us from commercial selection models and has been used successfully by a number of institutions.
The basic model assumes that some form of formal procurement process will take place although a possible outcome of following the model is that you discover ways of using and adapting existing data, applications and infrastructure to achieve the same result more efficiently and effectively (more on this in the section entitled 'buy, borrow or mend?').
This guide assumes that the decision to select an IT application or tool is being approached as a project and that some form of formal project management framework is in place. Further guidance is available in our project management guide covering portfolio, programme and project management.
To establish that you have a viable and worthwhile project
To establish a project management framework
To develop an outline plan.
Prepare the initial business case for the project
Identify risks and issues
Appoint a project team
Develop an initial plan.
Project Initiation Document (PID)
Any systems replacement project, however small, or any decision to acquire a new technology tool should be initiated on the basis of some form of business case. The preparation of a business case helps ensure that the project is worth doing and that there are clear objectives. At the very least this business case should contain a brief description of the proposed development and key objectives of the project including:
Anticipated outcomes of the project
Relationship to the organisation’s plan or overall strategy
Anticipated opportunities and benefits
Who will be affected by the proposed development
What risks are involved in implementing the proposed development − and what planning is in place to manage such risks
What are the investment costs – how much capital is required
What are the likely running costs once the development is off the ground
What about payback – how long will it take for the organisation to recoup projected costs as savings or benefits
What are the timescales for the proposed development.
It is also worth looking at some of the questions about business requirements raised in the “defining your requirements” section of this resource. You should at least be able to answer the questions in broad terms before you start-up a project.
Large and complex projects will need to undertake considerably more scoping work once the initial business case receives approval. It is expected that there will be a formal statement which may be variously known as a scoping document, quality plan, project charter or Project Initiation Document (PID).
Smaller scale projects may combine the initial overview document and the more detailed PID but all projects require some form of documentation clearly outlining the project management framework and methodology. As a minimum this should contain: roles and responsibilities; details of the project management approach; an outline plan; risks; assumptions; and constraints.
Scoping the project
It is important when scoping a project to ensure that details are clear with minimal possibility of misinterpretation or confusion on the part of stakeholders. It should be clear from the outset what is and is not included in the scope of the project. This will help avoid potential difficulties arising from any unscoped additions on the task load, ultimately causing slippage in the progress of the project.
It will also help you manage stakeholder expectations by communicating clearly what you will and won’t deliver. A project may meet all of its objectives yet still be perceived as a failure by stakeholders if they have misunderstood the scope of the project and expect to receive benefits you are unable to deliver.
An example of this could include the scoping of a timetabling system: will the system hold courses or examinations? Or both? If the project scope includes only courses then the possibility of inclusion of examinations would be outside of the original scope.
It can often prove quite difficult to keep within scope given that additional suggestions can be closely linked to the original initiative. Changing the scope of the project frequently poses risks in terms of keeping to budget and deadlines but it is important to realise that scope change may also offer opportunities to derive previously unforeseen benefits.
Conversely you may also need to descope items from time to time in order to keep within project tolerance limits. The key to managing your project scope is to include a formal change control mechanism within your project methodology.
Change control is vital in keeping your project on track. Any change to the agreed deliverables of your project should be subject to an impact analysis. There are various means of undertaking such analyses and all recognised project methodologies have a preferred approach.
Key to any approach is to consider the impact in terms of time, cost and quality.
By this stage in the project you already have an agreed plan and deliverables and an agreed budget. By considering the proposed change under the above headings you should be able to establish whether the change is within acceptable tolerance limits or whether it has a significant impact on any of these areas. Generally a project manager may approve changes that are within tolerance but will refer to a higher authority such as a project steering board where the proposal amounts to a significant scope change.
In these instances the Board should be presented with some form of business case if it is being asked to approve a change.
To continue with the system selection example given earlier − let’s assume we had chosen to purchase a course timetabling system but excluded examination timetabling. Ideally we should have consulted all relevant stakeholders about the project scope before getting the project underway but let’s assume the registrar suddenly thinks it would be a good idea to include exam timetabling. If we make this change it will set our deadlines back because we will need to define our requirements in this area and it is likely to result in additional cost.
We must make a business case for why the change (in this case presumably an improvement in quality) is worth doing.
Projects that involve the purchase of software applications or certain types of IT tools can pose additional issues in relation to change control when formal tendering is involved. You must have a watertight scope before starting the procurement process otherwise regulations dictate that you may have to start again and re−advertise.
Changing the goalposts after you have gone out to tender not only reduces your chances of attracting the right suppliers it may also leave you open to legal action by suppliers.
A major issue for an organisation of our type is who to involve in any project. This may be glossed over in many commercial approaches on the assumption that it is generally obvious who should be allocated a particular job. Things aren’t quite the same in the education world which is why we focus here on involvement rather than simply setting up a team.
Most project methodologies will take you through identifying your key stakeholders, assessing their likely attitudes to the project and designing strategies to keep them on board. In education you ignore this at your peril. There are various approaches to involving stakeholders and you must think carefully about the best approach for your particular circumstances in order to get input from the right people at the right time.
It is important that the analysis is shared with colleagues and preferably signed off at project sponsor level to ensure that you do not get a ‘rabbit−out−of−the−hat’ stakeholder emerging unexpectedly in the middle of your project. This can derail a project.
In drawing up this sort of schedule it sometimes helps to assess the ‘potential impact on the project’ heading if you consider the type of involvement various stakeholders have on complex projects. If the project has been set in a strategic context it will follow that most members of the organisation will be seen to some extent as stakeholders exercising some sort of influence or control as follows:
Strategic - determining the strategy which this system underpins − may sponsor the project
Managerial - executes managerial control over elements of the system being implemented
Operational - is involved in operating the system or parts of it
Direct influence - is directly affected by outputs of the system but is not engaged in inputting to it
Indirect influence - is only indirectly affected by the system if at all
This is not an exhaustive list and you can create your own types to help you analyse your own organisation. However it is particularly important for you not to ignore the last two types of stakeholder. Although it could be argued that the last type is not a stakeholder at all, it is a particular characteristic of education organisations that particular interest groups have disproportionate negative power.
You need to acknowledge this and devise a management strategy for it. Typically, this often involves large−scale communication exercises just to ensure that people remain on-side. This is another reason why systems implementation in an educational environment is often so complex.
This covers organisational stakeholder analysis but you might ask “What do I do about directly involving people?” There are two basic approaches to this which can be summed up as representation versus delegation. Both have advantages and drawbacks.
Attempts to take in the full range of views, interest groups and organisational units as part of the full decision−making process. Characterised by democratic, committee−type decision−making. Covers full range of views.
An obvious route to gain widespread acceptance of decisions (?)
Involves people who may have limited knowledge of the subject area. Slows decision making Can result in compromises which don’t really represent ‘best fit’ in any particular area
Delegates responsibility to those identified as being best suited to the job. Work carried out by those with appropriate skills and knowledge permits project to move forward rapidly.
Acceptance relies on trust in those delegated – may be an alien approach in education. Needs care to ensure that all relevant issues are properly understood and covered.
As time is particularly constraining in the education world, with processes and policy moving on rapidly, our suggested model is to follow a delegation route with a small team of committed subject experts empowered to undertake work on behalf of the wider community. The empowerment aspect is crucial, as is (under either approach) a robust communication strategy, devised in accordance with your stakeholder analysis as outlined above.
A selection methodology very similar to that proposed here was used by the University of Sunderland in order to select a new student Administration system; a key difference being that it adopted a more representational approach to the evaluation team. The Sunderland case study dates to 2001 but remains relevant as it describes a generic approach to the decision-making process.
Whichever approach is most suitable to your organisation, the following key pointers apply:
To ensure a consistent approach, make sure the same people evaluate all of the solutions under consideration
Don’t assume that the ability to critically evaluate a solution can be taken for granted. The assessors may require some formal training. At the very least they should attend a briefing session where they have the opportunity to ask questions and resolve any differences of perception/approach
When planning the work allow time for training/briefing and thorough debriefing. Ensure your assessors are aware of the total time commitment required
If the project is particularly large or complex, it pays to think of your assessors as a project team. There is a vast body of literature on the development of teams. The phases of team development are commonly referred to as Forming, Storming, Norming and Performing (The Tuckman Model). If you need to bring together people from different backgrounds and experience in order to take major important decisions for the organisation, you need to allow some time for them to develop as a group. In small projects, very basic training or a detailed briefing as described above may be all that is required. In more major undertakings, time invested in developing your team will help ensure you make the right decisions
Ideally the people involved in assessing the potential solutions will be the same people who are responsible for defining the requirements in the first place. Where the constituencies are different there is again a need for briefing/explanation.
At this point in the project you should have at least an outline project plan. Even in a project where your organisation may be relaxed about the overall deadline, there are a number of factors to be considered. Your scoping exercise should allow you to answer the following questions:
Will you need to go through an EU procurement exercise and, if so, which route will you take as this affects the time required
Have you detailed all of the tasks that need to be carried out in order to define your requirements
Do you have sufficient experience to give realistic estimates of effort for these tasks
Who will be involved and what are the constraints on their availability
What resources do you need to carry out evaluations on potential solutions/suppliers (eg rooms for demonstrations etc) and what are the constraints on their availability
We consider planning a project in more detail in our project management guide. At this point it is important that you have basic outline which is understood by all those involved. Without this simple tool a project can very easily fail to make progress.
It is also a principle of sound project management/project planning that the project is broken into stages at which the business case is reviewed to ensure it is still valid. This may sound logical in theory but once a project is under way and gaining momentum, it is often difficult to stand back and look objectively at the business case.
To take the example of selecting an IT system to meet particular business requirements, you should be aware at the outset that a possible outcome of the project is that you can’t find anything that meets your requirements. This scenario may be unlikely and perhaps indicative of a flawed requirements specification but it could occur.
Perhaps more likely is that none of the solutions is quite ideal and you need to choose between best fit for different purposes, consider increasing your budget or look at changing your business processes.
It may be unpalatable to a team charged with selecting a new system to realise that the system will be far more costly than envisaged and cannot be justified in cost benefit terms but the project will have been a success in institutional terms if it reaches the right conclusion.
We will return to the subject of stage boundaries or key decision points as we progress through the model.
To ensure you end up with a system which meets the key objectives set out in the business case
Analysis and mapping of current business processes
Identify future business needs
Understand technical and other constraints.
Statement of requirements
Invitation to Tender (ITT)
Test scripts for demonstration events.
Defining what it is you are actually setting out to achieve is one of the most difficult stages in a project of this type. At the two extremes you run the risk of:
Describing exactly what you do at the moment and missing out on opportunities for change and development, or
Constructing a wish list of objectives that no system can match.
The creation of a statement of requirements is however crucial. A carefully considered and well−constructed list will be invaluable throughout subsequent stages of selection and implementation. As well as formally stating requirements arising from a process review, the statement of requirements can be used as the basis for constructing a formal tender and hence provides a benchmark for initial shortlisting of possible solutions.
It can also be used as the framework on which to base detailed evaluation criteria, for insertion into a negotiated contract as a formal document of customer requirements, and can be used further in the implementation stage as the basis for user acceptance testing (UAT).
In preparing your initiation or scoping document you should have identified in broad terms what it is you are trying to achieve and what, if any constraints are imposed upon you. Where the nature of the project is essentially reactive eg, the existing system is failing, the supplier is withdrawing support, the hardware requires replacement etc you may have constraints such as a fixed timescale or budget imposed.
Where the aim is to improve upon an existing system, there may be greater scope to explore a range of possible solutions.
In the education environment, most procurement is subject to a formal tendering process and many business system replacement projects will be of sufficient scale to be subject to EU procurement legislation. Later sections of this guide deal with the process of preparing a tender document and with EU legislation; this section is concerned with starting to define the requirements internally.
For smaller projects, the process of getting the requirements clear by means of a small team or focus group may give you all you need to move into the procurement stage. For larger projects this will be a first step in preparing a formal tender.
Your requirements will generally fall under three headings – general, technical and functional. The emphasis of our approach is on finding the best business solutions and to this end the model concentrates on functionality. We will however begin with the other areas as they may impose constraints on the rest of the project.
The first thing to consider is whether there are any general requirements which are a prerequisite for your decision. Examples of this may include requirements about the sort of company with whom your organisation would consider doing business.
Are you prepared to contract with a small supplier that offers good product functionality or do you feel there is greater security in dealing with a larger organisation? The answer to this question may depend on the scale of the development being considered and how central it is to the institution’s mission.
It is advisable to involve your Finance or Treasurer’s Department in decisions of this nature. They will normally be able to carry out an initial check on a company’s status at an early stage in the evaluation. Credit reporting can prove to be an invaluable tool at this stage.
Are you in the market for a specialist product which may have a limited user base or do you ideally want to identify a market leading product even if this means compromising in some areas of functionality?
Are you prepared to sign up to the company’s standard licensing terms, do you wish to impose your standard contract on them or do you want to negotiate a contract? Your answer to this question may depend on the size of the project and the extent to which you are undertaking any bespoke or developmental work. This does however have an impact on your procurement route especially where procurement is subject to EU regulations.
You also need to have a clear idea which of these options might be a realistic proposition for your potential suppliers. Our resource on working with commercial suppliers: establishing a contract will help you navigate this particular minefield. In particular we look at the differences between purchasing a software application and entering into a contract for the supply of cloud services.
Does your organisation have a defined IT architecture strategy and, if so, does this impose any constraints on your project? For example you may be committed to a single hardware or database platform which means you need to identify software which can run on that platform.
As of late 2011, many institutions do not yet have a defined strategy and policy for use of cloud services but it should not be assumed (as some over enthusiastic sales people would have you believe) that it is wise to enter into such arrangements without advice from your IT specialists.
In all cases it is advisable to involve those responsible for your IT infrastructure at an early stage. Even where there are no formal constraints on the technology, you need sound advice on whether the technology you are evaluating is sufficiently robust and scalable for your purpose and whether you have the in-house skills to maintain and support it or would need to buy in skills. The approach that Manchester Metropolitan University has taken to managing its technology portfolio using what it terms a ‘core plus’ model may be of interest in relation to this topic.
For a system that is intended to be used by students or staff remotely, such as a Virtual Learning Environment (VLE), it is important to consider browser and remote access requirements. For example, will your institution’s firewall allow offsite access? Anytime, anywhere access to college and university facilities is fast becoming the norm but it still pays to ask these questions – in some early VLE implementations the need for remote access was overlooked causing embarrassing issues after organisation-wide roll out of the software.
Nowadays you need also to consider remote access from a range of different end-user devices. For some types of application performance and usability via smartphone devices will be critical.
Archiving capability may also be required to meet legal and organisation requirements. Archiving and storage can be particular issues in relation to applications such as VLEs storing learning materials and e-portfolios and related applications that may store learner-generated content. Such content is becoming increasingly media-rich and the need to maintain an archive of resources rich in video and other forms of digital media demands significant storage capacity.
There are a range of external hosting options available. Cloud services may offer cost effective and convenient solutions but you need to remember that free or low cost services may not always remain so and that cloud storage may not always fulfil your requirements in relation to data protection and security for some types of information. There is a lot more information in our cloud computing guide.
Describing your functional requirements can be a very tricky issue. You need to strike the right balance between just describing what you do at the moment and coming up with ‘blue sky’ ideas that aren’t achievable.
It is an odd fact that, when considering requirements, even non-technical people are tempted to delve straight into describing the detail of how an IT system or application could/should work. Very often what they will describe is the way they do things at the moment with some minor improvements. This is not our recommended approach.
In order to give yourself the best chance of performing the function in a manner that is both efficient and effective you really need to stop thinking about an IT system and start thinking in terms of business activities or, even better, services.
Guidance on improving organisational efficiency will encourage you to think in terms of services changes the dialogue between IT specialists and other stakeholders. In particular our enterprise architecture guide looks at how enterprise architecture is being used as a strategic tool to align organisational vision and goals with the business processes and systems that support the organisation.
One away to approach your requirements definition is to think in terms of a pyramid with: strategy at the top – why you are doing something; tactics in the middle – what you are doing; and operations at the bottom – how you are doing it.
It is always worth starting by reviewing the strategy. If you can’t answer the question “why do you want to do this?” the project is already in trouble. We recommend taking a customer focused approach to answering this question. Consider “how does this help our relationship with the customer?” In most cases the customer will be the student.
If you either can’t identify the customer or can’t see how your project fits into the customer relationship maybe you need to rethink your objectives.
Next you can turn your attention to what it is that you need to do. Again you should start by looking at this in broad and simple terms. If you find yourself already embroiled in a huge amount of detail you are pitching your thoughts at the wrong level.
There are three components to any business process:
Having answered the question“why are you doing this?” you need to consider “what do you need the process to achieve?” ie, what are the outputs?
It all sounds straightforward but you may find, particularly where you are working with end users, that you ask either of these questions and the answer they give is, once again, to explain exactly what they do at the moment. They can easily confuse ‘what is‘ and ‘what should be‘.It is often useful to involve an outsider in these discussions to bring in greater objectivity.
Eventually you will get down to the detail of what data you need to collect and what kind of transformations you need to undertake to achieve the required outputs. You will probably need to undertake some form of process mapping and process analysis to aid you in setting out the requirements. We cover these topics in greater detail in the process improvement guide.
Having mapped your existing processes you may wish to apply some analytical tools to help you identify issues and problems that could be rectified when you implement a new system. It should be stressed that the sort of analysis we are talking about here is process analysis not IT system design but it can help you to identify the sorts of facilities you want to look for in a new system.
We advise leaving detailed process redesign until you have purchased your IT system and are working on implementation otherwise you could waste a lot of effort. Understanding the issues with your current processes however can only help you at this stage.
Creating a statement of requirements
To avoid the pitfalls previously outlined, we recommend that you address the following as a starting point for the statement of requirements:
Justify each item
You must be able to justify why you have included every item in the specification. To assist in this, it is helpful to state a purpose at the top of the requirements section. This helps to constantly focus on the strategic (why?) considerations that are often overlooked.
It is also worth listing definitions that will help understand the detailed requirements − for example, stating what is meant by terminology such as “course” and “unit” within the context of an academic programme.
Think “out of the box”
A process review should have already provided you with a list of requirements that encapsulate what you want to do rather than just what you do now (see our process review infoKit for further details). It is also worth stating any existing critical constraints alongside the purpose and definitions within the requirements section, again as a way of focusing on the new requirements.
Decide what is a must and then what (if any) requirements you could manage without for a brief period of time; which of these you may be able to produce yourself, agree with your prospective vendor or a third party for these to be written, or source a package that will interface readily with your other systems. This exercise can often help eliminate items if the justification isn’t there!
As well as considering requirements in the three broad categories (general, technical, functional) it can be helpful to further sub-divide up the requirements into smaller chunks – for example: data management, data input, data output, configuration and usability. This helps to focus appropriate input from the relevant constituents, balancing between business specialists, technical staff and end-users. It also promotes consistency between the separate strands of requirements – for example if your statement of requirements includes a section for admissions as well as enrolment you are likely to want data output in the same formats for both.
Defining the requirements for the support of learning, teaching and assessment can be carried out in much the same way as the requirements for administrative functions ie by defining what it is you want to do and why before investigating the tools that are available. Once you have this information you may find that comparative evaluations of different tools are more readily available for these types of application.
We have produced this brief checklist to assist in the compilation of a statement of requirements.
You might want to consider what type of company you want to engage with. Are you looking for a specific relationship, eg a development partnership? What information do you want to consider about the vendor – financial stability, market share, future plans? Any specifications you wish to make can be elaborated on within the Invitation to Tender (ITT) itself.
Access and security
What security model(s) do you need? A ‘horizontal’ model (access based on system functionality), a ‘vertical’ model (access based on data values) or both? Does it vary across different business components (eg, do student marks’ recording requirements differ from student application records?). Do you need access/sign-on to converge with other existing protocols, eg, are you looking for single sign-on to all organisation applications?
Consider the typical level of ‘set-up’ data volatility (how much it changes) and if this requires regular reconfiguration (eg changing organisational structures) think how you might draft this in as a requirement in terms of flexibility and streamlining of reconfiguration
What capabilities do you need to extract, transform and load data from other systems?
Does this include storage of items other than data, eg, scanned documents?
What are your data management requirements in terms of auditing, tracking, validation and error-trapping?
In relation to the point above, consider specific requirements for archiving and destruction of data (this will vary between business areas).
What interfacing capabilities do you need to other products?
Do you need to consider packages such as messaging/email facilities as well as other business and learning systems?
How do the interfaces needs to operate – do they need to pull data from other systems or push data out?
Do they need to be real time
Do you require workflow capabilities within the product or other automation requirements such as diary/scheduling or automatic triggering of reports?
Do you have any specific requirements for the user interface, navigation, report layouts, help and search facilities?
Do you require a web interface?
Are you considering self-service or portal functionality?
Consider the user base and training implications – does this lead to specific requirements such as drop-down lists containing intuitive descriptions rather than specialist codes
Do you need facilities for “fast track” data entry?
Consider the skillsets you have available to you to support the hardware and software. What are the limitations in terms of resources and training? Coupled with your IT architecture/strategy/policy, does this lead you to any specific requirements? What are your likely support requirements?
Do you have specific customisation/localisation requirements, and if so take care to investigate whether the methods proposed by the vendor are supported (ie are upgrade-proof) and are not localisations that require consultancy and recoding every time you upgrade
Do you have existing or desired service standards to which the product must comply? What are your requirements for backup and disaster recovery facilities?
Consider transaction and record volumes, peak processing times, and state scalability requirements
Are there resources or scheduling constraints whereby you need to state limits on frequencies and scale of upgrades and desired length of support for previous versions
Consider the number of environments you are likely to need (eg training, development, production and testing areas) and the resource implications of these
Are there any technical standards you require for interaction/interfacing/interoperability with other products
Do you want to specify any constraints about use of third party suppliers (subcontracting)?
Are there any stipulations you wish to make about licensing structures?
Are there limiting factors on running software relating to the kit available to users, eg specification of PCs?
These detailed requirements should fall naturally out of your process review, but it is worth paying particular attention to:
Any specialist areas of business – eg, specific data requirements or reporting for professional bodies within a subject specialism or legitimately differing processes for a satellite campus
New requirements arising from aspirations of greater interoperability between systems
Do you require all functionality outlined to be in one core product or set of products from one supplier? If so state this up front – there could be third-party “bolt-ons” hidden within the proposed solution with additional implications for licensing and ongoing maintenance
Remember to state expectations of coverage of statutory obligations eg, for agencies such as UCAS or HESA.
Experiences of defining requirements
Alongside the general guidance it is often useful to see how others have gone about doing this, so here are a few examples of requirements definitions.
In order to deliver the project the two partner universities had to find a way of defining their requirements for postgraduate research administration in a generic way. The project has produced a range of outputs that will be useful to others looking at this specific area of administration or looking for guidance on requirements definition:
The Student Timetabling Online Project (STOP) at the University of Central Lancashire used its requirements definition activities to develop a new university policy as well as informing the selection of a new IT system.
As-Is and To-Be processes show how the University described its current processes, identified issues with them and put forward a proposal to address the issues
Having determined what your requirements are, it pays to take a broad view of the possible means of meeting them. Gone are the days when your only options were the purchase of heavyweight, monolithic applications that were costly and cumbersome to customise or the purchase of a range of “best of breed” applications where you faced a considerable overhead in getting them to talk to one another.
These days you may be able to fulfil your requirements in a range of ways other than purchasing applications.
We have used the term “borrow” as an option. What we mean by this is looking at ways to achieve your objectives without necessarily having to purchase applications or infrastructure and/or by using the expertise of others so that you do not have to do the work yourself. You might think about:
Using cloud services
Sharing services with other organisations
Using open source products.
So-called cloud services are becoming an increasingly popular way of delivering infrastructure, administrative applications and services for learning, teaching and research. Part of the appeal is a different type of licensing model, referred to as “pay-per-use”, that can avoid the need for upfront capital investment.
It is possible to have services that are fully cloud hosted or to adopt a hybrid model where the use of cloud services at particular times can help deal with peaks in storage requirements. We discuss the use of cloud services in our cloud computing guide.
In some cases you may be able to deliver better quality or services that cannot be sustained through in-house delivery by sharing services with other organisations. There is a long history of successful shared services within the education sector and developments in technology are making it easier to operate across institutional boundaries. Find out more about this topic in the shared services guide.
One common form of sharing across different organisations is in contributing to the development of open and community source software. The open source advisory service OSS Watch provides more extensive guidance and resources on these topics.
Finally, rather than buying or borrowing a solution, we suggest you look at whether it is possible to adapt or “mend” resources already available to you in order to meet your requirements. Our resources on getting more from existing investments, or optimisation, cover a range of technical approaches that can facilitate this type of re-use. Key to this is the concept of Service Oriented Approaches (SOA): by applying certain technical standards, data from existing systems can be exported to a ‘service layer’ that can be used by a range of other applications.
In some cases using this type of approach to enable you to get a little bit more out of an existing system may allow you to avoid having to purchase a new application to carry out additional functions.
Prepare an invitation to tender
Assuming you have explored all of the options and do need to look externally for a solution then in most cases, apart from fairly small-scale pilots, your acquisition of new technologies will be subject to formal procurement procedures.
This means that the final stage of defining what you need is to set out your requirements in an Invitation to Tender (ITT). This document requires careful preparation if you are not to waste your own and suppliers’ time on unsuitable tenders. It should give suppliers a reasonable amount of background information about your organisation and your project as well as setting out the detailed requirements specifically and clearly.
One thing that you need to think through very carefully upfront is how you will evaluate the tenders you receive. You may choose simply to accept the lowest priced tender that meets your requirements but it is more usual to define a set of criteria by which you will identify the most economically advantageous tender (a phrase often abbreviated to MEAT).
You will need to state the criteria to be used and any weightings to be applied in the ITT. If you fail to do this then, under EU regulations, you will be obliged to accept the lowest priced tender.
It may seem very early to be doing this but you really need to develop your marking system and scoring template at the same time as the ITT. You should be absolutely clear about the mapping between your stated requirements and the evaluation criteria and it would be good practice to show this in an appendix to the ITT.
It pays to discuss the criteria and weightings with a range of key stakeholders and do some modelling just to ensure there are no hostages to fortune in your marking scheme that may skew the scores in ways that would be undesirable. You should also think about who will constitute your marking panel and ensure they are fully briefed at an early stage.
If you are making a major investment and are planning some kind of evaluation or demonstration event then you need to make absolutely sure that your procurement procedure allows for this, and that your ITT is clear about the basis for shortlisting and how shortlisted suppliers will be evaluated in the subsequent phase.
Details of interested suppliers and their offerings
This section assumes that one of the outputs of your earlier work on defining your requirements was an Invitation to Tender (ITT). Refer back to the 'defining requirements' section of this guide for ITT templates and examples.
Within the education environment it is likely that any IT systems procurement will be subject to some form of tendering exercise. This may be required in order to comply with your own financial regulations or, in the case of major purchases, to comply with EU regulations. It is all too easy to see the tendering process as a bureaucratic nightmare when, properly used, it can be of great help in the decision making process.
You are advised to seek early assistance from your own procurement officer and/or legal advisers to ensure that you select the most appropriate procurement route.
Should you be subject to the EU process there is a choice of open, negotiated and restricted procedures. Correct choice of procedure is important in determining what type of evaluation process you can undertake eg, whether you are legally able to shortlist or whether all suppliers who respond must be subject to the same evaluation.
It is in your own interest to undertake some form of market analysis at an early stage of project initiation in order to have an idea how many potential suppliers may be able to meet your needs. Project deadlines may be thrown substantially out if you are obliged to evaluate far more suppliers than you had planned for.
Our cloud service promotes the uptake of off-campus data centres and cloud services. This service formed the centre piece of a project by HEFCE’s University Modernisation Fund (UMF) to create efficiencies, make savings and improve services for the HE and FE sectors.
The intention was that, in practical terms, a core virtual server infrastructure would be deployed in data centres that offer discounted data management and storage services through shared virtual servers and data centre capacity to institutions.
Whichever procurement route you use it is important to ensure you understand the technicalities of the process. Failure to observe correct procedure can result in a legal challenge from a supplier which can cause delay to your project and ultimate financial loss.
In many cases there will be a “waiting” period between placing the advert and issuing tender documentation during which suppliers will prepare their responses. Don’t worry – your team will have plenty of work to do preparing for the evaluation stage of the project – read on.
To allow you to set the agenda for system demonstrations
To ensure you see how the system would operate your business scenarios
To empower you as an active tester not a passive viewer
To ensure fairness and objectivity in decision making
To avoid evaluators being diverted by sales talk.
Script test scenarios based on real life business scenarios
Prepare test data
Organise the evaluation events
Brief suppliers and evaluators.
Agenda for a set of tested scripts
Supplier briefing pack
Evaluators’ scores and product summaries.
Depending on the procurement process you have chosen you may either shortlist suppliers on the basis of their tender responses or move straight to a full evaluation of interested suppliers.
Where compilation of a shortlist is part of the process you are likely to look at:
The company’s financial standing
The company’s market position
What proportion of your requirements is the company able to meet? This is one of the most difficult areas to evaluate from a tender response alone. Company responses are understandably positive in their approach and will tend to gloss over any shortfalls against your requirements. Don’t worry – the next stage of the evaluation will help you differentiate the optimistic hopefuls from the real contenders
You should take care that any shortlisting decisions are fully documented and can be justified against your selection criteria should the decision be challenged.
In some cases you may be able to come to a selection decision on the basis of the responses to the Invitation to Tender (ITT) but, in the case of significant investment decisions, you are likely to wish to conduct some kind of evaluation event.
A formal evaluation event is an effective way to test the extent to which potential suppliers can meet your requirements. In the case of replacing a mission critical system the supplier evaluation is one of the most critical elements of the selection process. It is based on the premise that you as the customer should be able to see hard evidence of what the system can do and, in particular, what it can do for your business.
This means that, rather than simply attending a sales demonstration, you set the agenda based on your key business issues. All suppliers will be required to follow the same demonstration agenda making direct comparison far easier than would otherwise be the case.
Setting the agenda
The agenda will depend on what type of system you are buying and what the scale of the implementation is likely to be. This model was originally developed by the University of Northumbria during selection of a new integrated student and HR system. The scale of that project meant that the evaluation of each supplier took place over three days and had up to three streams running in parallel.
A simpler system will require a more straightforward evaluation event. The key thing is to ensure that your agenda covers your main business issues.
You may choose to look at: your most critical processes eg, in looking for an HR/payroll system you may decide that payroll and recruitment are critical to your organisation but training management and time and attendance are not hence you would focus the evaluation on the first two.
Your highest volume processes eg, in a student system might be admissions and enrolment.
Processes which give rise to particular problems in your organisation – this may stem from weaknesses in your current systems or the unique nature of your requirements.
Criteria which support your strategic objectives and/or which help differentiate between suppliers. For instance if your institution wishes to offer self-service functionality to students and only one potential supplier can meet this requirement this may distinguish between otherwise similar offerings.
Preparing test scripts
Having decided which are your key functional areas you can then start to devise test scripts for the evaluation. The script should follow the process through from start to finish and can be constructed using your statement of requirements as a starting point.
Again it is important to consider what you are trying to do rather than how you want to do it.
It is likely that the supplier will need to do a considerable amount of preparation and set up work in advance and you need to think carefully about what information they need to have in order to do this. For example they may require detailed information about a range of course structures in order to demonstrate course set up or you may wish to provide them with some example student data in order to see how this is mapped into their system.
The zip file includes an example of the sort of information on course structures.
The following example scripts cover general issues which may be of relevance to a variety of main student administrative processes:
Assessment and progression
Delivery of teaching and learning
The following example scripts cover general issues which may be of relevance to a variety of systems:
The following example scripts cover technical issues which may be of relevance to a variety of systems:
The technical example of scripts predate the advent of cloud computing. If you are considering a cloud option then you may need to consider all of these issues and more depending on the type of cloud deployment model you choose. In a hybrid model you may need to consider both on-campus and cloud infrastructure.
Your scripts should have an allotted time. It is often difficult to gauge the time required for each area when you are dealing with unknown systems. You will be aware that some tasks are bound to take longer than others eg setting up a new course is likely to take longer than changing a student’s address. As a general rule of thumb your script needs to allow at least a couple of minutes for straightforward tasks and should allow some time for questions.
Having prepared the agenda and scripts there is still a lot of work to do in actually organising the event. The scripts must be organised into a timetable. This should allow time for assessors to sum up their thoughts after each session and ideally to have a follow up session at the start or end of each day to pick up on issues from previous sessions.
If you are intending to run parallel sessions, the supplier needs to know this in good time to arrange for people to cover each session. Don’t underestimate how tiring the sessions will be for the assessors – the job demands a great deal of concentration and regular breaks will be required to maintain attention.
It is a good idea to have briefings with both assessors and suppliers before the event to set out the ground rules for the evaluation. Some of the points you may wish to cover include:
You need to provide the suppliers with some form of evaluation event pack that they can consider before attending the briefing. This should contain:
The test scripts
Copies of any test data they need to set up
Any relevant background information about the project.
This is the last chance for the supplier to ensure they understand your requirements before they demonstrate their product to you. A good supplier will have thought about the contents of the pack and come to the briefing prepared with a list of questions. Frequently they will take the opportunity to negotiate about the timetable ie they want to spend more time on this and less on that.
Our advice is “stay in the driving seat” – you have already thought about what is important to you and what you want to see. You may agree that some of the changes are reasonable but take care to ensure you are being fair to all of the suppliers involved.
In a tightly managed evaluation timing is of critical importance and it is in everyone’s interest to ensure that sessions do not overrun. You will not be in a position to make a fair comparison between two suppliers if one of them spent half an hour on a topic while the other demonstrated for two hours.
You need to beware of suppliers trying to emphasise the best features of their product at length (these may not necessarily be the most important features to you) then skate over weaknesses due to “lack of time“.
Similarly, you need to manage the input of your own evaluation team. If your scripts are well thought out and prepared (and the supplier is well prepared) the demonstration should give the team all the answers they need and there should be little need for ad hoc questioning. This isn’t always the case and you need to watch out for the risk of sessions being “hijacked” by someone with a particular interest in one area.
Thorough briefing of suppliers and evaluators will help but it is worth appointing someone to facilitate each session or at the very least to keep an eye on the time and ensure you are getting through the demonstration at the expected rate. Ideally this role should be undertaken by someone who isn’t scoring the session so they can give their full attention to the job. This person can also be helpful in picking up issues where suppliers do not stick rigidly to the order of the script and need to come back to it later.
The facilitator/timekeeper should have sufficient confidence to check with the team that a point has been adequately covered and move the supplier along or request further explanation where necessary.
Where your team has a lot of areas to evaluate you may wish to consider making a recording of the sessions. This can be helpful if you simply can’t remember the answer to a particular question or if there are differing interpretations of what was said. It can also help to tone down some of the more exuberant and optimistic sales promises if the supplier knows you have a full record of the discussion.
It is worth also appointing a team leader who is more of a functional expert to oversee the progress of the actual evaluation. This person will need to collect score sheets at the end of each session and do a rough check that there are no missing or anomalous results. Issues to look out for are where a number of people have failed to evaluate a point due to insufficient information or where the scores of individual team members differ greatly.
This can highlight areas which should be followed up in the recap/follow-up sessions.
The team leader may also facilitate the final summing up of the team scores. There are bound to be some genuine and valid differences of opinion about the different products and rather than take a purely quantitative approach and average out scores it is worth exploring the reasons for these differences. A simple average can give a compromise solution that isn’t a best fit in any area.
This facilitation role may equally well be carried out by an objective outsider provided they have facilitation skills and a reasonable level of subject knowledge. In practice most projects are unable to draw on an unlimited number of people and find it easier to use a team member.
You need to establish what scoring mechanism you will use for the evaluation to ensure consistency between evaluators. This may be a simple numeric score eg marks out of ten for each area but, as mentioned above, you need to consider the risk that a purely quantitative approach may smooth over some important issues.
Here is an example of a qualitative scoring system. This example works on the basis that a requirement is either “Met” or “Not Met”. There will however inevitably be grey areas which fall into the category “Partly Met”. It may be that a supplier tells you that this functionality is being developed and will be in a future release of the product or it may be that by changing your processes the system could achieve the desired output. In any major systems purchase there are likely to be many of these grey areas and you have to decide how important the gaps are and how you will compensate for them. It is these areas which may make or break your implementation project and you need to have a clear view of them before you can develop an effective implementation plan or set a realistic budget.
A suggested method of scoring is for each individual to complete an individual score sheet which is then input to a spreadsheet so that scores can be compared. Where all of the team members agree the score this can be taken as it stands. Where there are differences these need to be discussed and a final score agreed.
NB Where a large number of points are being scored it is possible to speed things up by automating the comparison process. This can be achieved in most spreadsheet packages by use of a simple macro.
Individual team members in the appropriate discipline areas covered by a system selection project complete test script score sheets at the actual supplier demonstration.
The scores and comments of individual team member score sheets are then entered into a summary spreadsheet.
The scores for individual suppliers are entered on to separate datasheets within the template – so supplier one appears on datasheet one, supplier two on datasheet two, etc. Team members are given numbers one, two, three, etc but these would usually be replaced with the members’ respective initials.
The datasheets record scores against each individual task set for the test script process – using a pre-determined scoring definition. A space should be left where no score has been awarded.
The information entered into the datasheets can then be seen in the specific discipline areas elsewhere in the template eg see worksheets for accommodation, general requirements, etc. Data should not be entered directly into these areas as they are formatted to reflect the information in the datasheets.
The information entered is also automatically used to produce graphical representations of the results in order to illustrate the strengths and weaknesses of individual suppliers when compared with other suppliers in the selection process. The graphs can be seen in the worksheets entitled “handout graphs” and “graphs” on the template.
Finally we come to the evaluation event itself. A lot of preparation work has gone into it. If you have pitched your scripts correctly and you have a good supplier demonstrating a good product, things should run very smoothly. Things become more difficult where the supplier hasn’t prepared well, hasn’t understood your requirements or is trying to sell you something that isn’t a good match to your requirements.
This is when your assessment starts to move into the grey areas. There are some key signs and phrases to watch out for:
I’ll need to switch to a different version to show you that.
Always ensure you know what version of the product they are demonstrating to you and what version they intend to sell you. As a cautionary tale it has been known for salesmen to demonstrate a product version that existed only on the salesman’s laptop. One particular salesman had undertaken a number of non-supported modifications in order to show that the product could meet the customer’s requirements.
That will be in the next version.
Don’t take their word for it – ask to see the evidence. If they are clear enough about the next release to be able to say definitively what is in it then they will already have detailed design documentation and confirmed release dates that they can show you. “That functionality is currently under development; we don’t yet have a release date for it” means nothing – treat it as vapourware.
You can configure it to do that.
You need to be clear about what is standard end-user configuration of the system, what requires a technical specialist and what is bespoke development. Don’t go in for bespoke development unless you absolutely have to. It will cost you more, it won’t be part of the standard support agreement and it will cause you ongoing maintenance headaches (see below).
It’s totally upgrade proof.
Be especially cautious of this one if it follows the one about configuration. As a general rule bespoke work is never upgrade proof. Be wary about changes to screens or forms. End user acceptance of screen layout and terminology is a major issue for most organisations. Many suppliers will tell you how easy it is to rename something that doesn’t suit your own terminology or change the layout of a screen but you could be creating a huge maintenance overhead for yourself.
This is an optional feature.
This one should ring a big warning bell. Firstly is it part of their product at all- many suppliers will demonstrate third party tools which bolt on to their product without making this clear to you. Third party tools mean additional licence costs, possible differences in the technology stack and possible contractual and implementation complications if the supplier doesn’t want to act as a one-stop-shop in providing the solution to you.
If the optional feature is part of their own suite make sure they are quoting you the full cost of everything they are demonstrating.
I’ll show you what X customer has done.
Integration with web technology allows many of these systems to offer great benefits but it also affords quite a few opportunities to pull the wool over unwary eyes. If they show you someone else’s flash website or portal be sure to establish what you are seeing of the core product and what you are seeing of another customer’s development efforts.
Yes – it does that.
This warning may sound overly cynical but if the phrase isn’t accompanied by a demonstration it isn’t worth much.
You will probably include some form of contact with reference sites as part of your evaluation. This may take the form of a questionnaire, a telephone call or a site visit. The value of such references can vary greatly and there are a number of factors you need to consider.
How similar are the reference site’s business processes to your own?
You will probably be looking for contacts within the sector (our willingness to share information is indeed one of our great strengths) but the difference between highly centralised and very devolved processing can be considerable.
What parts of the system is the reference site using?
It is worth checking this out with the reference site beforehand. We are aware of one example where a potential customer was sent by a supplier to see a “flagship implementation” of their new functionality only to find out the user had given up on the supplier’s product and was using an in-house development.
What did the reference site use beforehand?
There can be great differences in user expectations and satisfaction between sites that replace very old or non-automated systems and sites that try to keep at the leading edge of technological developments.
It is also important that you make the right contacts within the reference organisation. You won’t get an end-user perspective on admissions processing from a chief executive nor will you get a view on the company as a business partner from an admissions administrator. Make sure your questions are specific and targeted.
To ensure you have a clear idea of:
Costs and benefits of proposed solutions
Implementation timescales and resources requirements.
Decide on preferred supplier
Summarise product “fit” against your requirements and decide how you will handle any gaps
Negotiate with supplier.
Outline implementation plan and milestones
Agree payment terms.
Decide on preferred supplier
As a result of your evaluation exercise you may have a clear winner or you may have to come to a difficult decision between two or more products that match your requirements in different ways. If you have followed this methodology through you should be clear about your requirements and their relative priorities but it can still be a complex matter to weigh up the potential costs and benefits of different systems.
In the complex area of student administration it is often the case that no system will meet your needs fully and you must adopt a “best fit” approach. In this situation you should undertake a comprehensive analysis of the gaps in the product and decide how you intend to fill them. This is essential if you are to meet your user requirements, to produce an accurate costing for delivering your project and to produce realistic project timescales.
Some potential solutions to gaps, and key issues to consider, are noted below.
This should be avoided if at all possible. If you decide to undertake bespoke work make sure you are aware of the implications for design and development costs, support, maintenance and future product upgrades.
It may be that you can change your business processes to work in the same way as the system. If your project includes process change remember to allow additional resources for analysis, testing, communication, training and support.
You may decide to interface to a different system to carry out some functions. As systems become increasingly open integration becomes less of a technical issue and more about how you manage the information.
Technical considerations include the cost of writing, testing, maintaining and supporting interfaces. Business considerations include how you structure data within the different systems so that it can be integrated where necessary eg, for reporting purposes.
Negotiate with supplier
Having established which product best fits your needs you are in a position to negotiate with your preferred supplier. The extent to which you are free to negotiate on terms and conditions is determined to some extent by the procurement route you choose and what you specified in your Invitation to Tender (ITT).
One of the aims of this methodology is to put you in a good bargaining position at the end of the evaluation so you are advised to choose a route that allows you to negotiate after you have gained a thorough understanding of the product.
Contract negotiation is a minefield and your project plan should allow plenty of time for to-ing and fro-ing between your lawyers and the suppliers. Our resources on working with commercial suppliers cover most of the common issues with regard to traditional purchasing of software and infrastructure and highlights the differences that are emerging in the cloud environment, but you are nonetheless advised to seek legal advice at an early stage.
One point to bear in mind is that all of the large software companies will tell you that their standard contract terms are non-negotiable. The multi-nationals will say this very convincingly but all of them are able to exercise a greater or lesser degree of flexibility when it comes to winning your business in the end.
Two ways in which you can get a head start in preparing for this stage are to:
Include a sample set of your terms and condition with your ITT
Include a session on contract terms as part of your evaluation event – this will allow you to sum up the supplier’s different approaches to contracts as a factor in your final decision.
It is tempting after a possibly lengthy and wearing selection process to want to get this stage out of the way as quickly as possible and get on with implementation. Everyone believes at this point that things will go well and the signed contract will go into a drawer never to be looked at again.
In the real world this is all too often not the case. Cutting corners at this stage could be your biggest mistake. A well-drawn up contract should protect you (and the supplier) against unforeseen circumstances that throw your project off-track.
Sign the contract
It is assumed here that you are contracting with a single supplier for the purchase and implementation of your system. Our resources on working with commercial suppliers cover the additional complexities of third party involvement. We recommend that the contract should include the following schedules:
A list of the customer requirements that the supplier is undertaking to meet
This may be taken from your ITT but it is likely to require some amendment if there are areas where the supplier is unable to meet the requirements as originally stated. It is particularly important to include the specific agreements about areas where the supplier has promised to meet a requirement with a future release of the product.
An outline plan that highlights the key implementation milestones
The plan is bound to be subject to change but you should start with an agreed baseline and adopt formal change control procedures to handle deviation from the plan.
A payment schedule that references the milestones
An application or other types of tool is of no use to you unless it can be delivered as a working solution in your environment. Where you are partnering with a supplier on implementation, it is reasonable to pay against delivery of agreed objectives rather than simply pay up front.
This is particularly important in relation to implementation consultancy which can eat up a large proportion of any implementation budget. Staged payments give you some leverage in the event that the consultancy does not deliver the expected results or overruns the agreed budget.