10 Tips to Effectively Plan Performance Testing for a New Website

It’s rare for any major application or website to go live without going through performance testing. It’s an extremely important activity that requires effective planning and execution.  Based on our experience in implementing web applications including websites for global clients, we believe a well-planned approach to testing new web applications goes a long way in ensuring enhanced user experience. Below are 10 tips for planning and preparing for performance testing of a new website or a web application.

  1. Focus on performance – Performance should be integrated into the design from the very start of the new web application project rather than an afterthought. Performance testing should not be considered a one-time activity.
  2. Identify key pages to test – It’s not always possible to test every page, so it’s important to select the pages that are expected to have the highest hit rates. The IT or marketing departments usually have the web analytics stats for the site and will be able to provide information on the sections of the site that receive the most traffic.
  3. Know your target performance metrics – Well before embarking on any performance testing, you will need to identify realistic performance metrics. These may include page load times, number of concurrent users, etc. Typically you should aim to get these performance metrics before the solution is designed. This will ensure the architecture is able to cope with the expected loads and performance targets.
  4. Measure base line performance – Plan to measure baseline performance figures for the old site (using the old CMS if it’s a migration) and the new CMS without any customisations. The reason for this is that you will typically want to prove that the ‘new’ site offers the same performance levels as the old one. While migrating to a new CMS platform, it’s important to establish performance levels of the new CMS, without any of your customisations, so as to ensure that the customisations have not degraded performance of the site.
  5. Recreate the production environment – Plan to execute the tests on either the production hardware or a replica of production. If performance testing can’t be executed on production or hardware that replicates production, then the performance figures will need to be adjusted or extrapolated for production. It’s important to at least recreate any clustering and other configuration from the production environment to make the testing as realistic as possible. It’s important to ensure that the hardware used for testing doesn’t have any other applications running when the testing is due to be executed; this will ensure ‘clean’ performance figures.
  6. Avoid network congestion through off-peak testing – Ensure testing is planned for a period when there is a minimal load on the network such as during the weekend or sometime in the night. This is to avoid loading the network with testing traffic when the rest of the business is using the network for normal ‘business as usual’ activities.
  7. Team Management – Plan on having the right people available during the tests – e.g. DBA, Server Administrator, Technical Architect and, if possible, infrastructure architect to monitor network hardware such as load balancers.
  8. Application Configuration – Configure the application and servers for production – i.e. switch on caching, set an appropriate logging level, etc. Don’t forget to disable DEBUG level logging unless you want the additional detail, as excessive logging may slow performance of the application. Normal production level logging is usually set to INFO, so it’s logical to use the same for the performance test.
  9. Test the final production release – Ensure you’re running the production release software during the testing. It may be tempting to execute testing against an early release of the software, but unless this is the first of many performance tests, it is essential that the final production release is used. There is always a risk that the performance profile of the software will change from release to release.
  10. Use monitoring utilities – Switch on application and monitoring utilities to record key information such as CPU and memory load, database query times, etc. It’s important to understand what’s going on at the web server, application server, database and other key integration servers. For database centric applications and sites (and most are) effective database monitoring is essential. For example, an application executing around 100 queries for every web request, can place a huge strain on the servers and reduce the number of concurrent requests that the application can handle. It’s an extreme example, and one that I have encountered in the past, but certainly proves that it is important to keep an eye on the number of queries being executed for a given web request.

Performance testing of web applications offers some interesting insights on why an application can fail. I am sure you may have a different experience or issues to talk about, which we would be very interested to hear about. Please leave your comments or other tips you may want to share in the below comments box.

4 Comments

  • Scott Price September 15, 2011

    Excellent article Parvaze. You present some key thoughts that should be included in planning the performance testing process.

    While I understand and agree with your intention on #5, my experience in web dev has shown me that a “replica” never is truly the same as production. There are too many tiny variables to get exactly right. It’s hard to get hundreds of hardware and software configurations to be the same.

    Your point is still correct. My suggestion is to always run a performance test after you release code to production. That is the only way you know for sure. I’ve made the mistake of “extrapolation” a few times…it’s a pain when the expectations fall apart and you get a call from the CEO! 😉

    Thanks,
    Scott

    • Parvaze Suleman September 16, 2011

      Scott, thanks for your response. I do agree with you – performance testing should always be done on the production kit whenever possible. You’re right replica environments don’t always replicate the production environments 100% – I too have been caught out in the past. What are your thoughts on scenarios when the production environment is running a live site and it’s not possible to do any performance runs or stakeholders are reluctant to allow performance testing to take place on live? In the past, I have occasionally seen offline Disaster Recovery (DR) environments used for post-go-live performance testing.

  • Brad Johnson September 23, 2011

    Parvaze,

    Very good article! I would add that the pace of delivery and potentially huge scale of the modern web make all of this difficult, but critical. A few comments:

    To your opening sentence, we’ve actually found in our work with thousands of sites across every market segment that an alarming number of sites actually DO go live without ANY load or performance testing taking place – up to 75%. This MUST change!

    on 3) The challenge here is that clients often really don’t know what their high level or peak load expectations are. Considering the scale of many sites, and the multiplier of mobile access, it’s unclear what “success” will bring. It’s best to work upwards from the baseline (point 4) to understand the upper limits. We suggest establishing the baselines, then iterating higher scale tests as you tune the whole system until, 1) You reach limits that everyone agrees are acceptable, or 2) You do hit a pre-established limit with acceptable performance. From there, test even more, so you really do know where the “top” (breaking point) is!

    on 5) Adjusting and extrapolating performance numbers are too often way off or wrong and therefore invalid. Lab testing for scalability loses much credibility with dev and ops team if the numbers are not “real”. This alone helps the case to test in production environments and full scale, either (or both) in maintenance windows and live.

    on 6) When you test in prod, you will have NO PROBLEM getting everyone involved during the tests! It makes the sessions incredibly fun and dynamic, too! Many Issues get fixed during the session…not a day or 3 later.

    on 9) Of course, and running in the prod env to collect comparative reports of pre and post prod performance.

    on 10) This is the key. If the approach or tool does not enable you to have clear and correlated views of all the collected perf metrics, so that all the constituents can react to and implement changes — or completely control the test as it runs (i.e. stop, pause, adjust load, adjust scenarios)…your risks in production testing are higher.

    In response to your comment back to Scott, again, it’s the visibility and control you have of the test platform running the test as well as the speed and completeness of the views of all the monitored data that assure you will be able to react to any impact the test has – whether infra, code or database – on real users that would risk their experience. We have been pleasantly surprised at how common it is to see large (albeit sophisticated, and often cutting edge) companies who either have been testing in live production for years, or begin doing so once they see the rapid return and relatively low risk of the practice.

    To quote a leading retail customer of ours: “We’ve never had this happen during live prod testing, but even if we crashed the live system in off hours and affected 100 online users, it beats crashing on Cyber Monday with 350,000!”

    Brad
    (SOASTA CloudTest)

    • Parvaze Suleman September 26, 2011

      Brad, thank you for your insightful comments on this topic. I must say I am shocked that 75% of sites go live without any performance testing and I do agree, this does need to change.

      On your point about 3, my experience has been that customers tend to have unrealistic expectations about performance of their new site. I have come across a client who wanted their new site with lots of dynamic content and personalisation to offer the same or better performance levels than their old site, which was entirely static (i.e. HTML) and delivered through the web server. I agree with you that it is best to establish the baseline and tune the system until acceptable performance levels are reached or the system reaches its breaking point.

      On your point about using live production for performance testing, I will seriously consider using live production for some performance testing in the future. I was fortunate enough to work with a client recently who had invested in a replica test environment that was sized and configured to be the same as production but was offline as it was to be used for DR, so it ended up being used for performance and load testing for new functional releases. It is very unusual though especially in the current economic climate of cut backs!

      Thanks again for your comments.

      Parv

Comments are closed.