Continuous integration and testing of new features are at the heart of an agile software project. To implement them properly you need to set up environments for your app correctly. An environment is a copy of the app with all third-party services that are used either for development, testing or customer use.
Development
The development environment is the one where developers test features as they implement them during the sprint. It’s the least stable environment as new code is merged and deployed several times a day, but it also gives you the most up-to-date understanding of the current project state. As soon as development starts you need to set up the development environment so that developers can merge their code and test it after it has been deployed to the server. It's essential if you have multiple developers working on the project. There’s a huge difference between testing a feature locally on a developer's machine and testing it on the server.
Staging
After the sprint is completed, you need to start testing the newly implemented features to make sure they were implemented correctly, and there are no apparent bugs. There’s little point in doing that on the development server as it’s very unstable and the code is changing there frequently.
You need to set up a separate staging environment that will be used for internal testing and will be a copy of the development server and all related services. It’s important to keep both environments separate.
During the sprint, you need to test the features deployed to the staging environment that were implemented during the previous sprint. Use staging server to spot and document bugs, fix the most critical ones and apply hotfixes to staging.
It would be best if you made sure that the bug is fixed on staging environment without updating it with all code from development because development code is ever-changing and it will render staging unstable. That’s what hotfixes are about — they are small code updates that can be applied to the codebase without merging all of the other changes. The goal is to keep staging intact until the sprint finishes, test everything, fix the most critical bugs and then promote code to production.
You should never promote the code from development to staging if you're not happy with current state of staging and the code haven't been deployed to production. It's important to update all servers frequently. Otherwise, you will deploy too many features at once that will most likely result in a larger amount of bugs than your team typically handles.
Production
It's the environment where all your users and their data will be living in. While the data in staging and development environments is disposable, production data should be kept intact and secure. Developers should never work directly with the production environment because of the risk of data loss. The only exception is fixing critical bugs or investigating the incidents, but even in these cases, all work should be performed in a read-only mode if possible.
The typical pattern is to promote code to production once it has been tested and most critical issues have been fixed with hotfixes on stagign. The staging is then ready to be updated with development code and a new cycle of testing and fixing bugs begins. Once new code has been deployed to production user testing starts. Best way to perform user testing if you already have users is to deploy new features only to a given set of users (beta testers) and have them spot issues. However, if you are in a pre-launch mode or your user base is relatively small, you can deploy code for all users and notify them about an update and what has been changed. Just make sure that you are ready to fix bugs right after release to keep users happy.
Setting up support
As soon as you let users use the app on their own, you need to set up your customer support system. This support system will allow users to report bugs and ask questions, which is especially important if you want to implement continous integration and delivery process.
You can use one of the many apps to provide support, such as Zendesk or Intercom. A minimal setup will require you to set up a live chat so that whenever users have a problem, they will contact you immediately. Of course, you don’t need to be in front of the computer 24/7 — modern live chat apps allow you to reply to user messages via an app on your phone or they will email you the message and user’s contact details if someone is trying to contact you outside of working hours.
Monitoring the production app
To provide proper support and be confident about your production app health you need to set up basic monitoring. Monitoring will allow you to know if your app is running and if there are any backend or platform errors. You'll also get basic stats like the number of requests your app is serving, the time when your users are active and many more. The most significant benefit is the ability to see the health status of your servers — if there’s a high load or not enough memory, you ran out of disk space, or latest code update resulted in a massive amount of strange errors. This information is invaluable when you are trying to understand why users report that the app is ‘slow’, ‘not working correctly’ or 'buggy'. You can even set up alerts so that you are notified about critical issues with your app’s health.
Logging
Monitoring is great for getting a high-level view of your app’s state, but it gives you very little detail to debug and fix problems. To spot and fix bugs, developers need to understand what happened and what caused the problem users are experiencing. Typically this involves going back in time and looking through the log of events to understand the root cause. It means that if you don’t have proper logging set up, you will have a hard time debugging errors on production. Logs must contain as much information about events as possible. You need to collect and store them so that they are searchable when the need arises. Make sure though that you don’t include any sensitive information like user passwords in logs.
Analytics
Since users will be using the app on their own, you need to set up proper analytics to understand their behavior. To set up basic analytics, you need to decide what events or user actions you want to track and then collect the data about those events using tools like Segment.io. You can then analyze them using something like Amplitude. Such analytics will give you quantitive information — how many users log in daily, how many of them go to a particular screen or click a certain button. There’s also a way to get more detailed information using heat maps and session recordings. To get it you need to set up a tool like Hotjar.
Setting up proper continuous integration and delivery takes time, but once you have it and the team gets used to it, your development process becomes smooth and efficient. To get most out of each environment you need to set up monitoring, logging and error reporting for each one of them because only this way your team will be able to debug issues in the code deployed to the server.
