A Case From Russia: Building An Infrastructure For Financial Service Fondy

Team payment systems for online stores Fondy wrote vc.ru column about how the project infrastructure — with the kind of problems encountered in the design, how they were decided. The infrastructure of any it company can be roughly and quite crudely divided into two components. Technology (software, hardware, services) and business processes.

Now the infrastructure of our company is running quite stable, the processes allow us to grow rapidly and although no failures occurred they occur are orders of magnitude less than at the beginning: But it was not immediately. When we started to create a company three years ago, we had had a decade of practical experience with technology — both payment and information, but had no experience of building business processes from scratch. In this column we will tell you what mistakes we made, what challenges were faced and how they were solved in the process of building infrastructure we have today.

At the start everything seemed pretty simple and very clear. We know what technologies we will use and what kind of business model experience. At work, we have determined that the General Director and technical Director will close the primary goals of management:

And the team will hire such professionals: Next, we began to search for desired employees. The first issues we had pretty quickly.

Somehow from the beginning we had no luck with the search for a system administrator. We worked with three guys for very long periods of time, but only after a year was able to find a good and reliable specialist. All this year the deployment of the infrastructure of our servers (development environment, the food environment, build releases, system of tracking tasks, and control system version of the code) went on with varying success and with serious delays, delaying the release of MVP.

Additional complexity created by the fact that as a payment platform we need to be certified for safety standards, and the administrator must have the appropriate knowledge and experience to develop and implement hardware architecture. The Council. Treat search key specialists very seriously. Better hire a HR Agency which will be looking for candidates for the above-mentioned requirements and qualifications, and you will just have to interview.

If at the inception of the team you do not have enough competence to proconsulate specialist, ask some of your friends, acquaintances, and partners with the right qualifications to do it for you. It is worth remembering that the team runs with the speed of the slowest member, and hiring a low-skilled or no suitable experience specialist, you run the risk that the rest of the team will be very slow.

Run a minimal working product we have planned a four-month period. But as we detail not decomposable tasks and discussed the timing of implementation with the developers, any task is delayed for a period of 2 to 3 times more of the stated. We could not load the employees work at least 50%.

Always somewhere having a narrow neck. The task is not described, it is not ready for server deployment development and testing, the team does not understand the priority of tasks. In the accounting system task did not have clear processes, and all tasks would collapse one pile, from which the developer selects either a random problem, or one that he like best.

Most of the tasks hang at the “last mile” — like all subtasks are done, but to collect all the “bricks” does not work. At first you think that something is wrong with the team, and apparently you hired a weak specialists, and is many to dismiss or replace. But really, the main problem is the lack of business processes.

The Council. To build the infrastructure in which the development is fast, the challenges are every stage (production, development, testing, release) in the shortest possible time it is important to bring in project professionals with experience in teams working on agile methodologies, such as Agile, Scrum, XP. With a high degree of probability in its purest form, no development methodology you might not like it and will have to find a middle ground. Not to spend on getting your cones are too much valuable for business time, try to involve in the commissioning processes of experienced professionals.

When the process is gradually we began to build, to receive the first clients, the momentum to grow, the next test for us was the allocation of resources for automation of processes. A large part of startups close in the stage of rapid growth due to a disproportionate increase of costs for service business.

When we have automated system to connect new customers to the service, it turned out that our accounting Department can no longer cope with financial reports and information sheets, and lawyers with the formation of contracts and the elaboration of the legal aspects. We faced a dilemma — on the one hand we could not stop the development of the releases and throw the already small resources development for the automation of back-office operations, on the other, if the number of customers has increased ten times, wed have to hire another 10-20 professionals that would eat all of our revenues and drove in a big minus.

The Council. Sure, many Internet entrepreneurs faced a problem when an accountant or lawyer begins to adjust the vector of development of the business, requiring you to shift priorities from frontoffice and send them to the back office. For example, to abandon the development of meaningful functionality that clients need on the website or in the main product and focus on developing the automation of operational tasks.

At this stage, company need to understand that the back office cant run the business, but ignore it needs be. Need to adjust priorities and try to automate the mass routine operations — in the future when your competitors will manually send contracts and acts of acceptance to customers, avoiding mistakes of human factors and frittering away the trust, you will appreciate the importance of what he did. After a year we started to calculate how much cost us the server hardware and software. And here, we played into the hands of great experience and verified over the years the decision.

We decided not to buy their own iron and build the hardware infrastructure and server have chosen the Amazon cloud AWS, which is represented in 38 availability zones around the world and meets dozens of standards, rules, security certificates, has DDoS protection and physical access third-party individuals and organizations.

The total number of Amazon servers, according to third-party analysts in 2012 was nearly half a million, which gives almost unlimited possibilities for scaling. Scary to think how much time and resources to spend in order to provide at least a small part of what makes Amazon in the infrastructure of their project on their own. At the stage of product development we this choice gave big advantages — the first six months all server hardware cost us no more than $300 a month. This time we had to develop a MVP and hypothesis testing of our business model.

From the point of view of the cost of the software — we have always been supporters of the open source. There is a misconception that open-source products are less secure because of its open source code and carry large risks to business from the point of view of vulnerability, stability and quality technical support. But as the company which is subject to annual security audits and quarterly external scanning for intrusions, we can say open source software is not inferior to the famous proprietary products known to the software giant, and in some aspects even superior.

The Council. When you deploy your own hardware infrastructure note cloud solutions. They are usually cheaper to operate with smaller volumes and easier configuration and support. In our case, one system administrator is able to provide support to more than 30 servers without compromising overall efficiency. Also significantly reduce the cost of open-source products.

Databases, application servers, CMS, CRM, system, version control code and tracking tasks. In addition, should pay particular attention to corporate security. In recent years a large number of cyber attacks aimed at the business. From this point of view is to trust your safety to specialized companies and tools, such as DDoS defense systems, intrusion detection, external scanning vulnerability.

In fact, from drops applications, equipment failure, break and other force majeure it is impossible to get rid of. Even the most reliable systems fail. For example, a lightning strike in 2011 crippled piece of hardware Amazon and many websites went offline.

You should always expect that at any moment any part of your infrastructure may fail. Server, application telephone line call center, Internet backbone provider. Since we sign with our clients on a contract (Service Level Agreement) that guarantees a service level of 99.95%, that it is fully run we tried to reserve all critical components of our infrastructure and adhere to the strategy “let it come down” — in case of a fall we always have backup services, which in most cases included in the work of the automatic monitoring system.

The company has also developed Disaster recovery plan — document that describes the matrix the escalation of it incidents (where to go, what to do, what specialists to call) and the zones of responsibility of employees and top managers on business processes that were broken. Step 1. Configure monitoring for your services — there are several free applications which allow, for example, to notify a Telegram or Slack, if your website becomes unavailable.

Step 2. Try first to reserve those nodes that require the least resources to do so. The main site, application, database. If possible, make the main database and the backup were in different geographical zones or data centers (remember the story with zipper at Amazon).

Step 3. Work through the matrix escalation. Delimit different possible situations according to their criticality and assign responsible employees. Determine their maximum time of the reaction, for example, if the main company site is not available, then:

At the moment, the main problem that we, as a company, which became the “rails” of the business processes struggling is the development of innovations at the same pace that we scored at the start of. Increasing the momentum we gradually begin to take in daily operational tasks, which sometimes take so many resources that most of the time teams have to spend to support ongoing processes, not the creation of a new.

To adapt to rapidly changing business requirements and FINTECH industry we now share our team into two divisions. Innovation and Operation Team. Main tasks of the Operation Team are to provide level of service current business and generating income the existing business models.

In turn, the priority for the Innovation Team is to support rapid changes — generation of new ideas and products, implementing innovations, following market trends and business needs. We believe that with this problem we also will be able to cope. Send a private cases, in which you managed to improve (or, conversely, worsen) the performance of the project, [email protected] Interesting experiments are sure to get heading Growth Hacks.

Leave a Reply