Blockchain in the Real World
Google DevFest, 2017
Before research and development, I worked at Yandex in the security service, engaged in software development. In the Yandex security service, I was engaged in all sorts of different projects that are related to security. All sorts of X projects, so-called. A lot of interesting work.
About 3 years ago, I didn’t know anything at all, even somehow I didn’t really come across blockchain, but the world changed three years ago. My partners and I were among the first to offer the use of blockchain to the Central Bank and bankers at a banking conference in Kazan, at FINOPOLIS. This proposal caused some skepticism, then the investigative committee and others quickly created laws that it is necessary to punish everyone who is engaged in the crypt, and therefore the attitude towards us was very ambiguous.
And over these few years, the world has changed a lot, and the world, and our economy, the blockchain as a whole. And a year and a half later, after we proposed and met with skepticism, we heard another message, which is very important, in the same banking sector. The message from above, from the regulator, is that this is very interesting and has prospects. And then the active work went on.
Today I will tell you more about the technological part, we will not go into any super-details much, because I usually tell details on golang meetups, because most of our software is written in Go. Who writes on Go? Is there? Here, 3 people. It is ok. Usually less. This is not a marginal language.
Why does a blockchain business need
I will tell you about what qualities a blockchain system should have in order for its business to understand. Then I will tell you about our experience, because we, as a research and development center, act in the interests of one of the banks, and I will tell you some of the inside and interesting things, and maybe not very interesting, today. I will not name the client, thoughtful guys and girls will look, they will be able to Google and roughly understand who it is, but this is one of the big guys, probably about 20 million users and about a billion transactions a year pass through it. Go.
Blockchain is not only about cryptocurrencies. I’m not going to talk about how to fork Ethereum today. About what Bitcoin is. In general, who is mining? Who is mining, he will understand. By the way, it’s dangerous to talk now. One friend in Ukraine, it doesn’t matter what it is in Ukraine, his car was almost squeezed out. I bought it, and it was almost squeezed out by local bandits. So it’s really dangerous to talk in some regions.
Blockchain is really not only about cryptocurrencies. As, in fact, in the industry there, in Moscow, in St. Petersburg, I see such a trend that there are guys who have forked, made their own variation of some new cryptocurrency and are trying to somehow adapt it for business. Who has heard about Ethereum? Oh, great. And who thinks that Ethereum is cooler than bitcoin? Five people. Today we will not talk about what is cooler. But one quality that Ethereum has important is its flexibility. But we won’t talk about him today either.
When you come to a business, to a bank, for example, it has completely different tasks. The tasks are as follows: in a bank, for example, or in a payment system, just in business. I am an entrepreneur myself, not a startup, and this is the same thing that I encounter in some of my products, not only in the R&D services sector. For example, we have processing. And there is a system that tracks fraudulent activities. And so it turns out that they should be connected somehow, but they are developed by different teams, maybe even different product teams in different countries. And they need to exchange data and, simultaneously with the exchange of data, the question arises so that nothing is lost between the way they exchange.
About information security, business flexibility and speed
There are a lot of approaches in information security, how to ensure the formation of contours, data signing, etc., etc. We all know this, we have been through it all. But I want to simplify it all. And reduce paranoia. Paranoia slows down business, especially banking. The security guards say: no matter what happens. That is, it’s money, no one wants to see zero on their card on their payday. And then comes an important quality — a smaller service with a single point of failure. Unfortunately, we have very few companies that are able to make real services that are really experiencing the cutting of channels with data centers and so on. Because it costs really good money.
When we change, for example, some corporate database, usually Oracle, to something related to distributed registries, we fix it at several points, we add the so-called eventual consistency. Who knows what it is? And how to translate it in Russian? Some call it “consistency in the end.” I transferred money, transferred money from A to B, as a result, after a while, when the two systems are synchronized, it turns out that A has this money debited, we can immediately show it, and B will get it while bytes are on the wires, bits will reach another data center of some kind. It doesn’t happen right away. Not the point, it’s a technological thing. Absolutely not useful in the framework of our dialogue today. But it’s important. That is, when you develop it. Accordingly, a lot of repetitions. This system, it cannot be made strictly consistent, so that there is still some kind of distribution, because the CAP theorem. Can you say that? Can.
Information security and physical security are quite advanced. Because the Central Bank can take away the license if everything goes wrong. Therefore, the contour is usually protected. The server is in some kind of cage, under the key in the center. In good banks. And therefore, the idea of forking Ethereum or any other popular product, it does not quite suit us for the reason that, again, we do not need mining. And because there are no guarantees. I am not against this approach, but there is no guarantee that something will go wrong there. That is, they forked Ethereum, brought your salary to this forked ethereum, someone did not commit there to this open-source, and as a result you did not receive your money. Or you are standing in the evening, you want to buy groceries at a gas station or in a store, but you can’t do it. And at that moment, you start thinking about open-source that something else is needed.
Therefore, everything needs to be simplified as much as possible. Not to play some cool stuff that automates everything, but to simplify it to the maximum. To a screech, to a creak. In fact, we just need a chain where each subsequent record has a checksum, a link to the previous one. Such a chain. And, accordingly, so that we can quickly check these chains. For example, I unloaded anti-fraud from the transaction processing system and that’s it. It’s very hard to do this. Once again, it is difficult to apply bitcoin or Ethereum in real business. Because in a real business, it is important that the transaction is securely fixed.
Accordingly, the speed of fixation… what is the latest fixation speed there? Maybe who knows who has fresh data? What is the transaction commit rate for Bitcoin? 15 per second? Per minute? Not much can be driven away. When, for example, two banks will drive billions, they are unlikely to entrust it to such a system. Because 15 minutes. A lot can happen in 15 minutes. Especially with us. Accordingly, the question arises: what to do? Open solutions seem to be suitable, but not quite. To write something of our own — we usually return to the usual architecture. What we have there: a database, a table with accounts and an SQL query - translate from one to another.
Why blockchain, if there are already classic databases
These are already existing solutions, as a rule, all banks have them. And they hang themselves from the fact that there is one Oracle that is really hard and expensive. That everyone has attacked the blockchain? The exchange rate jumped and it became really expensive to service vendor solutions. There was a million dollars a year, it became two. Weighty. The question arises: what should we do, where should we store it? So we need some kind of storage. It was stored in MySQL, somewhere else. But a single base is a slow recovery. There is a big problem when we have everything at one point, even if it is a replicated database, there is still a joke that the replica has fallen, the master has fallen, you still need to restore it, for example, 200GB of data to pump, even if it happens on gigabit.
About the speed of recovery after an accident
How much is a gigabit? One hundred, a little less than 100 megabytes per second. Well, we’re counting. Accordingly, our customers will not wait for the application to say: “Sorry, the service is unavailable.” This is unacceptable for the financial system. It’s a shame. It seems that you keep your money, and you still have some kind of denial of service. Accordingly, the minimum is enough for us, we have big uptime requirements. Accordingly, there are different options for how people screw up solutions like bitcoin-like, and store, blockchain, this is, in fact, a binary chain. As a result, we need it for our product, which we have been developing for the last two years, which is already being integrated into more and more projects of our client, our bank. As a result, we come to the conclusion that we need some kind of our own solution.
How to raise your salary
What is our approach? Plus, what I’m telling you is an architectural pattern, it is replicated. Surely there are people who understand that everything related to the blockchain is well paid. And it will be even more heavily paid. That is, this is a good way to make yourself a level-up on salary. Why write something in web frameworks if you can do it.
About processing architecture and big data
So our pattern is this: instead of democracy, as in bitcoin, we work in a trusted environment. We obviously trust each other. We need to throw away a large number of chips related to the fact that we do not know each other and do not trust each other. Transactions in our case go through services called gates. Each gate has its own block, you can call it a chain, just all these transactions follow each other. And each account has its own chain. That is, we do not have a single chain. Gates negotiate with each other.
Self-organization: How processing nodes share responsibility
The idea is that we have initially one common range of accounts. For example, from zero to infinity. The first node appears. He looks at the current state of affairs and sees that he is the only one in this network. He takes over the entire range completely. Then the second node appears. He asks first. And he says, “And I want half.” If they agree, it is possible to agree when there are more than three of them, so that there is a quorum. I’m simplifying it so that it’s clear. Accordingly, each node works simply on the principle of “as I want, so give it to me.” Here I want to answer for one account, for example, there is some fat account to which the commission comes. And he can process it.
And this, by the way, is the main problem in sharded systems, where services are divided into shards, in pieces, each process has its own piece, a monster appears. For example, in Vkontakte, the data is also sharded and, for example, there is my page with ten posts, and there is Pavel Durov’s page, which has an insane number of friends, posts, comments, respectively, the service that processes him and processes me has a different load. Here we solve such a problem simply. We can dynamically say to each gate in real time: “OK, good.” And he asks, says, “I want this piece of responsibility,” and takes it, extends it periodically, once in some interval.
If he didn’t extend it, he’s back and anyone else can pick him up. Therefore, adding and removing nodes is very easy. And, accordingly, fault tolerance in the box. That is, a node has fallen, or a node needs to be updated, okay, they brought it out, entered it. If it was done in a second, for example, during this rental interval, no one will notice anything at all. Further. Since we have a trusted, reliable network, optics, sound system administrators, circuits, that is, at the network level it is also necessary to provide this somehow, there is no stability at once, when, good commercial fault tolerance, when you have several clusters connected via a copper wire, along which a chair periodically travels. This is strange. If you are given such a task: write a service so that everything dies like in Google, but everything works.
About expensive and cheap reliable infrastructure
We also need to invest in infrastructure. And sometimes it’s really easier to throw a few optical cables than to write a system that is experiencing, there, a disaster-resistant system, it’s really expensive. And programmers are likely to get confused after a while and will not understand whether it is really disaster-resistant, or not, or they were mistaken. This is based on the experience of observing different teams. All, accordingly, the chain on each gate is limited, that is, we do not store a copy of the chain like bitcoin, like related systems. We don’t need to download the whole story, we download it from the database, from the repository as we work with it. That is, right up to the moment you make the translation, you are told: “No, wait, I don’t have the data.” And he pumped it up in the background, repeated it, and was like, “Oh, it’s okay, I fixed it.” That’s why I said on the first slide about eventual consistency that there are a lot of repetitions, a lot of retries, these are the features of such systems. Don’t think it’s a bad thing. It is ok.
Why write your own specialized database
About reliable storage. When I wrote the title of this slide “reliable storage”, “reliable storage” sounds loud, that is, in fact, what is it? This is where the entire remaining chain of such a platform is stored. This approach can be used anywhere. And implement it in any language, and on any technology stack. It is important. Why we don’t use memcash or radish, because we have a limited data schema, we only have a transaction and account setup. That is, there are two entities. We don’t need to have some incredible flexibility. We can write a specialized engine. And it is important not to lose transactions in our case. That is, we have a hybrid approach: on the one hand, it is a distributed registry, on the other hand, it is a reliable storage with transactions. They lost, destroyed the blockchain and that’s it. Not everything, but for a particular account it can be sad.
How does it even work? These gates, they eventually processed the transaction, considered that the balance converges, and routed the received data, duplicated them, they write and write to the database. And what is it to write to the database? We have some controller, it’s just a service, it gets to it: from A was transferred to B and B had money. The transaction for A is outgoing and the transaction for B is incoming. Then we pack all this into a transactional model on shards. Accordingly, at the base at a low level, the data is sharded, divided, but already divided according to some logic of their own, which does not concern gates. That is, it is just a repository, it is located separately somewhere. Gates see only one address in fact behind the balance.
Next, shards. Each shard has its own replicas. For example, in several data centers. Accordingly, we are generally purple. Our data center will turn off — nothing will happen to us. It will connect back. Replicas will be restored from their two copies. As a rule, we have all the data on each shard stored in three copies. That’s how a fairly reliable preservation of transactions is provided. I am very critical of the word “reliable”, because there are a lot of databases, including commercial ones, from this new age, when they all write: “we are cool, we are wonderful, we are going through the shutdown of data centers, you can do anything and we will recover from any state.” The bankers won’t believe my word if I tell them that. So there is one, it’s not on the slide, but I’ll tell you.
Why is the processing really reliable, and the rest are not
There is such a test for databases, called Jepsen. Has anyone heard about this test, Jepsen? Oh. Have you written anything under Jepsen? Here. Jepsen is a kind of database testing framework written by some enthusiastic guy, Kyle. Kyle’s his name, right? Something like that. His nickname is aphyr. And what does this thing do? It runs any database on five virtual machines. Starts sending random requests to each machine. And in the process of sending requests, for example, to fix some data, to read, that is, some script, it begins to destroy these machines. To drive the system time back and forth. Freeze the process, defrost it. Kill this machine, lift it. That is, a complete destruction, as in the real world.
I am often told: “Here in NetFlix there is such a program ChaosMonkey, the monkey of chaos, they are also testing something similar.” And Kyle, with the help of Jepsen, practically broke most of the databases. That is, you can Google, he has a blog on this topic, he has a large number of bug reports on different databases. Who uses mongu? Well. Monga, some versions were broken, she lost data in such a situation, that is, if it was not enough to get into finance. What else was broken?
There were some jokes in Cassandra. That is, he broke a lot of things. When we wrote our repository, we immediately set the bar very high for ourselves: our task was to pass the Jepsen test. And recently he started walking. That is, our database runs completely at all three levels, at the level of a shard with replicas, we look, we make a destruction with replicas, then we do a level-up, then we fix it on shards, that is, we make a destruction not on replicas, but at the shard level, and at the top level, we make a destruction already at the level of the entire cluster. So that you can understand where something is wrong. This is really a big challenge. Huge respect to my colleague Nikifor, who implemented these tests on Jepsen. That is, he was doing it. It’s tin. They don’t do that here. While. And another important thing, if you ever start writing your database, it’s obvious that parallel access is a very big problem.
In our case, again, a chip, a cool chip. Since we only add data to our chain, we don’t overwrite, add, add, add, add. We do not have parallel access and write to the same key. Therefore, we can simply take and throw out a significant part of the transaction engine, or simplify it. Once. And two, it’s not really a problem for us. There is also a joke with a watch. With the moment when we have clocks leaving in different parts of the system and we need to somehow understand what event happened earlier. We paid a lot of attention to this problem, because the same Jepsen steals watches. And when we have requests coming to different points of the system from different ends, they cannot enter through one point, otherwise it will be a single point of failure, and in the end it is necessary to understand at the lowest level: what happened earlier, what happened later.
And if it is misunderstood, for example, relying on the wrong clock, then you can get in. And thanks to Jepsen. He blurts it out too. These are all the problems that any really distributed systems face. We have small shards, again, going back to the moment that, so as not to pump data for a long time. That is, in some setups, we may have a shard size of about half a gigabyte. This is an optimally calculated time so that if some kind of fakap occurred, the service rose, it managed to pump over the network, and we did not go beyond the limit of 8 seconds of permissible downtime. That’s where it’s coming from. Again, the size of the data, it really affects. That is, this number is not taken from the ceiling, 3, 10, 100. You can pay attention to this.
About sidechain, masterchain and the legal way of blockchain from the Central Bank
Then the question arises: we record this data at home, everything is fine, a bunch of servers, 3 data centers, how to make them really reliably verifiable? Our system, it falls under the category of the so-called sidechain. This is when we host a blockchain without entering global networks. And how to do it all the same with a reliable approach? We periodically take data from any sidechain, including ours, upload data, count the checksum from them, count the blocks and fix them, for example, in the Ethereum smart contract, or in bitcoin there is a hack with a bypass that allows arbitrary data to be written there. But, limited, of course, but allows you to write there. We did both.
Now the Central Bank is developing a legal masterchain with market participants, this is a system that will serve for some kind of anchoring. This term is new, it appeared just recently. This is when we unload the data from our system, in our circuit, and fix it somewhere at the top, where it is beyond our control. As a result, the data in the legal field will fly not to bitcoin, but to the central bank’s masterchain. Like this. I recommend paying attention to the masterchain. There is a landing page on RBC, recently it came out that the guys are very active, a very promising project. Because in the end, he will most likely have a legal status. And everything else, probably, will have some other status.
A universal tool for any bank: processing, anti-fraud and a dimensionless transaction database
About cases. Where we use it. First of all, in payment processing. In a variety of projects inside. This is one time. That is, it is not necessary to write data with a balance, you can record any events in our system, meaning that you cannot write arbitrary metadata to such a system, but you can record the fact of an event. Even without a balance, with zero balance. And they sold two sweaters to B. This is also a balance. Anti-fraud. She just works with the data, and she verifies it.
Verifies in about the same way as it works in bitcoin. Merkle-Tree. The stupidest algorithm. From business cases, this is also a credit history bureau. In fact, our system consists of the fact that there is a method to get the balance, make a transfer from A to B, get the history of payments, incoming, outgoing, and get the previous hash. That is, we first get the previous hash, add it to our request, sign it with our private key, and send it. There is a public key on the processing. He checks that the signature is correct, and that’s it. Gate. Gate does it, that’s all the secrets. Well, online sales. Accordingly, the sale of goods, I said about sweaters. Here the sidechain is used simply as a dimensionless database. In parallel, blockchain and I have also opened essentially dimensionless databases into which you can write, write, write. But with its limitations. She does not have a single point, everyone agrees, they agree distributed. That’s all, now the questions.
Questions: about the downed time on different nodes
Do you hear? Oh, good.
— Thank you for the report, I liked it.
— How did you solve the problem with the fact that transactions come with a downed time, how did you solve it?
— Google has such a paper about TrueTime. Have you read it? No? Will you tell me about the time? Come on.
— Nikifor: If two transactions come to the same account, that’s what we’re talking about, right? There are several approaches. Firstly, it is possible to exchange logical clocks, if they turn out to be disordered, then we synchronize the clocks and try again. That is, some event happened here, the counter, the clock was increased, sent to this one, the event started anew here, that is, we try again, this event happened here the counter was increased, on the third it all came, the counters were compared, they are different, comparable, the events were ordered. Another approach is to try to synchronize the clocks as accurately as possible with each other. Google has an atomic clock there so that everything is absolutely accurate. And it is guaranteed that our watches diverge by some millivelichina. Our system guarantees this, and we know it, we can rely on it. When different requests with different times came from two points to some third point. At the third point, we look at the timestamps and simply wait for this time interval. That is, if we waited for this time interval, nothing else came to us, then nothing else could have started more than this interval ago. Accordingly, no new event will come to us from the past or the future. Is that clear?
— In principle, yes.
— That is, we are not comparing timestamps, but comparing intervals, that is why I said about the paper, there is the main idea, if you don’t read it, this huge footcloth, that we compare intervals. And we have intervals, the permissible error discrepancy interval, it is set, for example, by the accuracy of atomic clocks, we approximately know plus or minus 20 milliseconds.
— That is, the event did not happen earlier than so much and not later than so much. And this is also. And if we can compare them so that the upper and lower boundaries do not intersect, then the events can be ordered. Basically, there is a struggle to reduce, so that there are fewer discrepancies.
Questions: when will the endless database end and why do we work well on cheap hardware
— Of course, I don’t know much about the blockchain, but this is the question: if you write to the database endlessly, then what? The volume will increase infinitely, it turns out. As we will in 20 years, there are billions of transactions per year, memory will increase, we will have to store it all somewhere.
— Well, yes. Therefore, sharding is actively used in this system, sharding into small pieces. If some shard swells, it is simply divided into 2 others. How the cell divides. With one two appeared. It’s the same with shards. That’s all. There will simply be more shards, not one data center, but 10. It’s not a big deal.
— Will it be more expensive?
— No, all this data, they are in a compact form and they can be stored on cheap hardware on “so-so” screws. That is, you can just buy cheap hardware in droves and put it under this storage. One of the main commercial features of this approach, such a base. Because expensive iron is expensive iron. That’s all. Because we, in fact, do not care if the machine dies, we will put a new one. The new one will be registered, it will pour over the same data for some time. Let’s move on. And all the other nodes, gates, base controllers, shards, and so on - they can all die. In fact. That’s all.
— Thank you.
Questions: about the release of processing to society
— What else? Any questions.
— Thank you for the report, it was very interesting. I understood correctly that you are making a repository with transactions for your large customer and at the same time this repository is not available anywhere on a paid or free basis. You didn’t even say its name.
— yes. Everything is so.
— Will it ever happen? On a commercial basis. It just sounds really good.
— There will be a paper.
— OK. And then we will finish something like that ourselves.
— Well, yes. We will try to burn more in the paper.
Questions: what does anti-fraud and blockchain have to do with it
— Hello, thank you for the report. I have 2 questions. Since you keep hashes of your data in your storage, yes? How much data volume differs from the use of some standard repositories, such as relational databases, by transactions. Second question: you mentioned that this technology can be used in antifraud, but I didn’t quite understand. Antifraud, as far as I know, implies the identification of anomalies in transactions, you somehow did not disclose this topic. Please explain, thank you.
- Let’s start with the anti-fraud. The anti-fraud system in the general sense is such a responsible piece in which some data anomalies are really analyzed. In order for us to analyze them correctly, we need the first thing: to make sure that our input is clean, that the administrator or the processing programmer has not put the “ifs” in the right places and does not drain the money to his friends. For example, or there is a modification on some other path. There are a lot of ways to figure out how to intervene in the middle. It turns out that the anti-fraud system with a clean entrance has worked, but we still need to ensure that its verdicts are carried out, that is, we have to store this log of what it does somewhere. So that there was no such thing that the anti-fraud blocked the account, then some kind of nonsense happened there, then these verdicts were rubbed in the database, nothing seemed to be blocked and everything seemed to be fine. In a large company with more than 20 products and many product teams, it is difficult to sit on top and observe everything, otherwise money will flow somewhere. Therefore, storage in immutable storage is a bonus. It’s just a bonus. Not all data is stored there, not all existing teams want to use this platform. Logically. They are like, “We put it in Oracle, why do we need your processing”. But this property, it is very valuable for them. There was also a question, I, I, when 2 questions are asked, I have an overflow. What was it about?
- About comparing your repository with more traditional Oracle-type repositories. That is, if you make transactions through the Oracle database and through your database. How much the amount of data will differ.
- Again, before sitting down to write a database, tests were conducted for this type of data. The most important problem there is not even in the size of the data, but in the fact that there is one master and a very heavy load is on him, because he alone holds the balance. And it is impossible to divide this master into 200 or 300 mini-masters. It’s not working anymore. It doesn’t work that way. There are hybrid approaches like Google Vitess, based on MySQL with sharding, an interesting approach, we also studied it. Someone is screwing Cassandra into this case. A very exotic solution, in my opinion. We tested it too. As a result, an architecture with a light service written in fast, using fast algorithms in a compiled language with a binary protocol, it provides everything we wanted from this repository. As for whether it takes a lot of data, we store data at the lowest level, in an optimized proto-safe. That is, there are also several variations in the protobaf. We store it in a version optimized for the size. What time is it there? Any more questions? Did I answer?
Questions: about the scope of development
— Thanks for the report. I have a question: do you have many people involved in development?
— Approximately? Okay
— Not a building exactly. A small team. More?
Questions: about the work of processing nodes, who is responsible for what
— Thanks for the report. As I understand it, on one account — one gate per unit of time?
— In different ways.
— That is, you can make several gates on one account?
— Yes, absolutely. And they are independent of each other. They don’t even know about each other.
— For example, to find out the balance of the client, who is responsible, a gate or a third-party entity?
— We go to the gate, ask for the balance. If he has his chain, in fact, he needs the last transaction on this account, because the balance is written in each transaction. We have made an action - the balance is so-and-so. If he has this transaction— he will give it away. If not, he will say: “Soryan, I don’t have it,” goes to the database, repeat the request, return the balance.
— And if there are several gates, how do they synchronize with each other? Through the master or among themselves?
— No, they process a specific range of accounts at one moment, that’s it. That is, they do not intersect.
— Thank you.
* * *
Presnenskaya embankment, house 12, floor 44,
MIBC “Moscow City”, Tower “Federation”, Moscow, 123317