In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.
Jumping back...
Miranda is composed of subsystems, each of which runs in its own thread. The threads communicate by means of Messages and BlockingQueues.
A thread simply takes the next Message off the queue and processes it.
The various subsystems of Miranda each have a State object. The states "know" how to process a Message (a method called processMessage), and common behavior, like shutting down or how to deal with a file change, is gathered into the superclasses of states.
Each state has a class it is partnered with that does the "real" work.
Friday, March 31, 2017
Introducing Miranda: Closing
In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.
Reasech shows that people tend to remember the opening and closing of a presentation the most; so I am going to go into my closing early on.
Hopefully this talk has given you an idea of what Miranda can do for you and the confidence that it can do it. Now the the decision you have to make is whether to pursue it further.
You download Miranda from GitHub at https://ClarkHobbie/miranda and perform a trial run. The tools that you will need are there as well. Setting up Miranda is relatively simple.
I would bid you to throw off the shackles of the Pager! While Miranda may not eliminate the need for a pager altogether, it will give you the confidence to know that, if you take your time in responding to a page, it's not the end of the world.
Reasech shows that people tend to remember the opening and closing of a presentation the most; so I am going to go into my closing early on.
Hopefully this talk has given you an idea of what Miranda can do for you and the confidence that it can do it. Now the the decision you have to make is whether to pursue it further.
You download Miranda from GitHub at https://ClarkHobbie/miranda and perform a trial run. The tools that you will need are there as well. Setting up Miranda is relatively simple.
I would bid you to throw off the shackles of the Pager! While Miranda may not eliminate the need for a pager altogether, it will give you the confidence to know that, if you take your time in responding to a page, it's not the end of the world.
Thursday, March 30, 2017
Introducing Miranda: How Miranda Works
In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.
So Miranda looks like something you might be interested in, but how does it work?
Mirada sits in front of your web service and accepts POST/PUT/DELETEs on its behalf. It then sends those events to your web service when it is up.
When a client sends a message to Miranda that is called an Event. The end point that they send the POST/PUT/DELETE to is called a Topic. An Event is delivered to a client as part of a subscription. When Miranda gets a 200 response as a result of sending an Event to a client, it records it as a Delivery. All these things are set up with Users.
Before doing anything, a User must login to the system and establish a Session.
Miranda operates as a cluster of nodes. To do anything, a qurom of 2 nodes needs to be established. Each Topic can have different rules about when to recognize an Event. The default policy is after Miranda receives a POST/PUT/DELETE and has forwarded the event to a qorum of other nodes, it responds to the client, telling them that it received the Event.
At the same time that a node is telling the other nodes about an Event, it writes the Event to the persistent store.
Events are kept for a configurable period of time but the default is a week. Deliveries are also kept for a configurable period of and this also defaults to a week.
Wednesday, March 29, 2017
Introducing Miranda: 9s of Reliability
In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.
What do the various "9s" of reliability mean? Here is a table with the amount of yearly downtime that each level allows:
So 1 9 any team could do if they have a working system. If they even have a pager the person with it does not pay a lot of attention to it.
If the company is serious about it 2 9s is no longer a joke. They definitely have a pager that people trade off on a weekly basis. When the pager goes off. the person tries to respond in 20 minutes or less. The person on call has the phone numbers of the rest of the team in case they need something.
At 3 9s, there is definitely someone with a pager and there have been very serious conversations about getting a control center, a la NASA, for the system. The system may or may not be distributed.
At 4 9s, there is a control room manned by very humor limited folks who have the numbers of each of the team members, and when something goes wrong, they call them. The person on call switches off weekly, reminiscent of wearing a pager, and if there isn't a hot standby then there have been very serious conversations as to why not.
At 5 9s, there is a control room, manned be people who were too serious for the previous level. The control person's duty is to switch over the system to a hot standby and to call the on-call person when a problem develops. Being on-call is no joke, and when the pager goes off, the on-call person must respond within 10 minutes.
At 6 9s, things are insane. The people who were too serious for 5 9s are manning the control room. There is a person whose sole duty is to decide whether to switch to the hot standby. There are two levels of people on call, and each of them must respond within 5 minutes.
Levels 7,8 and 9 require varying levels of hardware to support them, and there is still a control room and lists of on-call people. But the question becomes, if you are serious, what is this for? At 7 or 8 9s it becomes hard to tell if the system has been down. After all, people's internet connections are not 7 or 8 9s reliable.
What do the various "9s" of reliability mean? Here is a table with the amount of yearly downtime that each level allows:
| # of Nines | Percenatgae | Time |
|---|---|---|
| 1 | 90% | A month |
| 2 | 99% | 3 days |
| 3 | 99.9% | 8.8 hours |
| 4 | 99.99% | 52.6 Minutes |
| 5 | 99.999% | 5.3 Minutes |
| 6 | 99.9999% | 32 Seconds |
| 7 | 99.99999% | 3.2 Seconds |
| 8 | 99.999999% | 320 Milliseconds |
| 9 | 99.9999999% | 32 Milliseconds |
So 1 9 any team could do if they have a working system. If they even have a pager the person with it does not pay a lot of attention to it.
If the company is serious about it 2 9s is no longer a joke. They definitely have a pager that people trade off on a weekly basis. When the pager goes off. the person tries to respond in 20 minutes or less. The person on call has the phone numbers of the rest of the team in case they need something.
At 3 9s, there is definitely someone with a pager and there have been very serious conversations about getting a control center, a la NASA, for the system. The system may or may not be distributed.
At 4 9s, there is a control room manned by very humor limited folks who have the numbers of each of the team members, and when something goes wrong, they call them. The person on call switches off weekly, reminiscent of wearing a pager, and if there isn't a hot standby then there have been very serious conversations as to why not.
At 5 9s, there is a control room, manned be people who were too serious for the previous level. The control person's duty is to switch over the system to a hot standby and to call the on-call person when a problem develops. Being on-call is no joke, and when the pager goes off, the on-call person must respond within 10 minutes.
At 6 9s, things are insane. The people who were too serious for 5 9s are manning the control room. There is a person whose sole duty is to decide whether to switch to the hot standby. There are two levels of people on call, and each of them must respond within 5 minutes.
Levels 7,8 and 9 require varying levels of hardware to support them, and there is still a control room and lists of on-call people. But the question becomes, if you are serious, what is this for? At 7 or 8 9s it becomes hard to tell if the system has been down. After all, people's internet connections are not 7 or 8 9s reliable.
Miranda Takes Systems with 1 or 2 9s and Makes Them Appear to have Between 5 and 6 Nines
Miranda is a distributed, fault-tolerant system designed to run behind a load balancer that accepts messages on behalf of the underlying system and delivers them when the system is up.
The underlying system can be down for hours or days while Miranda accepts message for it. Therefore, the system appears to have between 5 and 6 9s of reliability, when it has fewer than that.
Tuesday, March 28, 2017
Introducing Mindanda: the Next Step
In preparation for a talk I'm giving at DOSUG I'm going to post my thinking as it develops.
The next step for Prospero is Miranda.
Miranda builds on many of the concepts that Prospero had. Things like capturing HTTP POSTs, forwarding them to subscribers, etc. Miranda does what Prospero does and takes the next step. In addition to capturing POSTs, it also capture PUT and DELETE.
Miranda also addresses Prospero's limitations. Prospero, for example, does everything in the clear. Miranda does everything in SSL/TLS. There are many areas that Miranda improves on Prospero.
Miranda works by sitting in front of, and recording for, web services. When a client sends a POST/PUT/DELETE to an endpoint, Miranda records it. Since Miranda is a distributed, fault-tolerant system it makes the underling web service to appear to be more reliable than it actually is.
Ideally, the service appears to be as reliable as Miranda is, and hopefully, that is very reliable.
The next step for Prospero is Miranda.
Miranda builds on many of the concepts that Prospero had. Things like capturing HTTP POSTs, forwarding them to subscribers, etc. Miranda does what Prospero does and takes the next step. In addition to capturing POSTs, it also capture PUT and DELETE.
Miranda also addresses Prospero's limitations. Prospero, for example, does everything in the clear. Miranda does everything in SSL/TLS. There are many areas that Miranda improves on Prospero.
Miranda works by sitting in front of, and recording for, web services. When a client sends a POST/PUT/DELETE to an endpoint, Miranda records it. Since Miranda is a distributed, fault-tolerant system it makes the underling web service to appear to be more reliable than it actually is.
Ideally, the service appears to be as reliable as Miranda is, and hopefully, that is very reliable.
Monday, March 27, 2017
Introducing Miranda: Prospero
OK, if a system with "9 9s of reliability" is unattainable with a sane budget, what is attainable?
The answer was Prospero.
Prospero began life as a "skunkworks" project with Erlang. While its initial goal may have been something else, it was used to make systems which only had 1 or 2 9s of reliability seem like they were up most of the time.
Clients would define a topic and then send HTTP POSTs to Prospero. Other clients would "subscribe" to these topics and register a URL that Prospero would associate with them. When a client POSTed Prospero would, in turn, POST to the URL that had been registered.
Prospero did its job pretty well, and Pearson later released it as an open source project with the name of "Subpub."
Chris Chew initially did most of the development. I did a little bit in the way of XML processing.
Over the course of about 3 years of support, the following limitations were discovered:
The answer was Prospero.
Prospero began life as a "skunkworks" project with Erlang. While its initial goal may have been something else, it was used to make systems which only had 1 or 2 9s of reliability seem like they were up most of the time.
Clients would define a topic and then send HTTP POSTs to Prospero. Other clients would "subscribe" to these topics and register a URL that Prospero would associate with them. When a client POSTed Prospero would, in turn, POST to the URL that had been registered.
Prospero did its job pretty well, and Pearson later released it as an open source project with the name of "Subpub."
Chris Chew initially did most of the development. I did a little bit in the way of XML processing.
Over the course of about 3 years of support, the following limitations were discovered:
- It was difficult to find people with Erlang experience
- Prospero was difficult to modify
- Prospero was limited to one data center
- Mnesia (the database that Prospero used) could become corrupted
- Prospero had difficulty with binary messages
Sunday, March 26, 2017
Update Properties
It is time, once again, to update the properties page. Since I haven't updated in a while, there is lots to do.
Subscribe to:
Posts (Atom)
