Thursday, June 22, 2017

Java and SSL:Welcome to the Dark Ages!


This post is part of a series about the worlds of Java and SSL.  I hope to do 1 post a day on this topic. The resulting posts will become the basis for another section of a talk that I am scheduled to give on August 10 at the Boulder/Denver Cybersecurity Meetup.
  • If you have an Java and SSL Expert..
  • SSL is difficult in Java NIO
  • Few tools are available
  • Even Stack Overflow was no help
All I can say about experts in the field of Java and SSL is that if you have one...KEEP THEM!  I found the combination of Java (NIO) and SSL to be very difficult.  And the Java world has had over 10 years (NIO was releases in 2006) to fix this!

I found SSL to be ridiculously difficult in Java NIO.  For something as ubiquitous as SSL I was hoping to find it an easier going.  Oh boy was I wrong.  

I had to fight SSL every step of the way.  If things became easy, I immediately became suspicious.  If I tried to do something "simple" in SSL, all the examples that I found generated warnings when I tried to use them.  When I found what I deemed a bug in one library, the person I worked with dismissed it as "not a bug," the list goes on and on.

I found very few libraries or frameworks for SSL.  The only real alternative to the classes in the JDK is BouncyCastle, but I found BC to be very poorly documented (there is a one page "User Guide" that basically points you to some examples and the JavaDoc).  

Two frameworks that implement SSL are Apache Mina and Netty.  Interacting with Netty was were I had the "this is not a bug" experience.  I am dreading the day that I have to work with the Mina folks.

Examples with SSL are few and far between.  Many problems I just couldn't find an answer to.  I even posted a problem on Stack Overflow, expecting a dozen message with title like "Try THIS, bonehead" but no one replied.

As with all my experiences, your own experience may vary from mine.

Java and SSL: What's in this Section?

This is the first of a series of posts about the worlds of Java and SSL.  I hope to do 1 post a day on this topic.  The resulting posts will become the basis for another section of a talk that I am scheduled to give on August 10 at the Boulder/Denver Cybersecurity Meetup.

SSL and Java

  • If you have a Java SSL expert...
  • How to create a new Certificate Authority
  • How to create a key pair in Java
  • How to create a certificate in Java
  • How to sign a certificate in Java
  • How to use a certificate in Java
  • Putting it all together: how to create a web service that uses SSL



Wednesday, June 21, 2017

Slides up on Slideshare

I just put my slides from my talk at DOSUG on Slideshare.  You can get them at https://www.slideshare.net/ClarkHobbie/miranda-77154415.

A Talk on Security

Recently, I asked to speak at the Denver/Boulder Cybersecurity Meetup. Pattie Kettle was kind enough to pencil me in for the August 10 meeting.

This leaves me with a problem.

My talk at DOSUG was around 40 minutes and doesn't address security directly.

So what I've decided to do is to add an "optional" section on SSL with Java.  I will use the five core topics (what is Miranda, why was it created, how it works, why it's reliable, and why it's secure) and then add on the "extra" topic. This should bring the talk up to 50 minutes, which should be acceptable.

I will blog about SSL and Java in the coming days.

Tuesday, June 6, 2017

Presented at DOSUG

(Finally) presented at DOSUG.  Slides available at ltsllc.com/talks as a PDF at http://ltsllc.com/talks/Miranda.pdf and as a PowerPoint presentation at http://ltsllc.com/talks/Miranda.pptx.

Tuesday, May 30, 2017

Slides up

For my legions of readers (hi Mom!) a pdf of my sides is up on the ltsllc site.

Sunday, May 7, 2017

Certificate Authorities

When a new instance of Miranda is created the user(s) need to create a new Certificate Authority (CA).  The CA can be self-singed or signed by a recognized authority like Verisign.  The CA signs the certificate that a Miranda node is required to present upon joining the cluster.

This authenticates the server to the clients and validates a client wanting to join the cluster.

The clients need the CA's certificate so that when they communicate with the server, to publish a new event for example, they can establish an SSL connection.  The other severs need the CA's certificate to verify other sever's certificates when a remote system tries to join the cluster.

The CA's certificate thus has to be added to all the client trustores  as well as all the server trustores.

Saturday, May 6, 2017

2 Phase Commits

Events and Deliveries  are controlled by remote policies, but creates/updates and deletes of User, Topics and Subscriptions go through a 2 phase commit.

The node that initiated the operation takes on the role of coordinator

During phase 1, nodes agree that a write is OK and prepare to go through with the operation (commit) or abort the operation.  If any node signals that they are not ready then the coordinator should abort the operation.

During phase 2 the coordinator tells the rest of the node to either commit or abort.  Assuming all the nodes responded that they were ready during phase 1, the coordinator signals that the nodes should commit.

As a concrete example, suppose an admin wants to add a new user to the system.  The node that the admin contacts acts as the coordinator.  The admin supplies the required information and the system announces it's intention to add the user to the rest of the nodes.  In this case, the other nodes check to see if they already have a user with the same name. Assuming they don't they signal that they are ready to add the user.  The coordinator issues a commit and the new user is added to the system.

Monday, May 1, 2017

Cross Origin Resource Sharing (CORS)

When I decided to split Miranda into two parts, I had to deal with CORS.  This is because the pages reside in one origin, but the servlets come from another.  It turns out that a servlet needs to implement the HTTP OPTIONS method and return the following headers:

Allow: *
Access-Control-Allow-Origin: *;
Access-Control-Allow-Headers: origin, content-type, accept, authorization, Access-Control-Allow-Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, HEAD
Access-Control-Max-Age: 1209600

The servlet itself needs to return the following header:

Access-Control-Allow-Origin: *

The following header needs to be added to the request:

Access-Control-Allow-Origin:





Sunday, April 30, 2017

To Divide or not to Divide?

I was thinking of making the part of Miranda that serves web pages separate from the rest of the system.

On the pro side this makes Miranda a bit more secure and reliable.  The web side can go down without bringing down the rest of the system.  If an attacker gains control of the web side the rest of the system is OK.

On the con side, Miranda will still have to host servlets, and they will probably be hosted by the same software as the web pages (Jetty).

Decisions, decisions.

I will divide Miranda into two parts because it offer more security and flexibility.  The Miranda side can use something that is optimized for servlets while the web side can use something optimized for HTML.

Saturday, April 29, 2017

Miranda & Logins

Miranda doesn't do passwords.

This is partially for security reasons (a system that deals in password may not be secure) but mostly because passwords are a pain.

Passwords are a pain because to do them properly, you can't store them in cleartext (an attacker who gained access to the system could get them) and, even encrypted or hashed they are still sensitive information.

Since Miranda already requires user to register a public key it uses that.  When a user "logs in" they simply provide their user name.  The system determines whether they already have an active session, If they do, the system returns it.  If they do not have an active session the system creates one.

When the system creates a session, it uses a random, 8-byte integer to identify it, the session id. When the system hands the session id back to the user, it first encrypts the value with the user's public key.  When the user gets this encrypted value, they decrypt it with their private key.

The user supplies the session id with all their requests, so the system validates their identity.

It is extremely unlikely that an attacker could guess a valid session id, and most operations could be modified to limit the number of failed attempts.  The exception are new events which require the system to process them as quickly as possible.

The alternative  is to detect when a large number of failed uses of session ids has occurred. Something for my todo list!

Tuesday, April 18, 2017

Introducing Miranda: Certificate Authorities

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop. At this point, I'm filling in the gaps so things may skip around a lot.

By default, you define a new certificate authority when you install Miranda.  The new certificate authority signs the certificates of all the Miranda nodes so that they can join the cluster.

When a node tries to join the cluster, it is asked to present a certificate. The cluster checks that the certificate is signed by the certificate authority before the node is allowed to join the cluster.  When clients contact the system with the web interface, this is also the certificate that is used by SSL/TLS.

The certificate authority itself can be signed by something like Verisign or it can be self-signed.  The default is to use a self-signed certificate.  This allows people to take Miranda for a "test drive" without requiring them to get a certificate first.


Sunday, April 16, 2017

Operations

Everything goes through the Miranda object.  This is because the Miranda object knows to tell the rest of the cluster about things like new Sessions and new Events. This also makes for a cluttered Miranda class since it has to know about the details of lots of operations.

The solution, as I see it, is to create a temporary subsystem for each operation: an Operation class. An object capable of receiving messages is required, hence a subsystem, and it knows about the details of the operation, thus the Miranda class is less cluttered.

The first of these operations is the login operation.  It is created when the Miranda object is asked to perform a login.  It looks up the user and creates a session for them.  If the user doesn't exist, then it signals this and terminates.

Saturday, April 15, 2017

Introducing Miranda: Frequently Asked Questions

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  I thought I would pose and answer some miscellaneous questions.
  • Why didn't you develop Miranda at Pearson?
  • Does Miranda work with AWS?
  • What is in store for the future of Miranda?
  • Miranda really needs feature X!

Why Didn't you Develop Miranda at Pearson?

My managers were always talking about doing this in the next release.  When that rolled around they would talk about the one after that and so on.

Does Miranda Work with AWS?

Miranda was designed with AWS in mind so it should work with AWS.

What is in Store for the Future of Miranda?

That depends on whether it takes off as open source.  For right now, I am planning on adding some sort of rewriting syntax for subscriptions.

Miranda Really Needs Feature X!

If a feature is asked for enough times, I will added it to the system.  If you want something added right now, then you can fork Miranda on github. If you want me to do it, then you can hire me as a consultant to move your feature to the front of the list.

Friday, April 14, 2017

Introducing Miranda: How it Works Revisted

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  After going through a dry run, it struck me that I talked too much about the nuts and bolts of Miranda, and not enough about how it work generally.  Therefore, I am redoing the "How it works" section to cover these issues better.

Miranda works by making the web service behind it appear to be more reliable than they are.  It does this by sitting in front of the web service and accept Events on its behalf.  Miranda itself is a distributed, fault-tolerant system and is very reliable.

Later, when the service is ready, Miranda delivers those events.  Thus the underlying service does not have to be functional all the time, Miranda will accept Events for it while it is down.  If an event causes the service to crash, Miranda can put the Event aside after the problem is fixed.

Miranda stores Events on a cluster of systems. If one node fails, the other nodes have a copy of the Event.  Nodes take responsibility for delivering Events during an "Auction".  If a node that is tasked with delivering events to a service goes down, the Subscription is Auctioned off again to another node.

Wednesday, April 12, 2017

Introducing Miranda: Miranda Asks for a Password at Startup

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

Passwords present a bit of a problem for the system.  Storing them in a file makes them insecure, but the system needs them to encrypt and decrypt files.  The solution that Miranda uses is if it lacks a password, to ask the user for it as part of its startup routine.

This does require a human being to start the system, but it will hopefully be infrequent enough that it will not be too much of a pain.

Introducing Miranda: Miranda is Distributed

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

One of the design goals of Miranda was to make it able to cross availability zones.  This was due to a limitation of the previous system, Prospero.  Prospero had trouble crossing availability zones because the database it used, Mnesia, did not like latency.  Furthermore, Prospero used RabbitMQ a lot and every round trip meant network delays.

Miranda does not use Mnesia or RabbitMQ, so it does not have these problems.

If a hurricane takes down a data center, the remaining nodes will keep going.  Subscriptions that the lost node was responsible for will be distributed among the remaining nodes.

When the down node comes back online, the system will "fill it in" on what happened while it was down. More specifically, when a node joins the cluster, it sends and receives the SHA1s of the cluster, users, topics, subscriptions, events and deliveries of the system.  If the SHA1 that it has locally does not match a remote SHA1, then the system tries to merge the remote file with its local file.

When a node joins or leaves the cluster the online nodes hold an election.  During the election, the subscriptions are distributed to the various nodes of the cluster.

When an Event or Delivery takes place it is shared among the members of the cluster. That way, any node can host any Subscription.

Changes to Users, Topics, and Subscriptions are also shared among the cluster members.

Tuesday, April 11, 2017

Introducing Miranda: How it Works

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop. At this point, I'm filling in the gaps, so things might jump around a bit.

Miranda works by

  • Sitting in front of a web service and accepting HTTP Events on its behalf.
  • It then delivers those Events to the web service when it is available.
  • The admin user defines other Users.
  • Users define Topics.
  • Other Users Subscribe to the Topics.
  • Users send HTTP Events to those Topics
  • Miranda delivers those Events as part of a subscription.

Introducing Miranda: Why Miranda Wont Crash

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

Miranda wont crash for the following reasons

  • Miranda is distributed.  It can lose one availability region and keep going.  When a node does come up, it will receive all the information it missed.
  • Each subsystem runs in its own thread. If one thread crashes, the other threads can keep going.
  • Miranda uses panics instead of System.exit.  That way, a lone method cannot terminate the system.
  • Miranda catches Exception.  So an uncaught exception wont crash the system.
  • Miranda has lots of tests. So a bug is unlikely to cause trouble.

Monday, April 10, 2017

Introducing Miranda: Why You Want Miranda

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

I have gone over a lot of topics concerning Miranda including:

  • The motivation for creating it
    • We want 9 9s of reliability
    • But we don't want to pay for it
    • Prospero gives us between 5 and 6 9s of reliablity but has problems
    • Miranda was created to address those problems
  • How it works
    • Admins create a local certificate authority
    • Admins create users
    • Users create Topics and Events
    • Users create Subscriptions
    • Miranda delivers Events to users
  • Why it wont crash
    • Miranda is distributed and fault-tolerant
    • Each subsystem runs in its own thread
    • Threads catch Exception
    • Miranda uses panics instead of System.exit
  • Why it's secure
    • Opensource
    • Miranda uses SSL/TLS for all communication
    • Miranda encrypts all files
    • Miranda asks for a password at startup
The conclusion is to use Miranda when your in the "we want 9 9s of reliability" situation.

Introducing Miranda: Why Miranda is Secure

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

At some point, people will need a reason to trust Miranda, this post gives that reason.

Miranda should be trusted because it is open source, because it uses HTTPS/TLS for communications, and because it encrypts its files.

Open source is more trustworthy than closed source because you can see the code that is performing the operations.  Users can build the system themselves and see that there is no malicious code.

Miranda uses SSL/TLS to communicate with other nodes.  That way, an attacker cannot see what is going between nodes.  Miranda also uses SSL/TLS when clients send new Events to the system, so attackers cannot see Events sent to the system.

Miranda encrypts all its files, so if an attacker gets a hold of one, it wont do them any good.

Finally, Miranda has the capability to ask for a passpharse at startup, so users don't have to store any secure information in files.

Sunday, April 9, 2017

Introducing Miranda: Prospero and Miranda Security

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Prospero supported limited security.  Admins could connect to it, log in, create new users and perform other administrative operations with a web browser.  It used HTTP for everything, however, and all its communications were in plain text. It also used plain text to talk to RabbitMQ.

Messages (POSTs) sent to it had to be signed with a symmetric key.  Admins therefore had access to all the user keys and they are stored in plain text in the database.

Miranda is more serious about security.  When the system is first installed a new certificate authority is created.  This CA is used to sign the certificates that the various nodes present when they join the cluster.

All users have a key pair and to do anything, they must first create a session.  The session is a random, 8 byte integer that is encrypted with the user's public key when it is handed back to the user.

All communication going into Miranda is encrypted using SSL/TLS.  Communications coming out of Miranda depend on the subscription: it can be HTTP or HTTPS.

Introducing Miranda: Unit Tests

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Miranda has many Junit tests.  316 at last count.  This does not guarantee that there are no bugs, but it does make it more likely.

A typical Miranda test checks that a Message gets to the method expected and is processed as expected.

I have also been pleasantly surprised by how smoothly Mockito has worked.  With it, I can focus on one class to test at a time and mock everything else out.

Miranda testing has also been simplified by the use of BlockingQueues and Messages.  A class method typically does a few things, then sends out a Message in response to an event.  Thus the test has to check that it took those actions.  The tests tend to be simple things because of this.

Thursday, April 6, 2017

Introducing Miranda: Exceptions and Panics

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

In Java, it is generally a bad idea to catch unchecked exceptions.  This is because a unchecked exception usually means something Very Bad has happened, like running out of memory, and it's time for the program to terminate.

So why does Miranda catch unchecked exceptions?

First of all, Miranda only does this in a few places.  Most notably, Miranda catches unchecked exceptions in the main loop, where it is getting the next message and processing it.  Since Miranda isn't supposed to crash, this is Miranda's last line of defense against a runaway subsystem taking everything down.

Secondly, Miranda's usual response to an exception is to create a panic.  I got the term from my days with Unix, where the operating system would, when it got into a bad state, "panic" and shut down.  In Miranda, when a panic occurs, the system can decide to halt immediately, shut down, or try to keep going.

If Miranda tries to keep going, it keeps track of these "recoverable" panics, and stops if the become too numerous.

Introducing Miranda: Threads and Subsystems

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Miranda is no supposed to crash, ever.  With a system whose downtime per year is measured in minutes, the entirety of the system needs to be looked at.  For this reason, all subsystems run in there own thread.  That way, if one thread crashes the rest can keep going.

Steps have to be taken to make sure that a stray unchecked exception doesn't take down the system, but it does make things more stable.

This represents a problem from a language standpoint, since languages like Java are not good at communicating between threads.  Things like synchronized methods make this easier, but still not ideal.

This is why Miranda uses BlocingQueues and Messages when subsystems need to communicate. Using this model, a subsystem simply takes the next Message from its queue and processes it.  If no Messages are available, then it waits.  Subsystems process each message before going onto the next, so there are no interrupts.

All this makes Miranda reliable, but cumbersome.  The difficulty in developing for Miranda is eclipsed by the need for reliability.

Wednesday, April 5, 2017

Introducing Miranda: How it all Fits Together: Deliveries

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

When a new Event is created, the Subscriptions examine it to see if they are interested in it (actually, they just look and see if they are subscribed to the topic).  If a Subscription is interested, it sends the event onto a delivery thread with the subscriber's URL.  The delivery thread sends the event to the client, who responds with a 2xx to signal that they got the event.

Graphically:

Subscriber      Delivery        Subscription    Event           Delivery
                Thread                          Manager         Manager
|               |               |               |               |
|               |               New Event       |               |
|               |               |<--------------- font="" nbsp="">
|               Deliver(Event)  |               |               |
|               |<--------------- font="" nbsp="">
|POST           |               |               |               |
|<--------------- font="" nbsp="">
(2xx)           |               |               |               |
--------------->|               |               |               |
|               (success!)      |               |               |
|               --------------->|               |               |
|               |               (success!)      |               |
|               |               --------------->|               |
|               |               |               New             |
|               |               |               Delivery        |
|               |               |               --------------->|


|               |               |               |               |

Introducing Miranda: How it all Fits Together: New Events

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Putting it all together, when a client sends an Event (we'll use a POST for this example) to a Miranda Topic, the system records the new Event and starts a write.  At the same time, it "tells" the Cluster about the new Event.

Graphically (primitively) it looks like this:

Client          Event           Event           Writer          Cluster
                Listener        Manager
|               |               |               |               |
POST            |               |               |               |
--------------->|               |               |               |
|               new POST        |               |               |
|               --------------->|               |               |
|               |               write events file               |
|               |               --------------->|               |
|               |               new POST        |               |
|               |               ------------------------------->|
|               (UUID)          |               |               |
|               |<--------------- nbsp="" span="">
(UUID)          |               |               |               |
|<--------------- nbsp="" span="">
|               |               |               |               |

Tuesday, April 4, 2017

Introducing Miranda: Deliveries

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

A Delivery represents an Event which has been sent to a subscriber and accepted.

A Delivery consists of:

  • The Event the Delivery is associated with.
  • When the Delivery occurred.
  • An attempt id (UUID)
Deliveries are "batched" together into groups of 100 (default) to form a delivery file.

Deliveries are created when the system is able to send an Event to a subscriber and the subscriber responds with a 2xx result.

All Deliveries go through the DeliverManger, which is in charge of all Deliveries and deliver files.

Introducing Miranda: Events

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Events are the POST/PUT/DELETE messages that are sent to topics.

Events have attributes they are:

  • The type of event (POST, PUT or DELETE).
  • When the Event occurred.
  • Who created the event.
  • The Event contents (for POST and PUT).
  • An id (a UUID).
Events are "batched" together into groups of 100 (default) to form an Event file.

An Event is created when a User performs a POST/PUT/DELETE to a Topic.

Events are managed by the EventManager. When a new event is created, it goes through the EventManager.


Monday, April 3, 2017

Introducing Miranda: Subscriptions

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Subscriptions ensure that Events sent to a Topic also get sent to the subscriber.

They consist of the user that created them, a URL where the events should be sent, a liveliness URL, and some attributes.

Users who can create subscriptions are called subscribers.  They can use the web site or the API to create a subscription.

When an event is delivered to the URL associated with a subscription and Miranda gets back a 2xx result, it is called a Delivery.

A subscription has attributes that control its behavior such as:

  • What to do with failed Events.  Does the system keep trying to deliver the Event, or does it put the failed Event into a list of failed Events.
  • How does the system handle recording Deliveries?  Does it move on after a Delivery has occurred, or does it wait for the delivery to be written to persistent store?
  • How does the system deal with other nodes?  Does it wait for acknowledgement or for  the other nodes to write the Delivery?
The owner of a subscription or the admin can change it through the web site or the API.

A subscription can be deleted by the owner or the admin.  All messages that have not been delivered are lost.

Elections determine which node handles deliveries for the subscription.

If the system cannot deliver an Event for a subscription because it did not receive a reply from the URL it tries to contact the subscriber via the liveliness URL. If it fails to contact the liveliness URL, it keeps trying, waiting twice as long between attempts until it reaches a limit of 5 minutes (default).

It keeps trying the liveliness URL every 5 minutes until it gets a 2xx result, at which time it resumes delivery of Events.  Miranda keeps Events for one week (default) before discarding them.

Introducing Miranda: Topics

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Topics are endpoints that users send POST/PUT/DELETEs to.  They each have the name that they are created with and a UUID to identify them.

Other users can subscribe to a Topic to get the messages sent to it.

Topics can have attributes that control how messages are managed including

  • Whether a response is sent when an Event is received or when it is written to persistent store.
  • Whether the topic waits for the other members of the cluster to receive/acknowledge/write an Event.
Topics are created by users with that ability from the website or via the API.

The owner of a topic can modify it.

To delete a topic the owner of the topic must request removal through the web site or the API.  The Topic must have no subscribers.

The admin user can delete a topic with subscribers.  In that case, the subscriptions are also removed.

Sunday, April 2, 2017

Introducing Miranda: Sessions

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Before a person can do anything in Miranda they must login and create a new Session. The user supplies a user name and the system responds with an encrypted, random long value, a session. The session is encrypted with the user's public key.  The user must accompany all their requests with the (decrypted) session.

The SessionManager keeps track of all the sessions.  It also is responsible for telling the cluster about new Sessions.  A session lasts for one hour (default).  The SessionManger checks for expired Sessions every 5 minutes (default).  If a Session is used, its expiration time is adjusted to give it an hour from that point until it expires.  When the SessionManager expires a Session, it tells the cluster about it, so the Session will expire on all nodes.

Introducing Miranda: the Cluster

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

The Cluster is the collection of all the Miranda nodes that the local system knows about.

It acts as a repeater: when the system wants to tell the other nodes about something, like a new user or a new topic, then a message is sent to the Cluster.

Elections, where ownership of the systems subscriptions is decided, is done by the Cluster.

When a new node connects to the system, the Cluster is notified.

Saturday, April 1, 2017

Introducing Miranda: the Network

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

The Network subsystem facilitates communication between the different nodes of the system.

It uses integers (handles) to identify the various connections.  When the local system wants to send some data to another node it sends the handle along with the message.  The network looks up the connection using the handle and sends the message.

Handles keep the rest of the system from knowing too many of the details of the networking library. They were created during the period when Miranda was switching from Netty to Mina.

The Network subsystems also covers the part of the system that listens for new nodes.  When a new node connects, the Network tells Miranda and the Cluster about it.

Introducing Miranda: the Major Subsytems

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

The major subsystems of Miranda include:
  • Miranda
  • Network
  • Cluster
  • SessionManager
  • UserManager
  • TopicManager
  • SubscriptionManager
  • EventManager
  • DeliveryManager
The Miranda subsystem servers multiple purposes.  It is the crossroads for a lot of messages and it maintains the state for the system.  When a new node joins the cluster, the Miranda subsystem maintain where in the process of syncing we are.

The Cluster represents the other Miranda nodes as a whole. When something happens locally that we want to tell the other nodes about, like the creation of a new session, a message is sent to the cluster.

The Network subsystem is used to communicate with the other Miranda nodes in the system. When a node wants to send another node some data, it sends a message to the Network.

The SessionManager is responsible for Sessions in the system.  A Session is created when a user logs in to Miranda.  When that happens the session manager also tells the Cluster about the new session.

Each of the remaining subsystems manages a set of objects.  The UserManager manages Users and so on.  The other subsystems also have a file associated with it.  The file holds the collection, and the associated manager monitors it for changes.

The UserManager is also consulted during the logon process.

The Events and Deliveries managers are different in that they are in charge of directories instead of single files.  Events and Deliveries are batched together into files that the Events and Deliveries managers are responsible for.

Friday, March 31, 2017

Introducing Miranda: How Miranda Works 2

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Jumping back...

Miranda is composed of subsystems, each of which runs in its own thread.  The threads communicate by means of Messages and BlockingQueues.

A thread simply takes the next Message off the queue and processes it.

The various subsystems of Miranda each have a State object.  The states "know" how to process a Message (a method called processMessage), and common behavior, like shutting down or how to deal with a file change, is gathered into the superclasses of states.

Each state has a class it is partnered with that does the "real" work.

Introducing Miranda: Closing

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Reasech shows that people tend to remember the opening and closing of a presentation the most; so I am going to go into my closing early on.

Hopefully this talk has given you an idea of what Miranda can do for you and the confidence that it can do it.  Now the the decision you have to make is whether to pursue it further.

You download Miranda from GitHub at https://ClarkHobbie/miranda and perform a trial run.  The tools that you will need are there as well.  Setting up Miranda is relatively simple.

I would bid you to throw off the shackles of the Pager!  While Miranda may not eliminate the need for a pager altogether, it will give you the confidence to know that, if you take your time in responding to a page, it's not the end of the world.

Thursday, March 30, 2017

Introducing Miranda: How Miranda Works

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

So Miranda looks like something you might be interested in, but how does it work?

Mirada sits in front of your web service and accepts POST/PUT/DELETEs on its behalf.  It then sends those events to your web service when it is up.

When a client sends a message to Miranda that is called an Event.  The end point that they send the POST/PUT/DELETE to is called a Topic.   An Event is delivered to a client as part of a subscription. When Miranda gets a 200 response as a result of sending an Event to a client, it records it as a Delivery.  All these things are set up with Users.

Before doing anything, a User must login to the system and establish a Session.

Miranda operates as a cluster of nodes.  To do anything, a qurom of 2 nodes needs to be established. Each Topic can have different rules about when to recognize an Event.  The default policy is after Miranda receives a POST/PUT/DELETE and has forwarded the event to a qorum of other nodes, it responds to the client, telling them that it received the Event.

At the same time that a node is telling the other nodes about an Event, it writes the Event to the persistent store.

Events are kept for a configurable period of time but the default is a week.  Deliveries are also kept for a configurable period of and this also defaults to a week.

Wednesday, March 29, 2017

Introducing Miranda: 9s of Reliability

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

What do the various "9s" of reliability mean?  Here is a table with the amount of yearly  downtime that each level allows:

# of Nines Percenatgae Time
1 90% A month
2 99% 3 days
3 99.9% 8.8 hours
4 99.99% 52.6 Minutes
5 99.999% 5.3 Minutes
6 99.9999% 32 Seconds
7 99.99999% 3.2 Seconds
8 99.999999% 320 Milliseconds
9 99.9999999% 32 Milliseconds

So 1 9 any team could do if they have a working system.  If they even have a pager the person with it does not pay a lot of attention to it.

If the company is serious about it 2 9s is no longer a joke.  They definitely have a pager that people trade off on a weekly basis.  When the pager goes off. the person tries to respond in 20 minutes or less.  The person on call has the phone numbers of the rest of the team in case they need something.

At 3 9s, there is definitely someone with a pager and there have been very serious conversations about getting a control center, a la NASA, for the system. The system may or may not be distributed.

At 4 9s, there is a control room manned by very humor limited folks who have the numbers of each of the team members, and when something goes wrong, they call them.  The person on call switches off weekly, reminiscent of wearing a pager, and if there isn't a hot standby then there have been very serious conversations as to why not.

At 5 9s, there is a control room, manned be people who were too serious for the previous level.  The control person's duty is to switch over the system to a hot standby and to call the on-call person when a problem develops.  Being on-call is no joke, and when the pager goes off, the on-call person must respond within 10 minutes.

At 6 9s, things are insane.  The people who were too serious for 5 9s are manning the control room. There is a person whose sole duty is to decide whether to switch to the hot standby.  There are two levels of people on call, and each of them must respond within 5 minutes.

Levels 7,8 and 9 require varying levels of hardware to support them, and there is still a control room and lists of on-call people.  But the question becomes, if you are serious, what is this for?  At 7 or 8 9s it becomes hard to tell if the system has been down.  After all, people's internet connections are not 7 or 8 9s reliable.

Miranda Takes Systems with 1 or 2 9s and Makes Them Appear to have Between 5 and 6 Nines

Miranda is a distributed, fault-tolerant system designed to run behind a load balancer that accepts messages on behalf of the underlying system and delivers them when the system is up.

The underlying system can be down for hours or days while Miranda accepts message for it. Therefore, the system appears to have between 5 and 6 9s of reliability, when it has fewer than that.

Tuesday, March 28, 2017

Introducing Mindanda: the Next Step

In preparation for a talk I'm giving at DOSUG I'm going to post my thinking as it develops.

The next step for Prospero is Miranda.

Miranda builds on many of the concepts that Prospero had. Things like capturing HTTP POSTs, forwarding them to subscribers, etc. Miranda does what Prospero does and takes the next step.  In addition to capturing POSTs, it also capture PUT and DELETE.

Miranda also addresses Prospero's limitations. Prospero, for example, does everything in the clear. Miranda does everything in SSL/TLS. There are many areas that Miranda improves on Prospero.

Miranda works by sitting in front of, and recording for, web services.  When a client sends a POST/PUT/DELETE to an endpoint, Miranda records it.  Since Miranda is a distributed, fault-tolerant system it makes the underling web service to appear to be more reliable than it actually is.

Ideally, the service appears to be as reliable as Miranda is, and hopefully, that is very reliable.




Monday, March 27, 2017

Introducing Miranda: Prospero

OK, if a system with "9 9s of reliability" is unattainable with a sane budget, what is attainable?

The answer was Prospero.

Prospero began life as a "skunkworks" project with Erlang. While its initial goal may have been something else, it was used to make systems which only had 1 or 2 9s of reliability seem like they were up most of the time.

Clients would define a topic and then send HTTP POSTs to Prospero.  Other clients would "subscribe" to these topics and register a URL that Prospero would associate with them.  When a client POSTed Prospero would, in turn, POST to the URL that had been registered.

Prospero did its job pretty well, and Pearson later released it as an open source project with the name of "Subpub."

Chris Chew initially did most of the development.  I did a little bit in the way of XML processing.

Over the course of about 3 years of support, the following limitations were discovered:
  • It was difficult to find people with Erlang experience
  • Prospero was difficult to modify
  • Prospero was limited to one data center
  • Mnesia (the database that Prospero used) could become corrupted
  • Prospero had difficulty with binary messages
There were other problems like one down subscriber could cripple the system, or that Prospero would stubbornly refuse to send any additional messages until all of the current messages were dealt with, but these were the issues that really stuck with us.

Sunday, March 26, 2017

Update Properties

It is time, once again, to update the properties page.  Since I haven't updated in a while, there is lots to do.

Saturday, March 25, 2017

Introducing Miranda: Motivation

In preparation for a talk that I'll give in June at DOSUG I'm going to post my thinking as it develops. The first section has to do with the motivation for Miranda.  The next section deals with how Miranda works and the last sections sums up.

So without further ado here's what I came up with:

The first section deals with the motivation for and the origins of the predecessor to Miranda: Prospero. It talks about how our boss wanted "9 9s of reliability" and how this works out to 30 milliseconds of downtime per year!  For the curious, I did some tests...

A ping of google.com yielded an average of time of 100ms.
micosoft.com and ibm.com timed out.

A system with "9 9s of reliability" is unattainable for the average company because it would cost too much.  You would need highly available hardware, several sites, a distributed, formally verified system, and hundreds if not thousands of developers.

Consider that the system would need to run on special hardware that has at least two "nodes" each of which has its own CPU, memory, disk and network connection.  Each node sends a "heartbeat" to the other nodes to ensure they are up. Remember that a failing node needs to switch over in less than 30 milliseconds when a fault occurs or the show's over!

Just the system that monitors the actual system would be challenging to design!

Such a system could, however, be built.

It would just probably cost billions of dollars, take years to develop, and require hundreds if not thousands of developers.



Friday, March 24, 2017

Why not Prosepo? Why not Modify Prospero?

Up until now, I have been talking about the limitations of the Prospero system.  But wouldn't it be easier to just modify Prospero instead of starting from scratch?  That's what this post is about.

I had to create a new system because:

  • A total rewrite was required to get away from Erlang
  • Prospero is tightly tied to Mnesia
  • Prospero is not well documented

A Total Rewrite was Required to get away from Erlang

A limitation of Prospero was that finding people with Erlang experience was difficult or impossible. Using a language like Java means a total rewrite anyways.

Prospero is tightly tied to Mnesia

The database that Prospero uses, Mnesia, is very tightly coupled to Prospero.  One of Mnesia's limitations is that it doesn't cross availability zones well.  Getting Prospero to work with multiple availability zones would require replacing Mnesia - a major rewrite.  Given that situation, a total rewrite would not require too much additional effort.

Prospero is not well Documented

While Prospero (Subpub in its latest incarnation) is widely used it is not well documented in terms of comments and the like.  The amount of effort required to document Prospero would approach the effort required for a rewrite.

Thursday, March 23, 2017

Why not Prospero? Miscellaneous

This is part of a multi-part discussion of why I started the Miranda project.  This post is a grab-bag of issues that I had with the original system.

Prospero Only Records HTTP POSTs

The original system only captures HTTP POST events. It didn't forward PUT and DELETE.

Prospero Only Uses HTTP for New Events

You could not send new events via HTTPS.  Furthermore, all communications between nodes was unencrypted, making it unsuitable for a non-secure environment.

Prospero Uses Symmetric Key Encryption

This meant that administrators knew all the keys; and if an attacker gains access to one table, they get all the client messages.

Miranda Addresses All these Issues

Miranda forwards HTTP PUT and DELETE, as well as POST.

Miranda does everything using SSL/TLS dealing with the 2nd issue.

Miranda uses public key encryption instead of secret key encryption. Thus administrators don't actually have client keys and an attacker gains no advantage by getting access to the keys that clients use.

Wednesday, March 22, 2017

Why not Prspero? Availability Zones

Prospero cannot work across availability zones: so you cannot have a West coast data center and an East coast data center work together.  This is, admittedly a rare problem: only a few companies even have a data center these days, but with the advent of services like AWS people want this capability for their systems.

Prospero relies on a distributed database called Mnesia, and Mnesia does not deal well with long delays, such as those you might run across in coast-to-coast operations.  In addition Prospero uses RabbitMQ, which also wants all its nodes to be in the same data center.  For these reasons, Propero is limited to one data center.

This was a problem.  For a mission critical application to go down if you lose one data center is bad. If a hurricane takes down a data center for a week this could be a very bad thing indeed.

We never did come up with a solution, but Miranda is designed for distributed use.

Tuesday, March 21, 2017

Why not Prospero? Erlang

This is the first part of a multi-part discussion of why I started the Miranda project.  This post goes into the limitations of the language that Prospero was written in, Erlang.

Erlang was created at Ericsson (a telecom company based in Stockholm, Sweden) in the 80s and released to the world in the 90s. It is used with "soft real-time" (where you can occasionally miss a deadline).

Two major drawbacks to Erlang are that it is hard to find people with Erlang experience and it can take several months for someone used to a language like Java to become proficient in Erlang.

It is Hard to Find People with Erlang Experience

Unlike C++, Java or C#, it is much harder to find people who have experience with Erlang.  At Pearson in Denver, for example, we had to contract with a firm in Europe to get support for Ejabberd, a chat program written in Erlang.

When we tried to find new members for the team to support Prospero, we had to dispense with Erlang experience as a requirement because nobody had it.

Erlang is basically a niche language in this country, with few adherents.

It is Hard to Train People in Erlang

As a rule of thumb, it would take several months before a developer was "up to speed" with Erleng.

In learning Erlang, one had to learn a different style of development called Functional Programming. An important difference with Functional Programming is that there are no variables - so a statement like "i++" should not be supported in a functional language. This is very different from traditional (imperative) languages, and takes awhile to get used to. 

Erlang syntax is also very different from the various "C-like" languages.  For example, if expressions in Erlang cannot have function calls and are seldom used.

The difference in programming styles and syntax combine to make Erlang a difficult language to pick up.

Erlang also has Good Points

Erlang also has its good points like light weight processes.  I once created a program that used several million threads (called processes in Erlang), but I couldn't do the same thing in Java.  At several thousand threads the VM wanted more memory than the system had.

Monday, March 20, 2017

Why not Prospero?

Prospero, the predecessor to Miranda, was a system written by Chris Chew (with a little help from me) in Erlang to capture HTTP POSTs and play them back to interested parties.

The question is this: why not just use Prospero?  Why go to the trouble of writing a new system?  I will answer this question in a series of posts.  This is partially because the answer is that complex, but also so that I have something to talk about for the next few days.

In broad strokes, here is the answer:


  • Prospero is written in Erlang
    • It is hard to find people who have experience with Erlang
    • It is hard to train people to use Erlang
  • Prospero cannot cross availability zones
    • Mnesia limitation
  • Prospero depends on many other systems
    • Erlang
    • RabbitMQ
    • Mnesia
  • Why not modify Prospero?
  • Misc reasons
    • Prospero only deals in POSTs
    • Prospero only runs over HTTP, not HTTPS

Saturday, March 18, 2017

Hierarchical States

Miranda use hierarchical states.

Hierarchical means that behavior can be shared across several classes.  This is very useful if you are sharing files, as Miranda does, and you don't want to write the same code over and over again.

I first heard about hierarchical states with Harel Statecharts, but I later used ROOMcharts (part of the late, Real-Time Object-Oriented Modeling methodology that later got absorbed into UML) because I liked them better.

The nice thing about hierarchical states are that you can define a behavior in a base state and all states that extend it get that behavior as well.  In the case of Miranda, the State class, that all state classes extend, responds to the stop message.  That way, all classes "know" how to stop.

Some classes do a bit more.  The ToipicFile class checks to see if it needs to be written, while the Node class, which represents different nodes in the cluster, needs to disconnect before shutting down. The state classes for these objects know to watch for a shut down message, and behave differently in those cases.

Friday, March 17, 2017

Jetty

I using Jetty for HTML and servlets for Minda.  So far it has been pretty. 

Uh oh, I probably just jinxed it.

I always envisioned a web interface for Miranda - to do things like check status, add or modify users and for administering topics.  

I'm using Goolge's angularjs with this - which is probably overkill.  But I took an online class on angular so I'm going to use it.  So far things have worked out well.  The main problem is that the web site looks like something I would create: clunky and lacking in grace.

Thursday, March 16, 2017

Mockito for the Win

I have not been doing a lot of work on Miranda this week.  I wanted to say that Mockito has worked out well, however.  It has simplified tests, particularly for states, by providing mock objects for Consumers.

Tuesday, March 14, 2017

It's Alive2: SSL

Miranda seems to work with SSL (actually TLS).

I didn't actually have to change anything, I just regenerated the serverkeystore and truststore files; and it seemed to work!

This is an important milestone for the Miranda project.  I have spent several weeks trying to get SSL/TLS working.  The key was being able to swap out Netty with Mina.  For some reason, Netty, was giving me the "Invalid signature" error and I couldn't get it to work.  Fortunately, Mina was a bit more forgiving.

Sunday, March 12, 2017

It's Alive!

Today I got two nodes to talk to each other.  Granted it was through an unencrypted channel but still...I felt like some sort of mad scientist: "Do you hear me Egor?  It's aaaaalliiiivveeeee!

Thursday, March 9, 2017

Well at least Mina Works

I  recently opened a defect on Netty. While they responded very quickly the person that I worked with insisted that it was not a bug and that something with the certificates was being messed up.

I don't know how to get things working with Netty, but things work with core Java and Mina, so it looks like Miranda will be going with Mina.

Wednesday, March 8, 2017

A Bug for Netty

I have put this off for as long as I can, it's time to post a bug to the Netty project.

As many readers of this blog will know, I have had a rocky relationship with Netty. The turning point came when I saw what a sorry state SSL/TLS was in when it comes to Java.  I then posted an apology to Netty.  Nevertheless Netty may have some problems.

In particular, my attempts to use Netty with a "local certificate authority" have met with failure.  I posted my problem to Stack Overflow in hopes of finding a solution there, but after a week no one has voted for the question or (aside from Jim Garrison) commented on the question.

Since that time I discovered another networking framework called Mina.  I have created a new test up on GitHub that uses Mina and seems to work.

I have been working with someone from Netty regarding the issue, so far without result.

Tuesday, March 7, 2017

Mina Seems to Work

I tired using apache mina in a test program (up on github at https://github.com/ClarkHobbie/ssltest3) and it worked.  I have posted a bug to the Netty project and see what they do with it.  In the mean time I will use apache mina.

This is not an indictment of Netty, I will be able to switch over if the folks there figure out what is going wrong.

UPDATE

The folks at Netty (specifically, normanmaurer) got back to almost immediately and asked me to try 4.1.9.  I did and it still had problems, but I am very impressed with how fast they got back to me.


Monday, March 6, 2017

Basic Use Cases


This represents the basic use cases that Mirada needs to implement in order to be useful.
  • Log in as admin
  • Create a user
  • Create a topic
  • Create a subscription
  • Create a message
  • Create a delivery

Log in as Admin

  • The user connects to the system with a browser
  • The system asks the user to log in
  • The user supplies their credentials
  • The system responds with the status page and gives the user a cookie

Create a User

  • The user goes to the users section of the app
  • The system presents the user with a menu of operations
  • The user indicates create a user
  • The system presents the user with the new user form
  • The user completes the form and presents it to the system
  • The system creates the new user 
  • The systems tells the other node about the new user
  • The system receives acknowledgement from the other node about the new user
  • The system reports success to the user

Create a Topic through the UI

  • The user goes to the topics section of the app
  • The system presents the user with the topics status screen
  • The user indicates they want to create a new topic
  • The system presents the user with the new topic form
  • The user completes the form and presents it to the system
  • The system create the new topic
  • The system tells the other nodes about the new topic
  • The system receives acknowledgement about the new topic from the other nodes
  • The system reports success to the user

 Create a Subscription by Sending a POST

  • The client posts a new subscription
  • The system checks the client's session
  • The client's session is valid
  • The system checks for duplicated subscriptions
  • The new subscription does not duplicate another topic
  • The system creates the new subscription 
  • The system tells the other nodes about the new subscription 
  • The system receives acknowledgement about the new subscription from the other nodes
  • The system reports success to the client

Create a Subscription through the UI

  • The user goes to the subscriptions section of the app
  • The system presents the user with the subsriptions status screen
  • The user indicates they want to create a new subscription 
  • The system presents the user with the new subscription form
  • The user completes the form and presents it to the system
  • The system checks for duplicated subscriptions
  • The new subscription does not duplicate another subscription 
  • The system creates the new subscription 
  • The system tells the other nodes about the new subscription 
  • The system receives acknowledgement about the new subscription from the other nodes
  • The system reports success to the user

Create a new Event

  • The client does an HTTP POST to a topic
  • The system checks the client's session
  • The client's session is valid.
  • The system records the data of the client's POST to a new event
  • The system tells the other nodes about the new event.
  • The system writes the new Event to the persistent store
  • The system reports success to the client

Deliver an Event

  • A new Event comes in
  • The system checks to see what subscriptions are interested
  • At least one subscription is interested
  • The system hands the Event off to the Subscription for delivery.
  • The Subscription notes the Event to persistent sore
  • The Subscription delivers the Event to the subscriber.
  • The system creates a new Delivery object
  • The system adds the Delivery to the subscription
  • The Subscription starts writing the Delivery to the persistent store
  • The system tells the other nodes about the Devlivery

Sunday, March 5, 2017

To Post or not to Post

I have been taking an online course on agularjs in the hopes that I could use it on the admin side of Miranda.  During my setup of the web site (I'm using Jetty btw) I discovered that Chrome doesn't like my certificate authority either.

I was planning on posting a bug to Netty regarding my difficulties in using a local CA, but I think I will wait until Chrome likes my web site.

Saturday, March 4, 2017

State Classes: Where the Brains are in the Miranda System

I decided to use external states in the Miranda system. Because Miranda is a multi-threaded system this is what I call a Big Deal.

What this means, in practical terms, is that each thread reacts differently depending on what state it's in.  For example, when a node is waiting for a "join" message, it reacts differently to a "join" message then when it is trying to connect.

What this boils down to is that, most of the behavior logic goes in the state classes and what I normally think of as the class itself does very little.

Time will tell if this was a good move or A Very Bad Idea.

Friday, March 3, 2017

Exceptions and Panics

Miranda should never stop.  Ever.

This means my blithe strategy of printing a stack trace and calling System.exit in response to an exception cannot hold.  Instead, most threads must continue and instead throw a Panic when things get really bad.

A Panic is like an exception, except that the system treats a panic as a request to shut down instead of automatically printing a stack trace and exiting.

I have also created a panic method on the Miranda class.  If the method returns at all (the default behavior is to call System.exit but this will change) it returns a boolean to indicate if the system as a whole is panicking, or if the caller should try and keep going.  The system may attempt to shutdown instead of calling System.exit so correctly handling the response to a call to Miranda.panic is important.


Thursday, March 2, 2017

Getting a Handle on Things

Given that

  • netty TLS is giving me grief*
  • Socket based SSl/TLS seems to work
I have decided the following:
  • All network communication (send/receive) will go through the network object
  • All network communications will use a handle: nothing will use the network directly
  • For sockets, all receives will use their own thread
I'm no too crazy about the last point, and in particular I don't know if you can ask a socket to send a message while you already issued a receive.  We shall see.

The good news is that this should make testing easier: just call a method and see what messages end up in the various queues.

* = After looking at the state of non-blocking, secure (i.e. SSL/TLS) I/O in Java, I changed my opinion of netty; but it still may have bugs.

Apologies to netty

Yesterday I saw just how bad SSL/TLS was for nio and java and realised just how much work the folks who put out netty really have to do.  Up until now, I have been frustrated with netty only to gain respect for anyone who will deal with SSL/TLS using non-blocking I/O in java.

So in summary I am sorry for my attitude up until now and will try to do better.

Wednesday, March 1, 2017

Who is Responsible for nio TLS?

And I thought netty was bad...

It doesn't hold a candle to nio TLS...

Consider this link. My god, you would need to be a TLS expert to use it!  And this is from Oracle...

Putting the reasons aside for the moment, it seems clear that

  • nio TLS is non-trival to use
  • There are very few libraries available
And this is after 10 years!

I am speechless.  Either developer are not using SSL/TLS with java, or I am missing something.

Tuesday, February 28, 2017

New Versions of SSLTest on GitHub

I got the netty and non-netty versions of SSL test up on github at

https://github.com/ClarkHobbie/ssltest

(netty version)

https://github.com/ClarkHobbie/ssltest2

(non-netty version)

All hail netty!

Excelsior!

I have finally gotten TLS to work.

With the example up on GitHub that you can get from:

https://github.com/ClarkHobbie/ssltest2

My SSL/TLS test works!

Admittedly, this is without netty but still.  If I can't get netty to work then at least I have that as a backup.

I will spend the rest of the day adapting my example to work with netty.

Monday, February 27, 2017

Wasting Away Again in TLSville


I spent (wasted) the day trying to get TLS working.

For the record, here are the commands for creating the keys:

openssl req -x509 -newkey rsa:2048 -keyout ca-key.pem.txt -out ca-certificate.pem.txt -days 365 -nodes
keytool -import -keystore truststore -file ca-certificate.pem.txt -alias ca  -storepass whatever
keytool –keystore serverkeystore –genkey –alias server -keyalg rsa -storepass whatever
keytool –keystore serverkeystore -storepass whatever –certreq –alias server  –file server.csr
openssl x509 -req -CA ca-certificate.pem.txt -CAkey ca-key.pem.txt -in server.csr -out server.cer -days 365 –CAcreateserial
keytool -import -keystore serverkeystore -storepass whatever -file ca-certificate.pem.txt -alias ca
keytool -import -keystore serverkeystore -storepass whatever -file server.cer -alias server

I have developed a simpler program that doesn't use netty.  For the interested, I have put it up on Github at

https://github.com/ClarkHobbie/ssltest2

SSL/TLS can have an overpowering lure to it, and cause me to waste time trying to fix it; hence the wasted day.  My posting to Stack Overflow has gotten neither votes nor help, leading me to believe that if anything is going to happen with this problem, I will have to do it.

All hail netty!

Sunday, February 26, 2017

Testing...Again

One situation that I came across while writing tests for Miranda is whether I should repeat myself. This is exemplified with in the file TestFileWatcherService.java.  The problem is that the methods testCheckFiles, testFireChanged and testWatch are all the same.

The thing is that testCheckFiles also tests the fireChanged and watch methods, so these additional tests are redundant.  The question then becomes whether to repeat tests that do the same thing, or just do them once.  At first, I replicated code, but now I'm not so sure.

Going forward, I will collapse tests that do the same thing into one, since I have found that less code is a Very Good Thing.

TLS still doesn't work (see my post on Stack Overflow).

All hail netty!

Saturday, February 25, 2017

To Mock or not to Mock

As part of the whole testing process, I find myself in situations where I could use a mock objects or the real thing.  I have already made some Big Mistakes by using other frameworks, and now I consider Mockito.

For those who are not familiar with it, Mockito is a framework for setting up mock objects.  It was made with my situation in mind.  The problem with it, is that you end up with blocks of code that have a lot to do with Mockito, but little to do with the system that you are testing.

On the other hand, I have ended up with large blocks of code that have a lot to do with testing Miranda, but little to do with Miranda itself. Thus the question remains: To use Mockito or  not.

I think that, at this point, I will use Mockito and if it causes any problems, I will take it out.

TLS still doesn't work. See my question on Stack Overflow for updates.

All hail netty!

Friday, February 24, 2017

The Invalid Signature Problem

For some time now, I have been dealing with a problem where connections don't work when I try to use a local certificate authority with netty and transport layer security (TLS).  The code for this problem is available on gihub at

    https://github.com/ClarkHobbie/ssltest

When I try and connect, I get the following exception:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLKeyException: Invalid signature on ECDH server key exchange message.

The complete commands are:

    java -cp target\ssl-test-1.0-SNAPSHOT.jar;netty-all-4.1.6.Final.jar Server

and

    java -cp target\ssl-test-1.0-SNAPSHOT.jar;netty-all-4.1.6.Final.jar Client

I have modified the program to work with "remote CAs" like google, running the program this way, doesn't work (google isn't setup to send messages), but I don't get the invalid signature exception either.

The complete command to run against google is:

    java -cp target\ssl-test-1.0-SNAPSHOT.jar;netty-all-4.1.6.Final.jar Client remote google.com 443

Turning off encryption entirely works, it can be done with the following commands:

    java -cp target\ssl-test-1.0-SNAPSHOT.jar;netty-all-4.1.6.Final.jar Server nossl

and

    java -cp target\ssl-test-1.0-SNAPSHOT.jar;netty-all-4.1.6.Final.jar Client nossl

I have tried a variety of things, none of which work.  If anyone knows of a solution, I'm all ears.  Till then I've posted a question on Stack Overflow at:

    http://stackoverflow.com/questions/42445115/invalid-signature-on-ecdh-server-key-exchange-message

All hail netty!

The complete exception is:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLKeyException: Invalid signature on ECDH server key exchange message
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:351)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:373)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:651)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:574)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:488)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:450)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:873)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLKeyException: Invalid signature on ECDH server key exchange message
        at sun.security.ssl.Handshaker.checkThrown(Unknown Source)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(Unknown Source)
        at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
        at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
        at javax.net.ssl.SSLEngine.unwrap(Unknown Source)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1097)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:968)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:902)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
        ... 16 more
Caused by: javax.net.ssl.SSLKeyException: Invalid signature on ECDH server key exchange message
        at sun.security.ssl.HandshakeMessage$ECDH_ServerKeyExchange.(Unknown Source)
        at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source)
        at sun.security.ssl.Handshaker.processLoop(Unknown Source)
        at sun.security.ssl.Handshaker$1.run(Unknown Source)
        at sun.security.ssl.Handshaker$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Unknown Source)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1123)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1008)
        ... 18 more

Miranda and Encryption

Why does Miranda want to use a certificate authority?  Why does Miranda use encryption at all?

Briefly, Miranda uses local certificate authorities to make it cheaper to use and easier to evaluate. Miranda uses encryption because events (messages) may contain things like personally identifiable information or other sensitive information.

The long answers are, well, longer.

First of all, if you don't have a requirement to encrypt things, then you can turn encryption off. Miranda was designed to be used on things like Amazon cloud, however, with traffic potentially going across the internet, so your events (messages) could be sent in the clear.  If you are comfortable with that arrangement, then you can simply turn off encryption.

Miranda uses local certificate authorities because all nodes in the cluster are required to have certificates.  It can quickly get expensive creating CERTs for every node in your cluster, not to mention inconvenient, with that approach.  Instead, you create your own certificate authority and use the local CA to sign all your node keys.

Miranda uses encryption because I found myself in situations where I wished that its predecessor, Prospero, did.  In particular, one of the obstacles to using Prospero in AWS was its lack of support for encryption.  Another problem was crossing availability zones.  If we had a node on the West coast, and another on the East coast, then they would probably talk across the internet.

As far as what to use, I thought SSL/TLS with their wide use, would be well supported, secure, cheap, and easy to use.  While they are indeed well supported, secure and inexpensive I have not found SSL/TLS to be at all easy to use.  I have run across a problem that has forced me to do all my development work "in the clear."  I refer to the dreaded "Invalid signature" problem that I posted on Stack Overflow about.

At any rate, that is why Miranda uses local certificate authorities and encryption in general.

All hail netty!