Java and SSL:Welcome to the Dark Ages!

This post is part of a series about the worlds of Java and SSL.  I hope to do 1 post a day on this topic. The resulting posts will become the basis for another section of a talk that I am scheduled to give on August 10 at the Boulder/Denver Cybersecurity Meetup.

• If you have an Java and SSL Expert..
• SSL is difficult in Java NIO
• Few tools are available
• Even Stack Overflow was no help

All I can say about experts in the field of Java and SSL is that if you have one...KEEP THEM!  I found the combination of Java (NIO) and SSL to be very difficult.  And the Java world has had over 10 years (NIO was releases in 2006) to fix this!

I found SSL to be ridiculously difficult in Java NIO.  For something as ubiquitous as SSL I was hoping to find it an easier going.  Oh boy was I wrong.  

I had to fight SSL every step of the way.  If things became easy, I immediately became suspicious.  If I tried to do something "simple" in SSL, all the examples that I found generated warnings when I tried to use them.  When I found what I deemed a bug in one library, the person I worked with dismissed it as "not a bug," the list goes on and on.

I found very few libraries or frameworks for SSL.  The only real alternative to the classes in the JDK is BouncyCastle, but I found BC to be very poorly documented (there is a one page "User Guide" that basically points you to some examples and the JavaDoc).  

Two frameworks that implement SSL are Apache Mina and Netty.  Interacting with Netty was were I had the "this is not a bug" experience.  I am dreading the day that I have to work with the Mina folks.

Examples with SSL are few and far between.  Many problems I just couldn't find an answer to.  I even posted a problem on Stack Overflow, expecting a dozen message with title like "Try THIS, bonehead" but no one replied.

As with all my experiences, your own experience may vary from mine.

Java and SSL: What's in this Section?

This is the first of a series of posts about the worlds of Java and SSL.  I hope to do 1 post a day on this topic.  The resulting posts will become the basis for another section of a talk that I am scheduled to give on August 10 at the Boulder/Denver Cybersecurity Meetup.

SSL and Java

• Welcome to the dark ages!
• How to create a new Certificate Authority
• How to create a key pair
• How to create a certificate signing request
• How to sign a CSR

Slides up on Slideshare

I just put my slides from my talk at DOSUG on Slideshare.  You can get them at https://www.slideshare.net/ClarkHobbie/miranda-77154415.

A Talk on Security

Recently, I asked to speak at the Denver/Boulder Cybersecurity Meetup. Pattie Kettle was kind enough to pencil me in for the August 10 meeting.

This leaves me with a problem.

My talk at DOSUG was around 40 minutes and doesn't address security directly.

So what I've decided to do is to add an "optional" section on SSL with Java.  I will use the five core topics (what is Miranda, why was it created, how it works, why it's reliable, and why it's secure) and then add on the "extra" topic. This should bring the talk up to 50 minutes, which should be acceptable.

I will blog about SSL and Java in the coming days.

Introducing Miranda: Certificate Authorities

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop. At this point, I'm filling in the gaps so things may skip around a lot.

By default, you define a new certificate authority when you install Miranda.  The new certificate authority signs the certificates of all the Miranda nodes so that they can join the cluster.

When a node tries to join the cluster, it is asked to present a certificate. The cluster checks that the certificate is signed by the certificate authority before the node is allowed to join the cluster.  When clients contact the system with the web interface, this is also the certificate that is used by SSL/TLS.

The certificate authority itself can be signed by something like Verisign or it can be self-signed.  The default is to use a self-signed certificate.  This allows people to take Miranda for a "test drive" without requiring them to get a certificate first.

Introducing Miranda: Frequently Asked Questions

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  I thought I would pose and answer some miscellaneous questions.

• Why didn't you develop Miranda at Pearson?
• Does Miranda work with AWS?
• What is in store for the future of Miranda?
• Miranda really needs feature X!

Why Didn't you Develop Miranda at Pearson?

My managers were always talking about doing this in the next release.  When that rolled around they would talk about the one after that and so on.

Does Miranda Work with AWS?

Miranda was designed with AWS in mind so it should work with AWS.

What is in Store for the Future of Miranda?

That depends on whether it takes off as open source.  For right now, I am planning on adding some sort of rewriting syntax for subscriptions.

Miranda Really Needs Feature X!

If a feature is asked for enough times, I will added it to the system.  If you want something added right now, then you can fork Miranda on github. If you want me to do it, then you can hire me as a consultant to move your feature to the front of the list.

Introducing Miranda: How it Works Revisted

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  After going through a dry run, it struck me that I talked too much about the nuts and bolts of Miranda, and not enough about how it work generally.  Therefore, I am redoing the "How it works" section to cover these issues better.

Miranda works by making the web service behind it appear to be more reliable than they are.  It does this by sitting in front of the web service and accept Events on its behalf.  Miranda itself is a distributed, fault-tolerant system and is very reliable.

Later, when the service is ready, Miranda delivers those events.  Thus the underlying service does not have to be functional all the time, Miranda will accept Events for it while it is down.  If an event causes the service to crash, Miranda can put the Event aside after the problem is fixed.

Miranda stores Events on a cluster of systems. If one node fails, the other nodes have a copy of the Event.  Nodes take responsibility for delivering Events during an "Auction".  If a node that is tasked with delivering events to a service goes down, the Subscription is Auctioned off again to another node.

Introducing Miranda: Miranda Asks for a Password at Startup

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

Passwords present a bit of a problem for the system.  Storing them in a file makes them insecure, but the system needs them to encrypt and decrypt files.  The solution that Miranda uses is if it lacks a password, to ask the user for it as part of its startup routine.

This does require a human being to start the system, but it will hopefully be infrequent enough that it will not be too much of a pain.

Introducing Miranda: Miranda is Distributed

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

One of the design goals of Miranda was to make it able to cross availability zones.  This was due to a limitation of the previous system, Prospero.  Prospero had trouble crossing availability zones because the database it used, Mnesia, did not like latency.  Furthermore, Prospero used RabbitMQ a lot and every round trip meant network delays.

Miranda does not use Mnesia or RabbitMQ, so it does not have these problems.

If a hurricane takes down a data center, the remaining nodes will keep going.  Subscriptions that the lost node was responsible for will be distributed among the remaining nodes.

When the down node comes back online, the system will "fill it in" on what happened while it was down. More specifically, when a node joins the cluster, it sends and receives the SHA1s of the cluster, users, topics, subscriptions, events and deliveries of the system.  If the SHA1 that it has locally does not match a remote SHA1, then the system tries to merge the remote file with its local file.

When a node joins or leaves the cluster the online nodes hold an election.  During the election, the subscriptions are distributed to the various nodes of the cluster.

When an Event or Delivery takes place it is shared among the members of the cluster. That way, any node can host any Subscription.

Changes to Users, Topics, and Subscriptions are also shared among the cluster members.

Introducing Miranda: How it Works

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop. At this point, I'm filling in the gaps, so things might jump around a bit.

Miranda works by

• Sitting in front of a web service and accepting HTTP Events on its behalf.
• It then delivers those Events to the web service when it is available.
• The admin user defines other Users.
• Users define Topics.
• Other Users Subscribe to the Topics.
• Users send HTTP Events to those Topics
• Miranda delivers those Events as part of a subscription.

Introducing Miranda: Why Miranda Wont Crash

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.  At this point, I'm filling in gaps, so things might jump around a bit.

Miranda wont crash for the following reasons

• Miranda is distributed.  It can lose one availability region and keep going.  When a node does come up, it will receive all the information it missed.
• Each subsystem runs in its own thread. If one thread crashes, the other threads can keep going.
• Miranda uses panics instead of System.exit.  That way, a lone method cannot terminate the system.
• Miranda catches Exception.  So an uncaught exception wont crash the system.
• Miranda has lots of tests. So a bug is unlikely to cause trouble.

Introducing Miranda: Why You Want Miranda

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

I have gone over a lot of topics concerning Miranda including:

• The motivation for creating it
• We want 9 9s of reliability
• But we don't want to pay for it
• Prospero gives us between 5 and 6 9s of reliablity but has problems
• Miranda was created to address those problems
• How it works
• Admins create a local certificate authority
• Admins create users
• Users create Topics and Events
• Users create Subscriptions
• Miranda delivers Events to users
• Why it wont crash
• Miranda is distributed and fault-tolerant
• Each subsystem runs in its own thread
• Threads catch Exception
• Miranda uses panics instead of System.exit
• Why it's secure
• Opensource
• Miranda uses SSL/TLS for all communication
• Miranda encrypts all files
• Miranda asks for a password at startup

The conclusion is to use Miranda when your in the "we want 9 9s of reliability" situation.

Introducing Miranda: Why Miranda is Secure

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

At some point, people will need a reason to trust Miranda, this post gives that reason.

Miranda should be trusted because it is open source, because it uses HTTPS/TLS for communications, and because it encrypts its files.

Open source is more trustworthy than closed source because you can see the code that is performing the operations.  Users can build the system themselves and see that there is no malicious code.

Miranda uses SSL/TLS to communicate with other nodes.  That way, an attacker cannot see what is going between nodes.  Miranda also uses SSL/TLS when clients send new Events to the system, so attackers cannot see Events sent to the system.

Miranda encrypts all its files, so if an attacker gets a hold of one, it wont do them any good.

Finally, Miranda has the capability to ask for a passpharse at startup, so users don't have to store any secure information in files.

Introducing Miranda: Prospero and Miranda Security

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Prospero supported limited security.  Admins could connect to it, log in, create new users and perform other administrative operations with a web browser.  It used HTTP for everything, however, and all its communications were in plain text. It also used plain text to talk to RabbitMQ.

Messages (POSTs) sent to it had to be signed with a symmetric key.  Admins therefore had access to all the user keys and they are stored in plain text in the database.

Miranda is more serious about security.  When the system is first installed a new certificate authority is created.  This CA is used to sign the certificates that the various nodes present when they join the cluster.

All users have a key pair and to do anything, they must first create a session.  The session is a random, 8 byte integer that is encrypted with the user's public key when it is handed back to the user.

All communication going into Miranda is encrypted using SSL/TLS.  Communications coming out of Miranda depend on the subscription: it can be HTTP or HTTPS.

Introducing Miranda: Unit Tests

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Miranda has many Junit tests.  316 at last count.  This does not guarantee that there are no bugs, but it does make it more likely.

A typical Miranda test checks that a Message gets to the method expected and is processed as expected.

I have also been pleasantly surprised by how smoothly Mockito has worked.  With it, I can focus on one class to test at a time and mock everything else out.

Miranda testing has also been simplified by the use of BlockingQueues and Messages.  A class method typically does a few things, then sends out a Message in response to an event.  Thus the test has to check that it took those actions.  The tests tend to be simple things because of this.

Introducing Miranda: Exceptions and Panics

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

In Java, it is generally a bad idea to catch unchecked exceptions.  This is because a unchecked exception usually means something Very Bad has happened, like running out of memory, and it's time for the program to terminate.

So why does Miranda catch unchecked exceptions?

First of all, Miranda only does this in a few places.  Most notably, Miranda catches unchecked exceptions in the main loop, where it is getting the next message and processing it.  Since Miranda isn't supposed to crash, this is Miranda's last line of defense against a runaway subsystem taking everything down.

Secondly, Miranda's usual response to an exception is to create a panic.  I got the term from my days with Unix, where the operating system would, when it got into a bad state, "panic" and shut down.  In Miranda, when a panic occurs, the system can decide to halt immediately, shut down, or try to keep going.

If Miranda tries to keep going, it keeps track of these "recoverable" panics, and stops if the become too numerous.

Introducing Miranda: Threads and Subsystems

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Miranda is no supposed to crash, ever.  With a system whose downtime per year is measured in minutes, the entirety of the system needs to be looked at.  For this reason, all subsystems run in there own thread.  That way, if one thread crashes the rest can keep going.

Steps have to be taken to make sure that a stray unchecked exception doesn't take down the system, but it does make things more stable.

This represents a problem from a language standpoint, since languages like Java are not good at communicating between threads.  Things like synchronized methods make this easier, but still not ideal.

This is why Miranda uses BlocingQueues and Messages when subsystems need to communicate. Using this model, a subsystem simply takes the next Message from its queue and processes it.  If no Messages are available, then it waits.  Subsystems process each message before going onto the next, so there are no interrupts.

All this makes Miranda reliable, but cumbersome.  The difficulty in developing for Miranda is eclipsed by the need for reliability.

Introducing Miranda: How it all Fits Together: Deliveries

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

When a new Event is created, the Subscriptions examine it to see if they are interested in it (actually, they just look and see if they are subscribed to the topic).  If a Subscription is interested, it sends the event onto a delivery thread with the subscriber's URL.  The delivery thread sends the event to the client, who responds with a 2xx to signal that they got the event.

Graphically:

Subscriber      Delivery        Subscription    Event           Delivery

                Thread                          Manager         Manager

|               |               |               |               |

|               |               New Event       |               |

|               |               |<--------------- font="" nbsp="">

|               Deliver(Event)  |               |               |

|               |<--------------- font="" nbsp="">

|POST           |               |               |               |

|<--------------- font="" nbsp="">

(2xx)           |               |               |               |

--------------->|               |               |               |

|               (success!)      |               |               |

|               --------------->|               |               |

|               |               (success!)      |               |

|               |               --------------->|               |

|               |               |               New             |

|               |               |               Delivery        |

|               |               |               --------------->|

|               |               |               |               |

Introducing Miranda: How it all Fits Together: New Events

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

Putting it all together, when a client sends an Event (we'll use a POST for this example) to a Miranda Topic, the system records the new Event and starts a write.  At the same time, it "tells" the Cluster about the new Event.

Graphically (primitively) it looks like this:

Client          Event           Event           Writer          Cluster

                Listener        Manager

|               |               |               |               |

POST            |               |               |               |

--------------->|               |               |               |

|               new POST        |               |               |

|               --------------->|               |               |

|               |               write events file               |

|               |               --------------->|               |

|               |               new POST        |               |

|               |               ------------------------------->|

|               (UUID)          |               |               |

|               |<--------------- nbsp="" span="">

(UUID)          |               |               |               |

|<--------------- nbsp="" span="">

|               |               |               |               |

Introducing Miranda: Deliveries

In preparation for a talk I'm giving at DOSUG I'm going to post my thoughts as they develop.

A Delivery represents an Event which has been sent to a subscriber and accepted.

A Delivery consists of:

• The Event the Delivery is associated with.
• When the Delivery occurred.
• An attempt id (UUID)

Deliveries are "batched" together into groups of 100 (default) to form a delivery file.

Deliveries are created when the system is able to send an Event to a subscriber and the subscriber responds with a 2xx result.

All Deliveries go through the DeliverManger, which is in charge of all Deliveries and deliver files.