Openfire Admin Login

How to change Openfire's admin login when using Windows Vista and an embedded database:

1. Shut down Openfire
2. As the Windows Administrator,
   Edit
   
   C:\Program Files (x86)\Openfire\embedded-db\openfire.script
   
   
3. Change the line that looks something like this
   
   INSERT INTO OFUSER VALUES('admin',NULL,'<hex string>', 'Administrator','admin','0','0')
   
   to
   
   INSERT INTO OFUSER VALUES('admin','<new password>',NULL, 'Administrator','admin','0','0')
   
   
4. Save the file, exit the editor
   
5. Restart Openfire

Here are the same instructions in a more annoying format:

Maybe this is just me but...well...I'd actually rather not deal with that alternative so I'm assuming that this is Openfire's problem problem and not mine.

OK, so for the upteenth time*, I lost the admin password on my Openfire installation.  Openfire is like the most user friendly XMPP server out there...that I know of...and I know of like 2 others so this one must be really, like, user friendly.  Anyhow, so I lost my admin login...again...and I tried to figure out how I could change that password without having to reinstall Openfire.

This applies to the Windows installation with an embedded database.  If you are using a real operating system and a real database, you probably don't have this problem.  If you do then it's because of you...not me.**

At any rate, if you're really unfortunate (read, me), then you are running on the the very interesting and very secure operating system that is Microsoft Vista.  In that case, you may find that Openfire is squatting in the following directory:

C:\Program Files (x86)\Openfire

Yes, it uses that font rather than courier.  Really.

What's more, the embedded database lives in

C:\Program Files (x86)\Openfire\embedded-db

Note that Vista took out the blank in just about everywhere except for the programs directory.

This makes it more secure.

I mean, why else would they have done it?

At this point, a stand-up comic would make an observation about how they recently had a child and wait politely for the audience to applaud.  The idea is to get the audience on your side by a) pointing out that you recently had a child and b) that you have been without sleep and c) are running Microsoft Vista and d) are on the edge and e) that not applauding could very well push you over the edge and f) whatever.

Now then, the embedded database does everything in memory except that it loads everything at startup.  Actually, I'm not sure if that's what it does at all, but it looks that way; and, since I'm the sort of person who forgets their admin login at the drop of a hat, I'm obviously the sort of person that you should listen to.***

The file

openfire.script

Has the various logins and passwords for the system.  In a bold move to ensure security, the passwords are encrypted.  Or rather, there is a field for the users table that contains a field called "encryptedpassword."  There is another password called "plainpassword."  Except both fields do not contain a period in the name.  I hate punctuation.

Now I know what you're thinking: "HA HA!  I shall just change the field that says 'email' to the new password and then the system shall let me in!"  This is why I make $300 an hour and you don't.****

What you actually need to do is change the "plainpassword" to the new password, and then change the "encryptedpassword" to "NULL!"******.

Except it doesn't work.

This is because the very interesting and extremely secure operating system, Windows Vista, will not let you save the file to that directory!  Instead you have to use the extremely secure step of editing the file as the administrator.

OK, so you do all the above and it still doesn't work.   At this point, if you are like me, you are saying Why is this clown using italics so much******* In addition you would not******** think to edit this file after shutting down Openfire, since that program rewrites the startup file every time it shuts down.

And then, only then, after all this enjoyable fun, would you be able to log onto Openfire.

* = which is, by the way, not because of me.
** = this has been verified by ltsllc.com.  Which is affiliated with me.  So therefore it's probably not objective.
*** = if you believe this and have a lot of money to waste on consultants, please contact me.
**** = if you believe or even if you don't, but you are willing to pay this amount for at least a year and you don't expect to get anything useful out of it, please contact me.*****
***** = P.S. to the previous note, you are also required to actually have that much money and be willing to give it to me in advance before contacting me.
****** = except that you leave off the exclamation point.  See, there really is a reason to pay me lots of money.
******* = except you were probably able to spell "italics" correctly on the first try.  Teacher's pet.
******** = see points above about giving me lots of money.
********* = does anyone else actually read footnotes?

CLIPC SMFIFOs

I posted some ideas for a new CLIPC feature that I am currently calling "shared memory FIFOs". For those who are interested, please take a look at the posting on the SourceForge site and post any thoughts, concerns, etc.

The link is:

http://sourceforge.net/apps/wordpress/clipc/2009/06/29/smfifos/

(Note: this link previously pointed to a page that asked you for a password. It has been updated so that it no longer does that.)

New CLIPC Site

I am gradually going to be moving all things CLIPC to the SourceForge site. In particular I am hoping to create some basic documentation on how to use CLIPC and its various components to the blog associated with that project.

The CLIPC site is at http://clipc.sourceforge.net
The CLIPC blog is at https://sourceforge.net/apps/wordpress/clipc/

CLIPC 0.2 up on SourceForge

The new and improved version of CLIPC is now available on SourceForge. This new version includes, among other things, non-blocking FIFOs.

Xming Locks!

Short Story

Starting up the gnome session manager from within a gnome-terminal avoids the annoying "Not Responding" problem that I get when going from Vista to Linux via XMing.

Long Story

Briefly, here is what I did to get it to work:

• Start Xming without access control
• SSH to the Linux system and set DISPLAY
• Start a gnome-terminal
• From within the gnome-terminal, start a gnome-session

Background

This is using XMing 6.9.0.31 to display something from Fedora Core 10 (2.6.27.19-170.2.35.fc10.i686) on Vista Home Permium.

For reasons unknown (anyone here with info, please comment), XMing seems to lock up on Vista frequently. After as little as 10 seconds, the main window stops responding to requests.

I would set up XMing using the configuration that I outlined in a previous post.

Xming after a few seconds

Start Xming without access control

I'm not exactly fond of this, but unfortunately I did not find a way around this. If I do, I will post this info in a followup.

XLaunch dialog

This results in the rather...blank Xming display:

Blank and not very useful

SSH to the Linux system and set DISPLAY

Using putty or whatever, log onto the Linux system. Note that this is using SSH, telnet, but not with something graphical like an xterm.

Next set the X-Windows environment variable, DISPLAY, to point back to the Vista system. I'm sure there's some way of doing this with a script, but I'm too lazy to figure it out.

Setup the DISPLAY variable

Start a gnome-terminal

I expect that xterm or kdeterm or others would do just as well, but my system only seems to have "gnome-terminal." This should result in an unadorned, ugly, terminal appearing on the Vista system:

Note the lack of border and resize.

From within the gnome-terminal, start a gnome-session

Emphasis on from within the gnome-terminal! I've tried doing this from the SSH session and it does not seem to work. Don't ask me why... it's just Linux!

Once again, perhaps this may work using the KDE equivalent, but this is what I use.

After starting up a session, the Xming display changes from the ugly default to something better looking:

Note the window now has a window bar, border and the background has changed.

gnome-session also spouts about a page and a half of scary-looking gibberish. The thing seems to work despite all this.

Conclusion

Using this approach, Xming is much more responsive than with the other approaches I've tried. It is somewhat cryptic, and various steps may not even be necessary. If you find other approaches that work, please post them.

JNI/Windows Gotchas: UnsatisfiedLinkError

The Short Story

This post describes a solution that has worked for me when I get a an UnsatisfiedLinkError on Windows using MingW, while developing a JNI library.

The Long Story

Java Native Interface (JNI) development on Windows is extremely annoying. There are all kinds of land mines strewn randomly about the landscape to snare the wary and unwary alike. Any one of these issues can require hours of hair-tearing in order to get things to a working state.

After dealing with enough of these, I decided to post some of the solutions I've found in order to save others from premature baldness. For today's entertainment, I have selected the infameous UnsatisfiedLinkError.

java.lang.UnsatisfiedLinkError: no found in java.library.path

This is caused by not having the DLL that contains your C/C++ code in the path that Java is using to find it. If you have this problem, try this:

• Bring up the properties for the project containing the native declarations
• From the properties dialog, bring up the build path for the project.
• From the build path properties, edit the native libraries location for the project.
• From the resulting dialog, select the folder where your DLLs live

Bring up the properties for the project containing the native declarations

For example:

For the CLIPC project, this is clipc-java. You can be sure by selecting the Java project that contains the .java files with declarations like this one:

From the properties dialog, bring up the build path for the project.

For example:

From the build path properties, edit the native libraries location for the project

For example:

From the resulting dialog, select the folder where your DLLs live

For example, this tells Eclipse to look in the workspace:

Then this tells Eclipse where to look for the DLL:

Conclusion

If you have an UnsatisfiedLinkError, you may need to tell Eclipse where your DLLs live. This post gives a step-by-step on how to do that.

XMing Rocks (yes, it does)!

The gnome windows manager desktop on Fedora 10

Short Story

If you need to develop on Windows and Linux, XMing will allow you to run programs on a Linux box and display them on a Windows machine.

Long Story

As the chief architect, engineer, and only developer on the CLIPC project, I need to be able to develop on Linux and Windows. Developing on Linux has been something of a chore up until now, because I had to either "rlogin" onto the machine, and give up any GUI capabilities, or physically sit in front of the system.

Cygwin-X

I've been trying to get a better setup for a couple of days and then I hit on the idea of using the ability of X-Windows to forward a display to another machine. One of the (many) problems that I encountered was getting an X-Windows server to actually run on Windows.

At first I tried to use Cygwin-X, the x-windows that comes with Cygwin, but I found it difficult to use. Any time I run a program and it seems to do nothing, I start getting testy. Specifically, I selected

 Start>Cyginw-X>Start XWin Server

After a bit of puttering about, I recalled running "startx" from the keyboard. I tried that from a Cygwin session and was "rewarded" with this:

Ugh...this actually makes Windows look good!

At this, I felt like I had been thrown into a time warp back to the 90's when you could peg the CPU of a machine running X by just bringing up a menu.

Mind you, I have tremendous respect and gratitude towards the folks who maintain cygwin. But there are limits.

Enter XMing

After glaring at Cygwin-X for a bit, I tried looking for something else. Google, after much prodding and use of something-or-other: directives turned up XMing.

I've been using MinGW for quite a while in general, and for the Windows portion of CLIPC in particular, so I am reasonably comfortable with things MING. I downloaded it and installed it in under 10min. Perhaps under 5min.

With most things Linux-like I expect stuff to take at least a weekend, so this was a very nice surprise.

At first I got the same, ugly X-Windows starting stuff, but with a little bit of reading, I determined how to fire up the gnome window manager. The way to do this is to specify "gnome-terminal" as the "start program" that you get from running XLaunch.

In the interest of being more helpful, here is what I used for the values in the XLaunch config screens:

For me the most important aspect is that it can run Eclipse. I want to have some vague confidence that my stuff will compile and run on the Linux version.

Eclipse running on Linux and forwarded to Windows

Conclusion

MingX is a user-friendly solution for those who want to forward X-Windows from a Linux box to a Windows box.

CLIPC File Naming

NOTE

The source code shown here may not be up on SourceForge until 4/23/2009 (Friday). This is due to some last minute problems found with the Linux code. I'll provide more information as it becomes available.

The Short Story

CLIPC uses file names to identify IPC resources. If two processes use the same file name for a Semaphore, they get connected to the same Semaphore.

The Long Story

You can find the source code and examples for CLIPC on SourceForge.

Naming: The Orphan Child of IPC

It seems to me that a great deal of effort has gone into ensuring that various IPC schemes are "correct" or that their waiting schemes are "fair," while somewhat less effort has gone into ensuring that they are easy to use. In particular, how do two processes that have no prior association connect in order to communicate?

Different platforms use different schemes for dealing with this problem. For semaphores, Windows has a "namespace" set aside for Semaphores and a bunch of other objects. In Linux there are several approaches. One of them is to use a restricted name space, another is to use integer values to identify the Semaphore.

File Names: a Ubiquitous Concept Across Platforms

A lot of effort during the design and evolution of Java to make file naming across platforms easier. For example, the path "/what/ever/one" will work on both Linux and Windows.

CLIPC builds on that strength in order to solve the naming problem for some IPC mechanisms.

In the case of Semaphores, CLIPC uses a file name to identify which Semaphore a client wants to use. If two processes use the same file name when creating a Semaphore, the two processes will end up connected together through that semaphore.

A nice side-effect of using file names to identify resources is that it solves the problem of who can use the resource. For example, if a client does not have permission to read a file associated with a semaphore, then they cannot access the associated semaphore.

Semaphore File Naming in CLIPC

In CLIPC, a client must specify a file name in order to connect to a Semphore. Here is an example where the file name is specified when the constructor is called:

Semaphore s = new Semaphore("/mysem");

Here is an example where the file name is specified when the Semaphore is connected to the underlying system resource:

Semaphore s = new Semaphore();
...
s.connect("/mysem");

CLIPC uses the name of the file to store a value that, when used in the underlying system call, will identify a Semaphore to the platform. If such a file exists at the time when the connection occurs, CLIPC will use the contents of the existing file instead of creating a new file.

File Creation and Race Conditions

Such a scheme has an inherent race condition in that if two processes decide to create the file at the same time, it is possible that both with think they were successful, yet each is using a different underlying identifier.

CLIPC handles this problem by using a two step process:

• Create a temporary file and populate it with an identifier
• Rename the temp file to the desired file name

For windows and Linux, renaming can be an atomic operation. In particular, it can be made to fail if the desired name is already being used. This ensures that in the case where two processes try to create the file at the same time, only one will succeed.

Conclusion

CLIPC uses file names for handling the issue of identifying which IPC Semaphore a client wants to use. File names are ubiuitous across platforms that Java supports, making them convenient for use as IPC "names." CLIPC uses files with semaphores to store the underlying platform ID for a semaphore.

CLIPC Semaphores in Action

The Short Story

How the CLIPC Semaphore class can be used to solve the lost update problem.

The Long Story

You can find the source code for CLIPC on SourceForge. The example that I'm showing here can also be found on SourceForge.

The Lost Update: How Not to Run a Bank

Here is some code that implements the bank application described in the previous blog entry on semaphores.

An account is implemented as a text file that contains the balance of the account. Clients use the following process:

1. Pause for a random period of time
2. Read the account balance from the file
3. Pause for a random period of time
4. Calculate the new balance/decision the withdraw
5. Write out the new balance to the file

The program randomly decides the amount to deposit/withdraw, with the limitation that it cannot withdraw more money than exists in the account.

   public void run(File file, Random r) throws Exception
   {
       ThreadUtils.sleep(r.nextInt(2000));
       int balance = readBalance(file);
       sleep(r.nextInt(4000));
       int amount = generateTransactionAmount(r, balance);
       printMessage(balance, amount);
       balance = balance + amount;
       writeBalance(file, balance);
   }

Here is a screenshot of the app not working quite as desired:

With Semaphores: the Lost Update Found!

By adding a semaphore, the lost update problem can be avoided. The previous process needs to be modified a little bit:

1. Lock the account (reserve the semaphore)
2. Read the account balance from the file
3. Pause for a random period of time
4. Calculate the new balance/decision the withdraw
5. Write out the new balance to the file
6. Unlock the account (release the semaphore)

The modified code:

   public void run(Semaphore sem, File file, Random r) throws Exception
   {
       ThreadUtils.sleep(r.nextInt(2000));
       sem.decrement();
       int balance = readBalance(file);
       sleep(r.nextInt(4000));
       int amount = generateTransactionAmount(r, balance);
       printMessage(balance, amount);
       balance = balance + amount;
       writeBalance(file, balance);
       sem.increment();
   }

The output from the revised sample program:

The code to connect to the semaphore or to create it if it does not already exist is:

 Semaphore(fileName);

The file is an actual file that is used to allow clients to connect to the same semaphore. I'll explain more in the next post.

This code creates a binary semaphore. To create a semaphore that can have a max value of "n" use this call:

 Semaphore(fileName, max value);

For this example, a binary semaphore was used. A command line argument passed the name of the semaphore file.

 sem = new Semaphore(argv[1]);

Conclusion

This posting contained a quick example of how to solve the "Lost Update" problem described in a previous post using a binary semaphore from the CLIPC library. The sample code can be obtained from SourceForge, along with the rest of the CLIPC code.

This example uses file naming for semaphores without really explaining what file naming is or what it is for. In the next exciting chapter of the ongoing CLIPC saga, I will explain how that works.

CLIPC Semaphores: Synchronization

The Short Story

A semaphore is a mechanism for synchronizing two processes. Synchronization is needed when two separate processes can interfere with each other. Semaphores work via decrement (reserve) and increment (release) operations.

The Long Story Part I: What is Synchronization?

Semaphores are a method of inter-process communication (IPC) for synchronizing multiple processes.

The Lost Update

Suppose you had a bank application that managed deposits and withdraws to one account. Here is a simple pseudo program for doing this:

1. Read the account balance
2. Calculate the new balance/decision the withdraw
3. Write out the new balance

Here is what the program, balance, etc. might look like if we wanted to deposit $40:

Action	Amount	Balance
Read balance	N/A	100
Calculate new balance	140	100
Write updated balance	140	140

This works fine so long as there is only one process working with the account, but what if two processes are trying to handle operations on the account? Suppose one of them is trying to deposit $40, while the other wants to withdraw $70. Here is how it might look:

Process	Action	Amount	Balance
Deposit 40	Read balance	N/A	100
Withdraw 70	Read balance	N/A	100
Deposit 40	Calculate new balance	140	100
Deposit 40	Write updated balance	140	140
Withdraw 70	Calculate new balance	30	140
Withdraw 70	Write updated balance	30	30

The $40 deposit has been lost because of concurrent update issues.

Fix for the Lost Update Found!

Suppose the pseudo program were changed so that it took other processes into account:

1. Lock the account
2. Read the account balance
3. Calculate the new balance/decision the withdraw
4. Write out the new balance
5. Unlock the account

Now if the same problematic situation were to arise, here is how things might play out:

Process	Action	Amount	Balance
Deposit 40	Lock account	N/A	100
Deposit 40	Read balance	N/A	100
Withdraw 70	Lock account	Operation blocks	100
Deposit 40	Calculate new balance	140	100
Deposit 40	Write updated balance	140	140
Deposit 40	Unlock account	N/A	140
Withdraw 70	Lock account	operation completes	100
Withdraw 70	Read balance	N/A	140
Withdraw 70	Calculate new balance	70	140
Withdraw 70	Write updated balance	70	70
Withdraw 70	Unlock account	N/A	70

By putting in the locks, the deposit is no longer lost.

Binary Semaphores as Exclusive Locks

In this situation, a binary semaphore could implement the "lock/unlock" operations used to synchronize access to the account.

A binary semaphore allows two states: reserved and available. If a process tries to reserve a semaphore that has already been reserved, as when the withraw process tries to perform its operation when the deposit processes has locked the account, the requesting process will block and go into a waiting/sleep state.

When the process that has reserved the semaphore releases it, the operating system chooses a process that is waiting for the semaphore and unblocks it. The above scenario depicts just such a situation.

The underlying mechanism for a semaphore is an integer value. When a process wants to reserve the semaphore, they decrement the semaphore's value. When the process is done with the resource, it releases the semaphore by incrementing its value.

With a binary semaphore, the object can have a value of 0 or 1. Therefore, if the semaphore has a value of 1 and a processes wants to reserve it by decrementing the semaphore, the semaphore's value becomes 0. When done with the resource, the process increments the semaphore and the value becomes 1.

If a process tries to decrement a semaphore whose value is already 0, the operation blocks until the other process that "owns" the semaphore increments it again.

Conclusion

That was a quick introduction to synchronization and semaphores. In future posts, I will go over how semaphores are used in CLIPC.

Introducing CLIPC

The Short Story

CLIPC is a new open-source java library for IPC. It provides new IPC primitives like semaphores and shared queues, and it makes existing primitives like shared memory easier to use. CLIPC currently supports the Windows and Linux platforms.

The Long Story

CLIPC is the com.lts.ipc library that I wrote because I could not find a Java library that did some Inter-Process Communications (IPC) functions that I was interested in. CLIPC has been the topic of several talks that I have given recently at the Boulder Java Users Group (BJUG) and the Denver Open Source Users Group (DOSUG).

Over the next couple of weeks, I am hoping to create some blog entries about CLIPC and some of the trials and tribulations I went through to write the library. This will be of interest to people who are interested in IPC and also those are are interested in the Java Native Interface (JNI).

CLIPC makes use of JNI because Java does not support certain IPC concepts like Semaphores and FIFOs. This requires the use of JNI to perform the required system calls and whatnot through C, a language that the two platforms that CLIPC currently supports uses.

Another aspect of CLIPC/JNI is that it requires the creation of a consistent interface across multiple platforms. Both Windows and Linux support First-In, First-Out messaging (FIFOs), but the similarities pretty much end there.

For example, on Windows there are Named Pipes. These are always bi-directionaly and function in a lot of ways like very fast TCP/IP connections. Named pipes are represented with files in a special directory.

Linux also supports FIFOs, but on that platform they are uni-directional. They appear to be files on the file system in that there are no naming requirements and they can appear anywhere that regular files can. Linux FIFOs require no special system calls in order to connect to, though they do require special calls to create.

How do you reconcile these differences in order to create a uniform interface to FIFOs? Should they really be more like the named pipes of Windows and allow bi-directional data flow or should they be uni-directional?

I don't know if I made the best decisions possible for CLIPC, but I can talk about why I made the decisions that I did. This will be the topic of future blog entries.

Linux, Samba, NTFS and Selinux

The Short Story

I had a problem where windows kept giving me an "access denied" error when I tried to access a file share from Linux. Using the following commands allowed it to work:

 
    semanage fcontext -a -t samba_share_t "<share>"
    setsebool -P samba_share_fusefs 1

The Long Story

After converting a Windows system to a Linux system (Fedora 10), I wanted to be able to use it as a file server via Samba and furthermore to be able to access my old hard drive.

The initial problem was that I could create the share, but I could not use it via windows. Whenever I tried I received a "permission denied" error. I tried making the share world readable, etc. but this did not help.

The "secure linux troubleshooter" --- SETroubleShooter, which I found from the application>system tools menu, had a good suggestion for the solution to the problem:

    semanage fcontext -a -t samba_share_t "<share>"

This allowed Samba to share the part of the disk that I had set aside for the file server. All well and good. Then I tried to share the old system disk.

This required some special mounting options because it was formatted with NTFS. Here is the entry from /etc/mtab:

     /dev/sda2 /export/whatever fuseblk rw,allow_other,blksize=4096 0 0

The system was able to mount the volume and I could ls around it etc., but trying to share it via Samba failed. I couldn't figure out what was going on until I tried disabling SELinux entirely via the following:

    echo 0 > /selinux/enforce

I then tried the share and it worked. I then reenabled SELinux:

    echo 1 > /selinux/enforce

At this point I felt like turning off SELinux and leaving it off, but some additional looking around on the web turned up this command:

    setsebool -P samba_share_fusefs 1

For whatever reason, SELinux needs to be told separately that it should let Samba share NTFS file systems as well as regular file systems.

All this may sound nice and easy, but it took hours of hair-tearing and glaring at the monitor. I found that SELinux/Samba interactions are not anywhere near as well documented as regular Samba problems.

I found the following pages to be useful in troubleshooting this problem:

• http://our.obor.us/node/2032
• http://www.mjmwired.net/resources/mjm-fedora-f10.html#samba

ByteBuffer is Fast Enough

The Short Story

• ByteBuffer.get/put is fast enough
• Concentrate data into chunks of 256+ bytes

The Long Story

Following up my posting about how ByteBuffer.get/put appeared to be about 3 times slower than simple array accessing, I did some testing.

ByteBuffer is Slow? Not so Fast!

ByteBuffer.put(byte[] b) is actually faster than using the array operator (e.g., b[0], b[1], etc.) ; for array sizes larger than 256 bytes. If you are moving around data with a ByteBuffer in chunks of less than 256 bytes, arrays may be faster. ByteBuffer can be much faster as the size of the array approaches and exceeds 4kB.

ByteBuffer becomes as fast as regular arrays at a buffer size of about 256 bytes

The Java array operator appears to have a bandwidth of 300MB/sec. This is the amount of time that it takes to read 1 byte from an array location and then write a value to the location. Put another way, the turn-around time for Java arrays is less than 5 nsec.

Performing the same operations with ByteBuffer.get/put at 1 byte per operation, the bandwidth could be as low as 7MB/sec; giving a turn-around time of less than 150 nsec. At first glance this is a huge difference: it suggests that ByteBuffer is 30 times slower than array access!

The bandwidth of ByteBuffer increases exponentially with the buffer size. When you hit about 256 bytes, array access and ByteBuffer are equivalent. ByteBuffer's performance continues to increase until you hit about 4KB in size --- at which point you are looking at over 1GB/sec.

Size Matters

The moral of the story is: try to concentrate your data. If you can put all the commonly accessed stuff into a block of 256+ bytes, access will be efficient. If you are in the situation where your data is small and scattered, things may get dicey.

Time Will Tell

Another important consideration is time. Ask yourself "do I really need a 5nsec response time?" The CPU cache is around 1 nsec and main memory is around 10 nsec. If you really need to turn one or two operations around that fast, you are pushing the performance boundaries of the platform.

If the answer is "yes, it really needs to be that fast," then it may be better to do this sort of thing entirely in JNI. That way Java index boundary checking can be avoided. Since we are talking about time scales where that could be a factor, it could make a difference.

If the answer is "no, 1 usec is easily fast enough," then ByteBuffer is probably the way to go. It will be much simpler and when you are trying to debug something, you can be much more confident that the problem is not in how shared memory is being read from or written to.

Your Mileage Will Vary

If you try the tests I performed on your own system, you will get different results. The nature of the tests makes them very sensitive to things like cache size, front-side bus speed, etc. The values I mention here like array access times of 300MB/sec are approximate.

Test Code

For those who are interested, you can find the test code here. The source is included in the executable JAR.

JRE and Shared Memory Overhead

Shared memory access in Java appears to be about 3 times slower than it could be.

In another post, I mentioned that ByteBuffer and its related classes provide a way for Java developers to access shared memory without having to resort to JNI. The problems come in when you try to perform operations on that segment.

One approach is to get a reference to an array of bytes that represents the data in the segment and then use the regular array syntax to access the data. The ByteBuffer.array() method appears to be the the way to do this, but unfortunately it does work.

Here is an example of what I mean:

// DOES NOT WORK!!
MappedByteBuffer mbb;
// code to initialize mbb omitted
byte[] ba = mbb.array(); // throws UnsupportedOperationException

After looking around a bit, I came to the conclusion that this was the intention of the original developers --- if you want to mess with the data in the segment, then you are supposed to use ByteBuffer.get/put.

This would be fine if get/put were about the same cost as using a straight byte array, but they appear to be 2 to 3 times slower. Here is a simple program that highlights the issue I'm running into. The basic difference is that one version uses:

b1 = bb.get(0);
bb.put(0, b2);

And the other that uses

b1 = bb[0];
bb[0] = b2;

The program performs these operations millions of times and then prints out the time (in milliseconds) they took to run. An example output:

Using get/put: 11578
Using array access: 3234

One thing this shows is that I really need to upgrade my system.

The basic point is that using get/put is a lot slower than using simple arrays. A program that reads and writes a lot of data in shared memory would be a lot faster if it could simply use an array rather than get/put.

Is there a way around this?

Shared Memory Using MappedByteBuffer

Java developers can access shared memory using the NIO class MappedByteBuffer. Here is an example:

RandomAccessFile rac = new RandomAccessFile("<some file name>", "rw");
FileChannel channel = rac.getChannel();
MappedByteBuffer buf = channel.map(MapMode.READ_WRITE, 0, 1024);

This will create a shared memory segment with the name of the file passed to RandomAccessFile. The segment will start out with the contents of that file, if it exists. If the file does not exist, then it will be created.

The segment can be changed or read using the get/put methods defined by the ByteBuffer class. A different Java process that uses the same set of of calls will get the same shared memory: if they change their instance you will see the changes and vice versa. What's more, this also applies to non-java processes that use the same file name and memory mapped system calls.

I have tried this on Windows and Linux and I have also taken a look at the implementation code on java.net It appears that both platforms are using memory mapped files.

As an aside, "memory mapped files" are an approach to using shared memory that appear to have originated with Unix. Unix tries to make most things look like files, so representing shared memory that way is pretty consistent with the Unix philosophy.

For those who are interested, here is a more complete example.

Welcome!

This site is about software development in general, with an emphasis on Java. It contains insights, discoveries, etc. that I have encountered while performing development on various projects.

"LTSLLC" is short for "Long Term Software, LLC." the consulting company that I created. I find blogger to be more convenient than Wordpress, otherwise I would host this blog off the main ltsllc site.