JNI/Windows Gotchas: UnsatisfiedLinkError

The Short Story

This post describes a solution that has worked for me when I get a an UnsatisfiedLinkError on Windows using MingW, while developing a JNI library.

The Long Story

Java Native Interface (JNI) development on Windows is extremely annoying. There are all kinds of land mines strewn randomly about the landscape to snare the wary and unwary alike. Any one of these issues can require hours of hair-tearing in order to get things to a working state.

After dealing with enough of these, I decided to post some of the solutions I've found in order to save others from premature baldness. For today's entertainment, I have selected the infameous UnsatisfiedLinkError.

java.lang.UnsatisfiedLinkError: no found in java.library.path

This is caused by not having the DLL that contains your C/C++ code in the path that Java is using to find it. If you have this problem, try this:

• Bring up the properties for the project containing the native declarations
• From the properties dialog, bring up the build path for the project.
• From the build path properties, edit the native libraries location for the project.
• From the resulting dialog, select the folder where your DLLs live

Bring up the properties for the project containing the native declarations

For example:

For the CLIPC project, this is clipc-java. You can be sure by selecting the Java project that contains the .java files with declarations like this one:

From the properties dialog, bring up the build path for the project.

For example:

From the build path properties, edit the native libraries location for the project

For example:

From the resulting dialog, select the folder where your DLLs live

For example, this tells Eclipse to look in the workspace:

Then this tells Eclipse where to look for the DLL:

Conclusion

If you have an UnsatisfiedLinkError, you may need to tell Eclipse where your DLLs live. This post gives a step-by-step on how to do that.

XMing Rocks (yes, it does)!

The gnome windows manager desktop on Fedora 10

Short Story

If you need to develop on Windows and Linux, XMing will allow you to run programs on a Linux box and display them on a Windows machine.

Long Story

As the chief architect, engineer, and only developer on the CLIPC project, I need to be able to develop on Linux and Windows. Developing on Linux has been something of a chore up until now, because I had to either "rlogin" onto the machine, and give up any GUI capabilities, or physically sit in front of the system.

Cygwin-X

I've been trying to get a better setup for a couple of days and then I hit on the idea of using the ability of X-Windows to forward a display to another machine. One of the (many) problems that I encountered was getting an X-Windows server to actually run on Windows.

At first I tried to use Cygwin-X, the x-windows that comes with Cygwin, but I found it difficult to use. Any time I run a program and it seems to do nothing, I start getting testy. Specifically, I selected

 Start>Cyginw-X>Start XWin Server

After a bit of puttering about, I recalled running "startx" from the keyboard. I tried that from a Cygwin session and was "rewarded" with this:

Ugh...this actually makes Windows look good!

At this, I felt like I had been thrown into a time warp back to the 90's when you could peg the CPU of a machine running X by just bringing up a menu.

Mind you, I have tremendous respect and gratitude towards the folks who maintain cygwin. But there are limits.

Enter XMing

After glaring at Cygwin-X for a bit, I tried looking for something else. Google, after much prodding and use of something-or-other: directives turned up XMing.

I've been using MinGW for quite a while in general, and for the Windows portion of CLIPC in particular, so I am reasonably comfortable with things MING. I downloaded it and installed it in under 10min. Perhaps under 5min.

With most things Linux-like I expect stuff to take at least a weekend, so this was a very nice surprise.

At first I got the same, ugly X-Windows starting stuff, but with a little bit of reading, I determined how to fire up the gnome window manager. The way to do this is to specify "gnome-terminal" as the "start program" that you get from running XLaunch.

In the interest of being more helpful, here is what I used for the values in the XLaunch config screens:

For me the most important aspect is that it can run Eclipse. I want to have some vague confidence that my stuff will compile and run on the Linux version.

Eclipse running on Linux and forwarded to Windows

Conclusion

MingX is a user-friendly solution for those who want to forward X-Windows from a Linux box to a Windows box.

CLIPC File Naming

NOTE

The source code shown here may not be up on SourceForge until 4/23/2009 (Friday). This is due to some last minute problems found with the Linux code. I'll provide more information as it becomes available.

The Short Story

CLIPC uses file names to identify IPC resources. If two processes use the same file name for a Semaphore, they get connected to the same Semaphore.

The Long Story

You can find the source code and examples for CLIPC on SourceForge.

Naming: The Orphan Child of IPC

It seems to me that a great deal of effort has gone into ensuring that various IPC schemes are "correct" or that their waiting schemes are "fair," while somewhat less effort has gone into ensuring that they are easy to use. In particular, how do two processes that have no prior association connect in order to communicate?

Different platforms use different schemes for dealing with this problem. For semaphores, Windows has a "namespace" set aside for Semaphores and a bunch of other objects. In Linux there are several approaches. One of them is to use a restricted name space, another is to use integer values to identify the Semaphore.

File Names: a Ubiquitous Concept Across Platforms

A lot of effort during the design and evolution of Java to make file naming across platforms easier. For example, the path "/what/ever/one" will work on both Linux and Windows.

CLIPC builds on that strength in order to solve the naming problem for some IPC mechanisms.

In the case of Semaphores, CLIPC uses a file name to identify which Semaphore a client wants to use. If two processes use the same file name when creating a Semaphore, the two processes will end up connected together through that semaphore.

A nice side-effect of using file names to identify resources is that it solves the problem of who can use the resource. For example, if a client does not have permission to read a file associated with a semaphore, then they cannot access the associated semaphore.

Semaphore File Naming in CLIPC

In CLIPC, a client must specify a file name in order to connect to a Semphore. Here is an example where the file name is specified when the constructor is called:

Semaphore s = new Semaphore("/mysem");

Here is an example where the file name is specified when the Semaphore is connected to the underlying system resource:

Semaphore s = new Semaphore();
...
s.connect("/mysem");

CLIPC uses the name of the file to store a value that, when used in the underlying system call, will identify a Semaphore to the platform. If such a file exists at the time when the connection occurs, CLIPC will use the contents of the existing file instead of creating a new file.

File Creation and Race Conditions

Such a scheme has an inherent race condition in that if two processes decide to create the file at the same time, it is possible that both with think they were successful, yet each is using a different underlying identifier.

CLIPC handles this problem by using a two step process:

• Create a temporary file and populate it with an identifier
• Rename the temp file to the desired file name

For windows and Linux, renaming can be an atomic operation. In particular, it can be made to fail if the desired name is already being used. This ensures that in the case where two processes try to create the file at the same time, only one will succeed.

Conclusion

CLIPC uses file names for handling the issue of identifying which IPC Semaphore a client wants to use. File names are ubiuitous across platforms that Java supports, making them convenient for use as IPC "names." CLIPC uses files with semaphores to store the underlying platform ID for a semaphore.

CLIPC Semaphores in Action

The Short Story

How the CLIPC Semaphore class can be used to solve the lost update problem.

The Long Story

You can find the source code for CLIPC on SourceForge. The example that I'm showing here can also be found on SourceForge.

The Lost Update: How Not to Run a Bank

Here is some code that implements the bank application described in the previous blog entry on semaphores.

An account is implemented as a text file that contains the balance of the account. Clients use the following process:

1. Pause for a random period of time
2. Read the account balance from the file
3. Pause for a random period of time
4. Calculate the new balance/decision the withdraw
5. Write out the new balance to the file

The program randomly decides the amount to deposit/withdraw, with the limitation that it cannot withdraw more money than exists in the account.

   public void run(File file, Random r) throws Exception
   {
       ThreadUtils.sleep(r.nextInt(2000));
       int balance = readBalance(file);
       sleep(r.nextInt(4000));
       int amount = generateTransactionAmount(r, balance);
       printMessage(balance, amount);
       balance = balance + amount;
       writeBalance(file, balance);
   }

Here is a screenshot of the app not working quite as desired:

With Semaphores: the Lost Update Found!

By adding a semaphore, the lost update problem can be avoided. The previous process needs to be modified a little bit:

1. Lock the account (reserve the semaphore)
2. Read the account balance from the file
3. Pause for a random period of time
4. Calculate the new balance/decision the withdraw
5. Write out the new balance to the file
6. Unlock the account (release the semaphore)

The modified code:

   public void run(Semaphore sem, File file, Random r) throws Exception
   {
       ThreadUtils.sleep(r.nextInt(2000));
       sem.decrement();
       int balance = readBalance(file);
       sleep(r.nextInt(4000));
       int amount = generateTransactionAmount(r, balance);
       printMessage(balance, amount);
       balance = balance + amount;
       writeBalance(file, balance);
       sem.increment();
   }

The output from the revised sample program:

The code to connect to the semaphore or to create it if it does not already exist is:

 Semaphore(fileName);

The file is an actual file that is used to allow clients to connect to the same semaphore. I'll explain more in the next post.

This code creates a binary semaphore. To create a semaphore that can have a max value of "n" use this call:

 Semaphore(fileName, max value);

For this example, a binary semaphore was used. A command line argument passed the name of the semaphore file.

 sem = new Semaphore(argv[1]);

Conclusion

This posting contained a quick example of how to solve the "Lost Update" problem described in a previous post using a binary semaphore from the CLIPC library. The sample code can be obtained from SourceForge, along with the rest of the CLIPC code.

This example uses file naming for semaphores without really explaining what file naming is or what it is for. In the next exciting chapter of the ongoing CLIPC saga, I will explain how that works.

CLIPC Semaphores: Synchronization

The Short Story

A semaphore is a mechanism for synchronizing two processes. Synchronization is needed when two separate processes can interfere with each other. Semaphores work via decrement (reserve) and increment (release) operations.

The Long Story Part I: What is Synchronization?

Semaphores are a method of inter-process communication (IPC) for synchronizing multiple processes.

The Lost Update

Suppose you had a bank application that managed deposits and withdraws to one account. Here is a simple pseudo program for doing this:

1. Read the account balance
2. Calculate the new balance/decision the withdraw
3. Write out the new balance

Here is what the program, balance, etc. might look like if we wanted to deposit $40:

Action	Amount	Balance
Read balance	N/A	100
Calculate new balance	140	100
Write updated balance	140	140

This works fine so long as there is only one process working with the account, but what if two processes are trying to handle operations on the account? Suppose one of them is trying to deposit $40, while the other wants to withdraw $70. Here is how it might look:

Process	Action	Amount	Balance
Deposit 40	Read balance	N/A	100
Withdraw 70	Read balance	N/A	100
Deposit 40	Calculate new balance	140	100
Deposit 40	Write updated balance	140	140
Withdraw 70	Calculate new balance	30	140
Withdraw 70	Write updated balance	30	30

The $40 deposit has been lost because of concurrent update issues.

Fix for the Lost Update Found!

Suppose the pseudo program were changed so that it took other processes into account:

1. Lock the account
2. Read the account balance
3. Calculate the new balance/decision the withdraw
4. Write out the new balance
5. Unlock the account

Now if the same problematic situation were to arise, here is how things might play out:

Process	Action	Amount	Balance
Deposit 40	Lock account	N/A	100
Deposit 40	Read balance	N/A	100
Withdraw 70	Lock account	Operation blocks	100
Deposit 40	Calculate new balance	140	100
Deposit 40	Write updated balance	140	140
Deposit 40	Unlock account	N/A	140
Withdraw 70	Lock account	operation completes	100
Withdraw 70	Read balance	N/A	140
Withdraw 70	Calculate new balance	70	140
Withdraw 70	Write updated balance	70	70
Withdraw 70	Unlock account	N/A	70

By putting in the locks, the deposit is no longer lost.

Binary Semaphores as Exclusive Locks

In this situation, a binary semaphore could implement the "lock/unlock" operations used to synchronize access to the account.

A binary semaphore allows two states: reserved and available. If a process tries to reserve a semaphore that has already been reserved, as when the withraw process tries to perform its operation when the deposit processes has locked the account, the requesting process will block and go into a waiting/sleep state.

When the process that has reserved the semaphore releases it, the operating system chooses a process that is waiting for the semaphore and unblocks it. The above scenario depicts just such a situation.

The underlying mechanism for a semaphore is an integer value. When a process wants to reserve the semaphore, they decrement the semaphore's value. When the process is done with the resource, it releases the semaphore by incrementing its value.

With a binary semaphore, the object can have a value of 0 or 1. Therefore, if the semaphore has a value of 1 and a processes wants to reserve it by decrementing the semaphore, the semaphore's value becomes 0. When done with the resource, the process increments the semaphore and the value becomes 1.

If a process tries to decrement a semaphore whose value is already 0, the operation blocks until the other process that "owns" the semaphore increments it again.

Conclusion

That was a quick introduction to synchronization and semaphores. In future posts, I will go over how semaphores are used in CLIPC.

Introducing CLIPC

The Short Story

CLIPC is a new open-source java library for IPC. It provides new IPC primitives like semaphores and shared queues, and it makes existing primitives like shared memory easier to use. CLIPC currently supports the Windows and Linux platforms.

The Long Story

CLIPC is the com.lts.ipc library that I wrote because I could not find a Java library that did some Inter-Process Communications (IPC) functions that I was interested in. CLIPC has been the topic of several talks that I have given recently at the Boulder Java Users Group (BJUG) and the Denver Open Source Users Group (DOSUG).

Over the next couple of weeks, I am hoping to create some blog entries about CLIPC and some of the trials and tribulations I went through to write the library. This will be of interest to people who are interested in IPC and also those are are interested in the Java Native Interface (JNI).

CLIPC makes use of JNI because Java does not support certain IPC concepts like Semaphores and FIFOs. This requires the use of JNI to perform the required system calls and whatnot through C, a language that the two platforms that CLIPC currently supports uses.

Another aspect of CLIPC/JNI is that it requires the creation of a consistent interface across multiple platforms. Both Windows and Linux support First-In, First-Out messaging (FIFOs), but the similarities pretty much end there.

For example, on Windows there are Named Pipes. These are always bi-directionaly and function in a lot of ways like very fast TCP/IP connections. Named pipes are represented with files in a special directory.

Linux also supports FIFOs, but on that platform they are uni-directional. They appear to be files on the file system in that there are no naming requirements and they can appear anywhere that regular files can. Linux FIFOs require no special system calls in order to connect to, though they do require special calls to create.

How do you reconcile these differences in order to create a uniform interface to FIFOs? Should they really be more like the named pipes of Windows and allow bi-directional data flow or should they be uni-directional?

I don't know if I made the best decisions possible for CLIPC, but I can talk about why I made the decisions that I did. This will be the topic of future blog entries.

Linux, Samba, NTFS and Selinux

The Short Story

I had a problem where windows kept giving me an "access denied" error when I tried to access a file share from Linux. Using the following commands allowed it to work:

 
    semanage fcontext -a -t samba_share_t "<share>"
    setsebool -P samba_share_fusefs 1

The Long Story

After converting a Windows system to a Linux system (Fedora 10), I wanted to be able to use it as a file server via Samba and furthermore to be able to access my old hard drive.

The initial problem was that I could create the share, but I could not use it via windows. Whenever I tried I received a "permission denied" error. I tried making the share world readable, etc. but this did not help.

The "secure linux troubleshooter" --- SETroubleShooter, which I found from the application>system tools menu, had a good suggestion for the solution to the problem:

    semanage fcontext -a -t samba_share_t "<share>"

This allowed Samba to share the part of the disk that I had set aside for the file server. All well and good. Then I tried to share the old system disk.

This required some special mounting options because it was formatted with NTFS. Here is the entry from /etc/mtab:

     /dev/sda2 /export/whatever fuseblk rw,allow_other,blksize=4096 0 0

The system was able to mount the volume and I could ls around it etc., but trying to share it via Samba failed. I couldn't figure out what was going on until I tried disabling SELinux entirely via the following:

    echo 0 > /selinux/enforce

I then tried the share and it worked. I then reenabled SELinux:

    echo 1 > /selinux/enforce

At this point I felt like turning off SELinux and leaving it off, but some additional looking around on the web turned up this command:

    setsebool -P samba_share_fusefs 1

For whatever reason, SELinux needs to be told separately that it should let Samba share NTFS file systems as well as regular file systems.

All this may sound nice and easy, but it took hours of hair-tearing and glaring at the monitor. I found that SELinux/Samba interactions are not anywhere near as well documented as regular Samba problems.

I found the following pages to be useful in troubleshooting this problem:

• http://our.obor.us/node/2032
• http://www.mjmwired.net/resources/mjm-fedora-f10.html#samba