Implementing FileSystem without MANIFEST.MF/META-INF/services

I'm trying to set up a file system (by implementing FileSystemProvider, FileSystem, FileStore, etc.) so that later a user could only call FileSystems.getFileSystem(uri), with a specific URI (custom scheme, host, path, etc).

This is normally done by putting an entry into jar's MANIFEST.MF/META-INF/services, so that a JVM - using that information - can set it up itself, but I don't have that option. I want to do it programmatically. How can this be done?

Did you try Google with this: java.nio.file.spi.DefaultFileSystemProvider

Quote:

---

Originally Posted by Norm

Did you try Google with this: java.nio.file.spi.DefaultFileSystemProvider

---

I did. But following virtually every found page/document I end up with "Providers are typically installed by placing them in a JAR file on the application class path or in the extension directory, the JAR file contains a provider-configuration file named java.nio.file.spi.FileSystemProvider in the resource directory META-INF/services" or words to that effect.

Like I said, I don't have that option. I want to

1. implement my own FileSystemProvider (and everything else it needs) and then
2. give it to JVM's disposal in such a way that a user could later
3. obtain instances of FileSystem it produces through FileSystems.newFileSystem(URI, ...) and FileSystems.getFileSystem(URI, ...).

I just don't know how to do #2. I'd like to know if it's possible and - if it is - how it can be done.

BTW: I don't really want to displace DefaultFileSystemProvider by setting system property, if that's what you meant. The FileSystem I need doesn't have much with disks (as in "file://" scheme/protocol). Think of it more like a set of distributed sources of data, including database(s) and network(s).

What about Jimfs?

Quote:

---

Originally Posted by Norm

What about Jimfs?

---

I would gladly use anything that would help me achieve the three points I described above. I can't seem to make that connection (using Jimfs facilities to make FileSystems.getFileSystem(URI) produce the FileSystem I need/implement). Would you help me with that, please? I don't mean implementing. Pointing me in the right direction would do. Thanks.

I've never written a FileSystem for java. I did write one for an IBM OS a long time ago and have an interest in how to do it for java.

The code for the jimfs system is on github. Download it and take a look at how it works.

Quote:

---

Originally Posted by Norm

I've never written a FileSystem for java. I did write one for an IBM OS a long time ago and have an interest in how to do it for java.

The code for the jimfs system is on github. Download it and take a look at how it works.

---

In Java, FileSystem is merely an abstract class one has to extend to make it suit ones own needs. It's a pretty high level abstraction of a file system. Implementing it is trivial in comparison with a low level stuff one has to do to implement a file system from hardware stand point, like I suspect you did.

I have taken a look at Jimfs, even before you had mentioned it. But I don't think it can meet my needs (I could be wrong, though). Like I said, I know how to implement FileSystemProvider, FileSystem, FileStore(s), etc. I just don't know how to seamlessly integrate it (the provider I mean) into a JVM so that the users could use it with no extra plumbing on their side, merely calling FileSystems.getFileSystem() with a proper URI.

Quote:

---

users could use it with no extra plumbing

---

I'm not sure how that would work. If the user doesn't tell the JVM about your URI, when and how would the JVM learn about it?

Quote:

---

Originally Posted by Norm

I'm not sure how that would work. If the user doesn't tell the JVM about your URI, when and how would the JVM learn about it?

---

User in a sense of a client. There is a platform, providing services incl. instances of FileSystem through FileSystems factory methods (FileSystems.getFileSystem(URI)) and there are clients on top of that platform. All the client needs to know is a correct URI. The data is then supplied to the clients using instances of FileSystem.

The platform would somehow register my implementation of FileSystemProvider and FileSystem (plus the URI specs) with the JVM, so that users (clients) would then be able to obtain them through FileSystems.getFileSystem(URI). At first I thought that this could be done through ServiceLoader, but then I realized that ServiceLoader relies on META-INF/services (inside jars) and only loads what is specified there.

What about the java -D option?

Quote:

---

Originally Posted by Norm

What about the java -D option?

---

What about it? That's an equivalent of setting a system property, right? Which property do you have in mind? And why?

Have you looked at what the JVM does with the jar's MANIFEST.MF/META-INF/services entries?

I'm a bit confused about where you'd put your code for the new file system. Using a jar file with the services folder seems like a convenient place.

Quote:

---

Originally Posted by Norm

Have you looked at what the JVM does with the jar's MANIFEST.MF/META-INF/services entries?

---

No, I haven't. I don't know where to look.

Quote:

---

Originally Posted by Norm

I'm a bit confused about where you'd put your code for the new file system. Using a jar file with the services folder seems like a convenient place.

---

It does, doesn't it. Well, I hate to repeat myself but... I don't have that option.

Quote:

---

I don't have that option.

---

Can you explain what the problem is with using that option?

Quote:

---

where to look.

---

Google.
The src.zip file that comes with the JDK.

I still wonder
where your code would be physically when its used
and what would make the JVM execute your code.

I've given up on trying to register a FileSystemProvider so that it could deliver instances of FileSystem through FileSystems.getFileSystem(URI).

I've settled for a semi-workaround, though: using FileSystems.newFileSystem(URI, Map<String,?>, ClassLoader). Of course that means that I need a way to pass the required ClassLoader around. I'm looking for something like a System.setProperty() and System.getProperty() but for Object/ClassLoader values instead of Strings. I would use the setter to "register" the ClassLoader and the user/client would use the getter to obtain it and use it to create the FileSystem.

What would you recommend?

I still wonder
where your code would be physically when its used
and what would make the JVM execute your code.

How would the client code use your code?

Quote:

---

Originally Posted by Norm

I still wonder
where your code would be physically when its used
and what would make the JVM execute your code.

How would the client code use your code?

---

OMG, instead of getting my questions answered I end up explaining stuff to people who are supposed to give me answers. Funny way to run a business.

But, here we go. My classes are loaded by the JVM from an SQL database and across the network, as I hinted above: "Think of it more like a set of distributed sources of data, including database(s) and network(s)." They come through as single entities, not packed in jars. Even if they were (in jars), I'd still end up with the same problem: how to tell the JVM to extract service data from META-INF/services. I'd expect that such a task would be done automatically (from a jar on a physical file system) merely as a convenience and that there would still be a chance to do it manually/programmatically. But that information is somehow very hard to come by. Nobody seems to know if it can be done at all, let alone how.

The client wouldn't use my code directly. It would use a FileSystem provided through FileSystems.getFileSystem(URI), as I've explained above on numerous occasions.

I've written hundreds of programs in java but have not covered all of its facilities. Not many people know it all.
The volunteers that work forums are probably not as knowledgeable as a professional that works a full shift.

You have a very interesting and very specialized problem.

Have you asked this question on: http://stackoverflow.com/questions/tagged/java

Quote:

---

Originally Posted by Norm

I've written hundreds of programs in java but have not covered all of its facilities. Not many people know it all.
The volunteers that work forums are probably not as knowledgeable as a professional that works a full shift.

You have a very interesting and very specialized problem.

Have you asked this question on: http://stackoverflow.com/questions/tagged/java

---

Yes. Yes, I have. Here it is: http://stackoverflow.com/questions/3...a-inf-services

Suprisingly, no answers there. Perhaps I should nag Larry directly. :)

I played with URLStreamHandlerFactory several years ago. It allows you to define your own protocol.

Code:

---

              URL.setURLStreamHandlerFactory(new TestFactory());
     
              try {
                  URL u = new URL("xttp://localhost:8080");
 
                  Object content = u.getContent();

---

There were several more classes involved, but nothing complicated.


There is probably a security concern that prevented Oracle from putting in a runtime interface for FileSystems.

============================================
I just noticed something when executing my test programs for your problem. I have built a META-INF folder with a services folder etc in the folder with my test program. I had put into a jar file for earlier testing and it worked, as expected. Now I tried executing my test program from a class file with the -cp . option. It worked -> the JVM is reading my META-INF folder and finding my FileSystemProvider. Ok. What if I rename the folder: META-INFXX.
The FS provider was not found. OK. Now what if my program renames the META-INFXX folder to META-INF before calling the ServiceLoader. It worked.
My test code

Code:

---


public class Norms_Test {
  public static void main(String[] args) throws Exception {

      File metaF =  new File("META-INFXX");
      System.out.println(metaF +" exists="+metaF.exists());
      metaF.renameTo(new File("META-INF"));  // Will this be found ???

        ServiceLoader<FileSystemProvider> sl = ServiceLoader.load(FileSystemProvider.class, ClassLoader.getSystemClassLoader());

        // ServiceConfigurationError may be throw here
        for (FileSystemProvider provider: sl) {
            System.out.println("0scheme="+ provider.getScheme());    // scheme=jar
        }


      java.util.List<FileSystemProvider> listfsp = FileSystemProvider.installedProviders();
      System.out.println(listfsp); // [sun.nio.fs.WindowsFileSystemProvider@12204a1, com.sun.nio.zipfs.ZipFileSystemProvider@a298b7]
      for(int i=0; i < listfsp.size(); i++) {
        System.out.println("1scheme="+listfsp.get(i).getScheme());    // file jar
      }

        /* Define File System Properies in HashMap */   
        Map<String, String> properties = new HashMap<>();
        /* set create to true if you want to create a new ZIP file */
        properties.put("create", "true");
        /* specify encoding to UTF-8 */
        properties.put("encoding", "UTF-8");       
        /* Locate File on disk for creation */

        URI uri = URI.create("norms://aFile");
        System.out.println("uri="+uri);  // uri=norms:///
       
        // http://jakubstas.com/file-system-api/#.Vs4t55xum00
        // To install a file system provider just place a jar file in your applications class path
        // this jar contains configuration file called java.nio.file.spi.FileSystemProvider in the resource directory
        // META-INF/services containing fully qualified names of one or more file system providers.

        // provider classes must have a zero-argument constructor so that they can be instantiated during loading.
        // A service provider is identified by placing a provider-configuration file in the resource directory META-INF/services.
        // The file's name is the fully-qualified binary name of the service's type.
        // The file contains a list of fully-qualified binary names of concrete provider classes, one per line.

        /* Create file System */
        try (FileSystem fs = FileSystems.newFileSystem(uri, properties)) {
            System.out.println("fs="+fs); //<<<<<<<
        }

  }
}

---

I didn't clean it up. It has stuff from several tests.

The console from when I executed:

Code:

---

Running: java.exe -client -cp . Norms_Test

META-INFXX exists=true
0scheme=jar
MyFSP empty constr
getScheme() called
0scheme=norms
MyFSP empty constr
getScheme() called
[sun.nio.fs.WindowsFileSystemProvider@14991ad, com.sun.nio.zipfs.ZipFileSystemProvider@d93b30, MyFileSystemProvider@16d3586]
1scheme=file
1scheme=jar
getScheme() called
1scheme=norms
uri=norms://aFile
getScheme() called
nFS uri=norms://aFile, env={create=true, encoding=UTF-8}
Exception in thread "main" java.lang.NullPointerException
        at MyFileSystemProvider.newFileSystem(MyFileSystemProvider.java:63)
        at java.nio.file.FileSystems.newFileSystem(Unknown Source)
        at java.nio.file.FileSystems.newFileSystem(Unknown Source)
        at Norms_Test.main(Norms_Test.java:70)

0 error(s)

---

The NPE is because I don't have a FileSystem defined.

Thanks, Norm.

Oh, and if someone wonders... I haven't opted for Norms workaround either. I think it's not something a professional solution can do. Instead, I made my own implementation of FileSystem available via JMX MBean. The client can still obtain it with a one-liner, albeit somewhat longer than the unrealized ideal FileSystems.getFileSystem(URI).