Java File System

This essay is about a suggested student project in Java programming. This essay gives a rough overview of how it might work. It does not describe an actual complete program. I have no source, object, specifications, file layouts or anything else useful to implementing this project. Everything I have to say to help you with this project is written below. I am not prepared to help you implement it; I have too many other projects of my own.

I do contract work for a living, which could include writing a program such as this. However, I don’t do people’s homework for them. That just robs them of an education.

You have my full permission to implement this project any way you please.

One problem with Java is it lacks a portable file system. Because every file system is different on every platform, Java implements the lowest common denominator. You can’t set hidden, archive or system attribute bits.

What you do is allocate a giant file say 10 gigabytes. It looks like one file to the host OS, but to Java it will be a file system containing many files. If you do your work well, eventually it may be the only file system of some future OS. You invent methods to allocate space inside this file for subfiles. You want to create all the ordinary file services:

Lookup file by fully qualified name.
Lookup file by high order part of name.
Lookup file by low order part of name.
Lookup files by last access or modify date. This allows a program like AltaVista Discovery to rapidly discover the files that have changed and need reindexing. Recently deleted files need to be logged too so that indexers can drop them from the indexes. Perhaps privileged application programs should be allowed to hook themselves in and monitor all file status changes system wide.
Enumerate files that match some filter composed of a number of positive and negative wildcards, or user-written.
Lookup file by handle.
read/write sequential and random.
caching, and delayed write (lazy write) caching.
associations — by extension is a bit dangerous. A file should be permitted to have several associations. You record the mime type of each file. None of the goofiness of Linux running file type guessers that sniff the first few bytes of the file or Windows file extensions that can have a multiplicity of meanings.
Which app installed/created this file? Is it user data, or part of an app package? URL where I can get an update of it if it is corrupt.
Filters so that you can rapidly search only files that contain your data, not vendor files.
Security — owners, read-only, hidden, system, file signatures/keys/checksums.
Dates — last access, last update.
locking.
defragging. See the notes on the defragging project. The whole key to a fast file system is to organise your disk layout to minimise arm movement, i.e. to keep it defragged as you go.
Marthaing — dynamically keeping the most active files in the prime real estate near the outer edge (near the beginning of the disk) near the swap file and the directory, and the least active files, and the bulk of the disk space in the inner tracks (near the end of the disk). This could be done on per cylinder basis, or per file or per cluster.
This file placement might presume a weekly reorg, or even better an ongoing low priority background reorg.
When you start using a file, it gets moved by a background process to prime real estate near the MFT, swapfile and directories. Background processes move files that have been not used in a while to less desirable real estate further away, freeing up space, but not too much free space in the desirable real estate territory. The bulk of your free space should be out in the least desirable territory. Roughly speaking files should be contiguous, and in order by last access date, and within that by directory with the most recently accessed files in the more prime locations.
The reorg process might be more integrated with ordinary I/O, so that whenever you write, that cluster gets moved to prime real estate as a side effect. You might do all your writes sequentially, interleaving writes from various files. If you had a two headed disk, you could dedicate one head to writes and one to reads. The write head would not move much. You write in a sequential circular squirrel cage pattern to a hot zone on disk. A background process moves the oldest data out of the hot zone so you don’t wrap around and overwrite. You can think of it as a second level of disk caching. You have a primary cache in RAM, and a secondary level in prime disk real estate.
CPUs are getting faster relative to hard disk. RAM for caching control information is getting cheaper. This means that, in future, it will pay to use more and more elaborate techniques to minimise disk head motions. You will be able to afford to do a lot of thinking to avoid a tiny bit of needless movement.
This leaves the problem of a giant file that you update only rarely. Perhaps the positioning algorithm needs to take into account both frequency of use and last access date. Perhaps you need decaying hit counter. You multiply previous hits by 0.9 and add today’s hits to determine priority for prime real estate. (The actual decay algorithm would be more sophisticated.) You also would also factor in size. A 1K file with 10 hits is much more deserving of prime real estate that an 100 MB file with 10 hits.
Temporary files — clean up of orphaned temporary files. Tracking who orphaned them.
Getting more advanced, you might want an API for persistent objects. Traditional file systems are hierarchical. You need something so you can find stuff in a million different ways.
Your file system would probably be designed with the idea of massive amounts of RAM being available, and would not worry quite so much about getting everything on rotating media immediately. You would only have to deal with graceful shutdown, perhaps with 10 minutes notice from your UPS. This would give you a big speed boost over traditional file and database systems that are overly paranoid.
Record which encoding was used to create a text file.
Record the associated DTD for an XML file.
hooks to file viewer and editor.
Ability to retrieve older versions of a file. Like a simplified CVS, it stores the differences between generations allowing it to reconstruct older versions.

You can start very simply. without a directory, requiring users to allocate files by absolute block numbers, and just ensuring you don’t allocate the same block twice. You have been bitching that Microsoft and IBM could not write a file system to save their lives — here’s you chance to see if you can write a faster one. Once you prove your structures, maybe the MS, IBM or Linux folk will consider supporting your new partition type.

I suggest that the file system directory be implementent as an SQL database. That handles integrity, and all manner of alternative indexing for you automatically. It also allows a low level hook where the user can query the directory using SQL code.

	Please email your feedback for publication, errors, omissions, broken/redirected link reports and suggestions to improve this page to Roedy Green :
	Canadian Mind Products
	mindprod.com IP:[65.110.21.43]
	Your face IP:[38.103.63.59]
	You are visitor number 13,725.
	You can get a fresh copy of this page from:	or possibly from your local J: drive (Java virtual drive/mindprod.com website mirror)
	http://mindprod.com/project/javafilesystem.html	J:\mindprod\project\javafilesystem.html