ByteBuffer : Java Glossary

What ByteBuffer is Not

• ByteBuffer is a baffling class. It is a bit like a RAM-based RandomAccessFile. It is also a bit like a ByteArrayList without the autogrow feature, to let you deal with partly filled byte[] in a consistent way.
• It looks at first as if it might be a traditional circular squirrel cage buffer but it is not. There is no circularity. Nor is it some kind of virtual moving window on a file.
• It is not like a pipe, designed to simulate an giant stream with a finite buffer. There is no queuing of any kind.
• It is not even as clever as COBOL double buffering, which you might suspect by it having a flip method.
• It has no asynchronous look-ahead.
• It is extremely low level. nio should probably have been call llio (low level io). You must explicitly clear and fill the buffer and explicitly read/or write it. It is up to you to avoid overfilling the buffer.
• It is a very mundane buffer. The only thing that makes it much different from a raw byte[] is the way may be backed something that only looks like a byte[] but is not really, e.g. an entire memory mapped file or direct I/O buffer containg part of a file.

How It Works

• You would expect ByteBuffer to have a length() method, it does not. Instead it has a several length-like concepts:
  mark <= position <= limit <= capacity.
  Just what do these term mean in English? Working right to left:

  • capacity

    Inside the ByteBuffer, there is a backing byte[] or something that behaves much like one. The capacity is its size. The capacity indexes the first slot past the end of the buffer.

  • limit

    When filling the buffer, the limit is the same as the capacity. When emptying the buffer, it is one past the last filled byte in the buffer.

  • position

    When filling the buffer,the position points just past the last byte filled in the buffer. When emptying the buffer, the position points just past the last byte written from the buffer.

  • mark

    The mark is an optional bookmark to let you record an interesting spot in the ByteBuffer that you want to return to later. When you take a mark() it records current position, and when you call reset() it restores that position.
• There nothing in the least automatic about reading a file with a ByteBuffer. Here is the barebones code to read a file using a ByteBuffer and FileChannel. If you run it and look at the log it produces, that should help you to get an understanding of how position, limit, remaining and capacity. work.
  1. Note how you must call ByteBuffer.clear to empty the buffer prior to the physical read.
  2. Then you call FileChannel.read to do the physical read.
  3. You then must call ByteBuffer.flip to convert from filling the buffer via physical I/O to emptying it via ByteBuffer. get.
  4. Then you use ByteBuffer.get to fetch the bytes out of the buffer.
• Similarly there nothing in the least automatic about writing a file with a ByteBuffer. Here is the barebones code to write to a file using a ByteBuffer and FileChannel. If you run it and look at the log it produces, that should help you to get an understanding of how position, limit, remaining and capacity. work.
  1. Note how you must call ByteBuffer.clear to empty the buffer prior filling it with ByteBuffer. put.
  2. Then you use ByteBuffer.put to fill the buffer with bytes.
  3. You then must call ByteBuffer.flip to convert from filling the buffer via ByteBuffer.put to emptying it via a physical write.
  4. Then you call FileChannel.write to do the physical write.
• You would expect ByteBuffer to have a constructor that takes a byte[]. It does not. ByteBuffer has no constructors at all. Instead you say ByteBuffer. wrap( byte[] ) or ByteBuffer. allocate( int ).
• You would expect ByteBuffer to have a toArray method. It does not. It has an array method, but that gives you the entire backing array, not a copy of the contents. Randolf Richardson randolf@inter-corporate.com pointed out you can roll your own toArray function like this:

  // how to simulate a ByteBuffer.toArray
  ByteBuffer bb;
  ...
  byte[] contentsOnly = Arrays.copyOf( bb.array(), bb.position() );

Operations

1. To get started you can clear. This sets the limit to the capacity and the position to zero, thus freeing up all the buffer space and making the buffer logically empty. It does not actually zero the backing array.
2. You can add data to the buffer with put. When the buffer is full, you get a BufferOverflowException. The call does not block until space is freed up. put has several variants to put just one or multiple bytes, to put the next bytes, or to specify the offset either absolutely (relative to the beginning of the buffer) or relatively (relative to the current position). It works analogously to the seek pointer manipulations in file I/O.
3. The way to prepare to read is to use flip rather than rewind. It sets the limit to the current position and then sets the position to zero. Using flip will prevent you reading with get out past where you have written. You must call flip exactly once. If you fail to call it or call it twice, the ByteBuffer will appear to be empty. To make things worse, ByteBuffer often calls flip for you automatically e.g. on FileChannel. map( FileChannel. MapMode. READ_ONLY,...) and on ByteBuffer. wrap. I consider this design grossly incompetent. The design maliciously attempts to trip up programmers.
4. You then read data out of the buffer with get. When you bang into the limit, you get a Buffer UnderflowException. Normally, you count your reads with a for loop up to the limit. The get call does not block until more data are available. get has several variants to get just one or multiple bytes, to get the next bytes, or to specify the offset either absolutely (relative to the beginning of the buffer) or relatively (relative to the current position). It works analogously to the seek pointer manipulations in file I/O.
5. After you have read data, you can go back to the beginning and start reading again with rewind, which leaves the limit unchanged and sets the position to zero. You must use flip before the first read pass and rewind before subsequent ones. If you screw this up you will either see an empty buffer or read gibberish out past the end of the data, all without error messages or exceptions!

Sample Code

Sequentially Reading a File

If you don’t want to allocate large hunks of your virtual address space, you could allocate a smaller MappedByteBuffer at some offset in the file other than 0, and read a decently large chunk of it. When done, allocate a new MappedByteBuffer. You can be considerably more generous in your chunk size than when allocating buffers.

Alternatively, you could do your I/O in a more conventional way using FileChannel. read( ByteBuffer dst ), to read the next chunk of the file into a pre-allocated ByteBuffer. This approach is clumbsier than traditional stream I/O, but can be more efficient, especially when you slew over most of the data, or access it via the backing array. It will pay off if for example you were processing just a 4-byte field in a 512-byte record, since only the bytes you need are copied from the buffer, not the entire record.

The effect is even more pronounced with MappedBuffers and large records where pages of records you don’t need are not even read in to RAM.

Handling Little Endian Files

Learning More