Mike`s Blog - Java

Booting a JVM

info@mhaller.de (Mike Haller) — Tue, 01 Mar 2011 23:49:00 +0100

I'm right in the middle of implementing an interpreter for JVM bytecode. Today was the first time the system printed out the magic "Hello World!" message.

The interpreted boot time for running on a JDK 1.6 host (i'm reusing their classes) is 4667ms. It takes 10ms to execute System.out.printlnt("Hello World!");.

Starting up the Virtual Machine and executing any class is pretty easy:

final VirtualMachine vm = new VirtualMachine();

final Classpath classpath = new Classpath();
classpath.addLoader(new HostVMResourceLoader()); // Re-uses JDK classes
classpath.addFolder("C:/Example/bin"); // The HelloWorld project
vm.setClasspath(classpath);

vm.start();

vm.execute("org/example/HelloWorld");

The output can be seen in the screenshot:

The following is an histogram of the bytecodes being executed, only to start up and initialize the JVM.
That includes booting up the System class, character sets, ThreadGroups, Reflection, NIO, and a lot of native methods which had to be replaced.

[01] aconst_null                3398 invocations
[02] iconst_m1                    31 invocations
[03] iconst_0                  33689 invocations
[04] iconst_1                  11795 invocations
[05] iconst_2                    445 invocations
[06] iconst_3                    181 invocations
[07] iconst_4                    121 invocations
[08] iconst_5                     57 invocations
[09] lconst_0                     18 invocations
[0a] lconst_1                      3 invocations
[0b] fconst_0                     19 invocations
[10] bipush                    58215 invocations
[11] sipush                      222 invocations
[12] ldc                        1287 invocations
[13] ldc_w                       494 invocations
[14] ldc2_w                       15 invocations
[15] iload                    362633 invocations
[16] lload                         5 invocations
[19] aload                     81878 invocations
[1a] iload_0                  100335 invocations
[1b] iload_1                  236106 invocations
[1c] iload_2                    1208 invocations
[1d] iload_3                    1383 invocations
[1e] lload_0                       3 invocations
[1f] lload_1                       4 invocations
[20] lload_2                       6 invocations
[22] fload_0                      15 invocations
[24] fload_2                      52 invocations
[25] fload_3                      24 invocations
[26] dload_0                       1 invocations
[2a] aload_0                  306476 invocations
[2b] aload_1                   43405 invocations
[2c] aload_2                   25744 invocations
[2d] aload_3                   83195 invocations
[32] aaload                      794 invocations
[33] baload                        6 invocations
[34] caload                    78421 invocations
[36] istore                   161005 invocations
[37] lstore                        2 invocations
[3a] astore                    26685 invocations
[3b] istore_0                      2 invocations
[3c] istore_1                    907 invocations
[3d] istore_2                    121 invocations
[3e] istore_3                    185 invocations
[3f] lstore_0                      1 invocations
[40] lstore_1                      2 invocations
[41] lstore_2                      3 invocations
[4b] astore_0                     35 invocations
[4c] astore_1                    742 invocations
[4d] astore_2                   2411 invocations
[4e] astore_3                  19006 invocations
[4f] iastore                      20 invocations
[53] aastore                    1654 invocations
[54] bastore                      19 invocations
[55] castore                     265 invocations
[57] pop                         808 invocations
[58] pop2                          4 invocations
[59] dup                        4418 invocations
[5a] dup_x1                        4 invocations
[5c] dup2                          1 invocations
[60] iadd                      86425 invocations
[61] ladd                          1 invocations
[64] isub                      61816 invocations
[68] imul                        814 invocations
[6a] fmul                         11 invocations
[70] irem                        135 invocations
[78] ishl                          8 invocations
[7a] ishr                          9 invocations
[7c] iushr                         8 invocations
[7e] iand                        163 invocations
[7f] land                          1 invocations
[80] ior                       25086 invocations
[81] lor                           1 invocations
[82] ixor                          8 invocations
[84] iinc                      29597 invocations
[85] i2l                           1 invocations
[86] i2f                          17 invocations
[8b] f2i                          11 invocations
[91] i2b                          13 invocations
[92] i2c                       25083 invocations
[94] lcmp                         13 invocations
[95] fcmpl                        19 invocations
[96] fcmpg                        19 invocations
[99] ifeq                      25358 invocations
[9a] ifne                      42109 invocations
[9b] iflt                     102593 invocations
[9c] ifge                      12405 invocations
[9d] ifgt                         27 invocations
[9e] ifle                       6280 invocations
[9f] if_icmpeq                 51280 invocations
[a0] if_icmpne                   813 invocations
[a1] if_icmplt                 77656 invocations
[a2] if_icmpge                 52068 invocations
[a3] if_icmpgt                    73 invocations
[a4] if_icmple                    52 invocations
[a5] if_acmpeq                  1612 invocations
[a6] if_acmpne                  2072 invocations
[a7] goto                      66816 invocations
[a8] jsr                           1 invocations
[a9] ret                           1 invocations
[ab] lookupswitch                  1 invocations
[ac] ireturn                  164404 invocations
[ad] lreturn                       6 invocations
[b0] areturn                   14314 invocations
[b1] return                     6938 invocations
[b2] getstatic                 39043 invocations
[b3] putstatic                   201 invocations
[b4] getfield                 310885 invocations
[b5] putfield                  15212 invocations
[b6] invokevirtual            102940 invocations
[b7] invokespecial              3603 invocations
[b8] invokestatic              65929 invocations
[b9] invokeinterface           13227 invocations
[bb] new                         950 invocations
[bc] newarray                     40 invocations
[bd] anewarray                   477 invocations
[be] arraylength                 764 invocations
[c0] checkcast                 23360 invocations
[c1] instanceof                   55 invocations
[c2] monitorenter                151 invocations
[c3] monitorexit                 151 invocations
[c6] ifnull                    15595 invocations
[c7] ifnonnull                 19950 invocations

Total: 3118655

Three million bytecode instructions executed to get a little message on the screen.

Spatial search with Lucene

info@mhaller.de (Mike Haller) — Sat, 16 Oct 2010 21:52:10 +0200

At the Hadoop Hackathon this weekend at Sybit, we've worked on a getting tiled images into HDFS and HBase. A side-story was how to search for these tiles based on GPS coordinates. I took the task to see how you can perform queries based on spatial coordinates. That is, searching for locations which are within a specified radius from an origin:

Continue reading "Spatial search with Lucene"

Visualizing Dataflow

info@mhaller.de (Mike Haller) — Mon, 21 Jun 2010 22:49:30 +0200

I've been playing around recently regarding visualization of dataflow in Java applications. It seems that there are tons of tools to inspect the control flow, but I had no luck yet finding something which can visualize the amount of data and the type of data flowing through complex systems.

Not being an AOP guru, i wondered if there was something else to use. Something, which is unobtrusive and can be applied to existing systems. The first thing i'm trying is Java's Debugging APIs, namely JDI, to automatically step through a program and record method entries.

Continue reading "Visualizing Dataflow"

Popularity of log frameworks

info@mhaller.de (Mike Haller) — Tue, 04 May 2010 08:43:44 +0200

Logging is some weird kind of aspect of software development. It's a necessary evil so to say. On the one hand, it's great to have the flexibility to either log in a file or to console, to roll the files, to enable certain log statements at runtime etc. But on the other hand, there are so many logging frameworks and so manyof them did wrong in their first encounter, it became quiet insane.

A few years ago, driven by the well-known problems of commons-logging, we've switched over to slf4j with log4j as engine. I don't regret that, as I like the slf4j-api and modularized design. I also kind of like log4j - its log4j.xml, its debug system parameters and that its practically spread all over the Java world. Yes, sure, there are some minor drawbacks, like the inability of the log4j committers to provide clean Maven pom.xml files for some version. Did they fix that btw?

Anyway, to cut a long story short: i've played around with Neo4j and Maven and came up with a quick Maven Repository scanner which counts the dependencies from reverse: it shows how often a certain artifact is referenced by another project as dependency. The data basis was my local repository (11988 artifacts) and it's surely biased to some degree. I'll try to get a clean Maven Repo mirror some day and rerun the experiment.

The winner is:

1162 times used (9,69%): org.slf4j
867 times used (7,23%): log4j
837 times used (6,98%): commons-logging

There is other interesting information in that experiment, like
20,33% of the artifacts are using junit, thereas 1,9% use testng. 21,13% use the Spring Framework and 6,45% of all the Maven artifacts in my repo use the javax.servlet spec. Easymock is used by 5,62% of the artifacts. Other popular libraries include com.sun.xml.bind, commons-lang, asm, Xerces, Apache Camel, Jetty, javax.xml.bind, javax.xml.soap, wsdl4j, commons-collections, Ant, Apache Cocoon, Apache Abdera, org.eclipse, commons-io, cglib, hibernate, velocity, woodstox and derby.

Consolas

info@mhaller.de (Mike Haller) — Sat, 23 Jan 2010 15:27:12 +0100

Dan writes about it, Sven & Vitaly are writing about it, Jeff, Scott and Steve found it years ago and are probably using it ever since. Many StackOverflow users do it as well. So why don't I?

Well, now I did and so far, it's promosing.
I switched from Courier New to Consolas.

Besides gaining some more space, it really looks much nicer.

But compare for yourself:

Consolas is a commercial font and if you're on Windows and have any of the Microsoft products installed, chances are that you already have Consolas installed on your system. If you don't, check out the other programming fonts features on Smashing Magazines articles.

Tracing with JDK Logging

info@mhaller.de (Mike Haller) — Sat, 16 Jan 2010 16:02:01 +0100

Imagine you're writing a small test case to check out some new library or version or cool new technology.

You're eager to try out that stuff, learn how it works and how to use it.
But, unfortunately, something goes wrong and you see the following:

Exactly, you see nothing.

You didn't quite read the quick start tutorial carefully and you must have missed something at the configuration file or at the code. You're not sure - all you get is a mysterious exception or the code just doesn't do anything visible, as it does here. At this point, you just want it to print out what's going on, anything - you would like to have some fine-grained logging output on your console.

You know it's a pain to set up all that logging facility stuff correctly and your loved pre-configured log4j.xml with all the handy shortcuts is on the other machine. Okay, all you needed was just the whole debug output of that library to get it to know better and see what's going wrong.

The project already has commons-logging on it's classpath, but gladly no implementation, so it's using JDK Logging as a backend. Commons-logging should die anyway. Really. I mean it.. You can see that it's using JDK Logging since the console output is two-line formatted like the following snippet. The name of the logger is the adapter used to bridge between Commons-Logging and JDK Logging, and in this case it's iBatis 3 JakartaCommonsLoggingImpl.
16.01.2010 16:20:27 org.apache.ibatis.logging.commons.JakartaCommonsLoggingImpl debug
FEIN: PooledDataSource forcefully closed/removed all connections.
16.01.2010 16:20:27 org.apache.ibatis.logging.commons.JakartaCommonsLoggingImpl debug
FEIN: PooledDataSource forcefully closed/removed all connections.

Now, back to the example which is currently not printing out anything. To get decent logging output to your console with standard JDK logging, you either have to create a logging.properties and fill it with some obscure properties like .handlers and anyway you don't know how to configure that correctly. Isn't there anywhere a good and quick to find example for that?

I started using this code, which seems to be the simplest possible way of getting something to the console quickly.

Logger rootLogger = LogManager.getLogManager().getLogger("");
Handler handler = new ConsoleHandler();
handler.setLevel(Level.ALL);
rootLogger.addHandler(handler);
rootLogger.setLevel(Level.ALL);

As you can see, there's some output now.
But it's red (System.err stream) and there are too many newlines for my taste.

It uses the root logger, identified by the empty string (""), adds a Handler, which is responsible for printing the log messages to the console. Other Handlers write the log messages to the file system or send it over the wire. The Level is the log level, which determines which log messages are being sent to the Handler and which are ignored. For example, you can set it to INFO, so only Info, Warning, Error-level log messages are being printed. Log messages of level Debug and Trace are being ignored.

Since I don't like the output being spread over two lines (it makes it reading much harder, especially if your output is tabular like like the iBatis output for java.sql statements. So I ended up with something like this, to format the lines into a single line and get it to print out on System output console instead of System error:

Logger logger = LogManager.getLogManager().getLogger("");
class RedirectingConsoleHandler extends ConsoleHandler {
	public RedirectingConsoleHandler(OutputStream target) {
		super();
		setOutputStream(target);
	}
}
;
Handler handler = new RedirectingConsoleHandler(System.out);
handler.setLevel(Level.ALL);
// handler.setFormatter(new XMLFormatter());
Formatter newFormatter = new Formatter() {

	@Override
	public String format(LogRecord record) {
		return String.format("%tT %-40.40s %-8.8s %s\n", new Date(),
				record.getLoggerName(), record.getLevel().getLocalizedName(),
				formatMessage(record));
	}
};
handler.setFormatter(newFormatter);
logger.addHandler(handler);
logger.setLevel(Level.ALL);

This code will redirect the output of the Logging facility to a configure stream, in this case the System.out stream. That's the RedirectingConsoleHandler class is for. The anonymous Formatter class will format the log message, e.g. add a timestamp, the name of the logger, on which level it was logged and the actual log message of the log record and only use one single line. The output looks much more how I like it:

(Yeah, i know the SQL statements are truncated at the example screenshot. But in modern times, with larger screens, it's no problem since the consoles are much broader than the space available on my blog)

Note: this is only used for testing, not for production, not for performance measurements, not for audit logging. This kind of logging is only meant for technical debug tracing. Logging is among the Top10 unknown performance killers in enterprise applications.

Memory leaks et alii

info@mhaller.de (Mike Haller) — Wed, 30 Dec 2009 14:30:15 +0100

Leaky web application

One of the most common problems in building enterprise web applications are leaks. A leak is consumption of a resource by a program where the program is unable to release the resource. Leaks come in various types, such as

Memory leaks

Thread and ThreadLocal leaks

ClassLoader leaks

System resource leaks

Connection leaks

Continue reading "Memory leaks et alii"

Per-unit-test profiling

info@mhaller.de (Mike Haller) — Mon, 16 Nov 2009 23:01:55 +0100

As a preparation for long-term performance measurements, (sometimes called Continuous Performance Management or CPM), I tried to automate the creation of JProfiler profiling snapshots and, along the way, came up with the following side product: Automated per-test method profiling.

Basically, the idea is to go from here:

to here

"What's so great about this?", you wonder?

Read the full article for fancy screenshots and code snippets.

(Btw, the test cases are due to this question. Usually, i don't do String performance tests

Continue reading "Per-unit-test profiling"

Top10 programming controversial

info@mhaller.de (Mike Haller) — Wed, 04 Nov 2009 11:02:37 +0100

There's a Stackoverflow question about controversial programming opinions:
The top ten as voted by the SO community:

1. The only "best practice" you should be using all the time is Use Your Brain
2. Most comments in code are in fact a pernicious form of code duplication
3. Programmers who don't code in their spare time for fun, will never become as good as those that do
4. Not all programmers are created equal
5. XML is highly overrated
6. "Googling it" is okay!
7. I fail to understand why people think that Java is absolutely the best "first" programming language to be taught in universities
8. If you only know *one language*, no matter how well you know it, you're not a great programmer
9. Performance does matter
10. The Business Apps farce (You have to read it to understand)

Where have all the threads gone?

info@mhaller.de (Mike Haller) — Mon, 06 Jul 2009 15:37:04 +0200

Recently, we got a nasty error in the log file:

java.lang.OutOfMemoryError: unable to create new native thread

Although we had specified -XX:HeapDumpOnOutOfMemoryError, this does not include dumping the heap to disk when the heap is not involved in an OutOfMemoryError. In this case, the JVM was unable to create new threads as the limit of threads was reached.

So, the question is: How many threads can be created?
The answer is, as usual: It depends!

Continue reading "Where have all the threads gone?"

My Own Twitter Client

info@mhaller.de (Mike Haller) — Sun, 17 May 2009 22:01:31 +0200

Using twitter4j, it's pretty easy to hack together a new Twitter desktop client. What I wanted to do is to have search capabilities and have all the tweets in a local database, so I could have a look and work with the messages people write even when i'm offline (which never happens anyway, but just in case).

So, taking Eclipse RCP, munching it with twitter4j, adding a bit of Apache Derby and other stuff, I ended up with the following prototype in a couple of hours:

It has a virtual table, so it loads tweets on-demand from the local database when you scroll down. It only reloads the latest 20 tweets from Twitter.com in online mode. There is an offline-mode to decrease the amount of network bandwidth and service utilization. Twitter is already pretty slow. The client caches all the profile images for faster loading. It displays the time of the tweet and highlights URLs. I had a feature in it which resolves shortened URLs and displays the real ones on hover, but i disabled it as I've got problems displaying a nice hover. Hover's in Eclipse RCP/SWT/JFace just don't look very nice, compared to all the new Web 2.0 or Mac designs found in modern apps today.

What other features could be put into yet-another-twitter-client? Tag clouds? URL clouds? Charts? Google Maps? Automatic translations like Babelfish?

In-Memory Databases

info@mhaller.de (Mike Haller) — Tue, 05 May 2009 21:58:48 +0200

For testing and demo installation purposes, or for in-memory data aggregation operations, it may be suitable to use a "nearly full-blown" database. They're called in-memory databases and there are some implementation for the Java platform which are used since years:

hsqldb
One of the first 100% Java implementations of a SQL relational database management system. Like all the others, it can be run from a persistent file system or run in-memory. It's fast, it has a very low footprint and it can be embedded in any Java application. Hsqldb was my favourite for years to use in unit tests and integration tests. It can also read and write CSV files.

H2Database
The successor of hsqldb is H2. It's better documented, performs even better and has more options. There's even a web console where you can work with the data in the in-memory database like you can do with phpMyAdmin. Great thing. I'm currently using H2 at home and at work.

Derby
Derby, the Java database, now also supports the in-memory protocol. That's great to hear and an additional argument for switching to Derby as the default implementation for databases in embedded versions or for demo installations.

Monitoring code quality

info@mhaller.de (Mike Haller) — Sun, 26 Apr 2009 13:35:02 +0200

I've been looking for tools to measure and visualize code quality for some time now. Every once in a while, there's some tool which aggregates results from quality measurements and monitoring tools like PMD, Findbugs, Checkstyle etc. but I haven't yet found something which attracted my interest.

Read on to find out about Codehaus' Sonar.

Codehaus Sonar for code quality

Continue reading "Monitoring code quality"

Java Class Popularity

info@mhaller.de (Mike Haller) — Fri, 17 Apr 2009 08:00:00 +0200

Taken from a heap dump of a JBoss Application Server with some applications deployed, i've look at the number of objects in the heap and compiled a 'popularity list' accordingly:

java.util
1. ArrayList
2. HashMap
3. TreeMap
4. Vector
5. Hashtable
6. LinkedList
7. HashSet
8. WeakHashMap
9. TreeSet

Read on to see the full list. I've marked some classes which I find interesting enough to have a second look on them. Know your library.

Continue reading "Java Class Popularity"

Date ranges in Lucene

info@mhaller.de (Mike Haller) — Sat, 04 Apr 2009 13:23:59 +0200

Lucene is a very efficient and fast Java search engine. Once indexed, any object can be found by looking for attribute matches. An object in Lucene is called Document and its attributes are called Fields. Lucene uses Query and Filter objects to narrow down the search to what the user wants to find. A Query makes the amount of data to be searched smaller while a Filter is used for more fine-grained control over what a search result shall include. Queries are more memory-hungry than filters, but for the usual use cases, both are very good.

In this post, I'd like to show how you can search for Date and Time ranges, for example modification dates of files. I'd like to find all files on my local file system which were modified in April 2009.

Continue reading "Date ranges in Lucene"