Wednesday, January 26, 2011

A lightweight anonymous visitor for Java collections

I got myself into a situation where I really wished that Java collections accepted a visitor interface of some kind so I could define one on the fly and get my job done. Alas, I decided to implement the utility myself and here's the result.

public interface IVisitor<T> {
void visit(T item);
}

public class CollectionUtils {
public static<T> void applyVisitor(Collection<T> collection, IVisitor<T> visitor) {
for (T item : collection)
visitor.visit(item);
}
}
A typical usage example would be:
public static void main(String[] args) {
CollectionUtils.applyVisitor(Arrays.asList(1, 2, 3, 4, 5), new IVisitor<Integer>() {
public void visit(Integer x) {
System.out.println(x);
}
});
}

Sunday, October 24, 2010

Efficient enumeration of all integers with a given pop count

Population count is the number of '1' bits in the binary representation of an integer.

The basic trick we employ is to get the next higher number with the same number of 1-bits, which we borrow from the Hacker's Delight (whole book). Below is a Java implementation of all 4 version of the snoob function.

public static int snoob1(int x) {
int r = x + (x & -x);
return r | ((x ^ r) >>> (2 + Integer.numberOfTrailingZeros(x)));
}

public static int snoob2(int x) {
int s = x & -x, r = x + s;
return r | ((x ^ r) >>> (33 - Integer.numberOfLeadingZeros(s)));
}

public static int snoob3(int x) {
int r = x + (x & -x);
return r | ((1 << (Integer.bitCount(x ^ r) - 2)) - 1);
}

public static int snoob4(int x) {
int s = x & -x, r = x + s;
return r | (((x ^ r) >> 2) / s);
}
We utlize this formula to implement the method below, where the width parameter is added for convenience.
public static void printAllIntegersOfWidthAndPop(int w, int n) {
if (n <= 0 || n > w || w > 32) return;
int x = (1 << n) - 1, g = x << (w - n), c = 0;
while (true) {
String s = Integer.toBinaryString(x);
while(s.length() < w) s = "0" + s;
System.out.println(++c + "\t" + s);
if (x == g) break;
else x = snoob*(x);
}
}
It is easy to figure out there will be Choose(w, n) such numbers. We know where the sequence starts and since we also know where it stops, we don't need to compute this number.

If you'd like to learn how to find the number of leading/trailing zeros or the pop count or learn about more cool bit twiddling hacks you can consult the book above (which the java.lang.Integer implementation of these methods is based on) or this magnificent web page.

Saturday, July 17, 2010

Generating combinations in lexicographical order using Java

Below is a Java port of the algorithm by James McCaffrey as presented in the MSDN article Generating the mth Lexicographical Element of a Mathematical Combination.

Please do take into consideration that this implementation only uses int as Java does not allow array indexing with long. This means that the valid input range is more restricted. Watch out for overflows!

/**
* Based on the Combinadic Algorithm explained by James McCaffrey
* in the MSDN article titled: "Generating the mth Lexicographical
* Element of a Mathematical Combination"
* <http://msdn.microsoft.com/en-us/library/aa289166(VS.71).aspx>
*
* @author Ahmed Abdelkader
* Licensed under Creative Commons Attribution 3.0
* <http://creativecommons.org/licenses/by/3.0/us/>
*/
public class Combinations {
/** returns the mth lexicographic element of combination C(n,k) **/
public static int[] element(int n, int k, int m) {
int[] ans = new int[k];

int a = n;
int b = k;
int x = (choose(n, k) - 1) - m; // x is the "dual" of m

for (int i = 0; i < k; ++i) {
a = largestV(a, b, x); // largest value v, where v < a and vCb < x
x = x - choose(a, b);
b = b - 1;
ans[i] = (n - 1) - a;
}

return ans;
}

/** returns the largest value v where v < a and Choose(v,b) <= x **/
public static int largestV(int a, int b, int x) {
int v = a - 1;

while (choose(v, b) > x)
--v;

return v;
}

/** returns nCk - watch out for overflows **/
public static int choose(int n, int k) {
if (n < 0 || k < 0)
return -1;
if (n < k)
return 0;
if (n == k || k == 0)
return 1;

int delta, iMax;

if (k < n - k) {
delta = n - k;
iMax = k;
} else {
delta = k;
iMax = n - k;
}

int ans = delta + 1;

for (int i = 2; i <= iMax; ++i) {
ans = (ans * (delta + i)) / i;
}

return ans;
}
}
The code below produced the output that follows:
public static void main(String[] args) {
int n = 5, k = 3;
int total = choose(n, k);
for(int i = 0; i < total; i ++) {
for(int x : element(n, k, i))
System.out.print(x + " ");
System.out.println();
}
}

0 1 2
0 1 3
0 1 4
0 2 3
0 2 4
0 3 4
1 2 3
1 2 4
1 3 4
2 3 4

Monday, June 21, 2010

G/G/1 Queue Model Simulation in Java

In this post, we present an object-oriented design and implementation of an event-driven G/G/1 queue model simulation using Java. The lecture notes on Computer System Analysis by Raj Jain were very helpful and are highly recommended. In particular, we would like to reference the introductory lectures on Simulation Modeling and Queueing Theory.

Please note that this implementation is provided as a guide/starting point and is not, by any means, complete. We preferred a simple design while keeping in mind where you may wish to extend it.

Update: Download the Eclipse project here.

An event-driven simulation operates by generating and processing events through time. For our queue model, we have two types of events: arrivals and departures or service completion. It is also necessary to determine which events should happen first.

public class Event implements Comparable<Event> {
protected double time;
protected int code;

...

public int compareTo(Event e) {
return Double.compare(time, e.time);
}
}
While the design will not make any assumptions about the interarrival or service time distributions, we still need a suitable way to represent them and a couple of concrete implementations for testing.
public abstract class Distribution {
public abstract double generateRV();
}

public class UniformDistribution extends Distribution {
double a, b;

...

public double generateRV() {
return a + rand.nextDouble() * (b - a);
}
}

public class ExponentialDistribution extends Distribution {
double lambda;

...

// Generating exponential variates.
public double generateRV() {
return -1/lambda * Math.log(rand.nextDouble());
}
}

public class NormalDistribution extends Distribution {
double mu, segma;

...

public double generateRV() {
return mu + rand.nextGaussian() * segma;
}
}
Now, we should be ready to implement the queue class. We preferred to keep the queue simple by moving data collection into a separate arbitrary observer.
/**
* @author Ahmed Abdelkader
* Licensed under Creative Commons Attribution 3.0
* <http://creativecommons.org/licenses/by/3.0/us/>
*/
public class GG1Q {
protected Distribution arrivalDistribution;
protected Distribution serviceDistribution;
protected double t;
protected PriorityQueue<Event> eventQueue = new PriorityQueue<Event>();
protected int customers;
protected QueueObserver observer;

...

public void run(double duration) {
eventQueue.add(new Event(t + arrivalDistribution.generateRV(), Event.ARRIVAL));
while(t < duration) {
Event e = eventQueue.poll();
t = e.time;
switch(e.code) {
case Event.ARRIVAL:
customers++;
eventQueue.add(new Event(t + arrivalDistribution.generateRV(), Event.ARRIVAL));
if(customers == 1)
eventQueue.add(new Event(t + serviceDistribution.generateRV(), Event.SERVICE));
break;
case Event.SERVICE:
customers--;
if(customers > 0)
eventQueue.add(new Event(t + serviceDistribution.generateRV(), Event.SERVICE));
break;
}
if(observer != null)
observer.stateChanged(this, e);
}
}

...
}
We define the observer interface and implement two sample observers: one for estimating the queue length PDF and the other for the waiting time CDF. A composite observer allows more than one observer to collect data of a single run.
public interface QueueObserver {
void stateChanged(GG1Q q, Event e);
void printStats();
}

public class CompositeQueueObserver implements QueueObserver {
protected ArrayList<QueueObserver> observers = new ArrayList<QueueObserver>();

public void addObserver(QueueObserver observer) {
observers.add(observer);
}

public void stateChanged(GG1Q q, Event e) {
for(QueueObserver observer : observers)
observer.stateChanged(q, e);
}

public void printStats() {
for(QueueObserver observer : observers)
observer.printStats();
}
}

public class QueueLengthQueueObserver implements QueueObserver {
protected TreeMap<Integer, Integer> lengthFrequencies = new TreeMap<Integer, Integer>();
protected int total;

public void stateChanged(GG1Q q, Event e) {
if(e.getCode() != Event.ARRIVAL) return;
Integer length = lengthFrequencies.get(q.getLength());
if(length == null) length = 0;
lengthFrequencies.put(q.getLength(), length + 1);
total++;
}

public void printStats() {
System.out.println(getClass().getSimpleName());
for(int length : lengthFrequencies.keySet())
System.out.println(length + "\t" + 1.0*lengthFrequencies.get(length)/total);
}
}

public class WaitingTimeQueueObserver implements QueueObserver {
protected ArrayList<Double> waitingTimes = new ArrayList<Double>();
protected int arrivalIndex, serviceIndex;
protected HashMap<Integer, Double> arrivalTimes = new HashMap<Integer, Double>();

public void stateChanged(GG1Q q, Event e) {
switch(e.getCode()) {
case Event.ARRIVAL:
arrivalTimes.put(arrivalIndex++, e.getTime());
break;
case Event.SERVICE:
waitingTimes.add(e.getTime() - arrivalTimes.get(serviceIndex++));
break;
}
}

public void printStats() {
System.out.println(getClass().getSimpleName());
if(waitingTimes.size() == 0) return;
double acc = 0;
Collections.sort(waitingTimes);
for(int i = 1, j = 0; i <= 10; i++) {
while(acc < i/10.0 && j < waitingTimes.size()) {
acc += 1.0/waitingTimes.size();
j++;
}
System.out.println(waitingTimes.get(j-1) + "\t" + i/10.0);
}
}
}
The following test program produced the output below:
public class Main {
public static void main(String[] args) {
GG1Q q = new GG1Q(new ExponentialDistribution(100.0/3600), new UniformDistribution(1, 10));
QueueObserver observer = new QueueLengthQueueObserver();
q.setObserver(observer);
q.run(1000);
observer.printStats();
}
}

QueueLengthQueueObserver
Queue length PDF:
1 0.47058823529411764
2 0.23529411764705882
3 0.058823529411764705
4 0.058823529411764705
5 0.11764705882352941
6 0.058823529411764705

Wednesday, April 28, 2010

GeoTraffic & GeoKeywords: Google Analytics custom reports for fun and profit

I've been using Google Analytics for 2 years to track the traffic on this blog. It's really interesting to see which posts receive more attention and what traffic sources and keywords generated how much traffic on any one day. This is specially important to me since the majority of the traffic to my blog comes from search engines a.k.a. google (organic).

However, when you really look into the traffic tracking thing, you'll find that the default reports which come out-of-the-box with Google Analytics don't help answer certain questions. This is more evident on a modest personal blog than it would be for a hot website like stackoverflow, for example. For your blog, you'd actually want to know who visited the blog. I mean, you'd like to push the limits of anonymity.

The main missing information in the default Analytics reports have to do with location. While you're able to know how many visits were generated from each country in addition to some useful statistics like the average time on website and pages/visit, there is still something missing.

I wanted to be able to answer these 2 critical questios:
1) Where does the direct traffic come from? and which pages are requested?
2) Who is searching for what?

Thanks to the Google Analytics team, we are now able to create our own custome reports to project the data whichever way we want. Once I found out, I created 2 custom reports to answer my 2 questions.

For the first question, the GeoTraffic report is the answer. The main dimension would be the Source. Next, we drill down towards: Country/Territory -> City -> Page. The main metric is Pageviews, then Pages/Visit and Avg. time on page.

For the second question, the GeoKeywords report is the answer. This time, the main dimension is the Country/Territory then we drill down to: Keyword -> City. For this report, I preferred the main metric to be Avg. time on page then Pages/Visit and Unique Visitors.

Now, I'm able to answer my questions, and I've been mainly interested in the first one: where the direct traffic comes from. The reason for that is that I've been waiting for something. I may talk about that later. I guess you're going to find it useful ;)

Friday, February 5, 2010

Introducing Google App Engine - All in One

Google App Engine lets you run your web applications on Google's infrastructure e.g. GFS and BigTable. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow. With App Engine, there are no servers to maintain: You just upload your application, and it's ready to serve your users... more docs. An alternative introduction is available on Wikipedia.

I compiled a list of videos that should get your engines up and running on this exciting web framework:

Campfire One - Introducing Google App Engine:

  1. Pt. 1 (9:10)
  2. Pt. 2 (12:34)
  3. Pt. 3 (13:19)
  4. Pt. 4 (7:44)
  5. Pt. 5 (5:55)
  6. Pt. 6 (8:04)
Overviews:
  1. Google App Engine - Early Look at Java Language Support (7:38)
  2. Overview of Google Web Toolkit (4:10)
  3. Getting Started with App Engine in Eclipse (5:07)
Google I/O 2008:
  1. Google I/O 2008 - Working with Google App Engine Models (1:00:32)
  2. Google I/O 2008 - Building Quality Apps on App Engine (48:43)
  3. Google I/O 2008 - Engaging User Experiences with App Engine (45:32)
  4. Google I/O 2008 - Python, Django, and App Engine (57:09)
Google I/O 2009:
  1. Google I/O 2009 - A Preview of Google Web Toolkit 2.0 (1:00:53)
  2. Google I/O 2009 - App Engine: Now Serving Java (55:00)
  3. Google I/O 2009 - Groovy and Grails in App Engine (1:00:14)
  4. Google I/O 2009 - Java Persistence & App Engine Datastore (1:09:32)
  5. Google I/O 2009 - ThoughtWorks on App Engine for Java (1:04:18)
Check out the project homepage and the developer's guide for more and subscribe to the Google App Engine Blog.

Thursday, January 28, 2010

Auto-formatting phone number UITextField on the iPhone

In this post, we'll use our PhoneNumberFormatter to implement an auto-formatting phone number text field in an attempt to mimic the behavior of iPhone's native apps, like Phone and Contacts.

So, let's call the text field in question myTextField. We start by calling addTarget on the text field to make it call the autoFormatTextField method in our view controller whenever it gets updated, either by the user or some other piece of code. If your view is ready, you should declare the handler as an IBAction and bind through IB. The method would then update the contents of the text fields to the formatted string returned by the formatter. If we do it that way, we'll also need to use a semaphore to prevent the method from being called endlessly.

The implementation outline would be:

// declarations

UITextField *myTextField;
int myTextFieldSemaphore;
PhoneNumberFormatter *myPhoneNumberFormatter;
NSString *myLocale; //@"us"

// init semaphore
myTextFieldSemaphore = 0;

// bind events programatically, skip when using IB.
[myTextField addTarget:self
action:@selector(autoFormatTextField:)
forControlEvents:UIControlEventValueChanged
];

// handle events
- (void)autoFormatTextField:(id)sender {
if(myTextFieldSemaphore) return;
myTextFieldSemaphore = 1;
myTextField.text = [phoneNumberFormatter format:myTextField.text withLocale:myLocale];
myTextFieldSemaphore = 0;
}
Update: As pointed out in the comments, for newer SDKs you may need to bind to UIControlEventEditingChanged. Thanks for your feedback!
//bind events

[myTextField addTarget:self
action:@selector(autoFormatTextField:)
forControlEvents:UIControlEventEditingChanged
];