Monday, June 21, 2010

G/G/1 Queue Model Simulation in Java

In this post, we present an object-oriented design and implementation of an event-driven G/G/1 queue model simulation using Java. The lecture notes on Computer System Analysis by Raj Jain were very helpful and are highly recommended. In particular, we would like to reference the introductory lectures on Simulation Modeling and Queueing Theory.

Please note that this implementation is provided as a guide/starting point and is not, by any means, complete. We preferred a simple design while keeping in mind where you may wish to extend it.

Update: Download the Eclipse project here.

An event-driven simulation operates by generating and processing events through time. For our queue model, we have two types of events: arrivals and departures or service completion. It is also necessary to determine which events should happen first.

public class Event implements Comparable<Event> {
protected double time;
protected int code;

...

public int compareTo(Event e) {
return Double.compare(time, e.time);
}
}
While the design will not make any assumptions about the interarrival or service time distributions, we still need a suitable way to represent them and a couple of concrete implementations for testing.
public abstract class Distribution {
public abstract double generateRV();
}

public class UniformDistribution extends Distribution {
double a, b;

...

public double generateRV() {
return a + rand.nextDouble() * (b - a);
}
}

public class ExponentialDistribution extends Distribution {
double lambda;

...

// Generating exponential variates.
public double generateRV() {
return -1/lambda * Math.log(rand.nextDouble());
}
}

public class NormalDistribution extends Distribution {
double mu, segma;

...

public double generateRV() {
return mu + rand.nextGaussian() * segma;
}
}
Now, we should be ready to implement the queue class. We preferred to keep the queue simple by moving data collection into a separate arbitrary observer.
/**
* @author Ahmed Abdelkader
* Licensed under Creative Commons Attribution 3.0
* <http://creativecommons.org/licenses/by/3.0/us/>
*/
public class GG1Q {
protected Distribution arrivalDistribution;
protected Distribution serviceDistribution;
protected double t;
protected PriorityQueue<Event> eventQueue = new PriorityQueue<Event>();
protected int customers;
protected QueueObserver observer;

...

public void run(double duration) {
eventQueue.add(new Event(t + arrivalDistribution.generateRV(), Event.ARRIVAL));
while(t < duration) {
Event e = eventQueue.poll();
t = e.time;
switch(e.code) {
case Event.ARRIVAL:
customers++;
eventQueue.add(new Event(t + arrivalDistribution.generateRV(), Event.ARRIVAL));
if(customers == 1)
eventQueue.add(new Event(t + serviceDistribution.generateRV(), Event.SERVICE));
break;
case Event.SERVICE:
customers--;
if(customers > 0)
eventQueue.add(new Event(t + serviceDistribution.generateRV(), Event.SERVICE));
break;
}
if(observer != null)
observer.stateChanged(this, e);
}
}

...
}
We define the observer interface and implement two sample observers: one for estimating the queue length PDF and the other for the waiting time CDF. A composite observer allows more than one observer to collect data of a single run.
public interface QueueObserver {
void stateChanged(GG1Q q, Event e);
void printStats();
}

public class CompositeQueueObserver implements QueueObserver {
protected ArrayList<QueueObserver> observers = new ArrayList<QueueObserver>();

public void addObserver(QueueObserver observer) {
observers.add(observer);
}

public void stateChanged(GG1Q q, Event e) {
for(QueueObserver observer : observers)
observer.stateChanged(q, e);
}

public void printStats() {
for(QueueObserver observer : observers)
observer.printStats();
}
}

public class QueueLengthQueueObserver implements QueueObserver {
protected TreeMap<Integer, Integer> lengthFrequencies = new TreeMap<Integer, Integer>();
protected int total;

public void stateChanged(GG1Q q, Event e) {
if(e.getCode() != Event.ARRIVAL) return;
Integer length = lengthFrequencies.get(q.getLength());
if(length == null) length = 0;
lengthFrequencies.put(q.getLength(), length + 1);
total++;
}

public void printStats() {
System.out.println(getClass().getSimpleName());
for(int length : lengthFrequencies.keySet())
System.out.println(length + "\t" + 1.0*lengthFrequencies.get(length)/total);
}
}

public class WaitingTimeQueueObserver implements QueueObserver {
protected ArrayList<Double> waitingTimes = new ArrayList<Double>();
protected int arrivalIndex, serviceIndex;
protected HashMap<Integer, Double> arrivalTimes = new HashMap<Integer, Double>();

public void stateChanged(GG1Q q, Event e) {
switch(e.getCode()) {
case Event.ARRIVAL:
arrivalTimes.put(arrivalIndex++, e.getTime());
break;
case Event.SERVICE:
waitingTimes.add(e.getTime() - arrivalTimes.get(serviceIndex++));
break;
}
}

public void printStats() {
System.out.println(getClass().getSimpleName());
if(waitingTimes.size() == 0) return;
double acc = 0;
Collections.sort(waitingTimes);
for(int i = 1, j = 0; i <= 10; i++) {
while(acc < i/10.0 && j < waitingTimes.size()) {
acc += 1.0/waitingTimes.size();
j++;
}
System.out.println(waitingTimes.get(j-1) + "\t" + i/10.0);
}
}
}
The following test program produced the output below:
public class Main {
public static void main(String[] args) {
GG1Q q = new GG1Q(new ExponentialDistribution(100.0/3600), new UniformDistribution(1, 10));
QueueObserver observer = new QueueLengthQueueObserver();
q.setObserver(observer);
q.run(1000);
observer.printStats();
}
}

QueueLengthQueueObserver
Queue length PDF:
1 0.47058823529411764
2 0.23529411764705882
3 0.058823529411764705
4 0.058823529411764705
5 0.11764705882352941
6 0.058823529411764705

Wednesday, April 28, 2010

GeoTraffic & GeoKeywords: Google Analytics custom reports for fun and profit

I've been using Google Analytics for 2 years to track the traffic on this blog. It's really interesting to see which posts receive more attention and what traffic sources and keywords generated how much traffic on any one day. This is specially important to me since the majority of the traffic to my blog comes from search engines a.k.a. google (organic).

However, when you really look into the traffic tracking thing, you'll find that the default reports which come out-of-the-box with Google Analytics don't help answer certain questions. This is more evident on a modest personal blog than it would be for a hot website like stackoverflow, for example. For your blog, you'd actually want to know who visited the blog. I mean, you'd like to push the limits of anonymity.

The main missing information in the default Analytics reports have to do with location. While you're able to know how many visits were generated from each country in addition to some useful statistics like the average time on website and pages/visit, there is still something missing.

I wanted to be able to answer these 2 critical questios:
1) Where does the direct traffic come from? and which pages are requested?
2) Who is searching for what?

Thanks to the Google Analytics team, we are now able to create our own custome reports to project the data whichever way we want. Once I found out, I created 2 custom reports to answer my 2 questions.

For the first question, the GeoTraffic report is the answer. The main dimension would be the Source. Next, we drill down towards: Country/Territory -> City -> Page. The main metric is Pageviews, then Pages/Visit and Avg. time on page.

For the second question, the GeoKeywords report is the answer. This time, the main dimension is the Country/Territory then we drill down to: Keyword -> City. For this report, I preferred the main metric to be Avg. time on page then Pages/Visit and Unique Visitors.

Now, I'm able to answer my questions, and I've been mainly interested in the first one: where the direct traffic comes from. The reason for that is that I've been waiting for something. I may talk about that later. I guess you're going to find it useful ;)

Friday, February 5, 2010

Introducing Google App Engine - All in One

Google App Engine lets you run your web applications on Google's infrastructure e.g. GFS and BigTable. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs grow. With App Engine, there are no servers to maintain: You just upload your application, and it's ready to serve your users... more docs. An alternative introduction is available on Wikipedia.

I compiled a list of videos that should get your engines up and running on this exciting web framework:

Campfire One - Introducing Google App Engine:

  1. Pt. 1 (9:10)
  2. Pt. 2 (12:34)
  3. Pt. 3 (13:19)
  4. Pt. 4 (7:44)
  5. Pt. 5 (5:55)
  6. Pt. 6 (8:04)
Overviews:
  1. Google App Engine - Early Look at Java Language Support (7:38)
  2. Overview of Google Web Toolkit (4:10)
  3. Getting Started with App Engine in Eclipse (5:07)
Google I/O 2008:
  1. Google I/O 2008 - Working with Google App Engine Models (1:00:32)
  2. Google I/O 2008 - Building Quality Apps on App Engine (48:43)
  3. Google I/O 2008 - Engaging User Experiences with App Engine (45:32)
  4. Google I/O 2008 - Python, Django, and App Engine (57:09)
Google I/O 2009:
  1. Google I/O 2009 - A Preview of Google Web Toolkit 2.0 (1:00:53)
  2. Google I/O 2009 - App Engine: Now Serving Java (55:00)
  3. Google I/O 2009 - Groovy and Grails in App Engine (1:00:14)
  4. Google I/O 2009 - Java Persistence & App Engine Datastore (1:09:32)
  5. Google I/O 2009 - ThoughtWorks on App Engine for Java (1:04:18)
Check out the project homepage and the developer's guide for more and subscribe to the Google App Engine Blog.

Thursday, January 28, 2010

Auto-formatting phone number UITextField on the iPhone

In this post, we'll use our PhoneNumberFormatter to implement an auto-formatting phone number text field in an attempt to mimic the behavior of iPhone's native apps, like Phone and Contacts.

So, let's call the text field in question myTextField. We start by calling addTarget on the text field to make it call the autoFormatTextField method in our view controller whenever it gets updated, either by the user or some other piece of code. If your view is ready, you should declare the handler as an IBAction and bind through IB. The method would then update the contents of the text fields to the formatted string returned by the formatter. If we do it that way, we'll also need to use a semaphore to prevent the method from being called endlessly.

The implementation outline would be:

// declarations

UITextField *myTextField;
int myTextFieldSemaphore;
PhoneNumberFormatter *myPhoneNumberFormatter;
NSString *myLocale; //@"us"

// init semaphore
myTextFieldSemaphore = 0;

// bind events programatically, skip when using IB.
[myTextField addTarget:self
action:@selector(autoFormatTextField:)
forControlEvents:UIControlEventValueChanged
];

// handle events
- (void)autoFormatTextField:(id)sender {
if(myTextFieldSemaphore) return;
myTextFieldSemaphore = 1;
myTextField.text = [phoneNumberFormatter format:myTextField.text withLocale:myLocale];
myTextFieldSemaphore = 0;
}
Update: As pointed out in the comments, for newer SDKs you may need to bind to UIControlEventEditingChanged. Thanks for your feedback!
//bind events

[myTextField addTarget:self
action:@selector(autoFormatTextField:)
forControlEvents:UIControlEventEditingChanged
];

Saturday, January 23, 2010

On the Fairness of Reservoir Sampling

Reservoir sampling is a family of randomized algorithms for randomly choosing K samples from a list of S items, where S is either a very large or unknown number.

A very elegant algorithm titled Algorithm R by Alan G. Waterman, is summarized as follows:

Fill up the 'reservoir' with the first k items from S
For every item S[j] where j > K
Choose an integer r between 0 and j
If r < K then replace element r in the reservoir with S[j]
One very interesting discussion with a friend led me to this problem. In this post we are going to prove that at all times, the probability of each processed item to be in the reservoir is the same for all items. In other words, after each iteration and when the algorithms terminates, all processed items will have the same probability of being included.

To initialize the reservoir, the first K items are included. At this point of time, each item is included by a probability of 1.

Later on, to process item number A, where A > K, we include it by a probability of K / A. (1)

For any of the K items in the reservoir, it will remain in the reservoir after item A is processed only in 2 cases:
1) Item A does not get included. This has a probability of ( A - K ) / A = 1 - K / A
2) Item A gets included but it does not replace the item in question, i.e. it can replace any of the other ( K - 1 ) items in the reservoir. This has a probability of ( K / A ) * [ ( K - 1 ) / K ] = ( K - 1 ) / A

So each of the K items remains in the reservoir by a probability of 1 - K / A + ( K - 1 ) / A = 1 - 1 / A

This is the same as the probability of A not replacing our item. Since the probability of A replacing any one item is 1 / A. The complement is directly found as 1 - 1 / A

Now, let P( A, B ) be the probability of item A still being included in the reservoir after item B is processed, where B >= A. Without loss of generality, we may assume that B is the next item to be processed.

For item A to remain in the reservoir after item B is processed, item A must have been included up to the point where B is to be processed. Equivalently, item A was still included after item (B - 1) was processed. In addition, for A to remain in the reservoir B must not replace it

From the discussion above, it follows that:
P( A, B ) = [ 1 - 1 / B ] * P( A, B - 1 )

Which can be expanded as:
P( A, B ) = [ 1 - 1 / B ] * [ 1 - 1 / ( B - 1 ) ] * ... * [ 1 - 1 / ( A + 2 ) ] * [ 1 - 1 / ( A + 1 ) ] * p( A, A )

But from (1):
P( A, A ) = K / A

As a result:
P( A, B ) = [ 1 - 1 / B ] * [ 1 - 1 / ( B - 1 ) ] * ... * [ 1 - 1 / ( A + 2 ) ] * [ 1 - 1 / ( A + 1 ) ] * K / A

But,
[ 1 - 1 / ( A + 1 ) ] = A / ( A + 1 )

Therefore:
P( A, B ) = [ 1 - 1 / B ] * [ 1 - 1 / ( B - 1 ) ] * ... * [ 1 - 1 / ( A + 2 ) ] * [ A / ( A + 1 ) ] * K / A
P( A, B ) = [ 1 - 1 / B ] * [ 1 - 1 / ( B - 1 ) ] * ... * [ 1 - 1 / ( A + 2 ) ] * K / ( A + 1 )

Again:
P( A, B ) = [ 1 - 1 / B ] * [ 1 - 1 / ( B - 1 ) ] * ... * K / ( A + 2 )

We can see that this recurrence reduces to:
P( A, B ) = K / B

This is the same as the probability of item B being included. In addition, P( A, B ) is not a function of A, it only depends on B, which is the number of elements processed so far.

This means that for the next item to be processed it will have the same probability of being included as all the items that were processed before it. By maintaining this property, when the algorithm terminates, all items will have the same probability of being included in the reservoir, regardless of the size of the list.

Friday, January 22, 2010

Locale-aware phone number formatting on the iPhone

iPhone developers often find themselves trying to stick to the standards set by Apple. Most of the time, doing so is facilitated by the SDK and the results are better. Sometimes though, it's not easy at all, mainly because some APIs are not open.

One example is phone number formatting. The Contacts application utilizes an auto-formatting UITextField for phone number entry. Many business applications would make use of such functionality but it's not provided by the current SDK.

I've been through that and was not happy about the task. After procrastinating for a while, I thought of a solution and later on, I actually implemented it, and it worked well for me. I wished I could make it into the sought after NSPhoneNumberFormatter, but I was done with the task. I thought I should share it with fellow iPhone developers. I hope many will be able to use it right away as-is, and it would be great if someone could finish the job and make the formatter.

I acquired the predefined localized phone formats from the UIPhoneFormats.plist. I was mainly interested in the us locale, but I kept the implementation generic. The main idea is to build some sort of a finite state machine (FSM) for the phone format and use it to process the input string. The FSM will both validate and add formatting characters to the string as needed. If the whole input could be processed successfully, then it is a valid format and the output is returned.

The problem that I didn't solve yet, is when the string can be matched to more than one format. I had to manually sort the formats from the most restrictive to the least restrictive, so the one that comes first is always selected. This hack was okay for US formats, but there was nothing I could do with the last 3 UK formats since, to me, they are essentially the same. I guess this can be worked around somehow.

I'll post the code here, and will try to add more comments later. Please feel free to add your comments or ask for clarifications.

Update 02/14/2010: I'm so glad to have habermann24 join in and create his phoney ruby lib. If your Ruby/Rails application deals with phone numbers, you got to check it out!

Update 09/08/2010: Check out libphonenumber: Google's common Java, C++ and Javascript library for parsing, formatting, storing and validating international phone numbers. The Java version is optimized for running on smartphones. You'll need to look for the AsYouTypeFormatter.java. Update++, check out the comment below by +SpoofApp.

Update 08/15/2011: Updates on libphonenumber: New development of the library will be presented in the 35th Internationalization and Unicode Conference in a session titled: libphonenumber - The Swiss Army Knife of International Telephone Number Handling. See session description for details.

Update 02/06/2015: libphonenumber on Github by Google Internationalization (i18n).

PhoneNumberFormatter.h

//  Created by Ahmed Abdelkader on 1/22/10.

// This work is licensed under a Creative Commons Attribution 3.0 License.

#import <Foundation/Foundation.h>

@interface PhoneNumberFormatter : NSObject {
//stores predefiend formats for each locale
NSDictionary *predefinedFormats;
}

/*
Loads predefined formats for each locale.

The formats should be sorted so as more restrictive formats should come first.

This is necessary as the formatting code processes the formats in order and
selects the first one that matches the whole input string.
*/
- (id)init;

/*
Attemps to format the phone number to the specified locale.
*/
- (NSString *)format:(NSString *)phoneNumber withLocale:(NSString *)locale;

/*
Strips the input string from characters added by the formatter.
Namely, it removes any character that couldn't have been entered by the user.
*/
- (NSString *)strip:(NSString *)phoneNumber;

/*
Returns true if the character comes from a phone pad.
*/
- (BOOL)canBeInputByPhonePad:(char)c;

@end

PhoneNumberFormatter.m
//  Created by Ahmed Abdelkader on 1/22/10.

// This work is licensed under a Creative Commons Attribution 3.0 License.

#import "PhoneNumberFormatter.h"

@implementation PhoneNumberFormatter

- (id)init {
NSArray *usPhoneFormats = [NSArray arrayWithObjects:
@"+1 (###) ###-####",
@"1 (###) ###-####",
@"011 $",
@"###-####",
@"(###) ###-####", nil];

NSArray *ukPhoneFormats = [NSArray arrayWithObjects:
@"+44 ##########",
@"00 $",
@"0### - ### ####",
@"0## - #### ####",
@"0#### - ######", nil];

NSArray *jpPhoneFormats = [NSArray arrayWithObjects:
@"+81 ############",
@"001 $",
@"(0#) #######",
@"(0#) #### ####", nil];

predefinedFormats = [[NSDictionary alloc] initWithObjectsAndKeys:
usPhoneFormats, @"us",
ukPhoneFormats, @"uk",
jpPhoneFormats, @"jp",
nil];
return self;
}

- (NSString *)format:(NSString *)phoneNumber withLocale:(NSString *)locale {
NSArray *localeFormats = [predefinedFormats objectForKey:locale];
if(localeFormats == nil) return phoneNumber;
NSString *input = [self strip:phoneNumber];
for(NSString *phoneFormat in localeFormats) {
int i = 0;
NSMutableString *temp = [[[NSMutableString alloc] init] autorelease];
for(int p = 0; temp != nil && i < [input length] && p < [phoneFormat length]; p++) {
char c = [phoneFormat characterAtIndex:p];
BOOL required = [self canBeInputByPhonePad:c];
char next = [input characterAtIndex:i];
switch(c) {
case '$':
p--;
[temp appendFormat:@"%c", next]; i++;
break;
case '#':
if(next < '0' || next > '9') {
temp = nil;
break;
}
[temp appendFormat:@"%c", next]; i++;
break;
default:
if(required) {
if(next != c) {
temp = nil;
break;
}
[temp appendFormat:@"%c", next]; i++;
} else {
[temp appendFormat:@"%c", c];
if(next == c) i++;
}
break;
}
}
if(i == [input length]) {
return temp;
}
}
return input;
}

- (NSString *)strip:(NSString *)phoneNumber {
NSMutableString *res = [[[NSMutableString alloc] init] autorelease];
for(int i = 0; i < [phoneNumber length]; i++) {
char next = [phoneNumber characterAtIndex:i];
if([self canBeInputByPhonePad:next])
[res appendFormat:@"%c", next];
}
return res;
}

- (BOOL)canBeInputByPhonePad:(char)c {
if(c == '+' || c == '*' || c == '#') return YES;
if(c >= '0' && c <= '9') return YES;
return NO;
}

- (void)dealloc {
[predefinedFormats release];
[super dealloc];
}

@end

Testing against a web server that doesn't respond, aka infinite delay

I wanted to simulate the situation when the server doesn't respond, without changing much. A friend of mine suggested I use any of the unassigned IPs on our LAN. Exactly what I was looking for! The request timed-out and I was able to see how my application would act in this case. We may also add an entry in the hosts file to name this black-hole-server, so it can be used later as needed. The hosts file will also enable us to override the DNS look-up and route the requests to the black hole, in case the host name was hard-coded/fixed in a binary, and we don't want to/can't modify the code.