Sunday, December 16, 2012

Python Script to Download XKCD webcomics (Code sample)

 I am a fan of XKCD webcomics. I decided to write a python script that will let you download the entire XKCD archive. You are free to use my code to download the comics for yourself. You can run this code periodically by specifying the number of the comic from which you want the download to begin. Just save this code with a .py extension and run it from a command line or a python interpreter. (Your python machine must have httplib2 to run this. Else you will have to download it from here. Enjoy!)


import httplib2

import os

h = httplib2.Http('.cache')

html = "<html><head></head><body><div id=\"comic\">"
end_html = "</div></body></html>"
i = raw_input("Enter the comic number from which you want the download to begin:")
i = int(i)
if (i < 1):
i = 1

if not os.path.exists("xkcd_archive"):

if not os.path.exists("xkcd_archive/images"):

while True:
url = ""
url = url + str(i)
response, content = h.request(url)
if response.status == 404:
index   = content.find("<div id=\"comic\"",0)
src = content.find('src=',index)
end_src = content.find('title',src)
comic = content[index : index + 1000]
start_index = comic.find('alt=')
end_index = comic.find('/>',start_index)
alt = comic[start_index + 5 : end_index - 2]
img_body =  "<h2> " + alt + "</h2>" +"<img " + "src =\"images/" + alt  + ".jpg\""

caption = comic[comic.find("title",0):comic.find('alt',0)]
image_url = content[src + 5:end_src]
content = html + img_body + caption + " />" + end_html
print "Downloading " + str(i) + "th comic --> " + alt,"\n"
content = bytes(content)
with open('xkcd_archive/' + alt + '.html','wb') as f:
print "\n"
i += 1
image_url = image_url[0:-5] + "jpg"
response, image = h.request(image_url,headers = {'type' : 'text/html'})
print response
print "\n",image_url
if response.status != 200:
image_url = image_url.replace(".jpg",".png")
print image_url,"\n"
response, image = h.request(image_url,headers = {'type' : 'text/html'})
print "png" + "\n"
print response
with open("xkcd_archive/images/" + alt + ".png", 'wb') as I:
with open("xkcd_archive/" + alt + ".html",'r+') as f:
   contents =
   contents = str(contents)
   temp = contents.find(".jpg")
   contents = contents[:temp] + ".png" + contents[temp+4:]
   print contents
with open("xkcd_archive/images/" + alt + ".jpg", 'wb') as I:


Sunday, November 25, 2012

Humor in Software

After coding for around 5 hours straight, the other day I clicked on the small little applet called
Wanda on my gnome desktop. Now, wanda is a  little program actually that chooses pseudo-random text from various databases. It was originally intended to diffuse project tensions when hackers used to work on unix command line. [Or at least that's what I read about it when I googled it anyway :) ].  Every time you  click on that small animated fish, it responds with something funny, a nice quotation, a proverb or something similar. It doesn't serve any purpose. It is not needed for the Linux OS and certainly doesn't relate to what I work on in any  way whatsoever. But I still think of it as one hell of a feature. In fact, when I have to program anything that's not OS specific, I almost always choose to do it on Linux. All because of the rhythmbox music player which you can "close" entirely from your view [unlike itunes on windows where you can use it in "mini-mode" but not entirely "close" it] and Wanda the Fish. iTunes is great  but i prefer rhythmbox simply because it doesn't stop playing music when i close it.

So when I clicked it the other day, it responded with this particular text :

Hack placidly amidst the noisy printers and remember what prizes there
may be in Science.  As fast as possible get a good terminal on a good system.
Enter your data clearly but always encrypt your results.  And listen to others,
even the dull and ignorant, for they may be your customers.  Avoid loud and
aggressive persons, for they are sales reps.
If you compare your outputs with those of others, you may be surprised,
for always there will be greater and lesser numbers than you have crunched.
Keep others interested in your career, and try not to fumble; it can be a real
hassle and could change your fortunes in time.
Exercise system control in your experiments, for the world is full of
bugs.  But let this not blind you to what virtue there is; many persons strive
for linearity and everywhere papers are full of approximations.  Strive for
proportionality.  Especially, do not faint when it occurs.  Neither be cyclical
about results; for in the face of all data analysis it is sure to be noticed.
Take with a grain of salt the anomalous data points.  Gracefully pass
them on to the youth at the next desk.  Nurture some mutual funds to shield
you in times of sudden layoffs.  But do not distress yourself with imaginings
-- the real bugs are enough to screw you badly.  Murphy's Law runs the
Universe -- and whether or not it is clear to you, no doubt <Curl>B*n dS = 0.
Therefore, grab for a piece of the pie, with whatever proposals you
can conceive of to try.  With all the crashed disks, skewed data, and broken
line printers, you can still have a beautiful secretary.  Be linear.  Strive
to stay employed.
-- Technolorata, "Analog"
The thing I was working was actually stuff I had to submit and I had put it off  till the deadline. Therefore, when the time came, I had barely finished half of the project and was pretty worried and trying to do everything as fast as possible. I was very tense as well.  After reading what Wanda said, I relaxed a bit and slowed down a little. I got it done eventually.

This situation reminded me of  an article David Pogue wrote for Scientific American  once. It talks about how humor used to be embedded in software and delighted users in the most unexpected ways. Nowadays, all you see are "serious industrial strength applications". The name itself sounds so mundane. Google does it still with their doodles. but it is an exception.  In reality most "industrial strength applications" have little or no room for humor or innovation for that matter. Paul Graham once said that "If you are suffering from Insomnia try reading the literature on relational databases. I have no doubt that the top gaming companies in the world are engaged in more intellectually stimulating research than bottom ten universities."

I like products that are lively. I presume others like it too.  A great way to make software livelier is by adding a sense of humor to your software. In fact, it may be all that's needed for users to switch to your software rather than a rival's.  (I am not saying this is strictly true, but it may just raise the odds in your favour when all other features are similar. ) After all, everyone in today's busy world could use a laugh once in a while and it will be doubly entertaining when a machine makes a joke.

Tuesday, July 31, 2012

The Gambler (Short story- fiction)

When Sir Charles inherited the estate from his grandfather, he was twenty three years old. He had just completed his education and was regarded by all as a fine young man. The fact that he was still unmarried didn't hurt either. Many a woman had tried to entice him and had failed.

When the Sepoy mutiny threatened England's rule, he enlisted in the army and was sent off to India.
He fought valiantly and rose through the ranks eventually becoming a colonel. He returned to London after two years with many medals adorning his uniform and became popular as a war hero. He looked forward to a quiet life and settled comfortably into his ancestral home. He lead the typical life of  a peer. His estate bestowed on him an enormous income and he prospered. When he turned twenty-eight, he married Victoria, the daughter of Lord William. Lord William was a famous gambler. He had made more from the casinos than his ancestors had made at the seas generations before him. Since his ancestors had shared the wealth with the Queen, they had become Lords.

Victoria was a lovely maiden who was three years younger than Charles. She was as graceful as a queen and looked magnificent in a white robe on the day of her wedding. She was educated and intelligent. And all those who were present at the wedding exclaimed how perfect Charles and Victoria were for each other.After their month-long honeymoon, the happy couple returned to Charles's estate and lived peacefully. They had everything they wanted and it looked as though nothing but many years of happiness lay ahead of them.

 A year later, Lord William expressed a desire to visit Sir Charles. Lord William had been away for a year to attend to some of his affairs abroad and he had just returned to England. He wished to see his daughter and son-in-law. Accommodations were made for his stay and three days later, Lord William was at the entrance of the Stony Mansion.The visit went pleasantly. His daughter was happy and his son-in-law was a devoted husband. On the third day,  Sir Charles and Lord William were sitting in the courtyard drinking a bottle of port, talking about this and that. Suddenly Charles asked,

 "how come you win at the casinos all the time? I have heard you never lose. Is there a trick?" 

Lord William gave him a gentle smile and his eyes were twinkling. 
"There is no trick my boy. I just play like anybody else. Luck has been kind to me. I have lost many times but people do not notice your losses if your profits are high enough and that is what has happened to me."
Charles did not pursue the matter and they spent the rest of the evening talking about wines.

That night, Charles lay awake in bed thinking  of schemes that could give one an unfair advantage over other gamblers. All the schemes that occurred to him were transparent and he could not think of any satisfactory scheme that could fool people all the time. He did not believe his father-in-law. he was certain that no man could be that lucky. He knew that one evening, Lord William had made around four million pounds in one of the richest and largest casinos in England. A casino that was so exclusive that you had to be invited there to play.

He resolved to find out the old man's secret and use it. The best way to learn the trick would be to observe Lord Williams. He would observe and learn. He could not directly gamble against his father-in-law so he conceived an elaborate plan. he would disguise himself as an oil merchant and visit the casinos Lord William frequented. He would always observe and learn from the master gambler. He was confident that if he observed enough, he could learn what the old man had up his sleeve.

Sir Charles was rich. But the rich have a need to possess more. To become richer. They are afraid that they might end up with nothing one day. So they try to insure against any such possibility by acquiring more. Charles was irritated that Lord William had not shared his secret. After all, Victoria was his only daughter and would it not benefit her if her husband made money gambling?

 Three months passed before Sir Charles was ready to hit the casinos. He familiarized himself with all  the rules, the etiquette, the system and learnt to play all the games which Lord William played.
When he entered the casino the next evening, Lord William was already playing poker. Charles was delighted and he observed him closely. Lord William lost two games and two hundred and fifty thousand. Charles was disappointed.

When Lord William moved on to play blackjack, Charles was able to secure a seat at the table and was watching him, fully alert. Never in his life had he been so alert. not even when he had gone to hunt tigers in the Indian jungles. With every nerve in his body observing Lord William, he began to play against the master. When Lord William left the casino that evening. He had made roughly half-a-million pounds. half of which was Charles' money. Charles did not mind losing. He considered his losses as nothing more than a fee that must be paid to learn from the very best.

But after two months, he still didn't have a clue what was happening. He sensed a pattern though. Whenever Lord William gambled, he would lose money as well as win. But somehow when he finished, he would have made up for his losses and he would have a small profit.

When gambling became an obsession with Sir Charles, Victoria sensed that something was wrong. her husband was not his usual cheery self and had changed in his manner. he always was in deep thought and was irritated whenever he was disturbed. He had started spending a lot of time alone in his study, brooding over something. When she asked him about the matter, he simply said, "Darling I am perfectly fine. Don't worry about me. There is no problem at all anywhere and no reason for concern. I just have a few business plans that need careful consideration. I assure you that once I settle this minor affair, there will be nothing to worry about. I love you deeply Victoria and I have always. I just need a little time  for myself."

Victoria had never been nosy. A rare trait in women. So she let the matter rest. But things were not rosy. When Lord William was not playing, Charles would still play. He had lost millions of pounds and had even borrowed to finance his new vice. He regretted it and tried to stop. But he always ended up  in his usual spot in the casino. When  he mortgaged his estate, he resolved to have one final crack at the puzzle.

He waited and watched Lord William play for two hours and then he finally made his move. he sat opposite and began to play. It was a bad move. He lost his estate. Charles walked home in despair that night and couldn't face anybody for three days. His wife grew very concerned. His health started deteriorating. And he vowed never to gamble again. He begged God for a second chance.

A month later, Lord William died of a heart attack. He named his son-in-law the sole heir to his enormous fortune. Also enclosed was a letter that only Charles was authorized to open. Sir Charles grieved the death of Lord William but thanked God for giving him his second chance. His health improved. He would never waste money again. After the funeral, he paid off the debts and mortgages. Throughout this time, nobody had known anything at all about his vice. His wife was clueless and  he was enormously relieved. he would be ashamed if Victoria ever learnt that he had borrowed money to gamble.
He opened the letter his father-in-law had left him and began to read:
Dear Charles,
    I have left everything to you. Consider it returns on your investment. Yes, I knew you were playing against me. Did you think I wouldn't recognize my own daughter's husband? Anyway, all's well that end's well. I really have no trick or secret. I was merely lucky. and if there is one thing I can tell you about gambling. It's this: Don't do it. It will rob you of your wealth. If it didn't, then there would be no casinos in existence today. Also, I have a tip for you: if you don't see a sucker at the poker table, leave quickly. You're the sucker.
With best wishes
Lord William

An enormous feeling of delight, satisfaction and relief surged through Charles. He felt overjoyed. Nothing had been lost. His father in law had saved the day, his family's name and honour and above all, Victoria's happiness. When he woke up next morning, Victoria was standing over him with a breakfast tray.
"I have something to tell you. I thought you might want to have breakfast in bed. It's been a long time since you have done that."
"Darling, I am so happy for us. You are the most wonderful woman I have ever met."
"I'm pregnant."
Charles nearly dropped his tray as he tried to set it down on the nightstand. He gave a joyous shriek and took Victoria in his arms. He hugged her tightly. She smiled and kissed him gently.

It looked as though nothing but many years of happiness lay ahead of them.

Sunday, July 29, 2012

The Mistress (Short-story- fiction)

It was just after five o clock. Arthur finally relaxed. It had been a long and grueling day. He loosened his tie and sighed before collapsing on his plush chair. Despite his weariness, he felt elated. The takeover had been a success. His bank now would be the largest in the city if not in the state, and he would make millions. His shares would fetch him a fortune when the rest of the city learnt about the takeover. With these thoughts in mind, he gazed out of his window savoring the moment.  His phone rang. It was Carla. "Hello Honey!", he said in a joyous voice.
 "Hello Darling, are you gonna stop by later?"
  "I promised I'd take Michele to dinner sweetheart. I'm afraid I cannot. I'll take you away this weekend.     I promise"
  Carla sighed. Arthur felt uncomfortable. To hell with it! he thought.
  " I will stop by on my way. I'm coming.", He said.
  "I will be waiting", Carla purred like a satisfied cat.
New plans for the evening. He thought. He got up, left the office, hailed a taxi and asked the driver to take him to 21st and third.
Arthur was a successful banker. He had also been a devoted husband and had never laid his eyes upon another woman after marrying Michele. That was until six months ago. Six months ago, he had met Carla at a party thrown by one of his banker friends. Though he had come to the party with his wife, his eyes kept searching for Carla. That night, he had felt really uncomfortable and was full of guile. He loved his wife. She had borne him two children. She was everything he had always  wanted. But lately, his marriage was not going that well. He had started spending more time at the bank and had paid little attention to her. His wife failed to show any enthusiasm as his successes grew. It irritated him. When they slept together at night, they just slept.  She still waited for him to come home every night before she had dinner, but the dinners usually were silent and neither of them spoke as much as they used to.
Also, his wife was nearly forty and had lost her looks. Arthur himself was forty-two. But he had remained in shape by exercising regularly and he looked no older than thirty-five. Carla was twenty-seven. She was a beauty. Her long brunette hair was shiny and soft. Her rainbow eye-brows, perfectly shaped nose and thin lips would always turn heads. Her body was slim and perfectly proportioned. In short, she was the ideal mistress for any successful guy going through a bad marriage.

Arthur paid the fare and got out of the taxi. He climbed the stairs and rang Carla's doorbell.
She answered it at once. He walked into her apartment and told her about his day at the bank over a steaming mug of hot chocolate. Carla was wearing a black negligee and looked gorgeous. He devoured her with his eyes. She didn't miss it. After spending an hour with her doing things steamier than the hot chocolate, he got ready to leave. "We'll go away for the weekend darling. I promise. But I must get back now. Also, I have a surprise for you!"
"Oh Arthur! I love you!", She said and kissed him on the lips.
When he finally reached home, his wife was waiting impatiently and he apologized nervously. "Those damned lawyers would not let me go! They kept going over every single line and I really couldn't leave. I'll make it up to you!"

He showered and dressed quickly. Around eight, they went to the Brundage's and the maitre de' showed them to their table. The evening passed amiably. His wife didn't seem to mind his delay and Arthur was just glad that she didn't ask him too many questions. After talking about the bank and the takeover, he had nothing left to say. Michele talked about getting some new furniture, re-modeling the drawing room and other trivia. He pretended to listen. When they were leaving, he took her to a jewelry store and bought her a magnificent string of pearls. Michele was genuinely surprised but very pleased and kissed him on the cheek and hugged him.

When the weekend came, Michele told him that she would go and visit her mother in Florida. He was delighted but said faking utmost sincerity, "I was hoping you'd join me for a trip to Vermont. I have a lot of meetings to attend there but I thought we could have a little time for ourselves."
"I'm sorry Arthur. I already told mother I'd come and she'll be expecting me."
"We'll go on a long vacation next month my darling. I have it all planned out. I'm taking you to Lausanne and We'll spend a month there."
She smiled and kissed him. "Take care."
"You too".
Once she had left, he dressed and asked Carla to meet him at the airport. In precisely one hour, they were on a plane to Vermont. After they had checked into the hotel under false names, they spent the rest of the day in bed but neither of them slept. The next morning, Arthur went to his bank's branch and got some paperwork out of the way. He returned to their suite around mid-day and they spent the rest of the weekend dining out, partying, skiing and taking long walks. Arthur felt like a teenager when he was with Carla.

When he returned home on Monday, he went straight to the bank. The takeover had made him a multimillionaire and though he had always been well-to-do, he began to think of himself as "rich" for the first time. The day passed as usual and he answered all the congratulatory letters he had received over the weekend feeling happy and buoyant. He even bought flowers to his wife while going home. When he arrived at the gate, Michele was waiting for him. He handed her the flowers and she took them brusquely. She didn't smile or in fact make any gesture of acknowledgement. He started rambling about how good the day had been and of the important position his bank was now in. She lead him directly to the living room and sat down on the sofa. She looked at him like a cop looking at a suspect.
"Arthur, sit down. I have something I must ask you."
He felt dread when he heard those words. He slowly sat down and was unable to meet her eyes. Thoughts were racing through his mind. had she found out about his mistress? what would he do now? Or was it something else she wanted to ask him?. He felt like a young school boy caught in 
wrong-doing and waiting outside the Principal's office. All his exuberance had vanished.
"How long have you and Carla been sleeping together?"
He felt as if someone had punched him in the face. He tried to breathe and relax, but his heart was pounding and he began to  sweat. For several moments, he could not speak. He took out his handkerchief with a shaking hand and wiped the sweat off his brow. She knew. How did she know? he had been careful. He had missed something. It was all over. He decided that the best thing to do would be to tell her everything and beg her to forgive him.
"Six months", He whispered and covered his face in his hands.
He told her the whole story. And when he had finished he begged her to forgive him.Michele did not speak at first. She considered him for a moment and then said acidly,
" I am afraid this marriage is over Arthur" and left without a word.
A month and a half later, Arthur and Michele were divorced. Arthur's guilty conscience had gnawed at his soul and he had given away millions of dollars to her as compensation and alimony. It still did not bring him peace. Michele moved to Florida and settled there permanently.

On a lovely sunday afternoon, Carla showed up at Michele's new summer home. She was taken directly to Michele by the butler. The butler left quickly. Michele took out her checkbook and wrote out a sum of ten million dollars to Carla and silently handed it to her. Carla accepted it without a word and carefully placed it inside her purse. Michele then poured Carla a drink.
Michele said," I could never have done it without you. Cheers."

Saturday, May 19, 2012

Why Google earns more from its ads

                     Google earns most of its revenue from online advertising. Despite the rise of Facebook, Linkedin, Quora, and a plethora of other websites and social networks, it's place in online advertising has remained unchallenged. Claims of Facebook being "the next Google" have been around for a while. But I seriously doubt whether those claims will come true in near future.

                   The investors' reaction to the highly anticipated Facebook IPO illustrates that many investors still feel that Facebook is a risky bet. They wonder if the company really matches the hype it generated. The closing stock price was $38.23, up just 0.61 from its starting price.
                   So why does Google earn more compared to its rivals? Well, because, in the online world, the relevance of the ads matters. People use Facebook to connect with friends and family. They use Linkedin for developing professional contacts, but users do not visit any of the social networking sites to obtain information.

                   When you are looking for something, you still go to Google. And that's when you pay attention to advertisements that appear along with the search results. In contrast, you rarely notice the ads Facebook places on its site. This is partly due to the fact that engineers and graphic designers try to be discreet when displaying ads. But Facebook has become a bit too discreet in that regard. Also, in a social setting, where there are people to talk to and you are reading the updates of your friends, chances are remote of you following an ad.

                   The new move by Google to integrate all the data from its services to build a profile of its users is a master stroke. It leads to personalized ads. While this move has been a troubling one and Google has faced a lot of flak over this new policy, especially from privacy advocates, when it's seen through a financial lens, the move makes perfect sense. Online advertising is a great source of revenue only if people see and follow your ads. Otherwise, it's a loss.
                   Does this mean that Facebook cannot beat Google? It's hard to say. Especially because both companies are constantly innovating and finding new ways to retain users longer. Google's browser Chrome now is second only to IE and I expect it will become first soon. The Chrome web browser also leads users to the Google's site- Yet another example of how Google ensures that users visit its site.
   So, while Facebook has a huge potential with its treasure trove of data, Google still retains it's edge when it comes to ads. There is still a substantial way to go before Facebook can become "The next Google".

Tuesday, May 15, 2012

Pragmatic Programming can beat Vendor Lock in.

Programming is a tough job. Part of what makes it tough is the fact that programmers make other programmers depend on them. It's all  part of survival. Disruptive technologies emerge more often in software than in any other kind of industry. Back in 1996 and 1997, the first website people would visit when they opened their browsers was Yahoo!. Later Google came into prominence and captured the top spot. It remained there for many years. Now, that position is becoming precarious and facebook has emerged as a serious competitor to Google. Today, several websites are vying for the top spot. Youtube, twitter, Linkedin, MySpace, flickr, and so many more.

 We have only been looking at how the web has changed so far. But the story continues beyond that. Adobe's pdf documents are fast becoming commonplace. Google chrome is eating into the market owned by IE, Object relational and Object oriented databases are a looming threat to companies that sell databases that still follow the relational model.

So how does a company ensure that it will stay afloat even though it's rivals make better products? The answer lies in economics: Vendor Lockin. It is a terrible thing. The software vendors create specialised sections of code or incorporate unresolvable compatibility issues into their software so that it will be extremely hard to switch from one service provider to another. The switching costs for software systems will be so high for some systems, that even if the company wants to switch, it will not be ready to pay that huge a sum. Also, there is an increased risk while switching: the new software may require additional training and processes, special skills and so on.. All these factors prevent companies from becoming better.

Now, from the point of view of the programmer, what's vendor lockin? Simple: you use any special feature provided by your environment and your system will become dependent on that environment. Microsoft does it. Google does it and Apple has been doing it and a lot of other companies want to get in on it. They want to lock people in. They want to make their platform dominant. They do not want to lose out to a rival simply because the rival's product is better.
 Standardization is one remedy to the problem. But then, everyone adds their own extensions to the standards and they will be so different that vendor lockin inevitably results leading to a dependence on the provider's platform and ensuring that the provider continues to make money even if his product is lousy. Pragmatic programming is the way to go against this prison. Using open-source libraries and creating scalable and re-usable code that depends as little as possible on the host's system (if at all), and following the principle of separating the logic clearly from the implementation details can considerably reduce the penalties while switching. Programmers need to use all their skills to make the system as cross compatible, upward compatible and "switchable" to ensure they don't get trapped by the vendors.
Serious programmers always think in terms of re-use. They are the laziest people on the planet and want to put in as little effort as possible. Being lazy this way makes them efficient. So the next time you import that library, make sure it's not proprietary and if it is, look for a similar open-source alternative. In the worst case, at least make sure that it is available everywhere. After all, you don't want someone else to control your work.