Shadow IT via the Cloud

Are we heading back to the days of Shadow IT, now with hidden infrastructure out on the cloud rather than in secret server rooms or under desks? Photo credit: Google’s First Production Server by Jurvetson on Flickr.


There is a very interesting article Bernard Golden in CIO magazine about what really runs on Amazon’s public cloud. He challenges the common perception that “public cloud computing is not being used by big enterprises… if large companies are using public clouds, it’s for “unimportant” applications.”

He argues that not only are enterprises using Amazon, and for important applications, but that use is increasing, despite the rise if private clouds.

Golden attributes this to the claim that

“…developers are the true decision makers in organizations…IT organizational decisions are actually made bottom up (i.e., by developers), and that senior management perception lags reality by a significant margin”.

He goes on to quote Stephen O’Grady of analysts Redmonk

“We are founded upon the idea that developers are the single most important constituency in technology. Open source dramatically lowers the barriers to adoption, such that developers may build upon what they want rather than what they’re given.”

Golden continues,

“The implication for organizations…is that decisions made by developers create commitments for the organizations they are part of — commitments that the organization does not recognize at the time they are made by the developer and may, in fact, be decisions that, had the organization understood them at the time they were made by the developer, it would have eschewed them. The result is that two or three years down the road, these organizations “discover” technology decisions and applications that are based on choices made by developers without organizational review.

This reminds me of a quote from Tolstoy quoted in “The Inner Ring” by CS Lewis:

When Boris entered the room, Prince Andrey was listening to an old general, wearing his decorations, who was reporting something to Prince Andrey, with an expression of soldierly servility on his purple face. “Alright. Please wait!” he said to the general, speaking in Russian with the French accent, which he used when he spoke with contempt. The moment he noticed Boris he stopped listening to the general who trotted imploringly after him and begged to be heard, while Prince Andrey turned to Boris with a cheerful smile and a nod of the head. Boris now clearly understood-what he had already guessed-that side by side with the system of discipline and subordination which were laid down in the Army Regulations, there existed a different and a more real system-the system which compelled a tightly laced general with a purple face to wait respectfully for his turn while a mere captain like Prince Andrey chatted with a mere second lieutenant like Boris, Boris decided at once that he would be guided not by the official system but by this other unwritten system.

Bill Jensen calls attention to this phenomenon of staff breaking rules to get things done in his new book “Hacking Work: Breaking Stupid Rules for Smart Results”. In an interview on AMA Edgewise her says:

“The number 1 source of work complexity is the infrastructure we find inside our companies. The tools, the procedures,  the processes – everything that is designed ton help us get our work done –  to be an  enabler –  is actually a problem, but senior executives own that and they are not willing to addresses it.”

So within organisations there are the official hierarchies  of the Org chart, and then there are the real IT hierarchies, with developers at the top, dictating real-world strategy by taking substantial decisions for their organisations almost by accident, as an orthogonal by-product of just trying to get things done, usually by breaking or bending rules.  What we have on the ground is a return to a form of Shadow IT, this time enabled by the cloud. As Golden notes,

One CIO noted that he had gone through the expense reports turned into him for reimbursement and found 50 different cloud computing accounts being used by developers in his organization. The reason? It’s a lot easier for a developer to obtain computing resources from a public cloud provider than to undergo the extended waits typical of the existing compute environments.

This is undoubtedly true. I see this all the time in enterprises I work with. In my experience it is usually test environments, proof of concepts for project justifications and sneaky ways of meeting requirements (Requirements checklist item #44: Offsite back-up? Check….backed up to Amazon S3)that end up in the public clouds, not core application components.

Developers and other IT professionals are taking the path of least resistance to the public cloud due to lack of internal agility (speed) or capacity is also one of the main reasons I hear from CIOs for why they want to adopt private clouds. They want to offer their developers (and other users) the sort of speed, elasticity and cost benefits of public clouds, but in a controlled private cloud.

“The upshot of all this is that today decisions are being made by developers about use of cloud computing that upper levels of the organization are unaware of (the CIO just cited is undoubtedly an exception to the general rule). A corollary to this is that IT organizations will discover new public cloud-based applications in a year or two that they had no idea were being placed in those cloud environments. “

I am not convinced of this. CIOs might wake up to find – to their relief – that their developers adopted “cloud thinking”, that their applications architectures are cloud optimised, that their applications can use cloud APIs. I do not think we will find that large submerged sections of their are running on public clouds. I think private clouds will fill this gap quickly, just as I think that eventually hybrid clouds and the applications they enable, are the future. Right now I see hybrid, but it is usually a hybrid of public cloud and traditional bare metal within the firewall.

Golden sees three ramifications from all this:

“1. Organizations will be taking on operational responsibility for applications located in cloud environments for which they are unprepared.”

He exhorts organisations to

“quickly get up to speed on the cloud environment and learn how to operate an application within it. This will include the need to retrofit monitoring, security, and management into applications, as developers often fail to address these areas because they are not required for application functionality. “

This is a very good point. Developers may be conversant with Cloud, but the Operations and Support people need time to integrate cloud applications into their portfolios even if they are very familiar with the cloud technologies in question.

“2. Compliance requirements will need to be analyzed for these “discovered” applications.”

Golden rightly identifies  a massive “tail-end” problem of cloud adoption – controlled or uncontrolled. The front end problem is often of of encapsulating or abstracting ones application into the cloud. Once that is successful, the tail-end problems hit. These are the compliance and governance issues, the modifications to internal procedures, the challenges of defining ownership and maintaining cloud bases systems. He advices have a multi-disciplined rapid response team ready to react to the discovery of a renegade application (or part of an application) running on  a public cloud.

“3. Investment patterns of IT spend will be modified in a stealthy fashion”

Golden reiterates  point made beautifully by Simon Wardley in his absolutely brilliant “”Situation Normal, Everything Must Change” talk at OSCON 2010 in which he points out that whilst Cloud Computing will unleash innovation and IT backlog, but it is unlikely to lower costs thanks to Jevons Paradox, Componentization and Creative disruption.

I will let Simon explain. This talk is one of the best I have seen on Cloud Computing and very well worth watching end-to-end:

I am not convinced that Simon is entirely correct about Cloud Computing not reducing IT spending. Cloud Computing introduces efficiencies. There is more bang for buck.

Simon contends that instead of CIOs being able to reduce costs, the new efficiencies and innovations will give rise to greater demand (Jevons Paradox) which will wipe out the cost savings.

I think that the IT backlog caused by today’s problems will be eroded by Cloud Computing and may generate more demand, but that demand will still be backlogged. The key constraint will moved from cost and infrastructure to processes constraints like workload capacity of  IT solution architects, support staff or even the developers themselves.

In other words, demand will be constrained by human factors, whilst the infrastructure costs of satisfying demand will drop, leading to a net drop in costs (i.e. a reduced IT spend).

That said, In suppose one could reasonably argue that those cost savings would be diverted into easing the key contraint(s), the net result still being no reduction in IT spend, just more to show for it.

Nasty supermarket tricks

This video was filmed in the relatively new Idea supermarket in the up-scale Usce shopping centre. It illuminates the downright dangerous and immoral tricks played by some Serbian supermarkets.

It is not only Idea. I have had two incidents at the big Maxi at Gospodar Jovanova 39 where I ended up with rotten food.

One one occasion I was sold rotting Skate (fish). On another, I bought a kilo of marinaded chicken, only to spot that it was green and putrescent whilst preparing it.

I am told the supermarkets use the marinade to disguise the rotting, so be very careful when buying any meat in a marinade or sauce.

Koran burning bunk


“How could a Christian pastor plan to do something so wicked as burn the Koran, everyone has been asking over the past week. A more pertinent question might be: why did virtually every world leader, from Obama to the UN secretary-general to Blair, bring attention to that pastor by publicly pleading with him not to carry out his dastardly, potentially world-destuctive act? (He has now apparently agreed to call it off.) The pastor is daft, but the international fear that one book-burning would unleash civilisational conflict and seriously harm American integrity and security is the real story here, speaking to a serious malaise in the West’s corridors of power.” – Brendan O’Neill, Spiked Online.

“I think the Koran-burning preacher is very similar to the GZ mosque backers except that the GZM [Ground Zero Mosque]  folks want a more permanent provocation” Kelly Loony, Twitter

Frank Furedi examines the media created furore over a nutty pastor Terry Jones’ plan (no on hold) to ritually burn copies of the Koran on 9/11.

I was embarrassed as everyone from the US military’s General Petraeus to the President Obama  begged the pastor not to do it because it will “endanger lives of US servicemen by unleashing Muslim anger on them”.

I would have cheered if Obama et al had denounced the planned Koran burning as a deeply offensive, thuggish and completely unnecessary act of gratuitous provocation and bad faith. But to denounce it because they are afraid of the backlash simply animates the myth of Muslim outrage and the power of the Muslim street.

CNN carried stories from Afghanistan where US soldiers are forbidden from eating or drinking in front of fellow Muslim soldiers during Ramadan, and their little chapel, deep in their own base, must remain unmarked. A solider explained that it is a “provocation” to have a church on “Muslim lands”.

The US military is portrayed as being scared witless not only of causing offence, but of facing the consequences of offensive acts by any Amercians. This affirms and emboldens the extremists who for 30 years have fought hard to intimidate the West into self-censorship and paralysis by mere threat.

Adding power to threats is the purpose of terrorism. And real power is the power to command with mere threat (As Slavoj Zizek points out, a father resorting to violence represents a failure of power and authority, an admission of impotence)

I wish Western leader were this animated and principled when it comes to defending Western civil liberties against the relentless attacks by Islamist activists seeking to undermine them.

By overreacting to Jones and indulging the myth of the Muslim backlash, we have sent a  clear message to the extremists says that the terrorism, bullying, riots, murders (of people like Theo van Gogh) and threats are working. We are scared. We will do anything not to provoke or “offend”, even if we offend by practising  thousand year old civil rights in our own countries.

I also resent this nut Jones having been promoted way beyond his station. Now he is off to New York to “broker a deal over the Ground Zero mosque in return for cancelling his threat to burn the Koran“.

My fundamentalists will talk to your fundamentalists…