Shadow IT via the Cloud

Are we heading back to the days of Shadow IT, now with hidden infrastructure out on the cloud rather than in secret server rooms or under desks? Photo credit: Google’s First Production Server by Jurvetson on Flickr.


There is a very interesting article Bernard Golden in CIO magazine about what really runs on Amazon’s public cloud. He challenges the common perception that “public cloud computing is not being used by big enterprises… if large companies are using public clouds, it’s for “unimportant” applications.”

He argues that not only are enterprises using Amazon, and for important applications, but that use is increasing, despite the rise if private clouds.

Golden attributes this to the claim that

“…developers are the true decision makers in organizations…IT organizational decisions are actually made bottom up (i.e., by developers), and that senior management perception lags reality by a significant margin”.

He goes on to quote Stephen O’Grady of analysts Redmonk

“We are founded upon the idea that developers are the single most important constituency in technology. Open source dramatically lowers the barriers to adoption, such that developers may build upon what they want rather than what they’re given.”

Golden continues,

“The implication for organizations…is that decisions made by developers create commitments for the organizations they are part of — commitments that the organization does not recognize at the time they are made by the developer and may, in fact, be decisions that, had the organization understood them at the time they were made by the developer, it would have eschewed them. The result is that two or three years down the road, these organizations “discover” technology decisions and applications that are based on choices made by developers without organizational review.

This reminds me of a quote from Tolstoy quoted in “The Inner Ring” by CS Lewis:

When Boris entered the room, Prince Andrey was listening to an old general, wearing his decorations, who was reporting something to Prince Andrey, with an expression of soldierly servility on his purple face. “Alright. Please wait!” he said to the general, speaking in Russian with the French accent, which he used when he spoke with contempt. The moment he noticed Boris he stopped listening to the general who trotted imploringly after him and begged to be heard, while Prince Andrey turned to Boris with a cheerful smile and a nod of the head. Boris now clearly understood-what he had already guessed-that side by side with the system of discipline and subordination which were laid down in the Army Regulations, there existed a different and a more real system-the system which compelled a tightly laced general with a purple face to wait respectfully for his turn while a mere captain like Prince Andrey chatted with a mere second lieutenant like Boris, Boris decided at once that he would be guided not by the official system but by this other unwritten system.

Bill Jensen calls attention to this phenomenon of staff breaking rules to get things done in his new book “Hacking Work: Breaking Stupid Rules for Smart Results”. In an interview on AMA Edgewise her says:

“The number 1 source of work complexity is the infrastructure we find inside our companies. The tools, the procedures,  the processes – everything that is designed ton help us get our work done –  to be an  enabler –  is actually a problem, but senior executives own that and they are not willing to addresses it.”

So within organisations there are the official hierarchies  of the Org chart, and then there are the real IT hierarchies, with developers at the top, dictating real-world strategy by taking substantial decisions for their organisations almost by accident, as an orthogonal by-product of just trying to get things done, usually by breaking or bending rules.  What we have on the ground is a return to a form of Shadow IT, this time enabled by the cloud. As Golden notes,

One CIO noted that he had gone through the expense reports turned into him for reimbursement and found 50 different cloud computing accounts being used by developers in his organization. The reason? It’s a lot easier for a developer to obtain computing resources from a public cloud provider than to undergo the extended waits typical of the existing compute environments.

This is undoubtedly true. I see this all the time in enterprises I work with. In my experience it is usually test environments, proof of concepts for project justifications and sneaky ways of meeting requirements (Requirements checklist item #44: Offsite back-up? Check….backed up to Amazon S3)that end up in the public clouds, not core application components.

Developers and other IT professionals are taking the path of least resistance to the public cloud due to lack of internal agility (speed) or capacity is also one of the main reasons I hear from CIOs for why they want to adopt private clouds. They want to offer their developers (and other users) the sort of speed, elasticity and cost benefits of public clouds, but in a controlled private cloud.

“The upshot of all this is that today decisions are being made by developers about use of cloud computing that upper levels of the organization are unaware of (the CIO just cited is undoubtedly an exception to the general rule). A corollary to this is that IT organizations will discover new public cloud-based applications in a year or two that they had no idea were being placed in those cloud environments. “

I am not convinced of this. CIOs might wake up to find – to their relief – that their developers adopted “cloud thinking”, that their applications architectures are cloud optimised, that their applications can use cloud APIs. I do not think we will find that large submerged sections of their are running on public clouds. I think private clouds will fill this gap quickly, just as I think that eventually hybrid clouds and the applications they enable, are the future. Right now I see hybrid, but it is usually a hybrid of public cloud and traditional bare metal within the firewall.

Golden sees three ramifications from all this:

“1. Organizations will be taking on operational responsibility for applications located in cloud environments for which they are unprepared.”

He exhorts organisations to

“quickly get up to speed on the cloud environment and learn how to operate an application within it. This will include the need to retrofit monitoring, security, and management into applications, as developers often fail to address these areas because they are not required for application functionality. “

This is a very good point. Developers may be conversant with Cloud, but the Operations and Support people need time to integrate cloud applications into their portfolios even if they are very familiar with the cloud technologies in question.

“2. Compliance requirements will need to be analyzed for these “discovered” applications.”

Golden rightly identifies  a massive “tail-end” problem of cloud adoption – controlled or uncontrolled. The front end problem is often of of encapsulating or abstracting ones application into the cloud. Once that is successful, the tail-end problems hit. These are the compliance and governance issues, the modifications to internal procedures, the challenges of defining ownership and maintaining cloud bases systems. He advices have a multi-disciplined rapid response team ready to react to the discovery of a renegade application (or part of an application) running on  a public cloud.

“3. Investment patterns of IT spend will be modified in a stealthy fashion”

Golden reiterates  point made beautifully by Simon Wardley in his absolutely brilliant “”Situation Normal, Everything Must Change” talk at OSCON 2010 in which he points out that whilst Cloud Computing will unleash innovation and IT backlog, but it is unlikely to lower costs thanks to Jevons Paradox, Componentization and Creative disruption.

I will let Simon explain. This talk is one of the best I have seen on Cloud Computing and very well worth watching end-to-end:

I am not convinced that Simon is entirely correct about Cloud Computing not reducing IT spending. Cloud Computing introduces efficiencies. There is more bang for buck.

Simon contends that instead of CIOs being able to reduce costs, the new efficiencies and innovations will give rise to greater demand (Jevons Paradox) which will wipe out the cost savings.

I think that the IT backlog caused by today’s problems will be eroded by Cloud Computing and may generate more demand, but that demand will still be backlogged. The key constraint will moved from cost and infrastructure to processes constraints like workload capacity of  IT solution architects, support staff or even the developers themselves.

In other words, demand will be constrained by human factors, whilst the infrastructure costs of satisfying demand will drop, leading to a net drop in costs (i.e. a reduced IT spend).

That said, In suppose one could reasonably argue that those cost savings would be diverted into easing the key contraint(s), the net result still being no reduction in IT spend, just more to show for it.

Nasty supermarket tricks

This video was filmed in the relatively new Idea supermarket in the up-scale Usce shopping centre. It illuminates the downright dangerous and immoral tricks played by some Serbian supermarkets.

It is not only Idea. I have had two incidents at the big Maxi at Gospodar Jovanova 39 where I ended up with rotten food.

One one occasion I was sold rotting Skate (fish). On another, I bought a kilo of marinaded chicken, only to spot that it was green and putrescent whilst preparing it.

I am told the supermarkets use the marinade to disguise the rotting, so be very careful when buying any meat in a marinade or sauce.