As the major international showpiece for broadcasting and pay TV, IBC is a good place to go trend spotting and web speed emerged as one of the hot buttons this year. As it happens I have recently co-written a white paper on the subject with Ben Schwarz, based on interviews with various operators and so I felt well placed to engage in the discussions at IBC. One point that stood out in the paper, called Taking TV Innovation from Operator Speed to Web Speed, was how achieving web speed requires a complete change in corporate mind-set and organization, alongside new tools for automating programming and testing. When it comes to project development and software deployment the mind-set among many Telcos as well as pay TV operators has been dictated by the long standing fear of suffering reputational damage by releasing a product too soon before all the bugs are out. This anti-beta mentality imposed a sharp brake on innovation and proved a major handicap when the Internet brigade arrived spearheaded by Google and beta became a permanent state, the new alpha if you like.
This took me back to my early days as a programmer with almost sole responsibility over software development for a smallish engineering firm, which meant I did everything from structural analysis to payroll. I had neither the time nor the tools for elaborate integrated system testing so inevitably my users had bug-ridden pre-beta versions afflicted on them, which took months before they reached a state that would be deemed fit for market. The firm regarded IT as a necessary evil on which as little money should be wasted as possible but at least accepted my argument that live testing in the field was by far the most efficient and indeed the only way of ironing out the bugs and resolving the numerous exceptions, even if that meant that for months some staff failed to get their bonuses paid on time.
The discussion of DevOps as the new mantra for software development and systems integration in pay TV also resonated with me. As virtually the sole programmer I was by definition Mr DevOps meaning that operations and development were rolled into one under my own persona, which at least meant that many bugs were fixed almost immediately after they surfaced. I thought nothing of this at the time but then in subsequent jobs with larger organizations I saw only too well how the separation between software development and operational testing impeded innovation and slowed down deployment. The emphasis on DevOps is absolutely correct, but as our paper made clear it has to be done in an entrepreneurial spirit where not only operations and software development are merged but the distinction between them almost evaporates.
Such blending of operations and development is an essential prelude to the microservices approach service providers are now striving for, with a shift away from code-heavy monolithic applications to smaller, self-contained processes that can be introduced with minimal disruption and low risk. A team of one to five people will typically manage all aspects of a microservice, combining operational and development skills. The paper cited Pedro Miguel Bandeira, Head of Product Development for TV, Internet and Voice at Portuguese operator NOS, explaining how the ability to prototype and test new features with customers without having to conduct extensive market research had quickened the pace of innovation markedly. He compared NOS’s situation favourably with competitors dependent on solutions such as Ericsson’s Mediaroom, reducing time to develop and deploy new service modules from months to just a few days. Ericsson would point out it has fixed this with its later MediaFirst IP video platform but that is another story.
Admittedly such a sensational reduction in time to market cannot be achieved just by organizational changes. Bandeira made the point that the DevOps methodology can almost be a victim of its own success in shortening software development cycles by running into a bottleneck when it comes to final Quality Assurance. Indeed as consumers have come to expect smartphone-like experiences with intuitive UIs the emphasis is shifting back onto QA as a vital step in the process. We may live in a world of continuous beta but users want a premium alpha experience like never before. As a result of these expectations coupled with the faster time to deployment a lot of QA processes that used to be done manually are now by necessity being automated. As Bandeira pointed out it is impossible to do human testing as in the past because it then took three weeks just to complete the internal QA, which would make all the other time savings redundant. Now advanced operators such as NOS have scripts that interact with the set top box for example, analysing its output in the different software layers. These can plough through over 5,000 different use cases in just 4 or 5 hours for basic interactions and 48 hours for longevity tests.
How I wish such tools had been available for payroll software testing all those years ago.
