The Next Decade: Challenges in IT

http://www.infostor.com/backup-and_recovery/disaster-recovery/2010/the-next-decade-solving-the-big-3-it-problems.html

by Russ Fellows
INFOSTOR
November 29, 2010

The next decade will bring challenges in many areas of information technology. Although new ideas and technologies are constantly emerging, only those that can deliver real value to businesses and consumers will be successful.  In working with IT consumers, Evaluator Group has identified several issues facing businesses -- the so-called ‘big IT problems’ of the next decade. 


The issues outlined below are topics that Evaluator Group has engaged in detailed conversations about with both IT consumers and IT producers.  Nearly every business relies on information technology, and as a result will be affected by the arrival of these changes to the IT landscape.
The three big challenges facing information technology over the coming decade are:
·         Data Center Transformation – Enabling businesses to efficiently manage and deploy IT
·         Data Analytics – Deriving value and business insights from data that is captured
·         Integrated Data Management – Intelligently managing data placement, protection and archiving
Solving these challenges will require significant technology, business, and operational expertise.  One of the reasons why these problems are proving difficult is that existing technologies cannot solve these issues cost efficiently.  In some cases, the technology required is still emerging and will require integration with existing products and processes.
Evaluator Group has published studies on each of these topics, and the  papers are available at no charge on the Evaluator Group’s website. 
Enabling technologies
Solving business challenges will require new technologies that are now emerging.  Data center transformation, data analytics and integrated data management all require technologies that optimize cost and performance, while enabling massive scale and security.  Solving these challenges has been attempted in the past with various degrees of success.  However, the scope and scale of the problems have outgrown the capabilities that existing technologies are able to deliver.
Next generation solutions will require multiple technologies in order to be successful.  Convention says IT departments should standardize and consolidate their equipment in order to improve management efficiencies.  Next, they should virtualize components in order to improve efficiency without adding to management complexity.  With these moves, it is possible to create standard business catalogs of services offerings at specific quality and price levels.  Finally, these services can be integrated into ongoing operations through automation in order to maintain efficiencies and productivity gains.
A few of the technologies needed to solve these challenges include:
·         Virtualized infrastructure: The key technology that enables scale, efficiency and flexibility
·         Scale-out and scale-up: Required to support the growth in data and information processing
·         Efficiency: Achieved through process and product standardization and management
·         Flexibility: Ability to support changing requirements
·         Security: A requirement in hosted, cloud, or distributed work environments,
·         Multi –tenant design: Supports multiple clients/tenants simultaneously
The concept of virtualization is nearly as old as computing itself.  It has been applied with various degrees of success to computer memory access, processing, storage and networking.  However, virtualization is now beginning to be made visible at an external level and is being used to transform major elements of IT.
Other design requirements include efficiency, which requires standardization of interfaces and operational automation.  The ability to scale both up and down, while meeting workloads that change over time requires flexibility in both the technology and management of the infrastructure.
Securing information in a multi-user, distributed environment is both challenging and necessary.  Without adequate security, the promise of hosted cloud computing cannot succeed.  Even within public cloud settings, security and data governance is a growing issue, which must be solved in order for the next wave of IT solutions to deliver value.
Management of information must be performed holistically, across the enterprise regardless of time or place.  Data governance, protection levels, placement and security of information must be protected by polices that can span the virtualized environment.
Finally, the ability to request, configure, manage and consume IT resources will require a new wave of tools designed to allow for “virtual system management,” encompassing logical elements rather than physical products.  So-called multi-tenant management tools must support securely managing multiple clients and administrators, all with separate logical views, while using common infrastructure.
Data Center Transformation
At its core, transforming the data center is as much about business transformation as it is about technology.  Perhaps the biggest component driving this change is the movement to IT as a Service (ITaaS).  The most visible example of this is the emergence of the terms “cloud” and “cloud computing.”
Evaluator Group began talking with our clients about the business drivers behind cloud computing and ITaaS, along with the emerging technological changes, and quickly realized that what companies were seeking was a way to transform their data centers and operations, a “data center transformation.”
Updating corporate data centers has been an ongoing process since the inception of IT.  Evaluator Group began to use the term “data center transformation” in late 2008 as a way of explaining the fundamental shifts that were emerging in the way IT departments and CIOs were looking to deliver IT to their constituents.  The term “cloud computing” began to emerge during this time as well; however, the term meant very little to most people.
What is clear is that business users are looking to gain flexibility in how, when and where they consume IT resources.  Businesses in particular are now looking for a better alignment between the needs of their business and the cost and service offerings that IT departments can deliver.
Data Analytics
Traditional data processing and data warehousing are narrow examples of an emerging area known as data analytics.  Common techniques in place are relatively slow and unable to scale to solve the analytical processing of thousands of data streams in near real-time.  Business users are now looking to process information using multiple data sources concurrently.
“Big data” requires the ability to scale out information processing and management, while still providing information protection and security.  Standardization of components along with virtualization can help drive efficiencies.  Moreover, the technologies outlined are all required to meet the challenge of data analytics.
The challenge is to break problems into those that can be processed or analyzed in parallel. Techniques have emerged, including MapReduce and others, as methods for efficiently processing these types of problems.
Some of the techniques include massively parallel processing, high-speed data access coupled with hardware and software integrated appliances.  The first wave of products was typically based on commodity hardware and software with a significant amount of tuning and integration required.  More recently, integrated solutions, which rely in part on proprietary hardware or software, are now coming to market.
Integrated Data Management
Data protection, tiering, and archiving have all been topics of discussion within companies and IT organizations for decades.  Often, these discussions are independent, focused on solving a particular problem.  Business application owners and IT workers alike are now looking for a way to solve these challenges in an integrated fashion.
What is needed is a strategy that encompasses all of these topics holistically.  Evaluator Group began using the terminology “Integrated Data Management” (IDM) to discuss these areas of interest.  Past efforts have realized that these aspects were related but placed too much emphasis on particular tools or techniques to solve the problem.
Within IDM are three areas of focus:
·         Data protection (backup, point-in-time copies, replication, security, etc.)
·         Tiering (moving data within a system and between systems for cost, performance, and efficiency)
·         Archiving (storing information for long-term preservation)
These three aspects of IDM are all related.  They are part of a bigger picture of managing data cost effectively, while meeting business objectives.  Understanding their importance and relationships are critical for building and operating optimal IT operations.
The three areas outlined will certainly not be the only challenges within IT.  However, these are the topics that we feel have the potential to revolutionize how IT organizations provide and deliver information and how people consume and leverage that information within both personal and business settings.
Russ Fellows is a senior partner with the Evaluator Group research and consulting firm.
  

The Big Five IT Trends

http://www.zdnet.com/blog/hinchcliffe/the-big-five-it-trends-of-the-next-half-decade-mobile-social-cloud-consumerization-and-big-data/1811

The "Big Five" IT trends of the next half decade: Mobile, social, cloud, consumerization, and big data

Summary: In today's ever more technology-centric world, the stodgy IT department isn't considered the home of innovation and business leadership. Yet that might have to change as some of the biggest advances in the history of technology make their way into the front lines of service delivery. Here's an exploration of the top five IT trends in the next half decade, including some of the latest industry data, and what the major opportunities and challenges are.

By Dion Hinchcliffe for Enterprise Web 2.0 | October 2, 2011

0Comments

10 Votes

1145

2,645
inShare

more +

"Much or most of these topics are in back burner mode in many companies just now seeing the glimmerings of recovery from the downturn. Much has been written lately about the speed at which technology is reshaping the business landscape today. Except that's not quite phrasing it correctly. It's more like it's leaving the traditional business world behind. There are a number of root causes: The blistering pace of external innovation, the divergent path the consumer world has taken from enterprise IT, and the throughput limitations of top-down adoption. As a result, there's a rapidly expanding gap between what the technology world is executing on and what the enterprise can deliver. Many now think this gap may actually become untenable, and they may be right. Yet recent large surveys of CIOs continues to show an almost exclusively evolutionary and internal focus. Many feel that a technology emphasis is wrong right now, and they're certainly right, if it's not integrated with top priority business objectives. However, these days it's technology advancements and new digital markets that are often the key to an organization's future.
At the end of the day, businesses must be able to effectively serve the markets they cater to, and doing so means using the same channels and techniques as their trading partners and customers. Organizations must adapt to the evolving marketplace to succeed. Fortunately, I do believe there are approaches that can yet be adopted to address this increasingly significant challenge.

A tectonic technology shift

One only need look at what's on the mind of CIOs these days (60% believe they should be directly driving growth and productivity) versus what they're well known for delivering on. Or perhaps more problematically, what their IT organizations are able to deliver on. Never in my two decades of experience in the IT world have I seen such a disparity between where the world is heading as a whole and the technology approach that many companies are using to run their businesses.
The issues are legion: There are at least five major "generational scale" changes to the computing landscape happening at about the same time: Delivery platforms are shifting (mobility, cloud, social), communication and collaboration channels are being reinvented (Web, mobile, social), the consumer world of technology is driving innovation, and data is opening up and exploding out of the proliferating apps, devices, and sensors that organizations are deploying or are connecting to (but alas, are often not engaging with.) And as you might expect, much or most of these topics are in back burner mode in many companies just now seeing the glimmerings of recovery from the downturn.

Moreover, workers are now demanding many of these innovations and expecting their organizations to provide something close in capability to what they can get nearly for free (or actually for free) on their own devices and networks. Managers and executives, albeit mostly on the business side, are typically pushing for 1) service delivery on next-generation mobile devices like the iPad, 2) much easier to use IT solutions, and 3) access to better, more collaborative and useful intranet capabilities.
"Easy", highly mobile, and "social" are the mantras of this new generation of IT. So to is the rapid (read: instant) acquisition and delivery of business solutions. There is a growing realization amongst workers and management that technology, though increasingly complex in itself, can be wielded far more rapidly and efficiently than their currently parochial capabilities are providing.
But this is not a blame game. IT is not necessarily at fault, or at least only indirectly. Instead, it seems to be the entire structure and process through which organizations absorb and metabolize technology. It's centralized. It's controlled. It's top-down. There are exceptions, but in most organizations, technology decisions are made at high levels and then pushed across the organization. This transmission process is slow and unpredictable. It's also often not supported on the ground where reality reaches the business.
Unfortunately, the slow-pace of IT adoption, hindered by traditional project management practices, endless customization processes, IT backlogs, security concerns, and a dozen other drags on delivery performance, is only part of the problem. The fact that the technology world is largely no longer driven by the enterprise world (as it used to be for decades) is another major reason that technology and business is having a harder time these days aligning.
A few examples will suffice: The endless and seemingly real-time flow of useful and highly innovative new mobile and Web apps for managing travel, money, news, communication, productivity, and countless other key functions is only an inadequate trickle in the enterprise today. The ability to quickly connect, communicate, and collaborate via social conversations, photos, audio, video, and more with anyone in the world is much more limited currently in most businesses. Finding and acquiring new software is just the click of a button in an app store in the consumer world, but an arduous, manual, and failure prone process in most organizations now. User experiences are changing: The aging and slow-to-evolve graphical user interface is being uprooted by touch based interfaces in new consumer apps that work much better in many physical situations. In contrast, the same overhaul is happening an order of magnitude more slowly for business apps.

Where does technology and IT go from here?

If we project these trends forward, what will the outcome be? Is there going to be a final fork in the road for consumer and enterprise technology, with each side looking at each other through a diverging pair of windows, with minimal crossover between the two? Or will the two worlds continue to blur together, as technology cross-pollinates from the growing wall of innovation coming from the Web and consumer technology world? Given the virality and pervasiveness of consumer technology, the latter is by far the most likely scenario.
So what are the key IT trends of the next half decade? How will organizations adapt to them? In a conversation I had recently with the Editor-in-Chief of CIO Magazine, Maryfran Johnson, we discussed what I dubbed the "Big Five", the biggest technology influences of the next half decade. This includes next-gen mobility, social media (or more specifically social business), cloud computing, consumerization, and big data. We agreed that these five -- of all current tech trends -- are at top of the list for what most organizations need to be planning for in their current strategies and roadmaps as they update and modernize, as well as (hopefully) out-innovate their competitors.
Below I will explore the approaches that might break the logjam that's preventing much of the business world from becoming as current with the technology advances as they should. But first lets take a look at each of these technology trends with an eye towards the most up-to-date statement of the advantages they can provide. I'll also provide a key new insight on overcoming the challenges of adopting them more effectively and successfully.

1) Next-Gen Mobile - Smart Devices and Tablets

It's obvious to the casual observer these days that smart mobile devices based on iOS, Android, and even Blackberry OS/QNX are seeing widespread use. But comparing projected worldwide sales of tablets and PCs tells an even more dramatic story. Using the latest sales projections from Gartner on tablets and current PC shipment estimates from IDC, we can see that by 2015 the tablet market will be 479 million units and the PC market will be only just ahead at 535 million units. This means tablets alone are going to have effective parity with PCs in just 3 years. Other data I've seen tells a similar story.
So, while it's still early days yet, it's also quite clear that enterprises must start treating tablets as equal citizens in their IT strategies. So why won't they? For several reasons:
Challenges to smart device adoption

• Smart devices have a poor enterprise ecosystem today. Enterprise software vendors and IT departments have organized around older platforms such as Windows and LAMP. Their infrastructure, skills, and relationships are largely built around an older generation of IT. In the meantime, iOS and Android have a lot to learn and to build up to begin to match this world, though they are starting to make progress in this regard.
• Many of the inherent advantages of smart mobile are anathema to structured IT. From app stores to HTML 5, the large and easy to access application universes of next-gen mobile immediately triggers a security lockdown response (right reaction, wrong response) from IT. I've even seen IT departments desire to remove app stores from smart mobile devices entirely. The solution is probably policy-based screening of apps, but that's a solution a ways away.

Key adoption insight
A likely approach that will scale is to do as JP Rangaswami advocates, and "design for loss of control." This doesn't mean letting go of essential control such as robust security enforcement, but it does mean providing a framework for users to bring their own mobile devices to work in a safe manner, including use of apps with business data under certain prescribed conditions. This unleashes choice and innovation and vitally, splits the work of adoption and rollout with users that want to use their favorite mobile devices/app to solve a business problem.

2) Social Media - Social Business and Enterprise 2.0

While mobile phones technically have a broader reach than any communications device, social media has already surpassed that workhorse of the modern enterprise, e-mail. Increasingly, the world is using social networks and other social media-based services to stay in touch, communicate, and collaborate. Now key aspects of the CRM process are being overhauled to reflect a fundamentally social world and expecting to see stellar growth in the next year. As Salesforce's Marc Benioff was very clear in his dramatic keynote at Dreamforce last month, leading organizations are becoming social enterprises.
There now seems to be hard data to confirm this view: McKinsey and Company is reporting that the revenue growth of social businesses is 24% higher than less social firms and data from Frost and Sullivan backs that up across various KPIs. The message is that companies are going to -- and have every reason to -- be using social media as a primary channel in the very near future, if they aren't already. It's time to get strategic.
Challenges to social media adoption

• Social media is not an IT competency. Simply put, the human interaction portion of social computing is generally not IT's strong suit. It tends to be treated as just another application to roll out instead of being integrated meaningfully into the flow of work.
• The more significant value propositions of social requires business transformation. Maintaining a Facebook page and Twitter account is relatively straightforward and necessary, but it usually won't generate significant growth, revenue, or profits by itself either. The more profound and higher order aspects of social media including peer production of product development, customer care, and marketing require deeper rethinking of business processes.

Key adoption insight
There are a growing number of established social media adoption strategies, but probably one of the most effective is to engage by example. Both leadership inside the company as well as top representatives to the outside world must engage in social channels to show how they'd like change to happen.
Related: Reconciling the enterprise IT portfolio with social media

3) Cloud computing

Of all the technology trends on this list, cloud computing is one of the more interesting and in my opinion, now least controversial. While there are far more reasons to adopt cloud technologies than just cost reduction, according to Mike Vizard perceptions of performance issues and lack of visibility into the stack remain one of the top issues for large enterprises. Yet, among the large enterprise CTO and CIOs I speak with, cloud computing is being adopted steadily for non-mission critical applications and some are now even beginning to downsize their data centers. Business agility, vendor choice, and access to next-generation architectures are all benefits of employing the latest cloud computing architectures, which are often radically advanced compared to their traditional enterprise brethren.
Challenges to cloud computing adoption

• Concerns of control. When jobs depend on IT being up and working, then you can be sure there will be reluctance to adopt the cloud. There's also little question that not going the cloud route will mean short-term job security, but at what ultimate cost? Never mind that many CIOs and heads of IT just feel they can't yet trust the cloud, despite many cloud providers being more reliable than internal infrastructure (Google recently reported four nines across its Gmail and Google Apps services.)
• Reliability and performance perceptions. Widespread outages by Amazon and Microsoft in the past has set back cloud adoption a minor amount, yet uptime is still extraordinary good by most enterprise standards. More of an issue is moving the enormous datasets that enterprises now posses into and out of the cloud quickly enough. Backhaul and other methods will need to improve substantially to address this satisfactorily for large enterprises.

Key adoption insight
Until cloud computing workloads can be seamlessly transferred back and forth between a company's private cloud and public/hybrid cloud, adoption will be held back and favored largely for greenfield development. Technologies are now emerging to make this possible, however, and for now, companies should invest in cloud standards (to the extent they exist today) to build private clouds in order to be in position to start selectively transferring services out on a trial basis (and being able to bring them back in safely as needed.)
Related: Fixing IT in the cloud computing era.

4) Consumerization of IT

I've previously made the point that the source of innovation for technology is coming largely from the consumer world, which also sets the pace. Yet that's just one aspect of consumerization, which some like myself and Ray Wang are calling "CoIT" for short. Consumerization also very much has to do with its usage model, which eschews enterprise complexity for extreme usability and radically low barriers to participation. Enterprises which don't steadily consumerize their application portfolios are in for even lower levels of adoption and usage than they already have as workers continue to route around them for easier and more productive solutions. Another decentralized and scalable solution is, as with next-gen mobile, to help workers help themselves to third party apps that are deemed safe and secure.
Challenges to applying consumerization to IT

• Vendors provide the UX. Usability and low barriers to participation won't exist until 3rd party vendors, which provide a large percentage of IT (often on lengthy upgrade intervals), get the message and overhaul their apps.
• Consumer technology often isn't enterprise ready. At one point, neither was open source, but eventually an industry that provided value-added services emerged. The same pattern is likely to happen with popular consumer apps.

Key adoption insight
Consumerization seems especially pernicious to IT departments because it happens all the time, without their involvement. Stats vary on "shadow IT", which is in the lower double digits, but much of it is for consumer apps. IT departments can begin programs in partnership with other large companies (to distribute the work) to certify SaaS, cloud, and mobile apps and train workers on data safety, backup, and integrity for example. Longer term, companies will imbue their IT service design, solution acquisition, and delivery with user experience and design approaches and fresh ideas from the consumer world. This will drive more worker productivity, less user support, and higher innovation in IT solutions.

5) Big data

Businesses are drowning in data more than ever before, yet have surprisingly little access to it. In turn, business cycles are growing shorter and shorter, making it necessary to "see" the stream of new and existing business data and process it quickly enough to make critical decisions. The term "big data" was coined to describe new technologies and techniques that can handle an order of magnitude or two more data than enterprises are today, something existing RDBMS technology can't do it in a scalable manner or cost-effectively.
Big data offers the promise of better ROI on valuable enterprise datasets while being able to tackle entirely new business problems that were previously impossible to solve with existing techniques. While most companies are still addressing their big data needs with data warehousing, according to Loraine Lawson, one need only scan the impressive McKinsey report on Big Data to see the major opportunities it offers on the business side.
Related: The enterprise opportunity of Big Data: Closing the "clue gap"
Challenges to adopting big data

• Big data requires many new skills. There are a host of advanced technologies and new platforms to learn to be effective with big data, and the IT departments I've spoken with are concerned about the skills they must acquire or foster internally to take advantage of them.
• Meaningful use of big data requires considerable cross-functional buy-in. Big data requires tapping into silos, warehouses, and external systems using new techniques. SOA has similar challenges because it had to coordinate and align so many parts of the business. While some big data will be single function, many of the more intriguing possibilities requires a lot of cooperation across the business and with external vendors, not at easy task.

Key adoption insight
Big data requires a mindset change as much as a technology update. This means making open data a priority for the enterprise as well as an operational velocity that hasn't been a priority before. Big data enables solving new business problems in windows that weren't possible before. It also means infrastructure, ops, and development must be part of the same team and used to working together. This means organizational refinements must be made to tap into the greater potential.

How IT can evolve to meet the Big Five

I'm beginning to see that in order to stay relevant, and not become the PBX department, IT departments must be prepared to take a "Big Leap" to meet the Big Five. What this Big Leap looks like will be different for every organization, and their are multiple directions that can be taken. As I wrote on Twitter recently, the deeply transformational nature of most of the Big Five means IT must either start leading the business models and evolution of the organization, or become a commoditized utility while the business figures out the moves on their own. This almost certainly means open supply chains and enabling strategic IT abundance via designed loss of control coupled with emergent and agile approaches to IT. Now that I've explored the Big Five, I'll take a look at the Big Leap soon and see what the options are for IT -- such as "The Next Generation Enterprise Platform" that Michael Fauscette recently posited -- to not only remain relevant in the 21st century, but become the driver of business.
Can IT become the driver of business or will the function be absorbed by lines of business as their leaders become digital natives?

Dion Hinchcliffe is an expert in information technology, business strategy, and next-generation enterprises. 

What is Web 3.0 ?

Web 3.0. 

Using the same pattern as the above Wikipedia definition, Web 3.0 could be defined as: “Web 3.0, a phrase coined by John Markoff of the New York Times in 2006, refers to a supposed third generation of Internet-based services that collectively comprise what might be called ‘the intelligent Web’ — such as those using semantic web, microformats, natural language search, data-mining, machine learning, recommendation agents, and artificial intelligence technologies — which emphasize machine-facilitated understanding of information in order to provide a more productive and intuitive user experience.”

Web 3.0 Expanded Definition. 

I propose expanding the above definition of Web 3.0 to be a bit more inclusive. There are actually several major technology trends that are about to reach a new level of maturity at the same time. The simultaneous maturity of these trends is mutually reinforcing, and collectively they will drive the third-generation Web. From this broader perspective, Web 3.0 might be defined as a third-generation of the Web enabled by the convergence of several key emerging technology trends:

Ubiquitous Connectivity

• Broadband adoption
• Mobile Internet access
• Mobile devices

Network Computing

• Software-as-a-service business models
• Web services interoperability
• Distributed computing (P2P, grid computing, hosted “cloud computing” server farms such as Amazon S3)

Open Technologies

• Open APIs and protocols
• Open data formats
• Open-source software platforms
• Open data (Creative Commons, Open Data License, etc.)

Open Identity

• Open identity (OpenID)
• Open reputation
• Portable identity and personal data (for example, the ability to port your user account and search history from one service to another)

The Intelligent Web

• Semantic Web technologies (RDF, OWL, SWRL, SPARQL, Semantic application platforms, and statement-based datastores such as triplestores, tuplestores and associative databases)
• Distributed databases — or what I call “The World Wide Database” (wide-area distributed database interoperability enabled by Semantic Web technologies)
• Intelligent applications (natural language processing, machine learning, machine reasoning, autonomous agents)

Conclusion

Web 3.0 will be more connected, open, and intelligent, with semantic Web technologies, distributed databases, natural language processing, machine learning, machine reasoning, and autonomous agents.

Retrieved from

http://lifeboat.com/ex/web.3.0

The Work Ontology

Excerpt from Pull: The Power of the Semantic Web"

"I've mentioned the personal ontology often enough that you should now have a pretty good idea what it is. It specifies all your complicated tastes, requirements, constraints, beliefs, and desires as it learns about you by watching you go about your day. Because people wear many different hats, you'll build several ontologies. You'll have one for all your different roles - as a worker, a volunteer, a parent, a skydiver, etc.

You'll probably share ontologies with various groups of coworkers. Your semantic dashboard has different contexts, so as you enter each facet of your dashboard, it knows which ontology to use.

Suppose you own a factory that makes concrete bricks. If your company's ontology knows about all the assets and processes, then you can play what-if using the ontology as a modeling tool.

You can say, "What would it take to add another production line versus building a separate plant in another location?" and it will automatically go online, hook up to several brick-making ontologies, look up transportation costs from current tables, bring in the costing and estimating applications, search for everything necessary, cost out the machinery, installation, and additions to your infrastructure, and show you various scenarios.

You'll make changes to the overall picture by saying you want more bricks per hour, you have clients farther away; you want larger production runs, or you need to start making more cinder blocks. The ontology will handle all the details, from sourcing components to costing out custom molds to estimating how much space you'll need for curing and storage, etc.

In the spirit of the semantic web, you'll get work done by asking questions rather than assembling parts by hand."


Pull: The Power of the Semantic Web to Transform Your Business (2009)
by David Siegel

Entrepreneurs See a Web Guided by Common Sense by John Markoff

By JOHN MARKOFF

Published: November 12, 2006

The New York Times

SAN FRANCISCO, Nov. 11 — From the billions of documents that form the World Wide Web and the links that weave them together, computer scientists and a growing collection of start-up companies are finding new ways to mine human intelligence.

Their goal is to add a layer of meaning on top of the existing Web that would make it less of a catalog and more of a guide — and even provide the foundation for systems that can reason in a human fashion. That level of artificial intelligence, with machines doing the thinking instead of simply following commands, has eluded researchers for more than half a century.

Referred to as Web 3.0, the effort is in its infancy, and the very idea has given rise to skeptics who have called it an unobtainable vision. But the underlying technologies are rapidly gaining adherents, at big companies like I.B.M. and Google as well as small ones. Their projects often center on simple, practical uses, from producing vacation recommendations to predicting the next hit song.

But in the future, more powerful systems could act as personal advisers in areas as diverse as financial planning, with an intelligent system mapping out a retirement plan for a couple, for instance, or educational consulting, with the Web helping a high school student identify the right college.

The projects aimed at creating Web 3.0 all take advantage of increasingly powerful computers that can quickly and completely scour the Web.

“I call it the World Wide Database,” said Nova Spivack, the founder of a start-up firm whose technology detects relationships between nuggets of information by mining the World Wide Web. “We are going from a Web of connected documents to a Web of connected data.”

Web 2.0, which describes the ability to seamlessly connect applications (like geographic mapping) and services (like photo-sharing) over the Internet, has in recent months become the focus of dot-com-style hype in Silicon Valley. But commercial interest in Web 3.0 — or the “semantic Web,” for the idea of adding meaning — is only now emerging.

The classic example of the Web 2.0 era is the “mash-up” — for example, connecting a rental-housing Web site with Google Maps to create a new, more useful service that automatically shows the location of each rental listing.

In contrast, the Holy Grail for developers of the semantic Web is to build a system that can give a reasonable and complete response to a simple question like: “I’m looking for a warm place to vacation and I have a budget of $3,000. Oh, and I have an 11-year-old child.”

Under today’s system, such a query can lead to hours of sifting — through lists of flights, hotel, car rentals — and the options are often at odds with one another. Under Web 3.0, the same search would ideally call up a complete vacation package that was planned as meticulously as if it had been assembled by a human travel agent.

How such systems will be built, and how soon they will begin providing meaningful answers, is now a matter of vigorous debate both among academic researchers and commercial technologists. Some are focused on creating a vast new structure to supplant the existing Web; others are developing pragmatic tools that extract meaning from the existing Web.

But all agree that if such systems emerge, they will instantly become more commercially valuable than today’s search engines, which return thousands or even millions of documents but as a rule do not answer questions directly.

Underscoring the potential of mining human knowledge is an extraordinarily profitable example: the
basic technology that made Google possible, known as “Page Rank,” systematically exploits human knowledge and decisions about what is significant to order search results. (It interprets a link from one page to another as a “vote,” but votes cast by pages considered popular are weighted more heavily.)

Today researchers are pushing further. Mr. Spivack’s company, Radar Networks, for example, is one of several working to exploit the content of social computing sites, which allow users to collaborate in gathering and adding their thoughts to a wide array of content, from travel to movies.

Radar’s technology is based on a next-generation database system that stores associations, such as one person’s relationship to another (colleague, friend, brother), rather than specific items like text or numbers.

One example that hints at the potential of such systems is KnowItAll, a project by a group of University of Washington faculty members and students that has been financed by Google. One sample system created using the technology is Opine, which is designed to extract and aggregate user-posted information from product and review sites.

One demonstration project focusing on hotels “understands” concepts like room temperature, bed comfort and hotel price, and can distinguish between concepts like “great,” “almost great” and “mostly O.K.” to provide useful direct answers. Whereas today’s travel recommendation sites force people to weed through long lists of comments and observations left by others, the Web. 3.0 system would weigh and rank all of the comments and find, by cognitive deduction, just the right hotel for a particular user.

“The system will know that spotless is better than clean,” said Oren Etzioni, an artificial-intelligence researcher at the University of Washington who is a leader of the project. “There is the growing realization that text on the Web is a tremendous resource.”

In its current state, the Web is often described as being in the Lego phase, with all of its different parts capable of connecting to one another. Those who envision the next phase, Web 3.0, see it as an era when machines will start to do seemingly intelligent things.

Researchers and entrepreneurs say that while it is unlikely that there will be complete artificial-intelligence systems any time soon, if ever, the content of the Web is already growing more intelligent. Smart Webcams watch for intruders, while Web-based e-mail programs recognize dates and locations. Such programs, the researchers say, may signal the impending birth of Web 3.0.

“It’s a hot topic, and people haven’t realized this spooky thing about how much they are depending on A.I.,” said W. Daniel Hillis, a veteran artificial-intelligence researcher who founded Metaweb Technologies here last year.

Like Radar Networks, Metaweb is still not publicly describing what its service or product will be, though the company’s Web site states that Metaweb intends to “build a better infrastructure for the Web.”

“It is pretty clear that human knowledge is out there and more exposed to machines than it ever was before,” Mr. Hillis said.

Both Radar Networks and Metaweb have their roots in part in technology development done originally for the military and intelligence agencies. Early research financed by the National Security Agency, the Central Intelligence Agency and the Defense Advanced Research Projects Agency predated a pioneering call for a semantic Web made in 1999 by Tim Berners-Lee, the creator of the World Wide Web a decade earlier.

Intelligence agencies also helped underwrite the work of Doug Lenat, a computer scientist whose company, Cycorp of Austin, Tex., sells systems and services to the government and large corporations. For the last quarter-century Mr. Lenat has labored on an artificial-intelligence system named Cyc that he claimed would some day be able to answer questions posed in spoken or written language — and to reason.

Cyc was originally built by entering millions of common-sense facts that the computer system would “learn.” But in a lecture given at Google earlier this year, Mr. Lenat said, Cyc is now learning by mining the World Wide Web — a process that is part of how Web 3.0 is being built.

During his talk, he implied that Cyc is now capable of answering a sophisticated natural-language query like: “Which American city would be most vulnerable to an anthrax attack during summer?”

Separately, I.B.M. researchers say they are now routinely using a digital snapshot of the six billion documents that make up the non-pornographic World Wide Web to do survey research and answer questions for corporate customers on diverse topics, such as market research and corporate branding.

Daniel Gruhl, a staff scientist at I.B.M.’s Almaden Research Center in San Jose, Calif., said the data mining system, known as Web Fountain, has been used to determine the attitudes of young people on death for a insurance company and was able to choose between the terms “utility computing” and “grid computing,” for an I.B.M. branding effort.

“It turned out that only geeks liked the term ‘grid computing,’ ” he said.

I.B.M. has used the system to do market research for television networks on the popularity of shows by mining a popular online community site, he said. Additionally, by mining the “buzz” on college music Web sites, the researchers were able to predict songs that would hit the top of the pop charts in the next two weeks — a capability more impressive than today’s market research predictions.

There is debate over whether systems like Cyc will be the driving force behind Web 3.0 or whether intelligence will emerge in a more organic fashion, from technologies that systematically extract meaning from the existing Web. Those in the latter camp say they see early examples in services like del.icio.us and Flickr, the bookmarking and photo-sharing systems acquired by Yahoo, and Digg, a news service that relies on aggregating the opinions of readers to find stories of interest.

In Flickr, for example, users “tag” photos, making it simple to identify images in ways that have eluded scientists in the past.

“With Flickr you can find images that a computer could never find,” said Prabhakar Raghavan, head of research at Yahoo. “Something that defied us for 50 years suddenly became trivial. It wouldn’t have become trivial without the Web.”

Thinking about Clean Clouds

There is a trade-off between efficiency/resource utilization on the one hand and reliability/convenience on the other with regard to cloud computing.  Although use of the term 'cloud computing' provokes images of data, apps and programs residing in some nebulous non-place like a cloud, it is all just moved to a data center somewhere. In order to prevent service outages or downtime, most providers maintain excess capacity meaning that often server utilization rates are "6% to 15%, with 75% of servers using less than 10%." (Glanz, 2012). Many are 'comatose' servers, doing little more than burning electricity and "little if any computational work." (Glanz, 2012).  The studies found that up to 75% of the servers in a given 'farm' were basically idle. 

Servers are not the only energy consumers in data centers however, industrial cooling systems are needed to keep the massive spaces useable.  There are also backup battery installations and chargers necessary to prevent disruptions due to the local electric grid.  In fact, most data centers maintain a stable of diesel generators that kick in whenever the local electrical service grid goes down.  The explosion of consumer data creation, transmission and storage due to the relative cheapness of cloud based centralized computing is leading to an energy useage profile that is simply not sustainable. 

One possible avenue to redress this mismatch is virtualization, which effectively merges multiple servers into one large flexible computing platform that can host a variety of applications or data on a much more rapid scaling basis.  While this may reduce the need to maintain individual excess capacity in computing resources and help improve server utilization rates, it will not reduce the total energy footprint. "Nationwide, data centers used about 76 billion kilowatt-hours in 2010, or roughly 2 percent of all electricity used in the country that year," noted The new York Times.

Glanz, J. "Power, Pollution and the Internet."  New York Times.  (September 22, 2012). 
Retrieved from
http://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of-energy-belying-industry-image.html

Which is better -- real or artificial intelligence?

The question of which is better, artificial or real intelligence contains several concepts that need unpacking first in order to make a reasonable answer.

First what exactly do we mean by intelligence?

There are a number of qualitative characteristics that seem to constitute intelligence. The ability to make sense of various inputs from the environment as well as remembering and learning from past experience are considered signs of intelligence. Also, the ability to make sense of ambiguity or seemingly contradictory inputs and dealing with perplexity through rational inference are components of intelligence. Intelligence means using a reasoned or rational thought process to respond and react successfully to new situations or scenarios. Intelligence is complex and adaptive. Intelligence is recognizing and making judgments about the relative importance of different elements within a situation to arrive at reasonable and effective responses.

Second, what do we mean by artificial intelligence? (I am assuming that 'real' intelligence is that exhibited by human agents).

Artificial intelligence, also called machine intelligence, is behavior performed by a computer system that if done by a human would be considered intelligent behavior. AI differs from the typical computer system in that it focuses on symbolic as opposed to numeric manipulation and utilizes heuristic processing as opposed to algorithmic processing. Heuristics are basically a form of intuitive knowledge, often called rules of thumb. Rather than a specific and rigid algorithm, heuristics are flexible and adaptive. AI's represent knowledge symbolically and manipulate that knowledge using heuristics thereby becoming capable of responses that appear in context to replicate real intelligence. Perhaps the best example is the chess program Deep Blue developed by IBM which beat Chess Grand Master Gary Kasparov. (Chess has long been considered a yardstick of measuring intelligent behavior and seems to be a perfect replicator of symbolic heuristic manipulation.)

Lastly, what do we mean by better? Better in what sense?

If it is simply a matter of preference, not surprisingly we humans would appear to prefer real (or human) intelligence. But there are a number of areas in which AI is arguably superior. Because artificial intelligence operates within a computer platform, it is well documented in a way that real intelligence is not. It is also internally consistent and thorough. It does not suffer from fatigue or have bad days. AI tends to execute tasks much faster than a human could and perform those tasks with a higher degree of accuracy. It also has a characteristic of permanence that real intelligence unfortunately lacks, as well as being easier to duplicate and disseminate. For certain tasks, AI is considerably cheaper to develop and deploy. But even with all of these advantages, there are still areas in which it is inferior to real intelligence. AIs are as of yet not naturally creative and lack inspiration. There are no AI Mozarts or Picassos yet. They are also inferior to human intelligence with regard to sensing the environment directly and adapting quickly. AIs are inferior to human jet fighter pilots.

So it would probably be correct to say, each is good at certain things and not so good at other things.

Reference

Turban, E., Sharda, R., & Delen, D. (2010). Decision Support and Business Intelligent Systems (9th ed.). Upper Saddle River, NJ: Prentice Hall.