Understand Natural Language Processing in 5 questions



One of the many exciting futurisms predicted by the original Star Trek TV series was the notion that people could have conversations with computers. “Computer, show me a map of all Klingon bases in the galaxy,” Captain Kirk would say to some unseen microphone on the bridge of the Enterprise. In response, the computer would—as if by magic—show the requested map on a large screen.

It seemed so incredibly far-fetched when the series came out in 1966. At that point, human beings could only interact with computers using teletypes and, in some cases still, paper punch cards. What Star Trek viewers could not have known, however, was that even by 1966 computer scientists were already a decade into ambitious projects aimed at getting computers to understand human language. Progress was slow at first, but the last few years have seen remarkable breakthroughs in natural language processing (NLP).

Today, we take it for granted that we can ask Siri to show us a map, as if we were Captain Kirk, but it has taken half a century to get to this point where computers are capable of speech recognition. As NLP’s sophistication and potential increases, it’s worth taking a moment to understand how computer achieved the powers of natural language understanding and natural language generation. These concepts are likely to predominate the IT field in the coming years.

What is Natural Language Processing?

Simply speaking, natural language processing refers to any situation where a human-to-machine interaction where the computer is able to understand or generate human-like language. This is in contrast to what computers were previously limited to, which was some form of machine language. Simple as the end result may appear, the actual process of getting a computer to perform NLP represents an extremely complex synergy of different scientific and technical disciplines.

NLP brings together computer science, linguistics, machine learning and artificial intelligence (AI). Depending on how the NLP software is designed, the process may also involve statistics and data analytics. These elements, working in concert, make NLP capable of “hearing” or reading natural human speech and accurately parsing the words so the computer can take the action expected by the human user.

How does it work?

How does NLP work? There is no single way that NLP software functions. However, in general, almost all NLP tools have capabilities that enable them to distinguish syntactic and semantic rules in addition to recognizing many different words.  This is not an easy thing to do. Consider the following written enterprise search query: An employee wants to know if the company has declared December 26 a holiday. She types “Do we get Boxing Day off?” That’s a question that most human beings can easily understand, even if they have to figure out that December 26 is sometimes called “Boxing Day.”

An NLP program is going to have to deconstruct this question on syntactic and semantic levels in order to enable the enterprise search engine to return a result. First, it has to recognize the “Do we” is a question reflecting “we,” meaning employees of the company. Then, it has to parse the words “get” and “off” as referring to getting time off from work, versus say, getting off a plane. It needs to understand what Boxing Day is… to the point where the search engine can actually respond to the real query, which is “Is December 26 a holiday?”

Performing these tasks is a pretty gigantic challenge. Human language is fantastically complex, with English being arguably one of the most difficult of them all. NLP transforms data into something that a computer can interpret by starting with what is known as “data pre-processing.” In this stage, the NLP tool analyzes syntax and semantics to understand the grammatical structure of the text. It identifies how the words relate to one another in the specific context at hand.

Data pre-processing may utilize tokenization, which breaks text down into semantic units for analysis. The process then tags different parts of speech, e.g. “we” is a noun, “do” is a verb and so forth. It could then perform techniques called “stemming” and “lemmatization,” which reduce words to their root forms. The NLP tool might also filter out words like “a” and “the” that don’t any unique information.

The data pre-processing step generates a clean dataset for the actual linguistic analysis. The NLP tool has an algorithm that then interprets the dataset. The algorithm can take one of several forms. With a rule-based approach, the NLP tool uses grammatical rules created by expert linguists. Alternatively, and this is increasingly common, NLP makes use of machine learning algorithms. These models are based on statistical methods that “train” the NLP to get better and better at understanding human language. Going further, the NLP tool might take advantage of deep learning, sometimes called deep structured learning, which is based on artificial neural networks.


What is an NLP application?

NLP applications put NLP to work in specific use cases, such as intelligent search. The technology has many uses, especially in the business world where people need help from computers in dealing with large volumes of unstructured text data. For example, a company might benefit from understanding its customers’ opinions of the brand. However, whatever insights there are regarding the brand are hidden with millions of social media messages. No human being is going to read them all. However, an NLP tool that is tuned for “sentiment analysis” could get the job done.

Other notable NLP applications include:

  • Virtual Assistants and Chatbots—these familiar bits of software can answer questions and provide online help, among many use cases using NLP. They are usually configured to learn from every “conversation” they have.
  • Market Research—NLP is able to help marketers learn about their customers by analyzing human language contained in unstructured data such as chat threads and online comments. This process uses text classification, another NLP application. Market research could also require text extraction, wherein the NLP tool looks for specific words, such as a product name, and extracts the relevant text for use in customer analysis. The software may be able to infer purchase intent, among other capabilities.
  • Speech Recognition—NLP tools can recognize spoken language, such as is the case with virtual assistants like Amazon’s Alexa and Apple’s Siri. The technology can also be put to work transcribing recordings and voice messages.
  • Urgency Detection—An NLP tool can be trained to spot urgent issues in a stream of natural language. For example, if a company receives 100,000 support emails a day, it can use NLP urgency detection to find customer who need help right away by spotting phrases like “I’m locked out of my car” or “I am about to go into the hospital.”


What are the benefits of NLP?

As the NLP applications suggest, the technology can deliver an extensive array of benefits to businesses. With NLP, machines get smarter and more able to interact with human beings. In enterprise search, for example, NLP-enable search platforms make it easier for employees to find information and documents. They don’t have to know exactly what they’re searching for. They can write a more general query and the platform will use its human language skills to discover whatever it is the user needs. This saves time and ultimately, money.

Government and public sector organizations can similarly benefit from NLP. By making it possible for people to get information and services without having to wait for a person, NLP can potentially improve people’s lives. It helps keep costs down, which is also a constant issue in the public sector.


What are the challenges of NLP?

NLP is an impressive technology, but it’s still quite early in its lifecycle. In 2066, people will probably be amazed at how primitive 2020’s state of the art looks to them. Many challenges exist. These include making speech recognition better along with achieving a more consistent and accurate understanding of language. This problem is partly due to some current limitations of AI. There is more to intelligence than just language, after all. For instance, a computer may not understand the meaning behind a statement like, “My wife is angry at me because I didn’t eat her mother’s dessert.” There’s a lot of human culture embedded in the language. The technology to identity such nuances has not been invented so far.


NLP is a foundational aspect of intelligent search solutions used in the enterprise. By understanding natural language, an enterprise search platform is better able to give users the information they seek. The platform can achieve this goal by connecting NLP with data in all systems, formats, locations or languages. The technology has come a long way in recent years, with many more advances sure to come in the near future. The bridge of the starship Enterprise awaits.


+1Share on LinkedInShare on Twitter

The benefits of a digital workplace solution

digital-worplace-solution 2


Even before the pandemic forced us to reevaluate how we work, it was clear that the traditional workplace as we knew it wasn’t working. A Harvard Business Review study from 2019 found that open workplaces meant to facilitate collaboration actually reduced face-to-face interactions by 70% and increased emails and messaging by 50%. Despite office planners’ best efforts, people were driving to the office just to email and Slack their colleagues. At the same time, an Accenture survey from 2019 found that 71% of executives felt that their employees were more “digitally mature” than their organization and were waiting for their company to catch up.

“Catching up” became the goal for many and ushered in this current phase of digital transformation.


The trouble with transformation

Businesses around the world were already taking steps toward their digital transformation when the pandemic forced an evolution overnight. What most companies learned is that remote work on a large scale is possible, but far from perfect. The digital workplaces they built were often done so from the technology ground floor first: clouds, content platforms, apps, and analytics. The next step would be to transform processes using AI to augment human workers, but many aren’t there yet. In reality, what ends up happening is these technologies throw off inhuman amounts of data, which overwhelms the people doing the work.

How can companies achieve an effective digital workplace, without drowning their employees in a sea of information?


A digital workplace solution to remove distance barriers 

With remote employees spread across the globe in larger numbers than ever, enabling access to information regardless of location becomes both a necessity and a challenge. Security and compliance issues aside, helping employees find what they need across different languages, SharePoint sites, messaging applications, and more is no small task.

A digital workplace solution that connects data and information is the missing piece in many digital transformation plans. Enterprise search is the answer and provides a central place to look for all files, documents, presentations, spreadsheets, weblinks, and other rich media. And it can do so by leveraging your existing systems – no integration is required.


A digital workplace solution to engage your employees

Digital workplace articles talk a lot about fostering employee engagement, but what does that really mean?

An engaged employee is someone who feels connected to the company’s vision and mission, who is productive and consistently gets the job done, and who sees the potential for their own growth alongside the company’s growth. These are people who feel energized, committed, and valued. And they have a measurable impact on a businesses’ success. In fact, research shows that companies with more employee engagement see more revenue.

As companies shift to digital workplaces, employee engagement often suffers. One reason for this is difficulty finding the information they need to do their jobs. As tools and applications are added to enable the digital workplace, this leads to scattered and siloed information that employees spend hours searching through. The harder it is to find what they need, the more disengaged and frustrated employees become.

With a cohesive enterprise search solution, employees can gain a unified view of information, delivered in a familiar search-based experience. To put it plainly, they can stop wasting time looking for things. This in turn ensures they adopt digital workplaces more easily and increases employee engagement.


A digital workplace solution to improve productivity

The exploding data and hours of time that employees waste looking for what they need has other ramifications, as well. Without a reliable way to search through and contextualize all the structured and unstructured data that exists, insights are routinely missed, and the value of the data is lost. Employees often need to ask colleagues for help finding the information they need, wasting additional time and resources and slowing progress. And in the end, the digital workplace that was meant to facilitate more creative, nonroutine work actually ends up producing the inverse.

An enterprise search solution can solve this information crisis. It can search and retrieve data regardless of format, type, language, and location. But more than that, it can use AI to understand the context of each piece and match it to the search intent. And the more data it is fed, the more it learns, returning better results with each query. To the end user, it is a simple and familiar experience that delivers powerful results. For businesses overall, it’s a key building block in their digital transformation.

+1Share on LinkedInShare on Twitter

Enterprise Search in the Digital Workplace

easy-enterprise-searchThis article originally appeared on the APQC blog.

Any knowledge-intensive organization of significant size either has a digital workplace or is scrambling to establish one. As Gartner so clearly stated back in 2017, the goal of the digital workplace as a business strategy is to boost employee engagement and agility through a more “consumerized” work environment. Employees should be empowered and motivated to get the information they need and then act on it to further the goals of the organization. The tools and resources provided to employees should seem familiar—like those used outside of work — in order to drive adoption and maximize ongoing productivity.

Building the Foundation

Gartner’s “building block” diagram illustrates the components necessary to enable and sustain an effective digital workplace. Note the “Information” building block, described as data and content being delivered in context. As organizations continue to pursue initiatives around digitization and digital transformation, they often create new challenges for their employees to overcome. Many of these challenges emanate from content and data that is accumulating quickly and constantly across siloed repositories in different formats and languages. For employees, navigating this complexity means wasted time, missed insights, and lost opportunities. The answer to this problem is enterprise search.

Gartner's Building Blocks of The Digital Workplace

Collaboration that Scales

A modern enterprise search platform includes a combination of capabilities that work together to provide information from enterprise content and data. It provides information relevance that is tuned to user needs and can improve through self-learning over time. It can scale in multiple directions, enabling many end-users to simultaneously access relevant information and insights from huge, diverse volumes of content and data. Modern search provides speedy response times, even in the most complex and demanding environments where time is literally money. The user experience can be easily configured to accommodate specialty use cases. Analytics are automated wherever possible, allowing the machine to do analysis while users apply judgment and make decisions.

Driving Business Value

Depending on the environment and application of enterprise search, the business impact can take on different forms. In some cases, enterprise search can drive revenue.  This has been proven in large pharma companies that are able to bring new drugs to market faster and in manufacturing and service organizations that can generate more proposals without sacrificing quality. In other cases, enterprise search drives cost optimization, accelerated productivity, and responsive compliance.


What companies need now is a practical means of connecting the dots to tap the potential value of all the content and data that resides across enterprise systems.  Doing so will address all kinds of business challenges, including those that were unforeseen in the implementation of each individual system.  Intelligent search platforms such as Sinequa can enable organizations to rise to these challenges and help transform the way professionals, businesses, and industries interact and operate in the digital world.


+1Share on LinkedInShare on Twitter

Working Effectively while Working Remotely with Enterprise Search

COVID-19 Work From Home

The working world is experiencing an unprecedented spike in remote work. “We’re being forced into the world’s largest work-from-home experiment and, so far, it hasn’t been easy for a lot of organizations to implement,” says Saikat Chatterjee, Senior Director, Advisory at Gartner. “In a recent webinar snap poll, 91% of attending HR leaders indicated that they have implemented ‘work from home’ arrangements since the outbreak, but the biggest challenge stems from the lack of technology infrastructure and lack of comfort with new ways of working.”

At the center of these challenges are employees not having a consistent and reliable way to reach the information they need to be well-informed. In some organizations, this is happening quickly and even starting to threaten business continuity, especially as more employees begin to rely on the digital workplace to be productive.

Enterprise Search in the Digital Workplace

Any knowledge-intensive organization of significant size probably has a digital workplace that includes what could be referred to as enterprise search (even if they don’t call it that). Maybe they downloaded an open-source kit that provides employees with a rudimentary way to query across sources using keywords. Or maybe they’ve chosen the ecosystem of a large technology company like Microsoft, Google, or IBM, which tend to exclude content and data stored outside of the ecosystem.  Now, faced with a sudden surge in the importance of quickly accessing essential information, regardless of its source or format, companies are realizing that these solutions fall short.

Regardless of the initial path chosen, there are some fundamental requirements that must be seriously considered to maximize the value of an enterprise search investment. These requirements include the following:

  • All enterprise content and data across time, locations, and languages must be securely available for employees to access without the need for risky data migration projects
  • Data security and access control must be rigorously enforced by default
  • Relevance and information accuracy are a must for users to do their work properly and swiftly. This requires different types of linguistic analysis, preferably provided out-of-the-box to save time in implementing enterprise search.
  • Classification-by-example powered by machine learning algorithms must also be available out-of-the-box for scenarios where a rules-based approach does not suffice
  • The user interface must be flexible and agile to support solutions for multiple use cases across the organization

These capabilities provide significant benefits for employees in the digital workplace in several different ways. Let’s take a look at some of the key benefits.

Employee Productivity

Having a robust enterprise search solution in place allows employees to quickly find the document, content, and information they are looking for, rather than spending time trying to contact other employees and disturb everyone’s workflow. This enables people to save crucial time, which can be channeled into more productive work.

Knowledge Sharing

According to data collected prior to the current spike in remote work, Fortune 500 companies were already losing roughly $31.5 billion a year by failing to share knowledge. Much of this “hidden” knowledge could be extremely useful in providing new hires with information that is not widely known by other employees within the organization.  Making sure this knowledge is explicit and findable lays a foundation for a much more efficient workforce.

Enterprise search enables organizations to surface the know-how and experience of senior managers so that the knowledge of the organization does not remain hidden when the employee leaves.  With an enterprise search solution in place, your current or future employees can easily access this information and continue doing their work with ease.

Information Access

It’s difficult to know with any certainty how much productive time employees are leaving on the table just because they cannot find the desired information or content they are looking for.  According to a benchmarking survey done by the folks over at IntraTeam, users within only 25% of organizations surveyed are satisfied with the internal search functionality.  And that was before everyone was suddenly displaced from their offices and forced to use online tools for the majority of their work.

Having a robust digital workplace structure in place means easy access to information. Enterprise search in the digital workplace provides a central place to look for all files, documents, presentations, spreadsheets, weblinks, and other rich media. This makes it extremely easy for team members, irrespective of their location to access information from any device quickly.

Competitive Advantage

Consistently well-informed employees can also provide better service to customers and offer better turnaround times. Since they are saving a lot of time, they can focus on the things that really matter and contribute to the business’s success more effectively.


The old phrase “Make hay while the sun shines” reminds us to make the most of our opportunities while we have the chance. In the current world health climate, with travel restrictions becoming more prevalent and events being canceled or postponed, now might be the ideal time for organizations to invest in tools and technology that directly drive operational efficiency. The positive impacts in terms of business continuity, cost savings, and employee empowerment can be enormous.


+1Share on LinkedInShare on Twitter

Enterprise Search Development: Start With the User Interface

This article was originally published on CMSWire.

Enterprise Search Development: Start With the User Interface 

By Martin White | Mar 10, 2020


Start with the user interface (UI) and work backwards. That was the advice I shared with search managers developing their existing application or planning a new application during an enterprise search workshop at the recent IntraTeam event in Copenhagen, Denmark. Sinequa recently sent me some examples of user interfaces from its customers (thank you Laurent Fanichet), which showed the variety and inevitable complexity of enterprise search UIs. Too often businesses make a deliberate choice on the technology and give little thought to the UI (so much for the ethos of user-centric delivery!).

The topics outlined below cannot be left to the implementation stage. Most search applications (with the obvious exception of Office 365) are UI neutral and can support almost any UI development language. Early work around these topics is essential, even at the specification stage, to ensure the investment is fit for purpose, not just to specification.


Enterprise information collections are much larger than might be imagined and inevitably contain many near-identical documents. HR and related corporate policies are just one example of this. So delivering the “most relevant” document in response to a query is limited at best.

The rhetoric of personalization through AI usually fails to deliver for two reasons: First, it assumes the user is seeking the information for themselves. Second, AI works on the basis of prior searches, but many of the searches will be by people who are new to an organization or role.

sinequa-screenshot-patent-miningProvide users with filters and facets so they can refine a set of results. But keep in mind, providing filtering just by file format and last revised date is a waste of screen space. Ask people how they might want to filter (e.g. country, date of publication, department, language). With that valuable information in hand, work out how the metadata to drive these filters is going to be derived — either from the text of the document, through tags or a combination.

Snippet Options

Quite a lot of work has been undertaken into the format of snippets. One size does not fit all. This is especially the case in enterprise search where the primary assessment of results is through information foraging. The format of the result and what ancillary information can be switched on or off by the user is important to consider. For some searches an expanded snippet with highlighted query terms might be invaluable, but this will limit the number of results displayed per page.


Designing search pages that scroll is a seriously bad idea. Even if the results are scrolled, the ancillary filters and facets will remain stationery, and in any case people will want to see the results in the context of a page of results. When usability testing happens later in the project, it will start with a discussion about which elements of the UI have been the subject of continuing discussion without a clear resolution and need real-life testing.


In the digital workplace, accessibility is very important as there will be few workarounds. At the outset you should be working with accessibility consultants to consider how voice browsers will work with the proposed UI and what the implications are for staff on the dyslexia spectrum.

Federated Search/Multilingual Search

The current interest in presenting the results from multiple repositories seems to ignore the challenges in how to present the final results. When there are only two applications (or languages) then two windows might be the best option, but as the number increases so does the complexity of the user interface. This becomes even more acute when results from text searches need to be interleaved with results from enterprise databases.

Training and Support

No matter how well you design a user interface, enterprise search is never going to be intuitive. This is due to the variable quality of the content and the metadata and the wide range of queries. Any discussion about a search UI has to take into account the extent to which training might be required for one or more aspects which will be a challenge to use.

To read the full article please visit https://www.cmswire.com/information-management/enterprise-search-development-start-with-the-user-interface/

+1Share on LinkedInShare on Twitter