In a recent post, we looked at how AI in ITSM can either accelerate service delivery or add more work if the data’s not ready. One recurring issue ive had is AI chatbots serving up irrelevant or outdated knowledge articles. If you’ve ever wondered why your chatbot can’t get it right, this post is for you.
Here’s a problem I’ve experienced first hand which may sound familiar
- We had a single IT knowledge base with over 200 knowledge custodian groups, nearly impossible to manage or govern effectively.
- The organisation had more than 1,000 assignment groups, making it complicated to align knowledge ownership and accountability.
- Governance was missing. Articles were poorly maintained, rarely updated, and often lacked review cycles.
- The knowledge base was dis-organised, badly categorised, hard to search, and difficult for both humans and AI to navigate.
In our previous post on AI in ITSM, we explored how AI is often blamed when the real culprit is poor data readiness. Nowhere is this more obvious than in knowledge bases. Before rolling out any AI-powered assistant, especially in platforms like ServiceNow, you need to fix your foundation.
Here’s how.
Why Knowledge Quality Matters
Generative AI and chatbot assistants rely on structured, well tagged, and accurate knowledge articles to function. If the articles are outdated, irrelevant, or inconsistently formatted, AI won’t magically overcome those flaws. Instead, it will amplify them, often in front of end users.
Some common issues include.
- Irrelevant or duplicate content surfacing
- Inconsistent article formats confusing Natural Language Processing (NLP)
- Poor use of tags or metadata
- Long, bloated articles with no clear summary
- Knowledge gaps leading to fallback answers like “Contact the Service Desk”
In the below example Ive referenced ServiceNow as this is something i have worked with on a regular basis. I would expect most tools to be similar.
Five Ways to Fix Your Knowledge Base in ServiceNow
1. Audit for Relevance
Use ServiceNow’s knowledge analytics dashboards to identify
- Articles with low helpfulness ratings
- Articles with zero recent views
- Content past its Valid To expiry date
These should be archived, rewritten, or removed.
2. Standardise Article Structure
Use templates in ServiceNow to enforce consistency across your knowledge base. Best practice:- include headings like..
- Summary
- Applies To
- Resolution Steps
- Related Articles
This improves both human readability and AI interpretation.
3. Tag with Purpose
The “Meta” field in ServiceNow can drastically improve search performance if used correctly. Tips:-
- Avoid generic tags like issue | tech | help | support
- Use precise terms (e.g., VPN_timeout, Outlook_config, SSO_login_failure, Zoom_audio_setup)
- Repeat important tags to boost their search weighting (In ServiceNow’s AI Search and legacy Zing search engines, keyword frequency is a factor in ranking relevance. Repeating a tag (like
"VPN_timeout") in the Meta field, article body, and summary makes it more likely that this article will surface when users search for related issues. It trains the search to associate the article strongly with that topic.)
4. Add Metadata That Matters
Enrich your articles with contextual metadata:
- Product or service
- Issue type
- Keywords
This helps Now Assist and AI Search narrow results and avoid incorrect matches.
5. Test with Real Queries
Before turning your chatbot live, test it against real user phrases. Use ServiceNow’s AI Search Preview or Now Assist suggestions. Here are three real-world example queries you could use to test your chatbot and knowledge base, along with the reasoning behind each..
1. Can’t log in to VPN from home
Why? This is a typical end-user phrase, phrased conversationally rather than technically.
What you’re testing:
- Does it surface an article tagged with
VPN_timeoutorremote_access_failure? - Does the article address home access specifically, or is it too generic?
2. Outlook keeps asking for password
Why? It’s a frequent issue with cached credentials or expired authentication tokens.
What you’re testing:
- Does the article explain both the cause and the fix (e.g. clearing Windows Credential Manager)?
- Is it linked to
SSO_login_failureorOutlook_configtags?
3. Set up email on new phone
Why? It’s a mobile related, device agnostic query and useful to test for articles with platform specific guidance.
What you’re testing..
- Does it surface a device specific article (e.g., Android vs iOS)?
- Is there a step-by-step article linked with tags like
mobile_email_setup,BYOD_support, orActiveSync?
Each of these simulates a natural end-user query rather than a technical keyword, allowing you to evaluate if your AI can interpret intent, context, and specificity the three pillars of relevance for chatbots and AI search.
Always ask yourself..
- Do the right articles surface?
- Are they helpful?
- Do they match the user’s intent?
If not, refine the tags, content, or structure until they do.
Final Thoughts
If AI is misfiring, your articles might not be ready. It’s not a tool problem, it’s a content one. Fixing your knowledge base isn’t glamorous, but it’s essential groundwork.
Want to see how this fits into the bigger picture of AI in ITSM? Read the original post here.
🔗 Further Information