ServiceNow Instance Troubleshooter Application - SUPPORT

 https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0870978

Instance Trouble shooter Application

• Provided free by ServiceNow
• Needs activation of Plugin "Instance Troubleshooter".
• Any user with Scan User (scan_user) / System Admin (admin) role can run it on a Instance.
• Once plugin is installed it is available under "Instance Troubleshooter" Menu
• "Get Started" menu would allow us to see the 19 troubleshoot categories
• Access Control Lists
• Authentication
• Clone
• Core Platform
• Currency
• Discovery
• Email
• Forms and Fields
• Import / Export
• MID Server
• Mobile
• Performance
• Performance Analytics
• Reporting
• Service Catalog
• Service Portal
• SLA
• User Experience
• Workflow
• Each Category has set of checks that can be performed under that category
• Access Control Lists checks for
• High Security Settings plugin disabled
• ACLs using getRowCount may cause performance issue
• ACLs without roles or conditions or scripts
• Authentication
• "Auto Redirect IdP" is Enabled and SSO is down
• Auth SAML 2.0 certificate is either missing the X509 data or invalid
• Check for CMS with SSO (Failed)
• Clone could overwrite SAML SSO on target
• Identify invalid Multi-SSO configuration in the instance
• IdP certificate has changed or expired
• LDAP Authentication using MID Servers
• X.509 Cert for SAML Authentication
• Clone

Cannot exclude database views

Clone: Check for invalid/Incorrect Clone Targets

Prod Check if clone target for prod is set to true (Failed)

Tables to not exclude in clone

• Core Platform
• Attachments with '&' in name changes to '&amp' (Failed)
• Change management 'Cancel' UI action doesn't work
• Check activity stream on sys_user
• Check for home_refresh user preference (Failed)
• Check for invalid roles, groups and inheritance
• Check for PRB1437770
• Check glide.set_x_frame_options
• Check glide.ui.escape_text
• Check if default admin is inactive (Failed)
• Check if system timezone is inactive (Failed)
• Check if text index is running on a table or hasn't been run.
• Client scripts defined on the global table
• Database debugging on
• Detect any ^NQ in business rules (Failed)
• Enabling debug properties may cause disk size issues (Failed)
• Enforce strict user image upload
• Escape HTML should be enabled
• Identify superfluously text indexed fields (Failed)
• Important Deleter jobs disabled
• JSON script include should be functional
• Language picker not showing any language
• Large attachment size may cause performance issues
• Longer session time out may cause performance issues (Failed)
• No user has security_admin role
• Restrict unauthenticated access to images
• Slowness accessing user records
• Unable to edit/create sysauto_script execution in Quebec
• Unsupported globals and API in Portal client scripts (Failed)
• Validate if rollback is available on fix script form.
• Currency
• Multi currency mode - validate currency records match locale (Failed)
• Validate exchange rates exist for active currencies
• Discovery
• Number of active credentials per class should not be more than 20
• sys_mod_count should not be more than 1000 for Credentials
• Table for credential class should exist
• Email
• All outbound email is directed to one address for debugging purposes (Failed)
• Email Reader job does not exist
• Email Reader job not running
• Email Reader stuck
• Ensure the OOB POP3/SMTP accounts match the instance name
• Guest account does not exist
• Guest account is not active
• Inbound email disabled (Failed)
• Outbound email disabled (Failed)
• Outbound email with large recipient list
• Securing Inbound actions (Failed)
• SMTP Sender job does not exist
• SMTP Sender job not running
• SMTP Sender stuck
• There is only one active SMTP account
• Forms and Fields
• @mentions not working for Journal Input type fields
• Check for empty sys_ui_section reference
• Delete UI action not working on form
• Deleting attachment loops forever
• Forms with large number of activities loading slowly
• HTML fields don't show all actions/options
• HTML tags visible in activity stream
• Items per page has options over 100 (Failed)
• Priority data lookup rules does not work on problem table.
• Public tags are not visible to everyone (Failed)
• Reference click through not available
• Table Check "Short Description" label changed to "Problem Statement"

 

• Import / Export
• Data Source record must have a value for the Import Set table name field
• Easy Import to import date column check
• Excel Export Cell Limit 1 (Failed)
• Excel Export Cell Limit 2
• Export Column Length Check for CSV
• Export CSV line break check
• Export results encoding Check
• Export Row Limit check 1 for UI
• Export Row Limit check 2 for UI (Failed)
• Export Row Limit check 3 for UI (Failed)
• Export to PDF Column Check
• Export to PDF issue
• Export to PDF Row Check
• ImportSetRow Field Addition check
• ISET At least one record exists in import set table (Failed)
• ISET Data Source: Attachment not present (Failed)
• ISET Data Source: MID Server must be associated to a Data Source of type JDBC
• ISET Data Source: MID Server must be associated to the Data Source of type LDAP
• ISET record has at least one import set run record completed per transform map (Failed)
• ISET record has at least one import set run record per transform map
• MID Server
• MID Server down
• MID Server is down or failed to upgrade - OCSP cert revocation verification
• MID Server
• MID Server validated
• MID Server validated should match instance version
• Mobile
• 'Desktop only' catalog items are showing up on the ServiceNow Mobile apps (Failed)
• Agent app only shows approvals from 3 tables (Failed)
• Check for invalid item paramters
• Check MRVS UI Policy Support
• Disable the ServiceNow Classic mobile app
• Grouped Incidents applets on Mobile app does not show the records
• Push notification 'Approval Assigned to Me' does not work
• Users are not able to login to Now/Agent Mobile Apps
• Performance
• Archive Consumer Jobs
• Performance Analytics
• Check properties values related to PA job collections
• Reporting
• Reports without a filter condition (Failed)
• Scheduled reports for inactive users
• Service Catalog
• Attachments don't show on RITM records
• Cascade Variable is not working in Order Guide
• Catalog business rule customizations (Failed)
• Catalog item is not accessible although there are no roles or user criteria
• Lookup Select Box variable causing catalog page slow
• Macro/Custom variable is not displayed (Failed)
• Rule base is not opening the catalog item in Order Guide
• Securing Record Producers (Failed)
• Unexpected behaviour when inactive mandatory variables (Failed)
• Service Portal
• Check sp_instance records for isServiceWorkspace option
• Date fields are broken in Service Portal languages other than English
• g_navigation does not work in Service Portal
• HTML fields does not show up on community portal
• Internal page will not be editable in portal designer
• KB attachments missing in Contextual Search
• Knowledge Base link is not working on Service Portal
• Knowledge search does not work for non-admin user on portal
• Menu item drop-down window is cut off when clicked onto
• Multi-row Variable Set does not show up on RITM records (Failed)
• No snc_external access to "Data Table from URL Definition" widget (Failed)
• Order Guide Widget missing tabs (Failed)
• Recommended value for $sp in sys_public table
• Search returns 404 for customer contact users.
• Service Portal Announcements Broken (Failed)
• Unable to remove attachment from catalog item
• window.open breaks the service portal page in Internet Explorer
• SLA
• SLA customizations
• Task SLA timings not updating (Failed)
• User Experience
• 'Favorite' Lists without a filter condition (Failed)
• Agents not getting notified of work items in Agent Workspace
• Asynchronous AJAX call in on-submit client scripts
• Avoid custom Global UI Scripts (Failed)
• AWA presence states are not configured to receive work items
• Certain fields missing from portal instance options
• Check if ImpersonateEvaluator was customized
• Check Max Entity Expansion Value
• Check Search Suggestions Enabled
• Check VTB Boards for Valid Tables (Failed)
• Checks for invalid table name in push content
• Escape Jelly should be true
• Forms not loading in Agent workspace
• glide.security.strict.actions should be true
• GlideRecord usage on Client Scripts (Failed)
• If post chat survey is set, then interaction transcript might generate late (Failed)
• Invalid UI policy configurations (Failed)
• Jelly debugging on
• Jelly/JS interpolation protection
• List optimize set to false
• Multiple Chat setup records are present in the instance
• On the initial load, UI 16 loads slow
• On the initial load, workspace loads slow
• Synchronous AJAX call in client scripts (Failed)
• Synchronous AJAX call in Service Portal client scripts
• UI Actions using getRowCount may cause performance issues
• UI Actions without conditions (Failed)
• User Row Count Preference (Failed)
• Workflow
• Approval Workflow Activity Definition should not be customized (Failed)
• Business rule and script for catalog task activity should not be customized
• Custom workflow activity definition should exist
• There must be an active and published workflow on the Request table
• There should be a workflow version for all executing workflow contexts
• Workflows Activities should not use current.update() function (Failed)

 

 

Resolution

 

• 'Desktop Only' catalog items are showing up on the ServiceNow Mobile apps
• As property: glide.sc.mobile.include_desktop_only_items is set to 'true', the 'Desktop only' catalog items are added to the mobile applications.
• To change this behavior, set the property to false.
• 'favorite' Lists without a filter condition
• We strongly recommend using condition filters on any list transactions where users have saved the list as a 'favorite' for frequent access.
• 1. Review the lists saved as favorites that have no filter condition
• 2. Re-create the list favorite with a useful condition to reduce the result set to be retrieved.
• Agent app only shows approvals from Request, Requested Item and Change Request
• - Per OOB configuration, the Approval applet launcher in the Agent app is to "Approve catalog requests, requested items, or change requests"
• This is due to the fact that OOB, only these 3 tables have their corresponding Item configurations (sys_sg_master_item) available under the ITSM Mobile scope.
• - In case it is needed to have approvals from a table other than the above three on the mobile app, it would require customization by creating a new Item configurations (sys_sg_master_item) record with the required conditions and associating it with the same Item stream (sys_sg_item_stream) that the other three sys_sg_master_item are associated to.
• All outbound email is directed to one address for debugging purposes
• Empty the value for system property glide.email.test.user
• Approval Workflow Activity Definition should not be customized
• Revert all Workflow Activity Definition (wf_activity_definition) with name containing "approval" back to their out-of-the-box version.
• Also WorkflowApprovalUtils script include back to its out-of-the-box version.
• Attachment name should not contain '&'
• Rename the attachments to remove '&'.
• Avoid custom Global UI Scripts
• Make the UI Script non-global, and include the UI Script on the appropriate page by referencing the script as needed.
• Catalog business rule customizations
• It is documented that you should not create Before Business rule customizations on catalog related tables.
• Please disable, remove, or configure this business rule to not use before for the business rule.
• Checks if CMS is properly configured for SSO
• 1) Navigate to sys_public.list
• 2) Find the record for view_content
• 3) Set the Active field to false
• Users noticing page refresh
• The recommended value for this is 'off'. If set to an integer value, please make sure that the value is the number of seconds after which the page loads.

If this user preference is set at the system level, please be aware that all users will experience unexpected page refresh unless overridden by a user-specific preference.

• Check for invalid roles, groups and inheritance
• Review the invalid records and determine if they are valid to be removed. If roles are not inherited then you can safely delete them via the UI.
• If some of the roles are inherited, try adding and removing the inheritance to a user and remove to fix, otherwise, raise the case to Support to fix this issue.
• The KB attached will also provide more information on this issue.
• Check if default admin is inactive in instance
• Activate the default admin user.
• Check to ensure the system timezone is active on the instance
• Please uncheck the inactive field and save this record.
• Checks to ensure VTB (visual task boards) are associated with a valid table
• Remove this record or associate it with a valid table
• Using operator (^NQ) in encoded queries causes incorrect reference links
• This is expected behavior. The NQ (top level OR) operator will 'OR' all previous query terms with following query terms. When there are no previous query terms, it is equivalent to regular 'OR'. Adjust the business Rule to not use the NQ Operator or disable the Business rule. Check documentation for more information.
• Enabling debug properties may cause disk size issues
• Disable the debug properties from system properties.
• This check is to verify the allowed number of rows and columns while exporting
• We recommend a value of 500,000 or less for glide.xlsx.max_cells (the number of cells to export for excel files)
• This check is to verify the allowed number of rows and columns while exporting
• We recommend a value of 10,000 or less for glide.xlsx.export.limit
• This check is to verify the allowed number of rows and columns while exporting
• We recommend a value of 10,000 or less for glide.excel.export.limit
• GlideRecord/g_form.getReference usage on Client Scripts
• Use client data as much as possible to eliminate the need for time-consuming server lookups. The best way to get information from the server are g_scratchpad, and asynchronous GlideAjax lookup.
• Background script to identify fields which are being superfluously text indexed
• Review these dictionary entries, if you do not wish to have them text indexed you can add attribute: no_text_index=true to each dictionary entry.
• If post chat survey is set, then interaction transcript might generate late

The job(Time Out Abandoned VA Conversations) currently scheduled to run once a day.

Below system property can be created to change the default timeout period for idle Virtual Agent conversations so that the conversation record gets closed earlier, concluding the conversation and generation of "transcript" in the interaction record.

Name: com.glide.cs.conversation_idle_timeout

Type :Integer

Value : (in seconds)

 

Enter the number of seconds that abandoned Virtual Agent conversations remain open, after the requester's last response. This value must be less than 7200 seconds, since the Time Out Abandoned VA Conversations job runs every 3600 seconds (hourly) to close idle conversations.

 

For example, a value of 1800 seconds (30 minutes) means that an abandoned conversation remains open for 1800 seconds (30 minutes). When the Time Out Abandoned VA Conversations job runs, it closes any conversations that have been idle longer than 1800 seconds.

• Inbound email disabled
• Set glide.email.read.active system property to 'true'.
• Invalid UI policy action configurations
• The fields that are set as read-only via the dictionary entry options or the ACLs cannot be set as mandatory as it is ignored by the platform. So, the users can submit the form despite the fact that there is a UI policy that is set up to make the field mandatory. The action of the UI policy is ignored in that case.  Rectify by modifying field permission and UI Policy configuration.
• At least one record exists in import set table
• If you are expecting at least one record to be imported, then check the logs for any errors or warnings that may have prevented the platform from imported data into the import set table.
• ISET Data Source: Attachment not present
• Attach the data file to the Data Source record.
• ISET record has at least one import set run record completed per transform map
• If you are expecting at least one record to be transformed, then check the logs for any errors or warnings that may have prevented the platform from transforming the data into the target table. Keep in mind that this may be normal if the transform phase has simply not started yet.
• Items per page has options over 100
• Remove options above 100 from the property glide.ui.per_page
• Longer session time out may cause performance issues
• Set the system property to the system default value of 30 or less.
• Macro variables are not displayed in the service catalog or portal
• This will occur if the macro variable fields macro and widget are both empty. To appear on the portal, the widget field must contain a valid sp_widget. For the macro to appear in the service catalog in the platform UI, the macro field should contain a valid sys_ui_macro.
• Multi-row Variable Set does not show up on RITM
• Deprecated catalog widget is being used instead of the latest catalog widget. Change the SC Catalog Item Deprecated to the SC Catalog Item on the affected page.

SC Catalog Item :uri=sp_widget.do?sys_id=3c29786e87133200e0ef0cf888cb0bdf

SC Catalog Item Deprecated:uri=sp_widget.do?sys_id=0fd6a6f247230200ba13a5554ee490b3

• Multi currency mode enabled validate records are using correct currency
• Records were found where the reference currency does not match locale. Perform an update on the related records so they reflect the locale in the reference currency field
• No snc_external access to "Data Table from URL Definition" widget
• If Service Portal User Criteria plugin is enabled, Add the user criteria with "snc_external" role to the canView related list of the Data Table widget.

OR

Add the snc_external role to the roles field in the Data Table widget.

• Order Guide Widget missing tabs
• Deprecated order guide widget is being used please replace with the "sc order guide" widget.
• Outbound email disabled
• Make sure the glide.email.smtp.active property is set to 'true' to enable outbound email.
• Check if clone target set to true for prod instance.
• This can be safely ignored if the instance is not production
• If this is prod, If you do not want to clone over production then please adjust property to false. This will stop the unlikely situation of you requesting a clone over prod.
• Reports without a filter condition
• We strongly recommend using condition filters on any reports that are created on the instance.  The idea is that by using a filter condition, the query can use an existing index to make the query/s and therefore the report load faster, and reduce the load on the database. If a filter condition is subsequently added to a report and there is no improvement, considering adding an index to improve the query execution.
• Securing Inbound actions
• All inbound actions should have roles mentioned in "Required roles" field to secure actions triggered from inbound actions, only users with the required roles should be allowed to trigger inbound action.
• Securing Record Producers
• Record producers should be assigned appropriate role, record producer script should secure GlideRecord and/or GlideSystem API calls.
• Announcements in the Service Portal are not displayed
• 1) Navigate to the scheduled job [AppSec] Daily Data Management: /sysauto_script.do?sys_id=2b98d54f53332300628eddeeff7b120b

2) Ensure the Run As field contains a valid, active admin user

3) Navigate to the list of announcements created by the scheduled job and deactivate them: /nav_to.do?uri=%2Fannouncement_list.do%3Fsysparm_query%3Dactive%3Dtrue%5Ename%3DISC:Invalid%20Run%20As%20User%26sysparm_first_row%3D1%26sysparm_view%3D

• Synchronous AJAX call in client scripts
• If using getXMLWait, replace with getXML and a callback function.
• Task SLA timings not updating
• SLA Engine system property "glide.sla.calculate_on_display" is currently set to "false".

This property recalculates Task SLA records when a Task form is displayed. This ensures that the task SLAs calculations are up to date but, this may increase form load time.

Please review and if you wish to have SLA timings refresh set this property to "true"

• UI Actions without conditions may cause unintended results
• Define conditions on UI Action to control its visibility to the intended users.
• Catalog submission issues if there are mandatory variables that are inactive
• Uncheck mandatory if the variable is inactive
• Unsupported globals and API in Portal client scripts
• The following globals are not supported for client scripts running in Service Portal
• $
• $$
• $j
• angular
• control
• document
• jQuery
• window

Modify the scripts to not have the logic using these globals for Service Portal specific client script.

• User Row Count Preference

1. Instance Administrators can limit how many rows a user can set as described in KB0750152. This is a global setting -  but we do recommend that this feature is used to prevent users from displaying too many rows at once.

2. If a user would like to reduce their row count, they can do so by modifying their row count setting like this  :

- Open a list

- Click on the 'additional actions menu' (3 horizontal lines) on the top left of the list/form

- Show

- Select x rows per page

Here we should always encourage 20 to be used

3. If users do have a requirement to view more than 50 rows at once, then perhaps the filter condition is not precise enough and we should encourage best practice.  For example, if you are looking for a particular active record on the task table, instead of just using a condition like this "active = 'true'" and then having to scroll through pages of lists, why not add additional parameters to narrow down the result set. For example, "active='true' AND assigned to 'John' and created on 'this year'". By doing this, the result set will be greatly reduced therefore making it much quicker & easier for the target record to be found.

• Workflows Activities should not use current.update() function
• Remove unnecessary current.update() calls from custom scripts.

 

Action

• The actual records that need to fixed is available in the "Source" Column of the Scan Findings
• These have to be actioned OR
• Mute Rule has to be provided